Abstract: Causal representation learning is the task of identifying the underlying causal variables and their relations from high-dimensional observations, such as images. Recent work has shown that one can reconstruct the causal variables from temporal sequences of observations under the assumption that there are no instantaneous causal relations between them. In practical applications, however, our measurement or frame rate might be slower than many of the causal effects. This effectively creates ``instantaneous' effects and invalidates previous identifiability results. To address this issue, we propose iCITRIS, a causal representation learning method that allows for instantaneous effects in intervened temporal sequences when intervention targets can be observed, e.g., as actions of an agent. iCITRIS identifies the potentially multidimensional causal variables from temporal observations, while simultaneously using a differentiable causal discovery method to learn their causal graph. In experiments on three datasets of interactive systems, iCITRIS accurately identifies the causal variables and their causal graph.

BibTeX

@inproceedings{lippe2023causal,
	title        = {{Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems}},
	author       = {Phillip Lippe and Sara Magliacane and Sindy L{\"o}we and Yuki M Asano and Taco Cohen and Efstratios Gavves},
	year         = {2023},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=itZ6ggvMnzS}
}

Abstract: A number of problems in learning can be formulated in terms of the basic primitive of sampling k elements out of a universe of n elements. This subset sampling operation cannot directly be included in differentiable models and approximations are essential. Current approaches take an order sampling approach to sampling subsets and depend on differentiable approximations of the Top-k operator for selecting the largest k elements from a set. We present a simple alternative method for sampling subsets based on conditional Poisson sampling. Unlike order sampling approaches, the parallel complexity of the proposed method is independent of the subset size which makes the method scalable to large subset sizes. We adapt the procedure to make it efficient and amenable to discrete gradient approximations for use in differentiable models. Furthermore, the method also allows the subset size parameter k to be differentiable. We demonstrate our approach on model explanation, image sub-sampling and stochastic k-nearest neighbor tasks outperforming existing methods in accuracy, efficiency and scalability.

BibTeX

@inproceedings{pervez2023scalable,
	title        = {{Scalable Subset Sampling with Neural Conditional Poisson Networks}},
	author       = {Adeel Pervez and Phillip Lippe and Efstratios Gavves},
	year         = {2023},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=p8hMBcPtvju}
}

Abstract: We propose topology-aware feature partitioning into k disjoint partitions for given scene features as a method for object-centric representation learning. To this end, we propose to use minimum s-t graph cuts as a partitioning method which is represented as a linear program. The method is topologically aware since it explicitly encodes neighborhood relationships in the image graph. To solve the graph cuts our solution relies on an efficient, scalable, and differentiable quadratic programming approximation. Optimizations specific to cut problems allow us to solve the quadratic programs and compute their gradients significantly more efficiently compared with the general quadratic programming approach. Our results show that our approach is scalable and outperforms existing methods on object discovery tasks with textured scenes and objects.

BibTeX

@inproceedings{pervez2023differentiable,
	title        = {{Differentiable Mathematical Programming for Object-Centric Representation Learning}},
	author       = {Adeel Pervez and Phillip Lippe and Efstratios Gavves},
	year         = {2023},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=1J-ZTr7aypY}
}

Abstract: We aim for image-based novelty detection. Despite considerable progress, existing models either fail or face dramatic drop under the so-called ``near-distribution" setup, where the differences between normal and anomalous samples are subtle. We first demonstrate existing methods could experience up to 20\% decrease in their AUCs in the near-distribution setting. Next, we propose to exploit a score-based generative model to produce synthetic near-distribution anomalous data. Our model is then fine-tuned to distinguish such data from the normal samples. We make quantitative as well as qualitative evaluation of this strategy, and compare the results with a variety of GAN-based models. Effectiveness of our method for both near-distribution and standard novelty detection is assessed through extensive experiments on datasets in diverse applications such as medical images, object classification, and quality control. This reveals that our method significantly improves upon existing models, and consistently decreases the gap between the near-distribution and standard novelty detection AUCs by a considerable amount.

BibTeX

@inproceedings{mirzaei2023fake,
	title        = {{Fake It Until You Make It : Towards Accurate Near-Distribution Novelty Detection}},
	author       = {Hossein Mirzaei and Mohammadreza Salehi and Sajjad Shahabi and Efstratios Gavves and Cees G. M. Snoek and Mohammad Sabokrou and Mohammad Hossein Rohban},
	year         = {2023},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=QWQM0ZwZdRS}
}

Abstract: Object-centric representations form the basis of human perception, and enable us to reason about the world and to systematically generalize to new settings. Currently, most works on unsupervised object discovery focus on slot-based approaches, which explicitly separate the latent representations of individual objects. While the result is easily interpretable, it usually requires the design of involved architectures. In contrast to this, we propose a comparatively simple approach – the Complex AutoEncoder (CAE) – that creates distributed object-centric representations. Following a coding scheme theorized to underlie object representations in biological neurons, its complex-valued activations represent two messages: their magnitudes express the presence of a feature, while the relative phase differences between neurons express which features should be bound together to create joint object representations. In contrast to previous approaches using complex-valued activations for object discovery, we present a fully unsupervised approach that is trained end-to-end – resulting in significant improvements in performance and efficiency. Further, we show that the CAE achieves competitive or better unsupervised object discovery performance on simple multi-object datasets compared to a state-of-the-art slot-based approach while being up to 100 times faster to train.

BibTeX

@article{loewe2022complexvalued,
	title        = {{Complex-Valued Autoencoders for Object Discovery}},
	author       = {Sindy L{\"o}we and Phillip Lippe and Maja Rudolph and Max Welling},
	year         = {2022},
	journal      = {Transactions on Machine Learning Research},
	url          = {https://openreview.net/forum?id=1PfcmFTXoa}
}

Abstract: In this work we propose a batch Bayesian optimization method for combinatorial problems on permutations, which is well suited for expensive-to-evaluate objectives. We first introduce LAW, an efficient batch acquisition method based on determinantal point processes using the acquisition weighted kernel. Relying on multiple parallel evaluations, LAW enables accelerated search on combinatorial spaces. We then apply the framework to permutation problems, which have so far received little attention in the Bayesian Optimization literature, despite their practical importance. We call this method LAW2ORDER. On the theoretical front, we prove that LAW2ORDER has vanishing simple regret by showing that the batch cumulative regret is sublinear. Empirically, we assess the method on several standard combinatorial problems involving permutations such as quadratic assignment, flowshop scheduling and the traveling salesman, as well as on a structure learning task.

BibTeX

@inproceedings{oh2022batch,
	title        = {{Batch Bayesian Optimization on Permutations using the Acquisition Weighted Kernel}},
	author       = {Changyong Oh and Roberto Bondesan and Efstratios Gavves and Max Welling},
	year         = {2022},
	booktitle    = {Advances in Neural Information Processing Systems},
	url          = {https://openreview.net/forum?id=LODRFJr96v},
	editor       = {Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho}
}

Abstract: Cryo-Electron Microscopy (Cryo-EM) is an important imaging method which allows high-resolution reconstruction of the 3D structures of biomolecules. It produces highly noisy 2D images by projecting a molecule's 3D density from random viewing directions. Because the projection directions are unknown, estimating the images' poses is necessary to perform the reconstruction. We focus on this task and study it under the group synchronization framework: if the relative poses of pairs of images can be approximated from the data, an estimation of the images' poses is given by the assignment which is most consistent with the relative ones. In particular, by studying the symmetries of cryo-EM, we show that relative poses in the group O(2) provide sufficient constraints to identify the images' poses, up to the molecule's chirality. With this in mind, we improve the existing multi-frequency vector diffusion maps (MFVDM) method: by using O(2) relative poses, our method not only predicts the similarity between the images' viewing directions but also recovers their poses. Hence, we can leverage all input images in a 3D reconstruction algorithm by initializing the poses with our estimation rather than just clustering and averaging the input images. We validate the recovery capabilities and robustness of our method on randomly generated synchronization graphs and a synthetic cryo-EM dataset.

BibTeX

@inproceedings{cesa2022on,
	title        = {{On the symmetries of the synchronization problem in Cryo-EM: Multi-Frequency Vector Diffusion Maps on the Projective Plane}},
	author       = {Gabriele Cesa and Arash Behboodi and Taco Cohen and Max Welling},
	year         = {2022},
	booktitle    = {Advances in Neural Information Processing Systems},
	url          = {https://openreview.net/forum?id=owDcdLGgEm},
	editor       = {Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho}
}

Abstract: Equivariant networks capture the inductive bias about the symmetry of the learning task by building those symmetries into the model. In this paper, we study how equivariance relates to generalization error utilizing PAC Bayesian analysis for equivariant networks, where the transformation laws of feature spaces are determined by group representations. By using perturbation analysis of equivariant networks in Fourier domain for each layer, we derive norm-based PAC-Bayesian generalization bounds. The bound characterizes the impact of group size, and multiplicity and degree of irreducible representations on the generalization error and thereby provide a guideline for selecting them. In general, the bound indicates that using larger group size in the model improves the generalization error substantiated by extensive numerical experiments.

BibTeX

@inproceedings{behboodi2022a,
	title        = {{A PAC-Bayesian Generalization Bound for Equivariant Networks}},
	author       = {Arash Behboodi and Gabriele Cesa and Taco Cohen},
	year         = {2022},
	booktitle    = {Advances in Neural Information Processing Systems},
	url          = {https://openreview.net/forum?id=6dfYc2IUj4},
	editor       = {Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho}
}

Abstract: Learning high-level causal representations together with a causal model from unstructured low-level data such as pixels is impossible from observational data alone. We prove under mild assumptions that this representation is however identifiable in a weakly supervised setting. This involves a dataset with paired samples before and after random, unknown interventions, but no further labels. We then introduce implicit latent causal models, variational autoencoders that represent causal variables and causal structure without having to optimize an explicit discrete graph structure. On simple image data, including a novel dataset of simulated robotic manipulation, we demonstrate that such models can reliably identify the causal structure and disentangle causal variables.

BibTeX

@inproceedings{brehmer2022weakly,
	title        = {{Weakly supervised causal representation learning}},
	author       = {Brehmer, Johann and de Haan, Pim and Lippe, Phillip and Cohen, Taco},
	year         = {2022},
	booktitle    = {Advances in Neural Information Processing Systems},
	url          = {https://arxiv.org/abs/2203.16437}
}

Abstract: The goal of this paper is to bypass the need for labelled examples in few-shot video understanding at run time. While proven effective, in many practical video settings even labelling a few examples appears unrealistic. This is especially true as the level of details in spatio-temporal video understanding and with it, the complexity of annotations continues to increase. Rather than performing few-shot learning with a human oracle to provide a few densely labelled support videos, we propose to automatically learn to find appropriate support videos given a query. We call this self-shot learning and we outline a simple self-supervised learning method to generate an embedding space well-suited for unsupervised retrieval of relevant samples. To showcase this novel setting, we tackle, for the first time, video instance segmentation in a self-shot (and few-shot) setting, where the goal is to segment instances at the pixel-level across the spatial and temporal domains. We provide strong baseline performances that utilize a novel transformer-based model and show that self-shot learning can even surpass few-shot and can be positively combined for further performance gains. Experiments on new benchmarks show that our approach achieves strong performance, is competitive to oracle support in some settings, scales to large unlabelled video collections, and can be combined in a semi-supervised setting.

BibTeX

@inproceedings{yang2022less,
	title        = {{Less Than Few: Self-shot Video Instance Segmentation}},
	author       = {Yang, Pengwan and Asano, Yuki M. and Mettes, Pascal and Snoek, Cees G. M.},
	year         = {2022},
	booktitle    = {Computer Vision -- ECCV 2022},
	publisher    = {Springer Nature Switzerland},
	address      = {Cham},
	pages        = {449--466},
	isbn         = {978-3-031-19830-4},
	editor       = {Avidan, Shai and Brostow, Gabriel and Ciss{\'e}, Moustapha and Farinella, Giovanni Maria and Hassner, Tal},
	url          = {https://link.springer.com/chapter/10.1007/978-3-031-19830-4_26#citeas}
}

Abstract: Understanding the latent causal factors of a dynamical system from visual observations is a crucial step towards agents reasoning in complex environments. In this paper, we propose CITRIS, a variational autoencoder framework that learns causal representations from temporal sequences of images in which underlying causal factors have possibly been intervened upon. In contrast to the recent literature, CITRIS exploits temporality and observing intervention targets to identify scalar and multidimensional causal factors, such as 3D rotation angles. Furthermore, by introducing a normalizing flow, CITRIS can be easily extended to leverage and disentangle representations obtained by already pretrained autoencoders. Extending previous results on scalar causal factors, we prove identifiability in a more general setting, in which only some components of a causal factor are affected by interventions. In experiments on 3D rendered image sequences, CITRIS outperforms previous methods on recovering the underlying causal variables. Moreover, using pretrained autoencoders, CITRIS can even generalize to unseen instantiations of causal factors, opening future research areas in sim-to-real generalization for causal representation learning.

BibTeX

@inproceedings{lippe2022citris,
	title        = {{CITRIS: Causal Identifiability from Temporal Intervened Sequences}},
	author       = {Lippe, Phillip and Magliacane, Sara and L{\"o}we, Sindy and Asano, Yuki M and Cohen, Taco and Gavves, Stratis},
	year         = {2022},
	month        = {17--23 Jul},
	booktitle    = {Proceedings of the 39th International Conference on Machine Learning},
	publisher    = {PMLR},
	series       = {Proceedings of Machine Learning Research},
	volume       = {162},
	pages        = {13557--13603},
	url          = {https://proceedings.mlr.press/v162/lippe22a.html},
	editor       = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}
}

Abstract: Equivariance is becoming an increasingly popular design choice to build data efficient neural networks by exploiting prior knowledge about the symmetries of the problem at hand. Euclidean steerable CNNs are one of the most common classes of equivariant networks. While the constraints these architectures need to satisfy are understood, no practical method to parametrize them generally has been described so far, with most existing approaches tailored to specific groups or classes of groups. In this work, we generalize the Wigner-Eckart theorem proposed in (Lang et al.), which characterizes general G-steerable kernel spaces for compact groups G over their homogeneous spaces, to arbitrary G-spaces. This enables us to directly parameterize filters in terms of a band-limited basis on the base space, but also to easily implement steerable CNNs equivariant to a large number of groups. To demonstrate its generality, we instantiate our method on a large variety of isometry groups acting on the Euclidean space R^3. Our general framework allows us to build E(3) and SE(3)-steerable CNNs like previous works, but also CNNs with arbitrary G<=O(3)-steerable kernels. For example, we build 3D CNNs equivariant to the symmetries of platonic solids or choose G=SO(2) when working with 3D data having only azimuthal symmetries. We compare these models on 3D shapes and molecular datasets, observing improved performance by matching the model's symmetries to the ones of the data.

BibTeX

@inproceedings{cesa2022program,
	title        = {{A Program to Build E(N)-Equivariant Steerable CNNs }},
	author       = {Gabriele Cesa and Leon Lang and Maurice Weiler},
	year         = {2022},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=WE4qe9xlnQw}
}

Abstract: We present a method for training neural network models with discrete stochastic variables. The core of the method is \emphstability regularization which is a regularization procedure based on the idea of noise stability developed in Gaussian isoperimetric theory in the analysis of Gaussian functions. Stability regularization is method to make the output of continuous functions of Gaussian random variables close to discrete, that is binary or categorical, without the need for significant manual tuning. The method allows control over the extent to which a Gaussian function's output is close to discrete, thus allowing for continued flow of gradient. The method can be used standalone or in combination with existing continuous relaxation methods. We validate the method in a broad range of experiments using discrete variables including neural relational inference, generative modeling, clustering and conditional computing.

BibTeX

@inproceedings{pervez2022stability,
	title        = {{Stability Regularization for Discrete Representation Learning}},
	author       = {Adeel Pervez and Efstratios Gavves},
	year         = {2022},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=6tmjoym9LR6}
}

Abstract: Learning the structure of a causal graphical model using both observational and interventional data is a fundamental problem in many scientific fields. A promising direction is continuous optimization for score-based methods, which efficiently learn the causal graph in a data-driven manner. However, to date, those methods re-quire constrained optimization to enforce acyclicity or lack convergence guarantees. In this paper, we present ENCO, an efficient structure learning method for directed, acyclic causal graphs leveraging observational and interventional data. ENCO formulates the graph search as an optimization of independent edge likelihoods, with the edge orientation being modeled as a separate parameter. Consequently, we provide for ENCO convergence guarantees under mild conditions, without having to constrain the score function with respect to acyclicity. In experiments, we show that ENCO can efficiently recover graphs with hundreds of nodes, an order of magnitude larger than what was previously possible, while handling deterministic variables and discovering latent confounders.

BibTeX

@inproceedings{lippe2022efficient,
	title        = {{Efficient Neural Causal Discovery without Acyclicity Constraints}},
	author       = {Lippe, Phillip and Cohen, Taco and Gavves, Efstratios},
	year         = {2022},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=eYciPrLuUhG}
}

Abstract: The goal of this paper is to self-train a 3D convolutional neural network on an unlabeled video collection for deployment on small-scale video collections. As smaller video datasets benefit more from motion than appearance, we strive to train our network using optical flow, but avoid its computation during inference. We propose the first motion-augmented self-training regime, we call MotionFit. We start with supervised training of a motion model on a small, and labeled, video collection. With the motion model we generate pseudo-labels for a large unlabeled video collection, which enables us to transfer knowledge by learning to predict these pseudo-labels with an appearance model. Moreover, we introduce a multi-clip loss as a simple yet efficient way to improve the quality of the pseudo-labeling, even without additional auxiliary tasks. We also take into consideration the temporal granularity of videos during self-training of the appearance model, which was missed in previous works. As a result we obtain a strong motion-augmented representation model suited for video downstream tasks like action recognition and clip retrieval. On small-scale video datasets, MotionFit outperforms alternatives for knowledge transfer by 5%-8%, video-only self-supervision by 1%-7% and semi-supervised learning by 9%-18% using the same amount of class labels.

BibTeX

@inproceedings{gavrilyuk2021motionaugmented,
	title        = {{Motion-Augmented Self-Training for Video Recognition at Smaller Scale}},
	author       = {Kirill Gavrilyuk and Mihir Jain and Ilia Karmanov and Cees G M Snoek},
	year         = {2021},
	booktitle    = {ICCV},
	address      = {Montreal, Canada},
	url          = {https://isis-data.science.uva.nl/cgmsnoek/pub/gavrilyuk-motionfit-iccv2021.pdf}
}

Abstract: Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of input. We suggest that the hierarchical VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function. Building on this we generalize VAE neural networks by incorporating a smoothing parameter motivated by Gaussian analysis to reduce higher frequency components and consequently the variance in parameterizing functions and show that this can help to solve the problem of posterior collapse. We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KL divergence sharply drops to zero at a critical value of the smoothing parameter that is similar for the same model across datasets. We validate the phenomenon across model configurations and datasets.

BibTeX

@inproceedings{pervez2021spectral,
	title        = {{Spectral Smoothing Unveils Phase Transitions in Hierarchical Variational Autoencoders}},
	author       = {Pervez, Adeel and Gavves, Efstratios},
	year         = {2021},
	month        = {18--24 Jul},
	booktitle    = {Proceedings of the 38th International Conference on Machine Learning},
	publisher    = {PMLR},
	series       = {Proceedings of Machine Learning Research},
	volume       = {139},
	pages        = {8536--8545},
	url          = {https://proceedings.mlr.press/v139/pervez21a.html},
	editor       = {Meila, Marina and Zhang, Tong},
	pdf          = {http://proceedings.mlr.press/v139/pervez21a/pervez21a.pdf}
}

Abstract: Despite their popularity, to date, the application of normalizing flows on categorical data stays limited. The current practice of using dequantization to map discrete data to a continuous space is inapplicable as categorical data has no intrinsic order. Instead, categorical data have complex and latent relations that must be inferred, like the synonymy between words. In this paper, we investigate Categorical Normalizing Flows, that is normalizing flows for categorical data. By casting the encoding of categorical data in continuous space as a variational inference problem, we jointly optimize the continuous representation and the model likelihood. Using a factorized decoder, we introduce an inductive bias to model any interactions in the normalizing flow. As a consequence, we do not only simplify the optimization compared to having a joint decoder, but also make it possible to scale up to a large number of categories that is currently impossible with discrete normalizing flows. Based on Categorical Normalizing Flows, we propose GraphCNF a permutation-invariant generative model on graphs. GraphCNF implements a three step approach modeling the nodes, edges, and adjacency matrix stepwise to increase efficiency. On molecule generation, GraphCNF outperforms both one-shot and autoregressive flow-based state-of-the-art.

BibTeX

@inproceedings{lippe2021categorical,
	title        = {{Categorical Normalizing Flows via Continuous Transformations}},
	author       = {Phillip Lippe and Efstratios Gavves},
	year         = {2021},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/pdf?id=-GLNZeVDuik}
}

Abstract: A common approach to define convolutions on meshes is to interpret them as a graph and apply graph convolutional networks (GCNs). Such GCNs utilize isotropic kernels and are therefore insensitive to the relative orientation of vertices and thus to the geometry of the mesh as a whole. We propose Gauge Equivariant Mesh CNNs which generalize GCNs to apply anisotropic gauge equivariant kernels. Since the resulting features carry orientation information, we introduce a geometric message passing scheme defined by parallel transporting features over mesh edges. Our experiments validate the significantly improved expressivity of the proposed model over conventional GCNs and other methods.

BibTeX

@inproceedings{haan2021gauge,
	title        = {{Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs}},
	author       = {Pim De Haan and Maurice Weiler and Taco Cohen and Max Welling},
	year         = {2021},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=Jnspzp-oIZE}
}

Abstract: Group equivariant convolutional networks (GCNNs) endow classical convolutional networks with additional symmetry priors, which can lead to a considerably improved performance. Recent advances in the theoretical description of GCNNs revealed that such models can generally be understood as performing convolutions with $G$-steerable kernels, that is, kernels that satisfy an equivariance constraint themselves. While the $G$-steerability constraint has been derived, it has to date only been solved for specific use cases - a general characterization of $G$-steerable kernel spaces is still missing. This work provides such a characterization for the practically relevant case of $G$ being any compact group. Our investigation is motivated by a striking analogy between the constraints underlying steerable kernels on the one hand and spherical tensor operators from quantum mechanics on the other hand. By generalizing the famous Wigner-Eckart theorem for spherical tensor operators, we prove that steerable kernel spaces are fully understood and parameterized in terms of 1) generalized reduced matrix elements, 2) Clebsch-Gordan coefficients, and 3) harmonic basis functions on homogeneous spaces.

BibTeX

@inproceedings{lang2021winger,
	title        = {{A Wigner-Eckart Theorem for Group Equivariant Convolution Kernels}},
	author       = {Leon Lang and Maurice Weiler},
	year         = {2021},
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=ajOrOhQOsYx}
}

Abstract: A key requirement for graph neural networks is that they must process a graph in a way that does not depend on how the graph is described. Traditionally this has been taken to mean that a graph network must be equivariant to node permutations. Here we show that instead of equivariance, the more general concept of naturality is sufficient for a graph network to be well-defined, opening up a larger class of graph networks. We define global and local natural graph networks, the latter of which are as scalable as conventional message passing graph neural networks while being more flexible. We give one practical instantiation of a natural network on graphs which uses an equivariant message network parameterization, yielding good performance on several benchmarks.

BibTeX

@inproceedings{haan2020natural,
	title        = {{Natural Graph Networks}},
	author       = {de Haan, Pim and Cohen, Taco S and Welling, Max},
	year         = {2020},
	booktitle    = {Advances in Neural Information Processing Systems},
	publisher    = {Curran Associates, Inc.},
	volume       = {33},
	pages        = {3636--3646},
	url          = {https://proceedings.neurips.cc/paper/2020/file/2517756c5a9be6ac007fe9bb7fb92611-Paper.pdf},
	editor       = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin}
}

Abstract: For many of the physical phenomena around us, we have developed sophisticated models explaining their behavior. Nevertheless, measuring physical properties from visual observations is challenging due to the high number of causally underlying physical parameters -- including material properties and external forces. In this paper, we propose to measure latent physical properties for cloth in the wind without ever having seen a real example before. Our solution is an iterative refinement procedure with simulation at its core. The algorithm gradually updates the physical model parameters by running a simulation of the observed phenomenon and comparing the current simulation to a real-world observation. The correspondence is measured using an embedding function that maps physically similar examples to nearby points. We consider a case study of cloth in the wind, with curling flags as our leading example -- a seemingly simple phenomena but physically highly involved. Based on the physics of cloth and its visual manifestation, we propose an instantiation of the embedding function. For this mapping, modeled as a deep network, we introduce a spectral layer that decomposes a video volume into its temporal spectral power and corresponding frequencies. Our experiments demonstrate that the proposed method compares favorably to prior work on the task of measuring cloth material properties and external wind force from a real-world video.

BibTeX

@inproceedings{runia2020cloth,
	title        = {{Cloth in the Wind: A Case Study of Physical Measurement Through Simulation}},
	author       = {Runia, Tom F. H. and Gavrilyuk, Kirill and Snoek, Cees G. M. and Smeulders, Arnold W. M.},
	year         = {2020},
	booktitle    = {CVPR},
	doi          = {10.1109/cvpr42600.2020.01051},
	archiveprefix = {arXiv},
	arxivid      = {2003.05065},
	eprint       = {2003.05065}
}

Abstract: This paper tackles the problem of localizing actions in long untrimmed videos. Different from existing works, which all use annotated untrimmed videos during training, we learn only from short trimmed videos. This enables learning from large-scale datasets originally designed for action classification. We propose a method to train an action localization network that segments a video into in-terpretable fragments, we call ActionBytes. Our method jointly learns to cluster ActionBytes and trains the local-ization network using the cluster assignments as pseudo-labels. By doing so, we train on short trimmed videos that become untrimmed for ActionBytes. In isolation, or when merged, the ActionBytes also serve as effective action proposals. Experiments demonstrate that our boundary-guided training generalizes to unknown action classes and localizes actions in long videos of Thumos14, MultiThumos, and ActivityNet1.2. Furthermore, we show the advantage of Ac-tionBytes for zero-shot localization as well as traditional weakly supervised localization, that train on long videos, to achieve state-of-the-art results.

BibTeX

@inproceedings{jain2020action,
	title        = {{ActionBytes: Learning From Trimmed Videos to Localize Actions}},
	author       = {Jain, Mihir and Ghodrati, Amir and Snoek, Cees G. M.},
	year         = {2020},
	booktitle    = {CVPR},
	doi          = {10.1109/cvpr42600.2020.00125}
}

Abstract: The softmax and binary classifier are commonly preferred for image classification applications. However, as softmax is specifically designed for categorical classification, it assumes each image has just one class label. This limits its applicability for problems where the number of labels does not equal one, most notably zero-and multi-label problems. In these challenging settings, binary classifiers are, in theory, better suited. However, as they ignore the correlation between classes, they are not as accurate and scalable in practice. In this paper, we start from the observation that the only difference between binary and softmax classifiers is their normalization function. Specifically, while the binary classifier self-normalizes its score, the softmax classifier combines the scores from all classes before normalisation. On the basis of this observation we introduce a normalization function that is learnable, constant, and shared between classes and data points. By doing so, we arrive at a new type of binary classifier that we coin quasibinary classifier. We show in a variety of image classification settings, and on several datasets, that quasibinary classifiers are considerably better in classification settings where regular binary and softmax classifiers suffer, including zerolabel and multi-label classification. What is more, we show that quasibinary classifiers yield well-calibrated probabilities allowing for direct and reliable comparisons, not only between classes but also between data points.

BibTeX

@inproceedings{liao2020quasibinary,
	title        = {{Quasibinary Classifier for Images with Zero and Multiple Labels}},
	author       = {Liao, Shuai and Gavves, Efstratios and Oh, ChangYong and Snoek, Cees},
	year         = {2020},
	journal      = {International Conference on Pattern Recognition},
	url          = {https://isis-data.science.uva.nl/cgmsnoek/pub/liao-quasibinary-icpr2020.pdf}
}

Abstract: Stochastic neural networks with discrete random variables are an important class of models for their expressiveness and interpretability. Since direct differentiation and backpropagation is not possible, Monte Carlo gradient estimation techniques are a popular alternative. Efficient stochastic gradient estimators, such Straight-Through and Gumbel-Softmax, work well for shallow stochastic models. Their performance, however, suffers with hierarchical, more complex models. We focus on hierarchical stochastic networks with multiple layers of Boolean latent variables. To analyze such networks, we introduce the framework of harmonic analysis for Boolean functions to derive an analytic formulation for the bias and variance in the Straight-Through estimator. Exploiting these formulations, we propose FouST, a low-bias and low-variance gradient estimation algorithm that is just as efficient. Extensive experiments show that FouST performs favorably compared to state-of-the-art biased estimators and is much faster than unbiased ones.

BibTeX

@inproceedings{pervez2020low,
	title        = {{Low Bias Low Variance Gradient Estimates for Hierarchical Boolean Stochastic Networks}},
	author       = {Pervez, Adeel and Cohen, Taco and Gavves, Efstratios},
	year         = {2020},
	booktitle    = {ICML},
	url          = {http://proceedings.mlr.press/v119/pervez20a/pervez20a.pdf}
}

Abstract: Not available

BibTeX

@inproceedings{kilickaya2020self,
	title        = {{Self-Selective Context for Interaction Recognition}},
	author       = {Kilickaya, Mert and Hussein, Noureldien and Gavves, Efstratios and Smeulders, Arnold},
	year         = {2020},
	booktitle    = {ICPR}
}

Abstract: Not available

BibTeX

@inproceedings{gupta2020tackling,
	title        = {{Tackling Occlusion in Siamese Tracking with Structured Dropouts}},
	author       = {Gupta, Deepak and Gavves, Efstratios and Smeulders, Arnold},
	year         = {2020},
	booktitle    = {ICPR}
}

Abstract: Not available

BibTeX

@inproceedings{gavves2020model,
	title        = {{Model Decay in Long-Term Tracking}},
	author       = {Gavves, Efstratios and Tao, Ran and Gupta, Deepak and Smeulders, Arnold},
	year         = {2020},
	booktitle    = {ICPR}
}

Abstract: The big empirical success of group equivariant networks has led in recent years to the sprouting of a great variety of equivariant network architectures. A particular focus has thereby been on rotation and reflection equivariant CNNs for planar images. Here we give a general description of E(2)-equivariant convolutions in the framework of Steerable CNNs. The theory of Steerable CNNs thereby yields constraints on the convolution kernels which depend on group representations describing the transformation laws of feature spaces. We show that these constraints for arbitrary group representations can be reduced to constraints under irreducible representations. A general solution of the kernel space constraint is given for arbitrary representations of the Euclidean group E(2) and its subgroups. We implement a wide range of previously proposed and entirely new equivariant network architectures and extensively compare their performances. E(2)-steerable convolutions are further shown to yield remarkable gains on CIFAR-10, CIFAR-100 and STL-10 when used as drop in replacement for non-equivariant convolutions.

BibTeX

@inproceedings{Weiler2019General,
	title        = {{General E(2)-Equivariant Steerable CNNs}},
	author       = {Weiler, Maurice and Cesa, Gabriele},
	year         = {2019},
	booktitle    = {Advances in Neural Information Processing Systems},
	publisher    = {Curran Associates, Inc.},
	volume       = {32},
	pages        = {},
	url          = {https://proceedings.neurips.cc/paper/2019/file/45d6637b718d0f24a237069fe41b0db4-Paper.pdf},
	editor       = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett}
}

Abstract: We present a general theory of Group equivariant Convolutional Neural Networks (G-CNNs) on homogeneous spaces such as Euclidean space and the sphere. Feature maps in these networks represent fields on a homogeneous base space, and layers are equivariant maps between spaces of fields. The theory enables a systematic classification of all existing G-CNNs in terms of their symmetry group, base space, and field type. We also answer a fundamental question: what is the most general kind of equivariant linear map between feature spaces (fields) of given types? We show that such maps correspond one-to-one with generalized convolutions with an equivariant kernel, and characterize the space of such kernels.

BibTeX

@inproceedings{Cohen2019General,
	title        = {{A General Theory of Equivariant CNNs on Homogeneous Spaces}},
	author       = {Cohen, Taco S and Geiger, Mario and Weiler, Maurice},
	year         = {2019},
	booktitle    = {Advances in Neural Information Processing Systems},
	publisher    = {Curran Associates, Inc.},
	volume       = {32},
	url          = {https://proceedings.neurips.cc/paper/2019/file/b9cfe8b6042cf759dc4c0cccb27a6737-Paper.pdf},
	editor       = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett}
}

Abstract: This paper focuses on Bayesian Optimization (BO) for objectives on combinatorial search spaces, including ordinal and categorical variables. Despite the abundance of potential applications of Combinatorial BO, including chipset configuration search and neural architecture search, only a handful of methods have been pro- posed. We introduce COMBO, a new Gaussian Process (GP) BO. COMBO quantifies “smoothness” of functions on combinatorial search spaces by utilizing a combinatorial graph. The vertex set of the combinatorial graph consists of all possible joint assignments of the variables, while edges are constructed using the graph Cartesian product of the sub-graphs that represent the individual variables. On this combinatorial graph, we propose an ARD diffusion kernel with which the GP is able to model high-order interactions between variables leading to better performance. Moreover, using the Horseshoe prior for the scale parameter in the ARD diffusion kernel results in an effective variable selection procedure, making COMBO suitable for high dimensional problems. Computationally, in COMBO the graph Cartesian product allows the Graph Fourier Transform calculation to scale linearly instead of exponentially.We validate COMBO in a wide array of real- istic benchmarks, including weighted maximum satisfiability problems and neural architecture search. COMBO outperforms consistently the latest state-of-the-art while maintaining computational and statistical efficiency.

BibTeX

@incollection{Oh2019Combinatorial,
	title        = {{Combinatorial Bayesian Optimization using the Graph Cartesian Product}},
	author       = {Oh, Changyong and Tomczak, Jakub and Gavves, Efstratios and Welling, Max},
	year         = {2019},
	booktitle    = {Advances in Neural Information Processing Systems 32},
	publisher    = {Curran Associates, Inc.},
	pages        = {2914--2924},
	url          = {http://papers.nips.cc/paper/8557-combinatorial-bayesian-optimization-using-the-graph-cartesian-product.pdf},
	editor       = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett}
}

Abstract: Visual repetition is ubiquitous in our world. It appears in human activity (sports, cooking), animal behavior (a bee's waggle dance), natural phenomena (leaves in the wind) and in urban environments (flashing lights). Estimating visual repetition from realistic video is challenging as periodic motion is rarely perfectly static and stationary. To better deal with realistic video, we elevate the static and stationary assumptions often made by existing work. Our spatiotemporal filtering approach, established on the theory of periodic motion, effectively handles a wide variety of appearances and requires no learning. Starting from motion in 3D we derive three periodic motion types by decomposition of the motion field into its fundamental components. In addition, three temporal motion continuities emerge from the field's temporal dynamics. For the 2D perception of 3D motion we consider the viewpoint relative to the motion; what follows are 18 cases of recurrent motion perception. To estimate repetition under all circumstances, our theory implies constructing a mixture of differential motion maps: gradient, divergence and curl. We temporally convolve the motion maps with wavelet filters to estimate repetitive dynamics. Our method is able to spatially segment repetitive motion directly from the temporal filter responses densely computed over the motion maps. For experimental verification of our claims, we use our novel dataset for repetition estimation, better-reflecting reality with non-static and non-stationary repetitive motion. On the task of repetition counting, we obtain favorable results compared to a deep learning alternative.

BibTeX

@article{Runia2019Repetition,
	title        = {{Repetition Estimation}},
	author       = {Tom Runia and Cees G M Snoek and Arnold W M Smeulders},
	year         = {2019},
	journal      = {International Journal of Computer Vision},
	volume       = {127},
	number       = {9},
	pages        = {1361--1383},
	url          = {https://doi.org/10.1007/s11263-019-01194-0},
	date         = {2019-09-01},
	keywords     = {},
	pubstate     = {published},
	tppubtype    = {article}
}

Abstract: Many computer vision challenges require continuous outputs, but tend to be solved by discrete classification. The reason is classification's natural containment within a probability n-simplex, as defined by the popular softmax activation function. Regular regression lacks such a closed geometry, leading to unstable training and convergence to suboptimal local minima. Starting from this insight we revisit regression in convolutional neural networks. We observe many continuous output problems in computer vision are naturally contained in closed geometrical manifolds, like the Euler angles in viewpoint estimation or the normals in surface normal estimation. A natural framework for posing such continuous output problems are n-spheres, which are naturally closed geometric manifolds defined in the R (n+1) space. By introducing a spherical exponential mapping on n-spheres at the regression output, we obtain well-behaved gradients, leading to stable training. We show how our spherical regression can be utilized for several computer vision challenges, specifically viewpoint estimation, surface normal estimation and 3D rotation estimation. For all these problems our experiments demonstrate the benefit of spherical regression. All paper resources are available at https://github.com/leoshine/Spherical Regression.

BibTeX

@inproceedings{Liao2019Spherical,
	title        = {{Spherical regression: Learning viewpoints, surface normals and 3D rotations on n-spheres}},
	author       = {Liao, Shuai and Gavves, Efstratios and Snoek, Cees G.M.},
	year         = {2019},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2019.00999},
	isbn         = {9781728132938},
	issn         = {10636919},
	archiveprefix = {arXiv},
	arxivid      = {1904.05404},
	eprint       = {1904.05404},
	keywords     = {3D from Single Image}
}

Abstract: This paper focuses on the temporal aspect for recognizing human activities in videos; an important visual cue that has long been undervalued. We revisit the conventional definition of activity and restrict it to Complex Action: a set of one-actions with a weak temporal pattern that serves a specific purpose. Related works use spatiotemporal 3D convolutions with fixed kernel size, too rigid to capture the varieties in temporal extents of complex actions, and too short for long-range temporal modeling. In contrast, we use multi-scale temporal convolutions, and we reduce the complexity of 3D convolutions. The outcome is Timeception convolution layers, which reasons about minute-long temporal patterns, a factor of 8 longer than best related works. As a result, Timeception achieves impressive accuracy in recognizing the human activities of Charades, Breakfast Actions and MultiTHUMOS. Further, we demonstrate that Timeception learns long-range temporal dependencies and tolerate temporal extents of complex actions.

BibTeX

@inproceedings{Hussein2019Timeception,
	title        = {{Timeception for complex action recognition}},
	author       = {Hussein, Noureldien and Gavves, Efstratios and Smeulders, Arnold W.M.},
	year         = {2019},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2019.00034},
	isbn         = {9781728132938},
	issn         = {10636919},
	archiveprefix = {arXiv},
	arxivid      = {1812.01289},
	eprint       = {1812.01289},
	keywords     = {Action Recognition,Video Analytics}
}

Abstract: The principle of equivariance to symmetry transformations enables a theoretically grounded approach to neural network architecture design. Equivariant networks have shown excellent performance and data efficiency on vision and medical imaging problems that exhibit symmetries. Here we show how this principle can be extended beyond global symmetries to local gauge transformations. This enables the development of a very general class of convolutional neural networks on manifolds that depend only on the intrinsic geometry, and which includes many popular methods from equivariant and geometric deep learning. We implement gauge equivariant CNNs for signals defined on the surface of the icosahedron, which provides a reasonable approximation of the sphere. By choosing to work with this very regular manifold, we are able to implement the gauge equivariant convolution using a single conv2d call, making it a highly scalable and practical alternative to Spherical CNNs. Using this method, we demonstrate substantial improvements over previous methods on the task of segmenting omnidirectional images and global climate patterns.

BibTeX

@inproceedings{Cohen2019Gauge,
	title        = {{Gauge equivariant convolutional networks and the icosahedral CNN}},
	author       = {Cohen, Taco S. and Weiler, Maurice and Kicanaoglu, Berkay and Welling, Max},
	year         = {2019},
	booktitle    = {ICML},
	isbn         = {9781510886988},
	archiveprefix = {arXiv},
	arxivid      = {1902.04615},
	eprint       = {1902.04615}
}

Abstract: Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

BibTeX

@inproceedings{Louizos2019Relaxed,
	title        = {{Relaxed quantization for discretized neural networks}},
	author       = {Louizos, Christos and Reisser, Matthias and Blankevoort, Tijmen and Gavves, Efstratios and Welling, Max},
	year         = {2019},
	booktitle    = {ICLR},
	archiveprefix = {arXiv},
	arxivid      = {1810.01875},
	eprint       = {1810.01875}
}

Abstract: Deep neural networks are almost universally trained with reverse-mode automatic differentiation (a.k.a. backpropagation). Biological networks, on the other hand, appear to lack any mechanism for sending gradients back to their input neurons, and thus cannot be learning in this way. In response to this, Scellier \& Bengio (2017) proposed Equilibrium Propagation - a method for gradient-based training of neural networks which uses only local learning rules and, crucially, does not rely on neurons having a mechanism for back-propagating an error gradient. Equilibrium propagation, however, has a major practical limitation: inference involves doing an iterative optimization of neural activations to find a fixed-point, and the number of steps required to closely approximate this fixed point scales poorly with the depth of the network. In response to this problem, we propose Initialized Equilibrium Propagation, which trains a feedforward network to initialize the iterative inference procedure for Equilibrium propagation. This feed-forward network learns to approximate the state of the fixed-point using a local learning rule. After training, we can simply use this initializing network for inference, resulting in a learned feedforward network. Our experiments show that this network appears to work as well or better than the original version of Equilibrium propagation while requiring fewer steps to converge. This shows how we might go about training deep networks without using backpropagation.

BibTeX

@inproceedings{OConnor2019Initialized,
	title        = {{Initialized equilibrium propagation for backprop-free training}},
	author       = {O'Connor, Peter and Gavves, Efstratios and Welling, Max},
	year         = {2019},
	booktitle    = {ICLR}
}

Abstract: Backpropagation is almost universally used to train artificial neural networks. However, there are several reasons that backpropagation could not be plausibly implemented by biological neurons. Among these are the facts that (1) biological neurons appear to lack any mechanism for sending gradients backwards across synapses, and (2) biological “spiking” neurons emit binary signals, whereas back-propagation requires that neurons communicate continuous values between one another. Recently Scellier and Bengio [2017], demonstrated an alternative to backpropagation, called Equilibrium Propagation, wherein gradients are implicitly computed by the dynamics of the neural network, so that neurons do not need an internal mechanism for backpropagation of gradients. This provides an interesting solution to problem (1). In this paper, we address problem (2) by proposing a way in which Equilibrium Propagation can be implemented with neurons which are constrained to just communicate binary values at each time step. We show that with appropriate step-size annealing, we can converge to the same fixed-point as a real-valued neural network, and that with predictive coding, we can make this convergence much faster. We demonstrate that the resulting model can be used to train a spiking neural network using the update scheme from Equilibrium propagation.

BibTeX

@inproceedings{OConnor2019Training,
	title        = {{Training a spiking neural network with equilibrium propagation}},
	author       = {O'Connor, Peter and Gavves, Efstratios and Welling, Max},
	year         = {2019},
	booktitle    = {AISTATS}
}

Abstract: A major challenge in Bayesian Optimization is the boundary issue (Swersky, 2017) where an algorithm spends too many evaluations near the boundary of its search space. In this paper we propose BOCK, Bayesian Optimization with Cylindrical Kernels, whose basic idea is to transform the ball geometry of the search space using a cylindrical transformation. Because of the transformed geometry, the Gaussian Process-based surrogate model spends less budget searching near the boundary, while concentrating its efforts relatively more near the center of the search region, where we expect the solution to be located. We evaluate BOCK extensively, showing that it is not only more accurate and efficient, but it also scales successfully to problems with a dimensionality as high as 500. We show that the better accuracy and scalability of BOCK even allows optimizing modestly sized neural network layers, as well as neural network hyperparameters.

BibTeX

@inproceedings{Oh2018Bock,
	title        = {{BOCK: Bayesian Optimization with Cylindrical Kernels}},
	author       = {Oh, Chang Yong and Gavves, Efstratios and Welling, Max},
	year         = {2018},
	booktitle    = {ICML},
	isbn         = {9781510867963},
	archiveprefix = {arXiv},
	arxivid      = {1806.01619},
	eprint       = {1806.01619}
}

Abstract: Not available

BibTeX

@article{kicanaoglu2018estimating,
	title        = {{Estimating small differences in car-pose from orbits}},
	author       = {Kicanaoglu, Berkay and Tao, Ran and Smeulders, Arnold WM},
	year         = {2018},
	journal      = {BMCV},
	url          = {https://arxiv.org/abs/1809.00720}
}

Abstract: In many machine learning tasks it is desirable that a model's prediction transforms in an equivariant way under transformations of its input. Convolutional neural networks (CNNs) implement translational equivariance by construction; for other transformations, however, they are compelled to learn the proper mapping. In this work, we develop Steerable Filter CNNs (SFCNNs) which achieve joint equivariance under translations and rotations by design. The proposed architecture employs steerable filters to efficiently compute orientation dependent responses for many orientations without suffering interpolation artifacts from filter rotation. We utilize group convolutions which guarantee an equivariant mapping. In addition, we generalize He's weight initialization scheme to filters which are defined as a linear combination of a system of atomic filters. Numerical experiments show a substantial enhancement of the sample complexity with a growing number of sampled filter orientations and confirm that the network generalizes learned patterns over orientations. The proposed approach achieves state-of-the-art on the rotated MNIST benchmark and on the ISBI 2012 2D EM segmentation challenge.

BibTeX

@inproceedings{Weiler2018Learning,
	title        = {{Learning Steerable Filters for Rotation Equivariant CNNs}},
	author       = {Weiler, Maurice and Hamprecht, Fred A. and Storath, Martin},
	year         = {2018},
	month        = {June},
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}
}

Abstract: This paper strives for pixel-level segmentation of actors and their actions in video content. Different from existing works, which all learn to segment from a fixed vocabulary of actor and action pairs, we infer the segmentation from a natural language input sentence. This allows to distinguish between fine-grained actors in the same super-category, identify actor and action instances, and segment pairs that are outside of the actor and action vocabulary. We propose a fully-convolutional model for pixel-level actor and action segmentation using an encoder-decoder architecture optimized for video. To show the potential of actor and action video segmentation from a sentence, we extend two popular actor and action datasets with more than 7,500 natural language descriptions. Experiments demonstrate the quality of the sentence-guided segmentations, the generalization ability of our model, and its advantage for traditional actor and action segmentation compared to the state-of-the-art.

BibTeX

@inproceedings{Gavrilyuk2018Actor,
	title        = {{Actor and Action Video Segmentation from a Sentence}},
	author       = {Gavrilyuk, Kirill and Ghodrati, Amir and Li, Zhenyang and Snoek, Cees G.M.},
	year         = {2018},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2018.00624},
	isbn         = {9781538664209},
	issn         = {10636919},
	archiveprefix = {arXiv},
	arxivid      = {1803.07485},
	eprint       = {1803.07485}
}

Abstract: We consider the problem of estimating repetition in video, such as performing push-ups, cutting a melon or playing violin. Existing work shows good results under the assumption of static and stationary periodicity. As realistic video is rarely perfectly static and stationary, the often preferred Fourier-based measurements is inapt. Instead, we adopt the wavelet transform to better handle non-static and non-stationary video dynamics. From the flow field and its differentials, we derive three fundamental motion types and three motion continuities of intrinsic periodicity in 3D. On top of this, the 2D perception of 3D periodicity considers two extreme viewpoints. What follows are 18 fundamental cases of recurrent perception in 2D. In practice, to deal with the variety of repetitive appearance, our theory implies measuring time-varying flow Ft and its differentials \^a‡Ft, \^a‡\^A\textperiodcenteredFt and \^a‡ \~A - Ft over segmented foreground motion. For experiments, we introduce the new QUVA Repetition dataset, reflecting reality by including non-static and non-stationary videos. On the task of counting repetitions in video, we obtain favorable results compared to a deep learning alternative.

BibTeX

@inproceedings{Runia2018Real,
	title        = {{Real-World Repetition Estimation by Div, Grad and Curl}},
	author       = {Runia, Tom F.H. and Snoek, Cees G.M. and Smeulders, Arnold W.M.},
	year         = {2018},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2018.00939},
	isbn         = {9781538664209},
	issn         = {10636919},
	archiveprefix = {arXiv},
	arxivid      = {1802.09971},
	eprint       = {1802.09971}
}

Abstract: The vast majority of natural sensory data is temporally redundant. For instance, video frames or audio samples which are sampled at nearby points in time tend to have similar values. Typically, deep learning algorithms take no advantage of this redundancy to reduce computations. This can be an obscene waste of energy. We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data - not the rate at which we process the data. We do this by implementing a form of Predictive Coding wherein neurons communicate a combination of their state, and their temporal change in state, and quantize this signal using Sigma-Delta modulation. Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-fire neurons, and to a spike-timing-dependent weight-update similar to Spike-Timing Dependent Plasticity (STDP), a synaptic learning rule observed in the brain. We demonstrate that on MNIST, on a temporal variant of MNIST, and on Youtube-BB, a dataset with videos in the wild, our algorithm performs about as well as a standard deep network trained with backpropagation, despite only communicating discrete values between layers.

BibTeX

@inproceedings{OConnor2018Temporally,
	title        = {{Temporally efficient deep learning with spikes}},
	author       = {O'Connor, Peter and Gavves, Efstratios and Reisser, Matthias and Welling, Max},
	year         = {2018},
	booktitle    = {ICLR},
	archiveprefix = {arXiv},
	arxivid      = {1706.04159},
	eprint       = {1706.04159}
}

Abstract: We present VideoLSTM for end-to-end sequence learning of actions in video. Rather than adapting the video to the peculiarities of established recurrent or convolutional architectures, we adapt the architecture to fit the requirements of the video medium. Starting from the soft-Attention LSTM, VideoLSTM makes three novel contributions. First, video has a spatial layout. To exploit the spatial correlation we hardwire convolutions in the soft-Attention LSTM architecture. Second, motion not only informs us about the action content, but also guides better the attention towards the relevant spatio-temporal locations. We introduce motion-based attention. And finally, we demonstrate how the attention from VideoLSTM can be exploited for action localization by relying on the action class label and temporal attention smoothing. Experiments on UCF101, HMDB51 and THUMOS13 reveal the benefit of the video-specific adaptations of VideoLSTM in isolation as well as when integrated in a combined architecture. It compares favorably against other LSTM architectures for action classification and especially action localization.

BibTeX

@article{Li2018VideoLSTM,
	title        = {{VideoLSTM convolves, attends and flows for action recognition}},
	author       = {Li, Zhenyang and Gavrilyuk, Kirill and Gavves, Efstratios and Jain, Mihir and Snoek, Cees G.M.},
	year         = {2018},
	journal      = {CVIU},
	doi          = {10.1016/j.cviu.2017.10.011},
	issn         = {1090235X},
	archiveprefix = {arXiv},
	arxivid      = {1607.01794},
	eprint       = {1607.01794},
	keywords     = {Action recognition,Attention,LSTM,Video representation}
}

Abstract: Time-aware encoding of frame sequences in a video is a fundamental problem in video understanding. While many attempted to model time in videos, an explicit study on quantifying video time is missing. To fill this lacuna, we aim to evaluate video time explicitly. We describe three properties of video time, namely a) temporal asymmetry, b) temporal continuity and c) temporal causality. Based on each we formulate a task able to quantify the associated property. This allows assessing the effectiveness of modern video encoders, like C3D and LSTM, in their ability to model time. Our analysis provides insights about existing encoders while also leading us to propose a new video time encoder, which is better suited for the video time recognition tasks than C3D and LSTM. We believe the proposed meta-analysis can provide a reasonable baseline to assess video time encoders on equal grounds on a set of temporal-aware tasks1

BibTeX

@inproceedings{Ghodrati2018Video,
	title        = {{Video Time: Properties, encoders and evaluation}},
	author       = {Ghodrati, Amir and Gavves, Efstratios and Snoek, Cees G.M.},
	year         = {2018},
	booktitle    = {BMVC},
	archiveprefix = {arXiv},
	arxivid      = {1807.06980},
	eprint       = {1807.06980}
}

Abstract: The goal of this paper is to search and match the best rendered view of a texture-free 3D shape to an object of interest in a 2D query image. Matching rendered views of 3D shapes to RGB images is challenging because, 1) 3D shapes are not always a perfect match for the image queries, 2) there is great domain difference between rendered and RGB images, and 3) estimating the object scale versus distance is inherently ambiguous in images from uncalibrated cameras. In this work we propose a deeply learned matching function that attacks these challenges and can be used for a search engine that finds the appropriate 3D shape and matches it to objects in 2D query images. We evaluate the proposed matching function and search engine with a series of controlled experiments on the 24 most populated vehicle categories in PASCAL3D+. We test the capability of the learned matching function in transferring to unseen 3D shapes and study overall search engine sensitivity w.r.t available 3D shapes and object localization accuracy, showing promising results in retrieving 3D shapes given 2D image queries.

BibTeX

@article{Liao2018Searching,
	title        = {{Searching and matching texture-free 3D shapes in images}},
	author       = {Liao, Shuai and Gavves, Efstratios and Snoek, Cees G.M.},
	year         = {2018},
	journal      = {ICMR},
	pages        = {326--334},
	doi          = {10.1145/3206025.3206057},
	isbn         = {9781450350464},
	file         = {:C$\backslash$:/Users/admin/Downloads/LiaoICMR2018.pdf:pdf},
	keywords     = {3d shape retrieval,3d-2d matching,Texture-free}
}

Abstract: This paper aims for event recognition when video examples are scarce or even completely absent. The key in such a challenging setting is a semantic video representation. Rather than building the representation from individual attribute detectors and their annotations, we propose to learn the entire representation from freely available web videos and their descriptions using an embedding between video features and term vectors. In our proposed embedding, which we call Video2vec, the correlations between the words are utilized to learn a more effective representation by optimizing a joint objective balancing descriptiveness and predictability. We show how learning the Video2vec embedding using a multimodal predictability loss, including appearance, motion and audio features, results in a better predictable representation. We also propose an event specific variant of Video2vec to learn a more accurate representation for the words, which are indicative of the event, by introducing a term sensitive descriptiveness loss. Our experiments on three challenging collections of web videos from the NIST TRECVID Multimedia Event Detection and Columbia Consumer Videos datasets demonstrate: i) the advantages of Video2vec over representations using attributes or alternative embeddings, ii) the benefit of fusing video modalities by an embedding over common strategies, iii) the complementarity of term sensitive descriptiveness and multimodal predictability for event recognition. By its ability to improve predictability of present day audio-visual video features, while at the same time maximizing their semantic descriptiveness, Video2vec leads to state-of-the-art accuracy for both few- and zero-example recognition of events in video.

BibTeX

@article{Habibian2017Video,
	title        = {{Video2vec Embeddings Recognize Events When Examples Are Scarce}},
	author       = {Habibian, Amirhossein and Mensink, Thomas and Snoek, Cees G.M.},
	year         = {2017},
	journal      = {PAMI},
	doi          = {10.1109/TPAMI.2016.2627563},
	issn         = {01628828},
	keywords     = {Event recognition,Representation learning,Semantic video representation},
	pmid         = {27849523}
}

Abstract: This paper strives to track a target object in a video. Rather than specifying the target in the first frame of a video by a bounding box, we propose to track the object based on a natural language specification of the target, which provides a more natural human-machine interaction as well as a means to improve tracking results. We define three variants of tracking by language specification: one relying on lingual target specification only, one relying on visual target specification based on language, and one leveraging their joint capacity. To show the potential of tracking by natural language specification we extend two popular tracking datasets with lingual descriptions and report experiments. Finally, we also sketch new tracking scenarios in surveillance and other live video streams that become feasible with a lingual specification of the target.

BibTeX

@inproceedings{Li2017Tracking,
	title        = {{Tracking by natural language specification}},
	author       = {Li, Zhenyang and Tao, Ran and Gavves, Efstratios and Snoek, Cees G.M. and Smeulders, Arnold W.M.},
	year         = {2017},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2017.777},
	isbn         = {9781538604571}
}

Abstract: Event detection in unconstrained videos is conceived as a content-based video retrieval with two modalities: textual and visual. Given a text describing a novel event, the goal is to rank related videos accordingly. This task is zero-exemplar, no video examples are given to the novel event. Related works train a bank of concept detectors on external data sources. These detectors predict confidence scores for test videos, which are ranked and retrieved accordingly. In contrast, we learn a joint space in which the visual and textual representations are embedded. The space casts a novel event as a probability of pre-defined events. Also, it learns to measure the distance between an event and its related videos. Our model is trained end-to-end on publicly available EventNet. When applied to TRECVID Multimedia Event Detection dataset, it outperforms the state-of-the-art by a considerable margin.

BibTeX

@inproceedings{Hussein2017Unified,
	title        = {{Unified embedding and metric learning for zero-exemplar event detection}},
	author       = {Hussein, Noureldien and Gavves, Efstratios and Smeulders, Arnold W.M.},
	year         = {2017},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2017.225},
	isbn         = {9781538604571},
	archiveprefix = {arXiv},
	arxivid      = {1705.02148},
	eprint       = {1705.02148}
}

Abstract: Deep neural networks can be obscenely wasteful. When processing video, a convolutional network expends a fixed amount of computation for each frame with no regard to the similarity between neighbouring frames. As a result, it ends up repeatedly doing very similar computations. To put an end to such waste, we introduce Sigma-Delta networks. With each new input, each layer in this network sends a discretized form of its change in activation to the next layer. Thus the amount of computation that the network does scales with the amount of change in the input and layer activations, rather than the size of the network. We introduce an optimization method for converting any pre-trained deep network into an optimally efficient Sigma-Delta network, and show that our algorithm, if run on the appropriate hardware, could cut at least an order of magnitude from the computational cost of processing video data.

BibTeX

@inproceedings{OConnor2017Sigma,
	title        = {{Sigma-delta quantized networks}},
	author       = {O'Connor, Peter and Welling, Max},
	year         = {2017},
	booktitle    = {ICLR}
}

Abstract: The iterative nature of the expectation maximization (EM) algorithm presents a challenge for privacy-preserving estimation, as each iteration increases the amount of noise needed. We propose a practical private EM algorithm that overcomes this challenge using two innovations: (1) a novel moment perturbation formulation for differentially private EM (DP-EM), and (2) the use of two recently developed composition methods to bound the privacy “cost” of multiple EM iterations: the moments accountant (MA) and zero-mean concentrated differential privacy (zCDP). Both MA and zCDP bound the moment generating function of the privacy loss random variable and achieve a refined tail bound, which effectively decrease the amount of additive noise. We present empirical results showing the benefits of our approach, as well as similar performance between these two composition methods in the DP-EM setting for Gaussian mixture models. Our approach can be readily extended to many iterative learning algorithms, opening up various exciting future directions.

BibTeX

@article{Park2017DP,
	title        = {{DP-EM: Differentially private expectation maximization}},
	author       = {Park, Mijung and Foulds, James and Chaudhuri, Kamalika and Welling, Max},
	year         = {2017},
	journal      = {AISTATS},
	archiveprefix = {arXiv},
	arxivid      = {1605.06995},
	eprint       = {1605.06995},
	file         = {:C$\backslash$:/Users/admin/Downloads/1605.06995.pdf:pdf}
}

Abstract: Bayesian inference has great promise for the privacy-preserving analysis of sensitive data, as posterior sampling automatically preserves differential privacy, an algorithmic notion of data privacy, under certain conditions (Dimitrakakis et al., 2014; Wang et al., 2015b). While this one posterior sample (OPS) approach elegantly provides privacy "for free, " it is data inefficient in the sense of asymptotic relative efficiency (ARE). We show that a simple alternative based on the Laplace mechanism, the workhorse of differential privacy, is as asymptotically efficient as non-private posterior inference, under general assumptions. This technique also has practical advantages including efficient use of the privacy budget for MCMC. We demonstrate the practicality of our approach on a time-series analysis of sensitive military records from the Afghanistan and Iraq wars disclosed by the Wikileaks organization.

BibTeX

@inproceedings{Foulds2016On,
	title        = {{On the theory and practice of privacy-preserving Bayesian data analysis}},
	author       = {Foulds, James and Geumlek, Joseph and Welling, Max and Chaudhuri, Kamalika},
	year         = {2016},
	booktitle    = {UAI},
	isbn         = {9781510827806},
	archiveprefix = {arXiv},
	arxivid      = {1603.07294},
	eprint       = {1603.07294}
}

Abstract: In this paper we present a tracker, which is radically different from state-of-the-art trackers: we apply no model updating, no occlusion detection, no combination of trackers, no geometric matching, and still deliver state-of-theart tracking performance, as demonstrated on the popular online tracking benchmark (OTB) and six very challenging YouTube videos. The presented tracker simply matches the initial patch of the target in the first frame with candidates in a new frame and returns the most similar patch by a learned matching function. The strength of the matching function comes from being extensively trained generically, i.e., without any data of the target, using a Siamese deep neural network, which we design for tracking. Once learned, the matching function is used as is, without any adapting, to track previously unseen targets. It turns out that the learned matching function is so powerful that a simple tracker built upon it, coined Siamese INstance search Tracker, SINT, which only uses the original observation of the target from the first frame, suffices to reach state-of-theart performance. Further, we show the proposed tracker even allows for target re-identification after the target was absent for a complete video shot.

BibTeX

@inproceedings{Tao2016Siamese,
	title        = {{Siamese instance search for tracking}},
	author       = {Tao, Ran and Gavves, Efstratios and Smeulders, Arnold W.M.},
	year         = {2016},
	booktitle    = {CVPR},
	doi          = {10.1109/CVPR.2016.158},
	isbn         = {9781467388504},
	issn         = {10636919},
	archiveprefix = {arXiv},
	arxivid      = {1605.05863},
	eprint       = {1605.05863}
}

Abstract: Iteratively reweighted least squares (IRLS) is a widely-used method in machine learning to estimate the parameters in the generalised linear models. In particular, IRLS for L1 minimisation under the linear model provides a closed-form solution in each step, which is a simple multiplication between the inverse of the weighted second moment matrix and the weighted first moment vector. When dealing with privacy sensitive data, however, developing a privacy preserving IRLS algorithm faces two challenges. First, due to the inversion of the second moment matrix, the usual sensitivity analysis in differential privacy incorporating a single datapoint perturbation gets complicated and often requires unrealistic assumptions. Second, due to its iterative nature, a significant cumulative privacy loss occurs. However, adding a high level of noise to compensate for the privacy loss hinders from getting accurate estimates. Here, we develop a practical algorithm that overcomes these challenges and outputs privatised and accurate IRLS solutions. In our method, we analyse the sensitivity of each moments separately and treat the matrix inversion and multiplication as a post-processing step, which simplifies the sensitivity analysis. Furthermore, we apply the \$\backslash$it\concentrated differential privacy\\ formalism, a more relaxed version of differential privacy, which requires adding a significantly less amount of noise for the same level of privacy guarantee, compared to the conventional and advanced compositions of differentially private mechanisms.

BibTeX

@article{Park2016Note,
	title        = {{A note on privacy preserving iteratively reweighted least squares}},
	author       = {Park, Mijung and Welling, Max},
	year         = {2016},
	journal      = {ICMLW},
	pages        = {1--5},
	url          = {http://arxiv.org/abs/1605.07511},
	archiveprefix = {arXiv},
	arxivid      = {1605.07511},
	eprint       = {1605.07511}
}