Loading...
Vacancies

Master Thesis Intern | On the Generalizability of Causal Representations

Phillip Lippe, Sara Magliacane

Project period: Nov 2022 - Jun 2023
Application deadline: October 10, 2022

On the Generalizability of Causal Representations

Causal Representation Learning (CRL) is the task of identifying the underlying latent causal structure from high-dimensional inputs such as images or videos [Schölkopf et al., 2021]. Obtaining such causal representations has the promise of enabling Deep Learning models to reason about causal questions (“why did A happens?”, “what would have changed if B happened?”, etc.) and allow for more efficient and generalizable algorithms. For example, in the Atari environment Pong, we are given a sequence of images, from which we want to identify the ball, the player’s paddle, the opponent’s paddle, and the score as the causal variables of the environment that affect our game. Once the variables and their causal graph have been identified, we could use the learned representation to answer questions such as “How can I influence my score?”, “Why did I lose?”, etc.
So far, most works in causal representation learning focused on identifying the causal structure of a single environment. However, humans arguably reuse prior experience to quickly and accurately estimate causal relations in new settings. This project aims at investigating the generalizability of causal representations learned by neural networks. Thereby, we will focus on CITRIS [Lippe et al., 2022], a recent CRL method for videos with interventions. CITRIS disentangles the latent representation of a pretrained autoencoder into multidimensional causal factors, allowing it to eventually generalize to more settings for which only observational data is available. In some preliminary experiments, Lippe et al. showed that CITRIS can generalize to unseen object shapes, but a thorough investigation on its limitations and its precise capabilities is missing. This project will try to fill this gap by experimenting with new instances of causal factors, but also varying number of variables and factors. The overall concept of the project is reaching beyond a single architecture, and will be important for the whole research field of causal representation learning. Depending on the success, we can also reach for more ambitious generalization experiments, such as using autoencoders pretrained on ImageNet. Finally, a general question could be how we could create a good benchmark for 'generalizing causal representations' across different domains/environments, and what potential metrics are.

References:
[Schölkopf et al., 2021] Schölkopf, Bernhard, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 'Toward causal representation learning.' Proceedings of the IEEE 109, no. 5 (2021). (link)
[Lippe et al., 2022] Lippe, Phillip, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Stratis Gavves. 'CITRIS: Causal Identifiability from Temporal Intervened Sequences.' In International Conference on Machine Learning. PMLR, 2022. (
link)

Work Environment

The thesis will be jointly supervised by Phillip Lippe and Sara Magliacane with weekly meetings, and the student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

This project aims at submitting a research paper to one of the top-tier AI/ML conferences (NeurIPS, ICML, ICLR) in case of successful progress. You are expected to propose and implement novel methods for evaluating and improving the learning of generalizable causal representations. We are looking for students with exceptional coding skills in PyTorch or JAX, experience in tools such as git, and a strong study record in core courses such as Machine Learning 1&2 and Deep Learning. In-depth knowledge of causality, for example by taking the causality course, is a plus, but basic knowledge is sufficient for starting the project. In-depth knowledge of causal representation learning will be gained in the first weeks of the project by reading papers and discussions/explanations of your supervisors.

Contact

In case you are interested in the project, would like to chat about the project or have questions, please write an email to Phillip (p.lippe@uva.nl) and Sara (s.magliacane@uva.nl). The application deadline for the project is October 10. Please include your CV and your current study record in your application.

Master Thesis Intern | Hierarchical Causal Representation Learning

Phillip Lippe, Sara Magliacane

Project period: Nov 2022 - Jun 2023
Application deadline: October 10, 2022

Hierarchical Causal Representation Learning

Learning the causal factors of an environment from high-level observations like images is a crucial step towards understanding and reasoning of agents [Schölkopf et al., 2021]. For example, from images of a robotic environment with several objects, we first need to identify objects as abstract variables and their interactions. As shown in CITRIS [Lippe et al., 2022], one can identify multidimensional causal factors, i.e. causal variables that span over multiple dimensions, when we have access to interventions. In a realistic setup, we may not be able to directly perform interventions on atomic causal variables, but rather start with high-level actions that affect multiple causal variables at once. For example, a robotic arm might first be able to do coarse actions, such as hit multiple objects at once or only influence a subset of variables. From these interactions, one can block-identify causal variables, which means that we identify groups of causal variables, which, however, are not disentangled within the groups. For example, in the Atari environment Pong, we could block-identify the position of the player’s and the opponent's paddle as 2 latent variables, but would not know which dimension describes which paddle, or whether they describe some arbitrary combinations of both. Based on this initial disentanglement, the robotic system may be able to learn more precise skills and start influencing single causal variables. We then have to redefine our representation and disentangle the individual groups of causal variables. This overall leads to learning a hierarchical causal representation, adapted by the available skills at a time. To this end, there exists no method that allows/implements such a hierarchical learning structure.
This Master thesis project aims at developing a novel approach for learning hierarchical causal representations from interacting systems. The project is not limited to specific environments and can be specialized to the student’s interests. Further, the project will initially focus on empirical model development and experimentation, while a theoretical grounding of the approach can be included as well.

References:
[Schölkopf et al., 2021] Schölkopf, Bernhard, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 'Toward causal representation learning.' Proceedings of the IEEE 109, no. 5 (2021). (link)
[Lippe et al., 2022] Lippe, Phillip, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Stratis Gavves. 'CITRIS: Causal Identifiability from Temporal Intervened Sequences.' In International Conference on Machine Learning. PMLR, 2022. (
link)

Work Environment

The thesis will be jointly supervised by Phillip Lippe and Sara Magliacane with weekly meetings, and the student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

This project aims at submitting a research paper to one of the top-tier AI/ML conferences (NeurIPS, ICML, ICLR) in case of successful progress. You are expected to propose and implement novel methods for hierarchical causal representations. We are looking for students with exceptional coding skills in PyTorch or JAX, experience in tools such as git, and a strong study record in core courses such as Machine Learning 1&2 and Deep Learning. In-depth knowledge of causality, for example by taking the causality course, is a plus, but basic knowledge is sufficient for starting the project. In-depth knowledge of causal representation learning will be gained in the first weeks of the project by reading papers and discussions/explanations of your supervisors.

Contact

In case you are interested in the project, would like to chat about the project or or have questions, please write an email to Phillip (p.lippe@uva.nl) and Sara (s.magliacane@uva.nl). The application deadline for the project is October 10. Please include your CV and your current study record in your application.

Master Thesis Intern | Does 3D modeling help image representation learning?

Pengwan Yang, Yuki Asano

Project period: Nov 2022 - Jun 2023

Deep learning has achieved remarkable gains in 3D modelling recently with NeRF and similar works. The first of these works focus on novel view generation, in which new views can be generated from a set of existing photographs [1]. While these make for stunning visual demonstrations, so far their use has been limited, mainly due to the harsh requirements (knowing of camera location etc). In this project, our goal is to find out whether these models can be used for the task of self-supervised representation learning. In self-supervised representation learning, the goal is typically to obtain strong visual encoders by learning from raw data alone, without using labels. This field has seen great progress in recent years, e.g. works like DINO/MAE [2]. The idea of this project is to leverage these NeRF methods as additional augmentations to self-supervised learning methods. For this, we will leverage recent work that can generate 3D views from a single image and use these to learn representations self-supervisedly. We will study the task of single-image representation learning [3], in which neural networks are learned from scratch using self-supervised learning methods and only a single source image. This has the benefits of isolating the effect of dataset size and augmentations and 3D modelling, and also fast training time.

References:
[1] Nerf: Representing scenes as neural radiance fields for view synthesis.Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. ECCV, 2022.
[2] Emerging properties in self-supervised vision transformers. Caron, Mathilde, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. ICCV, 2021.
[3] A critical analysis of self-supervision, or what we can learn from a single image. Asano, Yuki M., Christian Rupprecht, and Andrea Vedaldi. ICLR, 2020.

Work Environment

The thesis will be jointly supervised by Pengwan Yang and Yuki Asano with weekly or biweekly meetings, and the student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

The aim will be to publish this work at a top ML/CV venue such as CVPR/ICCV/ICML/NeurIPS. The student should be well familiar with Python and PyTorch and have background knowledge in Deep Learning and Computer Vision. Further, the student should be eager to do research.

Contact

In case you are interested in the project or have questions, please write an email to Pengwan (p.yang3@uva.nl).

Master Thesis Intern | Self-Supervised Learning on Point Clouds by Free Form Deformation

Pengwan Yang, Yuki Asano

Project period: Nov 2022 - Jun 2023

Recognizing point cloud objects is important in countless applications, e.g. self-driving cars and robotics. Recently, deep learning models have shown promising results on various point cloud tasks using classic, supervised learning techniques. While large point cloud datasets can be captured fast and cheaply using modern sensors, manual labeling point clouds is expensive. A potential solution is self-supervised learning, in which neural networks are learnt with the raw data alone and which has shown great success in computer vision (mainly in video and image domains). Self-supervised learning methods typically rely on augmentations or input transformations to produce different views of datapoints that can be leveraged for learning without labels. Yet augmentation methods for point clouds are still in their infancy and very limited [1,2]. The goal of this project is to apply state of the art self-supervised learning methods on pointclouds and analyze their performance with regards to various augmentation techniques applied. The main focus of this project is to adapt a geometric computer graphics technique from the 80’s called Free Form Deformation (FFD) into a novel augmentation method for point clouds [3]. FFD is based on the idea of enclosing an object within a cube or another hull object, and transforming the object within the hull as the hull is deformed. This will likely keep the semantic information invariant, while augmenting the individual points clouds enough to generate useful variation. The goal of this project is thus to develop a technique that will be useful for many subsequent works that learn from point clouds.

References:
[1] Self-supervised deep learning on point clouds by reconstructing space. Sauder, Jonathan, and Bjarne Sievers. NeurIPS, 2019.
[2] PointGLR: Unsupervised Structural Representation Learning of 3D Point Clouds. Rao, Yongming, Jiwen Lu, and Jie Zhou. TPAMI, 2022.
[3] Free-form deformation of solid geometric models. Sederberg, Thomas W., and Scott R. Parry. Computer graphics and interactive techniques, 1986.

Work Environment

The thesis will be jointly supervised by Pengwan Yang and Yuki Asano with weekly or biweekly meetings, and the student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

The aim will be to publish this work at a top ML/CV venue such as CVPR/ICCV/ICML/NeurIPS. The student should be well familiar with Python and PyTorch and have background knowledge in Deep Learning and Computer Vision. Further, the student should be eager to do research.

Contact

In case you are interested in the project or have questions, please write an email to Pengwan (p.yang3@uva.nl).

Master Thesis Intern | Dense self-supervised test-time training for recycling large-scale pretrained visual encoders

Mohammadreza Salehi, Yuki Asano

Project period: Nov 2022 - Jun 2023

Self-supervision based methods have achieved astonishing results in image classification tasks. Thanks to models such as MAE [1] and DINO [2], we can now surpass supervised training in many downstream tasks, such as efficient object recognition or semantic segmentation. It is clear that these large-scale trained models contain many highly useful features in their outputs, as evidenced by their general applicability. Yet many of these features will be irrelevant for tasks sufficiently divorced from the original training dataset, e.g. an image of a kitchen scene, will not require all the fine-grained dog-breed discriminating features. Recently, works such as Leopart [3] have leveraged such general representations to fine-tune the networks in a spatially-dense, self-supervised manner, resulting in self-supervised semantic segmentation. In this project, we wish to go beyond [3] and develop a method to adapt these pretrained models for a dense prediction task using only a single training sample and no labels. While this might sound unreasonable, single-image based training [4] has shown success and is highly related to test-time-training, where networks are continuously fine-tuned after training, thus better adapting to novel domains. The resulting models will yield per-image self-supervised segmentation maps, which will allow for the setup of a novel benchmark for evaluating self-supervised models’ adaptability. The experimental setup would be similar to Leopart [1] and will be done on datasets like COCO, Pascal VOC and DAVIS.

References:
[1] He et al. Masked autoencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[2] Caron et al. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
[3] Ziegler and Asano. Self-Supervised Learning of Object Parts for Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[4] Asano, and Saeed. Extrapolating from a Single Image to a Thousand Classes using Distillation. arXiv preprint arXiv:2112.00725 (2021).

Work Environment

The thesis will be jointly supervised by Mohammadreza Salehi and Yuki Asano with weekly or biweekly meetings, and the student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

The aim will be to publish this work at a top ML/CV venue such as CVPR/ICCV/ICLR/NeurIPS. The student will be advised by the supervisor on a weekly or at least bi-weekly basis.

Contact

In case you are interested in the project or have questions, please write an email to Mohammadreza (s.salehidehnavi@uva.nl).

Master Thesis Intern | Spatial prompt tuning for temporal action recognition

Mohammadreza Salehi, Yuki Asano

Project period: Nov 2022 - Jun 2023

Video representation learning such as [1, 2] has achieved remarkable performances by training on large datasets of hundreds of thousands of videos. Yet, models such as CLIP [3] show that many video-related visual tasks such as action recognition or video-text retrieval can be well achieved with strong image-based encoders. Recently, [4] has shown the fantastic abilities of prompt tuning for manipulating the knowledge of a pre-trained model to tackle transfer learning or domain adaptation tasks. On the other hand, [5] has shown that action recognition could be efficiently addressed by simply making a big image grid (super image) out of sampled frames of a training video and using image-based vision transformers. In this research, we aim to investigate if it is possible to design a prompt tuning method to employ CLIP’s knowledge on such super images and improve the performance on action recognition tasks. If successful, the resulting method will be a simple drop-in replacement that will provide generalist image-text models such as CLIP the ability to understand videos.

References:
[1] On Compositions of Transformations in Contrastive Self-Supervised Learning. Patrick and Asano et al. ICCV 2021
[2] A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning. Feichtenhofer et al. CVPR 2021
[3] Learning Transferable Visual Models From Natural Language Supervision. Radford et al. ICML 2021.
[4] Lester, Brian, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021).
[5] Fan, Quanfu, and Rameswar Panda. An Image Classifier Can Suffice For Video Understanding. arXiv preprint arXiv:2106.14104 (2021).

Work Environment

The thesis will be jointly supervised by Mohammadreza Salehi and Yuki Asano with weekly or biweekly meetings, and the student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

The aim will be to publish this work at a top ML/CV venue such as CVPR/ICCV/ICLR/NeurIPS. The student will be advised by the supervisor on a weekly or at least bi-weekly basis.

Contact

In case you are interested in the project or have questions, please write an email to Mohammadreza (s.salehidehnavi@uva.nl).

Master Thesis Intern | Differential Privacy guarantees in Deep Learning

Rob Romijnders

Project period: Nov 2022 - Jun 2023

Machine learning models find increasing use in everyday society. Even in secure environments, sensitive data will be used to train neural networks. This project focuses on the privacy aspects of training neural networks. Existing literature has shown that actual training data can be reconstructed from neural networks and/or their gradients[1]. The research project addresses this vulnerability. The first question is how Differential Privacy (DP) provides protection against this vulnerability [2]. The second question is to quantify the DP guarantee, and compare against existing standards[3]. Finally, the challenge of this research project is to design a novel algorithm that achieves better performance at tighter (better) privacy guarantees.

References:
[1] Chen et al. 2021 Understanding Training-Data Leakage from Gradients in Neural Networks; Zhu et al. 2021 Recursive gradient attach on privacy
[2] Abadi et al. 2016 Deep Learning with Differential Privacy; Dwork, Roth Algorithmic Foundations of Differential Privacy
[3] Apple research 2017 Learning with Privacy at Scale

Work Environment

This project will be performed within the AMLab/QUVA group at UvA and supervised by ir. Rob Romijnders. We will have regular meetings (e.g. weekly or biweekly as required). The student might join the QUVA lab as an intern.

Expectations

Outcomes of this project could be published in Privacy workshops in top-tier conferences such as NeurIPS, ICLR, AISTATS, or ICML. Students are expected to have experience with neural networks and basic probability theory.

Contact

Reach out to romijndersrob@gmail.com for more information, to chat about my supervision style, and how you envision this project.

Master Thesis Intern | Sharing deep learning models with privacy guarantees

Rob Romijnders

Project period: Nov 2022 - Jun 2023

Machine learning models find increasing use in everyday society. Even in secure environments, sensitive data will be used to train neural networks. This project focuses on the privacy aspects of sharing such trained neural networks, “How can neural networks be publicly shared without implicating the training data?”. Existing literature has shown that samples can be reconstructed from the trained networks and/or its gradients [1], and differential Privacy provides protection against such attacks [2]. However, the most cited approach to DP in deep learning today turns out to be quite inefficient [3]. This approach assumes that gradients of the learning procedure are vulnerable to attacks and thus protects against this. In real life, however, neural networks are trained in secure environments (data centres, protected on-premise hardware). Thus releasing models only needs the protection of a fully trained model.

References:
[1] Chen et al. 2021 Understanding Training-Data Leakage from Gradients in Neural Networks; Zhu et al. 2021 Recursive gradient attach on privacy
[2] Zhang et al. 2022 A Survey on Gradient Inversion: Attacks, Defenses and Future Directions
[3] Abadi et al. 2016 Deep Learning with Differential Privacy

Work Environment

This project will be performed within the AMLab/QUVA group at UvA and supervised by ir. Rob Romijnders. We will have regular meetings (e.g. weekly or biweekly as required). The student might join the QUVA lab as an intern.

Expectations

The outcome of the project could have huge benefits to sharing models online (such as industrial models, or medical models). We expect to publish the research results on top-tier conferences like ICLR, NeurIPS, or AISTATS.

Contact

Reach out to romijndersrob@gmail.com for more information, to chat about my supervision style, and how you envision this project.

Details to the application process can be found here: Datanose

Master Thesis Intern | Contact tracing in a COVID-like pandemic: about privacy

Rob Romijnders

Project period: Nov 2022 - Jun 2023

The COVID19 pandemic caused major trouble in 2020 and 2021. One option to mitigate such a pandemic is contact tracing the virus. By knowing contacts between people and the test outcomes, statistical algorithms can infer who has a chance for COVID. In this project, we address the problem of privacy in such contact tracing: one wants to prevent at all costs that COVID scores of individuals become freely available. Differential privacy (DP) provides a guaranteed safeguard against this vulnerability. However, the requirements for DP will worsen the inference quality (and thus ability to mitigate a pandemic). The research question is thus “what is the best DP guarantee for an algorithm that can successfully infer virus contamination and mitigate the pandemic?”.

References:
[1] Herbrich et al. 2020 Crisp: A probabilistic model for individual-level covid-19 infection risk estimation based on contact data
[2] Baker et al. 2021 Epidemic mitigation by statistical inference from contact tracing data
[3] Abadi et al. 2016 Deep Learning with Differential Privacy
[4] Dwork, Roth Algorithmic Foundations of Differential Privacy

Work Environment

This project will be performed within the AMLab/QUVA group at UvA and supervised by ir. Rob Romijnders. We will have regular meetings (e.g. weekly or biweekly as required). The student might join the QUVA lab as an intern.

Expectations

This project involves analysing statistical algorithms [1,2] and developing privacy guarantees [3,4]. As such, students are expected to be familiar with statistics (expected values, mean, variance), and probability theory. Experience with inference algorithms like variational inference, MCMC, and belief propagation is a plus. The project has high visibility due to the impact of these pandemics. As such, we aim to submit this research to top-tier conferences like NeurIPS and ICML, or a science journal such as Nature. Students are expected to have familiarity with probability theory, statistics, and an interest in privacy.

Contact

Reach out to romijndersrob@gmail.com for more information, to chat about my supervision style, and how you envision this project.

Details to the application process can be found here: Datanose

Master Thesis Intern | Adaptive sampling for continuous Group Equivariant CNNs

Gabriele Cesa

Project period: Nov 2022 - Jun 2023

Group-Equivariant CNNs [1] (GCNNs) are generalizations of convolutional neural networks (CNNs) beyond translations to other groups of transformations (e.g. rotations). A GCNN is equivariant to a desired group of transformations, i.e. if the input of the network transforms (e.g. the input image is rotated or translated), its output transforms accordingly (think about an image segmentation task). Equivariance is a powerful inductive bias that allows a neural network to exploit the symmetries in the data, with no need to learn them (e.g. a rotation-equivariant model generalizes to rotated versions of the same image even if they have not been observed during training). In a CNN, a feature (i.e., a channel) is a function over all translations (the value stored at a pixel is the response of the filter translated to that location). Analogously, in a GCNN, a feature is a function over the elements of the group. The most interesting groups are continuous infinite groups (like the rotations in the 3D space). To be able to work with these groups, it is often necessary to perform some form of discretization by sampling features on a finite number of points [2,3] (e.g., in a CNN, images and features can be imagined as continuous functions on the plane which are sampled on the pixel grid). Such discretization generally harms the equivariance of the model to all continuous transformations (a CNN is not equivariant to sub-pixel translations) and requires a large number of samples to reduce this problem. The underlying issue is that the sampling points must be chosen in advance. Instead, by adapting (i.e., transforming) the sampling points when the input function is transformed, the measured values will not change. This allows for perfectly equivariant computations. Check this video for a visualization of this. In this project, you will study how the sampling points can be conditioned on the input of a GCNN to generate an adaptive grid and ensure all computations are perfectly equivariant. In particular, the project will focus on the group of 3D rotations, possibly with applications on molecular data (e.g. chemical property prediction).

References:
[1] Cohen, T., & Welling, M. Group equivariant convolutional networks. In International conference on machine learning, 2016.
[2] Cohen, Taco S., et al. Spherical CNNs. International Conference on Learning Representations, 2018.
[3] Bekkers, Erik J. B-Spline CNNs on Lie groups. International Conference on Learning Representations. 2019.

Work Environment

This project will be performed within the AMLab/QUVA group at UvA and supervised by Gabriele Cesa. We will have regular one-on-one meetings (ideally weekly). Additional meetings can be scheduled ad-hoc as required. The student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

The proposed research has the potential for publication. The student is expected to be comfortable working with advanced mathematics, primarily for understanding and converting math to code. Knowledge of group/representation theory is not a necessary prerequisite but the student is expected to learn about these advanced topics during the first part of the project.

Contact

You can contact Gabriele via g.cesa@uva.nl.

Details to the application process can be found here: Datanose

Master Thesis Intern | Learning the Degree of Equivariance in Steerable CNNs

Gabriele Cesa

Project period: Nov 2022 - Jun 2023

Group-Equivariant CNNs [1] (GCNNs) are generalizations of convolutional neural networks (CNNs) beyond translations to other groups of transformations (e.g. rotations). A GCNN is equivariant to a desired group of transformations, i.e. if the input of the network transforms (e.g. the input image is rotated or translated), its output transforms accordingly (think about an image segmentation task). Equivariance is a powerful inductive bias that allows a neural network to exploit the symmetries in the data, with no need to learn them from scratch (e.g. a rotation-equivariant model generalizes to rotated versions of the same image even if they have not been observed during training). Sometimes, the symmetry group to consider is not known a priori. For instance, different symmetries appear at different scales in natural images: while local patterns often have rotational symmetry, the whole image is not globally symmetric to rotations (e.g. an object can appear in any rotation within a photo, but the sky is always above and gives a fixed orientation of the photo). Similarly, when processing a 3D scan of a brain, local patterns can appear in any rotation but the whole brain has precise orientation with respect to the human body. Since each layer of a CNN has a different receptive field, the equivariance degree of each layer can be adapted to the symmetries appearing at its own scale. In this work, we want to explore some strategies to automatically determine the optimal equivariance level of a neural network module. It seems convenient to approach a relaxed version of this problem: enforce equivariance to a large symmetry group but allow the model to partially break equivariance with some penalty.
In this project, we will rely on the framework of Steerable CNNs [2, 3], which allows one to easily implement a wide variety of group equivariant architectures. This is achieved by explicitly restricting the learnable filters of a CNN to a subspace of equivariant filters, where equivariance to a larger group generally constrains the filters to live in a smaller subspace (hence, enforcing a stronger weight sharing).

References:
[1] Cohen, T., & Welling, M. (2016). Group equivariant convolutional networks. International Conference on Machine Learning.
[2] Cohen, T., & Welling, M. (2017). Steerable CNNs. International Conference on Learning Representations.
[3] Weiler, M., & Cesa, G. (2019). General E(2)-Equivariant Steerable CNNs. Advances in Neural Information Processing Systems.

Work Environment

This project will be performed within the AMLab/QUVA group at UvA and supervised by Gabriele Cesa. We will have regular one-on-one meetings (ideally weekly). Additional meetings can be scheduled ad-hoc as required. The student will join the QUVA lab, a collaboration between UvA and Qualcomm, as an intern. To become an intern, you will need to agree to a non-disclosure form that gives Qualcomm the right to patent the intellectual property that comes out of this project, but does not restrict your research opportunities. In return, you will receive an internship allowance of 200€ per month as compensation and additional compensation in case of a successful patent application of the thesis work.

Expectations

The proposed research has the potential for publication. The student is expected to be comfortable working with advanced mathematics, primarily for understanding and converting math to code. Knowledge of group/representation theory is not a necessary prerequisite but the student is expected to learn about these advanced topics during the first part of the project.

Contact

You can contact Gabriele via g.cesa@uva.nl.

Details to the application process can be found here: Datanose

Master Thesis Intern | Program synthesis and graph neural networks

Natasha Butt

Project period: Nov 2022 - Jun 2023

Program synthesis refers to a class of techniques that generate programs given a specification of semantic and syntactically requirements. This is an active area of research with papers published yearly in machine learning conferences (NeurIPS, ICLR, ICML), as well as programming systems and formal methods conferences. Exciting recent publications include [1], presenting OpenAI's Codex, where large language models are used to generate code from text prompts. There are also significant commercial applications of program synthesis, for example, FlashFill, a feature of Excel, performs string processing tasks through programming-by-examples [2]. Generally, program synthesis involves search in a language for programs that meet a specification. For example, in programming-by-examples, this specification is a set of input-output examples, and so synthezisers are searching for programs that generate a given output when executed on a given input. For more background on program synthesis, the lecture notes of [3] and the introduction of [4] are recommended.
Neuro-symbolic program synthesis, using deep learning methods for program synthesis, is an exciting and rapidly growing field. Generally, neural networks used in the literature tend to be recurrent or transformer models, where programs are represented as strings. However, string representations do not fully reflect the grammar of the language that programs are written in. As a result, graph neural networks with tree representations of programs may better represent program structure, however, these struggle with sequential reasoning which will be important for predicting the next term in a program. Recent work in neuro-symbolic program synthesis has looked at combining models for sequential data and graph neural networks such as Instruction Pointer Attention Graph Neural Networks [5], Graph Sandwiches [6] and Graph Relational Embedding Attention Transformers [6]. Further, [7] recently released their open-source Python library for creating, amongst other things, representations of Python programs as abstract syntax trees (ASTs). This project will explore the use of graph neural networks in program synthesis, with the aim of outperforming existing benchmarks based on recurrent or transformer models.

References:
[1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka-plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri,G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray,N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati,K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code; 2021.
[2] S. Gulwani, “Automating string processing in spreadsheets using input-output examples, 2011
[3] A. Solar-Lezama, “Introduction to program synthesis,” 2018.
[4] M. Nye, “Search and representation in program synthesis,” 2022.
[5] D. Bieber, C. Sutton, H. Larochelle, and D. Tarlow, “Learning to execute programs with instruction pointer attention graph neural networks,” 2020.
[6] V. J. Hellendoorn, C. Sutton, R. Singh, P. Maniatis, and D. Bieber, “Global relational models of source code,” 2020.
[7] D. Bieber, K. Shi, P. Maniatis, C. Sutton, V. Hellendoorn, D. Johnson, and D. Tarlow, “A library for representing python programs as graphs for machine learning,” 2022.

Work Environment

This project will be performed within the AMLab group at UvA with the option of also becoming a paid QUVA intern. Meetings will be in person research meetings at Lab42 up to once a week. The advisor will be Natasha Butt. Natasha Butt is a second year PhD student in QUVA Lab (Qualcomm and UvA) working on unsupervised learning for source compression with Max Welling, Taco Cohen and Auke Wiggers.

Expectations

Students are welcomed and encouraged to contribute their own ideas to the direction of this project. Prior knowledge of deep learning is useful. No prior knowledge of program synthesis is necessary. This research is expected to be publishable in a top tier machine learning conference workshop.

Contact

You can contact Natasha via n.e.butt@uva.nl.

Master Thesis Intern | Program synthesis and reinforcement learning

Natasha Butt, Matthew Macfarlane

Project period: Nov 2022 - Jun 2023

Program synthesis refers to a class of techniques that generate programs given a specification of semantic and syntactically requirements. This is an active area of research with papers published yearly in machine learning conferences (NeurIPS, ICLR, ICML), as well as programming systems and formal methods conferences. Exciting recent publications include [1], presenting OpenAI's Codex, where large language models are used to generate code from text prompts. There are also significant commercial applications of program synthesis, for example, FlashFill, a feature of Excel, performs string processing tasks through programming-by-examples [2]. Generally, program synthesis involves search in a language for programs that meet a specification. For example, in programming-by-examples, this specification is a set of input-output examples, and so synthezisers are searching for programs that generate a given output when executed on a given input. For more background on program synthesis, the lecture notes of [3] and the introduction of [4] are recommended.
Reinforcement learning is a promising approach to program synthesis where a policy can be trained without requiring labelled examples. The policy can simply interact with problems and be trained to maximise likelihood of solving tasks from a particular distribution. However, a particular challenge of train-ing a policy using reinforcement learning for program synthesis is that these problems are inherently sparse reward (reward of 1 for correct programs, 0 otherwise). A variety of methods have been proposed in the literature to deal with this. These include storing a replay buffer of previously found solutions[5][6], making use of policy improvement operators [7], or training via curriculum learning [8][9]. In this thesis, the student would investigate reinforcement learning methods that can cope with these extremely sparse environments, with a focus to outperforming current baselines in program synthesis.

References:
[1] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Ka-plan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri,G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray,N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati,K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code; 2021.
[2] S. Gulwani, “Automating string processing in spreadsheets using input-output examples, 2011
[3] A. Solar-Lezama, “Introduction to program synthesis,” 2018.
[4] M. Nye, “Search and representation in program synthesis,” 2022.
[5] C. Liang, M. Norouzi, J. Berant, Q. V. Le, and N. Lao, Memory augmented policy optimization for program synthesis and semantic parsing, Advances in Neural Information Processing Systems, vol. 31, 2018.
[6] D. A. Abolafia, M. Norouzi, J. Shen, R. Zhao, and Q. V. Le, “Neural program synthesis with priority queue training,” arXiv preprint arXiv:1801.03526, 2018.
[7] K. Ellis, M. Nye, Y. Pu, F. Sosa, J. Tenenbaum, and A. Solar-Lezama,“Write, execute, assess: Program synthesis with a repl,” Advances in NeuralInformation Processing Systems, vol. 32, 2019.
[8] X. Chen, C. Liu, and D. Song, “Towards synthesizing complex programs from input-output examples,” arXiv preprint arXiv:1706.01284, 2017.
[9] L. Kaiser and I. Sutskever, “Neural gpus learn algorithms,”arXiv preprint arXiv:1511.08228, 2015.

Work Environment

This project will be performed within the AMLab group at UvA, with the option of also becoming a paid QUVA intern. Meetings will be in person research meetings at Lab42 up to once a week. The advisors will be Natasha Butt and Matthew Macfarlane. Natasha Butt is a second year PhD student in QUVA Lab (Qualcomm and UvA) working on unsupervised learning for source compression with Max Welling, Taco Cohen and Auke Wiggers. Matthew Macfarlane is a second year student in AMLab focusing on reinforcement learning and search applied to combinatorial optimisation with Herke van Hoof.

Expectations

Students would be welcomed and encouraged to contribute their own ideas to the direction of this project. Prior knowledge of reinforcement learning and training of machine learning models is useful but not required. This research is expected to be publishable in a top tier machine learning conference workshop.

Contact

You can contact Natasha via n.e.butt@uva.nl.