Marta Garnelo

1Meta learning

A big part of my research falls under the umbrella of meta learning (which has been called many other things over the years like ‘learning to learn’, ‘few-shot learning’ or ‘in context learning’). I find the concept of explicitly training models to be able to quickly adapt at inference time fascinating and a key part of intelligence. Throughout my career I have found myself coming back to meta learning through different projects. They can generally be clustered into three:

Neural processes: NPs (CNP, ICML 2018, NP, 2018) were my first foray into the world of meta learning. The main idea was to train neural networks to approximate a distribution over functions rather than a single function. At inference time the model can then predict new query points given a few observations from a particular function within the distribution. Reminiscent of Gaussian Processes (which unsurprisingly motivated the name), by modelling a distribution over functions we not only get a mean prediction, but also learn to model the uncertainty from the data. Implemented as an autoencoder, the original CNP was deterministic, but our follow-up paper NP introduced a latent variable model that allowed us to sample entire functions in latent space.

Excitingly, NPs grew into somewhat of a research field of its own, with over 460 follow-up papers and counting (a likely incomplete list that I have put together can be found here). Unsurprisingly I was involved in several fun NP follow-up projects myself. For example, projects that aim to better understand NPs like characterising different training objectives (NP Objectives, NeurIPS BDL workshop 2018) or formally verifying properties of its predictions (Verification of Deep Probabilistic Models, 2018). We also worked on extensions to either its applications like Bayesian Optimisation (Meta-learning Surrogate Models, 2019) or to the architecture itself: adding attention (Attentive NP, ICLR 2019) and a memory component (Adaptive Posterior Learning, ICLR 2019) to the original model. NPs will always have a special place in my heart, so if your work involves them feel free to reach out!

Click the plot to add context points. These are the predictions of a small trained Conditional Neural Process, running in your browser: its mean and two standard deviations against the true function in white.

Intention: our work on Intention started out of interest for input convex neural networks and unexpectedly developed into an alternative to the classical Attention mechanism that has strong interpolation abilities. Instead of the universally used KVQ dot product of Attention, the Intention block computes the closed-form solution to linear regression. This gives the network a ‘free optimiser’ if it learns to project the problem into a space where it is convex. (KVQ Models with Intention, 2023)

AttentionIntentionQMLPKMLPVMLPtranspmatmulσmatmulhQMLPKMLPVMLPmatmulhtranspmatmulmatmul+αIinvmatmul+MLPw
A visualisation of how the computation graph of Intention varies from Attention. Highlighted: what Intention adds to Attention. Redrawn after Fig. 2, Key-Value-Query Models with Intention.

Tabular foundation models: third is an exciting family of models that complements LLMs and have been growing in popularity over the last years. While LLMs are extremely powerful with words, they can't be straightforwardly applied to predictive analytics. We investigate this in our paper on why LLMs fail at the tabular prediction task (Why LLMs Fail, 2026). As chief science officer at Fundamental I lead a team developing models that can tackle this task. (Fundamental white paper, 2025).

-0.500.510204060dimensionalitynormalised scoreLRAdaBoostNNRFLLM
Model performance on simple classification tasks as the dimensionality of the task increases. Unlike all other traditional models LLMs uniquely lose performance with dimensionality. Trend lines from Fig. 8, Why LLMs Fail at Tabular Prediction.

2Games, populations and mechanism design

Another big interest of mine is multiagent research. I am fascinated by its potential for reinforcement learning (RL) and how it turns many of our traditional frameworks on its head (including nullifying concepts as fundamental as agent performance itself). We talk about this paradigm change in our 2020 NeurIPS Tutorial Designing Learning Dynamics.

We have studied agents at population level and analysed learning dynamics, performance and diversity. Learning a diverse set of agents is particularly important (and challenging!) in non-transitive games. We propose different algorithms that increase diversity like rectified Nash response (Open-ended Learning, ICML 2019) and interaction graphs as a framework to express these algorithms. (Pick Your Battles, AAMAS 2021)

I have also collaborated on projects related to mechanism design, specifically on auction design (Auction Rules, ICLR workshop 2022, Neural Network Auction, IJCAI 2021) and multi-agent behaviour prediction applied to football (Game Plan, JAIR 2021, Off-screen Behaviour Prediction, Sci Rep 2022).

Visualising the effect of training against different subsets of adversaries in a population on agents' learning dynamics. Each step, an agent moves towards the average of its best responses to those it learns from. After Fig. 3, Pick Your Battles.

An arrow from one agent to another means the first learns from the second. Add your own arrows: tap an agent, then who it should learn from. Tap again to remove the arrow.

agents
pre-set examples

3Objects and relationsstructure in the representation is what makes learning cheap

Deep learning methods have produced incredible results but are famously very data hungry. Another line of my work asks whether structure in representation space can reduce the amount of training data needed. To this end we have considered different types of structures that take a more symbolic approach. They learn representations that explicitly capture objects and relations (Towards Deep Symbolic RL, 2016; Reconciling DL with Symbolic AI, 2019; PrediNet, ICML 2020; AlignNet, 2020)

When it comes to working with discrete objects I have collaborated on projects that detect discrete clusters in an unsupervised manner (GMVAE, 2016), models that learn compact representation of scenes and are able to count/localise objects as a result (GQN, Science 2018, Consistent GQN, 2018) and extracting individual protein atom coordinates from Cryo-EM images (Cryo-EM VAEs, 2021).

reward 0
The four game environments of Towards Deep Symbolic RL (Fig. 2). The agent is the +. An x is worth +1 and an o −1. Nothing is learning here: these are sampled paths.

references

Meta learning

  1. Conditional Neural Processes. Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J. Rezende, S. M. Ali Eslami. ICML 2018, PMLR 80, pp. 1704–1713.
  2. Neural Processes. Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, Yee Whye Teh. ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models.
  3. Empirical Evaluation of Neural Process Objectives. Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, Yee Whye Teh. NeurIPS 2018 Workshop on Bayesian Deep Learning.
  4. Verification of Deep Probabilistic Models. Krishnamurthy Dvijotham, Marta Garnelo, Alhussein Fawzi, Pushmeet Kohli. arXiv preprint, 2018.
  5. Meta-Learning Surrogate Models for Sequential Decision Making. Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, S. M. Ali Eslami, Yee Whye Teh. arXiv preprint, 2019.
  6. Attentive Neural Processes. Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, Yee Whye Teh. ICLR 2019.
  7. Adaptive Posterior Learning: Few-Shot Learning with a Surprise-Based Memory Module. Tiago Ramalho, Marta Garnelo. ICLR 2019.
  8. Exploring the Space of Key-Value-Query Models with Intention. Marta Garnelo, Wojciech Marian Czarnecki. arXiv preprint, 2023.
  9. Why Large Language Models Fail at Tabular Prediction. Marta Garnelo, Wojciech M. Czarnecki. arXiv preprint, August 2026.
  10. Developing Foundation Models for Real-World Tabular Data. Marta Garnelo, Wojciech Marian Czarnecki. Fundamental white paper, 2025.

Games, populations and mechanism design

  1. Designing Learning Dynamics. Marta Garnelo, David Balduzzi, Wojciech Czarnecki. NeurIPS 2020 Tutorial, Track 3, 7 December 2020 (video).
  2. Open-ended Learning in Symmetric Zero-sum Games. David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech M. Czarnecki, Julien Pérolat, Max Jaderberg, Thore Graepel. ICML 2019, PMLR 97, pp. 434–443.
  3. Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity. Marta Garnelo, Wojciech Marian Czarnecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gauthier Gidel, Hado van Hasselt, David Balduzzi. AAMAS 2021 (extended abstract); full paper on arXiv.
  4. Learning Truthful, Efficient, and Welfare Maximizing Auction Rules. Andrea Tacchetti, DJ Strouse, Marta Garnelo, Thore Graepel, Yoram Bachrach. ICLR 2022 Workshop on Gamification and Multiagent Solutions.
  5. A Neural Network Auction for Group Decision Making over a Continuous Space. Yoram Bachrach, Ian Gemp, Marta Garnelo, János Kramár, Tom Eccles, Dan Rosenbaum, Thore Graepel. IJCAI 2021 (Demo Track), pp. 4976–4979.
  6. Game Plan: What AI Can Do for Football, and What Football Can Do for AI. Karl Tuyls, Shayegan Omidshafiei, Paul Muller, Zhe Wang, Jerome Connor, Daniel Hennes, Marta Garnelo, et al.. Journal of Artificial Intelligence Research 71, pp. 41–88, 2021.
  7. Multiagent Off-screen Behavior Prediction in Football. Shayegan Omidshafiei, Daniel Hennes, Marta Garnelo, Zhe Wang, Adrià Recasens, Eugene Tarassov, et al.. Scientific Reports 12, 8638, 2022.

Objects and relations

  1. Towards Deep Symbolic Reinforcement Learning. Marta Garnelo, Kai Arulkumaran, Murray Shanahan. Deep Reinforcement Learning Workshop, NIPS 2016.
  2. Reconciling Deep Learning with Symbolic Artificial Intelligence: Representing Objects and Relations. Marta Garnelo, Murray Shanahan. Current Opinion in Behavioral Sciences 29, pp. 17–23, 2019.
  3. An Explicitly Relational Neural Network Architecture. Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell, Christos Kaplanis, David Barrett, Marta Garnelo. ICML 2020, PMLR 119, pp. 8593–8603.
  4. AlignNet: Unsupervised Entity Alignment. Antonia Creswell, Kyriacos Nikiforou, Oriol Vinyals, Andre Saraiva, Rishabh Kabra, Loic Matthey, Chris Burgess, Malcolm Reynolds, Richard Tanburn, Marta Garnelo, Murray Shanahan. arXiv preprint, 2020.
  5. Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders. Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, Murray Shanahan. arXiv preprint, 2016.
  6. Neural Scene Representation and Rendering. S. M. Ali Eslami, Danilo J. Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, et al.. Science 360(6394), pp. 1204–1210, 2018.
  7. Consistent Generative Query Networks. Ananya Kumar, S. M. Ali Eslami, Danilo J. Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, Murray Shanahan. arXiv preprint, 2018.
  8. Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs. Dan Rosenbaum, Marta Garnelo, Michal Zielinski, Charlie Beattie, Ellen Clancy, Andrea Huber, Pushmeet Kohli, Andrew W. Senior, John Jumper, Carl Doersch, S. M. Ali Eslami, Olaf Ronneberger, Jonas Adler. NeurIPS 2021 Workshop on Machine Learning in Structural Biology (oral).