Highly influential “dual-process” accounts of human cognition postulate the coexistence of a slow accurate system with a
fast error-prone system. But why would there be just 2 systems rather than, say, one or 93?
Here, we argue that a dual-process architecture might reflect a rational tradeoff between the cognitive flexibility
afforded by multiple systems and the time and effort required to choose between them. We investigate what the optimal set
and number of cognitive systems would depend on the structure of the environment.
We find that the optimal number of systems depends on the variability of the environment and the difficulty of deciding
when which system should be used. Furthermore, we find that there is a plausible range of conditions under which it is
optimal to be equipped with a fast system that performs no deliberation (“System 1”) and a slow system that achieves a
higher expected accuracy through deliberation (“System 2”).
Our findings thereby suggest a rational reinterpretation of dual-process theories.
…We study this problem in 4 different domains where the dual systems framework has been applied to explain human
decision-making: binary choice, planning, strategic interaction, and multi-alternative, multi-attribute risky choice. We
investigate how the optimal cognitive architecture for each domain depends on the variability of the environment and the
cost of choosing between multiple cognitive systems, which we call metareasoning cost.
The study of generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies
generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments.
Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment
will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We provide a unifying
formalism and terminology for discussing different generalisation problems, building upon previous works. We go on to
categorise existing benchmarks for generalisation, as well as current methods for tackling the generalisation problem.
Finally, we provide a critical discussion of the current state of the field, including recommendations for future work.
Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not
conducive to progress in generalisation, we suggest fast online adaptation and tackling RL-specific problems as some areas
for future work on methods for generalisation, and we recommend building benchmarks in underexplored problem settings such
as offline RL generalisation and reward-function variation.
Differentiable programming techniques are widely used in the community and are responsible for the machine learning
renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we
discuss a common chaos based failure mode which appears in a variety of differentiable
circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We
trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this
failure to spoil their differentiation based optimization algorithms.
Large pretrained language models such as GPT-3 have the surprising
ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt
consisting of input-output examples. Without being explicitly pretrained to do so, the language model learns from these
examples during its forward pass without parameter updates on “out-of-distribution” prompts. Thus, it is unclear what
mechanism enables in-context learning. In this paper, we study the role of the pretraining distribution on the emergence of
in-context learning under a mathematical setting where the pretraining texts have long-range coherence. Here, language
model pretraining requires inferring a latent document-level concept from the conditioning text to generate coherent
next tokens. At test time, this mechanism enables in-context learning by inferring the shared latent concept between prompt
examples and applying it to make a prediction on the test example. Concretely, we prove that in-context learning occurs
implicitly via Bayesian inference of the latent concept
when the pretraining distribution is a mixture of HMMs. This can occur despite the
distribution mismatch between prompts and pretraining data. In contrast to messy large-scale pretraining datasets for
in-context learning in natural language, we generate a family of small-scale synthetic datasets (GINC) whereTransformerand LSTM language
models both exhibit in-context learning. Beyond the theory which focuses on the effect of the pretraining distribution, we
empirically find that scaling model size improves in-context accuracy even when the pretraining loss is the same.
One of the key promises of model-based reinforcement learning is the ability to
generalize using an internal model of the world to make predictions in novel environments and tasks. However, the
generalization ability of model-based agents is not well understood because existing work has focused on model-free agents
when benchmarking generalization. Here, we explicitly measure the generalization ability of model-based agents in
comparison to their model-free counterparts. We focus our analysis on MuZero (Schrittwieser et al., 2020), a powerful
model-based agent, and evaluate its performance on both procedural and task generalization. We identify three factors of
procedural generalization—planning, self-supervised representation learning, and procedural data diversity—and show that by
combining these techniques, we achieve state-of-the art generalization performance and data efficiency on Procgen (Cobbe et
al., 2019). However, we find that these factors do not always provide the same benefits for the task generalization
benchmarks in Meta-World (Yu et al., 2019), indicating that transfer remains a challenge and may require different
approaches than procedural generalization. Overall, we suggest that building generalizable agents requires moving beyond
the single-task, model-free paradigm and towards self-supervised model-based agents that are trained in rich, procedural,
multi-task environments.
We introduce MetaICL (Meta-training for In-Context Learning), a new
meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learn-ing on a
large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test
time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment
on a large, diverse collection of tasks consisting of 142 NLP datasets including
classification, question answering, natural language inference, paraphrase detection and more, across seven
different meta-training/target splits. MetaICL outperforms a range of
baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We
find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and
that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target
task training data, and outperforms much bigger models with nearly 8× parameters.
The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models
such as transformers are being applied to a variety of domains. One important problem class that has remained relatively
elusive however is purposeful adaptive behavior. Currently there is a common perception that sequence models “lack the
understanding of the cause and effect of their actions” leading them to draw incorrect inferences due to auto-suggestive
delusions. In this report we explain where this mismatch originates, and show that it can be resolved by treating actions
as causal interventions. Finally, we show that in supervised learning, one can teach a system to condition or intervene on
data by training with factual and counterfactual error signals respectively.
The intersection between neuroscience and artificial intelligence (AI) research has created synergistic effects in both
fields. While neuroscientific discoveries have inspired the development of AI architectures, new ideas and algorithms from
AI research have produced new ways to study brain mechanisms. A well-known example is the case of reinforcement learning (RL), which has stimulated neuroscience research on how animals learn to
adjust their behavior to maximize reward.
In this review article, we cover recent collaborative work between the 2 fields in the context of meta-learning and its
extension to social cognition and consciousness. Meta-learning refers to the ability to learn how to learn, such as
learning to adjust hyperparameters of existing learning algorithms and how to use existing models and knowledge to
efficiently solve new tasks. This meta-learning capability is important for making existing AI systems more adaptive and
flexible to efficiently solve new tasks. Since this is one of the areas where there is a gap between human performance and
current AI systems, successful collaboration should produce new ideas and progress. Starting from the role of RL algorithms
in driving neuroscience, we discuss recent developments in deep RL applied to modeling prefrontal cortex functions. Even
from a broader perspective, we discuss the similarities and differences between social cognition and meta-learning, and
finally conclude with speculations on the potential links between intelligence as endowed by model-based RL and
consciousness.
For future work we highlight data efficiency, autonomy and intrinsic motivation as key research areas for advancing both
fields.
[Keywords: model-based reinforcement learning, meta-learning,
social cognition, consciousness]
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks.
It has been hypothesized that this is a consequence of implicit multitask learning in language model training. Can
zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we
develop a system for easily mapping general natural language tasks into a human-readable prompted form. We convert a large
set of supervised datasets, each with multiple prompts using varying natural language. These prompted datasets allow for
benchmarking the ability of a model to perform completely unseen tasks specified in natural language. We fine-tune a
pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks. The model attains strong
zero-shot performance on several standard datasets, often outperforming models 16× its size. Further, our approach attains
strong performance on a subset of tasks from the BIG-Bench benchmark,
outperforming models 6× its size. All prompts and trained models are available at
github.com/bigscience-workshop/promptsource/.
Collaborating with humans requires rapidly adapting to their individual strengths, weaknesses, and preferences.
Unfortunately, most standard multi-agent reinforcement learning techniques, such as
self-play (SP) or population play (PP), produce agents that overfit to their training partners and do not generalize well
to humans. Alternatively, researchers can collect human data, train a human model using behavioral cloning, and then use
that model to train “human-aware” agents (“behavioral cloning play”, or BCP).
While such an approach can improve the generalization of agents to new human co-players, it involves the onerous and
expensive step of collecting large amounts of human data first. Here, we study the problem of how to train agents that
collaborate well with human partners without using human data. We argue that the crux of the problem is to produce a
diverse set of training partners. Drawing inspiration from successful multi-agent approaches in competitive domains, we
find that a surprisingly simple approach is highly effective. We train our agent partner as the best response to a
population of self-play agents and their past checkpoints taken throughout training, a method we call Fictitious
Co-Play (FCP). Our experiments focus on a two-player collaborative cooking
simulator that has recently been proposed as a challenge problem for coordination with humans. We find that
FCPagents score significantly higher than SP, PP, and BCP when paired with novel agent and human partners. Furthermore, humans also report a
strong subjective preference to partnering with FCP agents over all
baselines.
The intertwined processes of learning and evolution in complex environmental niches have resulted in a remarkable
diversity of morphological forms. Moreover, many aspects of animal intelligence are deeply embodied in these evolved
morphologies. However, the principles governing relations between environmental complexity, evolved morphology, and the
learnability of intelligent control, remain elusive, because performing large-scale in silico experiments on evolution and
learning is challenging.
Here, we introduce Deep Evolutionary Reinforcement Learning(DERL): a computational framework which can evolve diverse agent morphologies to learn
challenging locomotion and manipulation tasks in complex environments.
Leveraging DERL we demonstrate several relations between environmental
complexity, morphological intelligence and the learnability of control. First, environmental complexity fosters the
evolution of morphological intelligence as quantified by the ability of a morphology to facilitate the learning of novel
tasks. Second, we demonstrate a morphological Baldwin effect i.e., in our simulations evolution rapidly selects morphologies that learn faster, thereby
enabling behaviors learned late in the lifetime of early ancestors to be expressed early in the descendants’ lifetime.
Third, we suggest a mechanistic basis for the above relationships through the evolution of morphologies that are more
physically stable and energy efficient, and can therefore facilitate learning and control.
Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification
of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage large-scale machine learning
techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning
tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem
with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their
labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input
of the rest of the data points. Presented with a set of samples from a new supervised learning task as input,
PFNs make probabilistic predictions for arbitrary other data points in a single
forward propagation, having learned to approximate Bayesian inference.
We demonstrate that PFNs can near-perfectly mimic Gaussian processes
and also enable efficient Bayesian inference for intractable problems, with over
200× speedups in multiple setups compared to current methods. We obtain strong results in very diverse areas such as
Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image
classification, demonstrating the generality of PFNs. Code and trained
PFNs arereleased under [ANONYMIZED;
please see the supplementary material].
The transformer architecture and variants presented a remarkable success across many machine learning tasks in recent
years. This success is intrinsically related to the capability of handling long sequences and the presence of
context-dependent weights from the attention mechanism. We argue that these capabilities suit the central role of a
Meta-Reinforcement Learning algorithm. Indeed, a meta-RL agent needs to infer the
task from a sequence of trajectories. Furthermore, it requires a fast adaptation strategy to adapt its policy for a new
task—which can be achieved using the self-attention mechanism. In this work, we present TrMRL (Transformers for Meta-Reinforcement Learning), a meta-RL agent that mimics the memory reinstatement mechanism using
the transformer architecture. It associates the recent past of working memories to build an episodic memory recursively
through the transformer layers. This memory works as a proxy to the current task, and we condition a policy head on it. We
conducted experiments in high-dimensional continuous control environments for locomotion and dexterous manipulation.
Results show that TrMRL achieves or surpasses state-of-the-art performance,
sample efficiency, and out-of-distribution generalization in these environments.
Lookahead search has been a critical component of recent AI successes, such as in the games of chess, go, and poker.
However, the search methods used in these games, and in many other settings, are tabular. Tabular search methods do not
scale well with the size of the search space, and this problem is exacerbated by stochasticity and partial observability.
In this work we replace tabular search with online model-based fine-tuning of a policy neural network via reinforcement
learning, and show that this approach outperforms state-of-the-art search algorithms in benchmark settings. In particular,
we use our search algorithm to achieve a new state-of-the-art result in self-play Hanabi, and show the generality of our
algorithm by also showing that it outperforms tabular search in the Atari game Ms. Pacman.
In complex systems, we often observe complex global behavior emerge from a collection of agents interacting with each
other in their environment, with each individual agent acting only on locally available information, without knowing the
full picture. Such systems have inspired development of artificial intelligence algorithms in areas such as swarm
optimization and cellular automata. Motivated by the emergence of collective behavior from complex cellular systems, we
build systems that feed each sensory input from the environment into distinct, but identical neural networks, each with no
fixed relationship with one another. We show that these sensory networks can be trained to integrate information received
locally, and through communication via an attention mechanism, can collectively produce a globally coherent policy.
Moreover, the system can still perform its task even if the ordering of its inputs is randomly permuted several times
during an episode. These permutation invariant systems also display useful robustness and generalization properties that
are broadly applicable. Interactive demo and videos of our results: https://attentionneuron.github.io/
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that
instruction tuning—finetuning language models on a collection of tasks described via instructions—substantially boosts
zero-shot performance on unseen tasks.
We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this
instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses
zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN evenoutperforms few-shot GPT-3 by a large
margin on ANLI, RTE, BoolQ,AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number
of tasks and model scale are key components to the success of instruction tuning.
AI and reinforcement learning (RL) have improved many areas, but are not yet
widely adopted in economic policy design, mechanism design, or economics at large. At the same time, current economic
methodology is limited by a lack of counterfactual data, simplistic behavioral models, and limited opportunities to
experiment with policies and evaluate behavioral responses. Here we show that machine-learning-based economic simulation is
a powerful policy and mechanism design framework to overcome these limitations. The AI Economist is a two-level, deep RL
framework that trains both agents and a social planner who co-adapt, providing a tractable solution to the highly unstable
and novel two-level RL challenge. From a simple specification of an economy, we learn rational agent behaviors that adapt
to learned planner policies and vice versa. We demonstrate the efficacy of the AI Economist on the problem of optimal
taxation. In simple one-step economies, the AI Economist recovers the optimal tax policy of economic theory. In complex,
dynamic economies, the AI Economist substantially improves both utilitarian social welfare and the trade-off between
equality and productivity over baselines. It does so despite emergent tax-gaming strategies, while accounting for agent
interactions and behavioral change more accurately than economic theory. These results demonstrate for the first time that
two-level, deep RL can be used for understanding and as a complement to theory for economic design, unlocking a new
computational learning-based approach to understanding economic policy.
“Open-Ended
Learning Leads to Generally Capable Agents”, Open Ended Learning Team, Adam Stooke, Anuj Mahajan,
Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese,
Nathalie Bradley-Schmieg, Nathaniel Wong… (2021-07-27; ai / scaling):
In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider
generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment
domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The
environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are
situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the
challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We
propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a
singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We
show that through constructing an open-ended learning process, which dynamically changes the training task distributions
and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The
resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising
to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on
Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of
our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use,
option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger
scale transfer of behaviour through cheap finetuning.
Generalization is a central challenge for the deployment of reinforcement learning (RL) systems in the real world. In
this paper, we show that the sequential structure of the RL problem necessitates new approaches to generalization beyond
the well-studied techniques used in supervised learning. While supervised learning methods can generalize effectively
without explicitly accounting for epistemic uncertainty, we show that, perhaps surprisingly, this is not the case in RL. We
show that generalization to unseen test conditions from a limited number of training conditions induces implicit partial
observability, effectively turning even fully-observed MDPs intoPOMDPs. Informed by this observation, we recast the problem of
generalization in RL as solving the induced partially observed Markov decision process, which we call the epistemic
POMDP. We demonstrate the failure modes of
algorithms that do not appropriately handle this partial observability, and suggest a simple ensemble-based technique for
approximately solving the partially observed problem. Empirically, we demonstrate that our simple algorithm derived from
the epistemic POMDP achieves significant gains in generalization over
current methods on the Procgen benchmark suite.
When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language
task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring
this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we
train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen
language model prompted with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot
learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a
sequence of multiple interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects
and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside
knowledge, by measuring a single model on a variety of established and new benchmarks.
A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To
address this, we describe a system designed from a generalization-first perspective, learning to update optimizer
hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system
outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2× speedups on
ImageNet, and a 2.5× speedup on a language modeling task using over 5 orders
of magnitude more compute than the training tasks.
The rapidly-changing ML model landscape presents a unique opportunity for building hardware accelerators optimized for
specific datacenter-scale workloads. We propose Full-stack Accelerator Search Technique (FAST), a hardware accelerator search framework that defines a broad optimization environment
covering key design decisions within the hardware-software stack, including hardware datapath, software scheduling, and
compiler passes such as operation fusion and tensor padding. Although FAST can be
used on any number and type of deep learning workload, in this paper we focus on optimizing for a single or small
set of vision models, resulting in significantly faster and more power-efficient designs relative to a general purpose ML
accelerator. When evaluated on EfficientNet, ResNet50v2, and OCR inference performance relative to a TPU-v3, designs
generated by FASToptimized for single workloads can improve
Perf/TDP (peak power) by over 6× in the best case and 4× on average. On a
limited workload subset, FAST improves Perf/TDP 2.85× on average, with a reduction to 2.35× for a single design optimized over the set
of workloads. In addition, we demonstrate a potential 1.8× speedup opportunity for TPU-v3 with improved scheduling.
In this article we hypothesize that intelligence, and its associated abilities, can be understood as subserving the
maximization of reward.
Accordingly, reward is enough to drive behaviour that exhibits abilities studied in natural and artificial intelligence,
including knowledge, learning, perception, social intelligence, language, generalisation and imitation. This is in contrast
to the view that specialised problem formulations are needed for each ability, based on other signals or objectives.
Furthermore, we suggest that agents that learn through trial & error experience to maximise reward could learn behaviour
that exhibits most if not all of these abilities, and therefore that powerful reinforcement learning agents could
constitute a solution to artificial general intelligence.
[Keywords: artificial intelligence, artificial general intelligence, reinforcement learning, reward]
This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from
July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the
hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning
emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This
competition has widespread impact as black-box optimization (e.g., Bayesian
optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many
applications outside of machine learning. The final leaderboard was determined using the optimization performance on
held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the
default settings of several open-source black-box optimization packages as well as random search.
Supporting state-of-the-art AI research requires balancing rapid prototyping, ease of use, and quick iteration, with the
ability to deploy experiments at a scale traditionally associated with production systems. Deep learning frameworks
such as TensorFlow, PyTorch and JAXallow users to transparently make use
of accelerators, such as TPUsand GPUs, to offload the more computationally intensive parts of training and inference in
modern deep learning systems. Popular training pipelines that use these frameworks for deep learning typically focus on
(un-)supervised learning. How to best train reinforcement learning (RL) agents at scale is still an active research
area.
In this report we argue that TPUs are particularly well suited for
training RL agents in a scalable, efficient and reproducible way. Specifically we describe two architectures designed to
make the best use of the resources available on a TPU Pod (a special
configuration in a Google data center that features multiple TPU
devices connected to each other by extremely low latency communication channels).
Figure 4: (a) FPS for Anakin, as a function of the
number ofTPU cores, ranging from 16 (ie. 2 replicas) to 128
(ie. 16 replicas). (b) FPS for a Sebulba implementation
ofIMPALA’s V-trace algorithm, as a function of the actor batch size, from 32 (as
in IMPALA) to 128. (c) FPS for a Sebulba implementation ofMuZero, as a function of the number of TPU cores, from 16 (ie. 2 replicas) to 128 (ie. 16 replicas).
Anakin: When using small neural networks and grid-world environments an Anakin architecture can easily
perform 5 million steps per second, even on the 8-core TPU accessible
for free through Google Colab.
This can be very useful to experiment and debug research ideas in the friendly Colab environment. In Figure
4a we show how, thanks to the efficient network connecting different TPU cores in a Pod, performance scales almost linearly with the number of cores;
the collective operations used to average gradients across replicas appear to cause only minimal overhead.
In a recent paper by Oh et al 2021 Anakin was used, at a much larger scale,
to discover a general reinforcement learning update, from experience of interacting with a rich set of environments
implemented in JAX. In this paper, Anakin wasused to learn a single
shared update rule from 60K JAX environments and 1K policies running and
training in parallel.
Despite the complex nature of the system, based on the use of neural networks to meta-learn not just a policy but the
entire RL update, Anakin delivered over 3 million steps per second on a 16-core TPU. Training the update rule to a good level of performance, required running
Anakin for approximately 24 hours; this would cost approximately $100 on GCP’s
preemptible instances
…Sebulba: Our second podracer architecture has also been extensively used for exploring a variety of RL
ideas at scale, on environments that cannot be compiled to run on TPU(eg. Atari, DMLaband MuJoCo). As both IMPALA and Sebulba are based on a decompositionbetween actors and learners, agents
designed for the IMPALA architecture can be easily mapped onto Sebulba; for
instance a Podracer version of the V-trace agent easily reproduced the results from Espeholt et al 2018.
However, we found that training an agent for 200 million frames of an Atari game could be done in just ~1 hour, by running
Sebulba on a 8-core TPU. This comes at a cost of approximately
$2.88, on GCP’s pre-emptible instances. This is similar in cost to
training with the more complex
SEED RL framework, and much cheaper than training an agent for 200 million Atari frames using either
IMPALA or single-streamGPU-based system such as that traditionally used by DQN.
…In addition to the trajectory length the effective batch size used to compute each update also depends on how many
times we replicate the basic 8-TPU setup. Sebulba also scales
effectively along this dimension: using 2048 TPU cores (an entire
Pod) we were able to further scale all the way to 43 million frames per second, solving the classic Atari videogame Pong in
less than 1 minute…Sebulba has also been used to train search-based agents inspired by MuZero
(Schrittwieser et al 2020). The workloads associated to these agents are very different from that of
model-free agents like IMPALA. The key difference is in the cost of action
selection. This increases because MuZero’s policy combines search with deep neural networks (used to guide and/or
truncate the search). Typically, search-based agents like MuZero required custom C++ implementations of the search to
deliver good performance. We could reproduce results from MuZero (no Reanalyse) on multiple RL environments, using
Sebulba and a pure JAX implementation of MCTS.
Training a MuZero agent with Sebulba for 200M Atari frames takes 9 hours on a 16-core TPU (at a cost of ~$40 on GCP’s
preemptible instances).
We found that scalability, via replication, was particularly useful in this context. Figure 4c reports
the number of frames per seconds processed by Sebulba when running MuZero on Atari, as a function of the number of
TPU cores. The throughput increased linearly with the number of
cores.
“Meta-Learning Bidirectional Update Rules”, Mark Sandler, Max Vladymyrov, Andrey
Zhmoginov, Nolan Miller, Andrew Jackson, Tom Madams, Blaise Aguera y Arcas (2021-04-10):
In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple
states. We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state
network where one state is used for activations and another for gradients, with update rules derived from the chain rule.
In our generalized framework, networks have neither explicit notion of nor ever receive gradients. The synapses and neurons
are updated using a bidirectional Hebb-style update rule parameterized by a shared low-dimensional “genome”. We show that
such genomes can be meta-learned from scratch, using either conventional optimization techniques, or evolutionary
strategies, such as CMA-ES. Resulting update rules generalize to unseen tasks
and train faster than gradient descent based optimizers for several standard computer vision and synthetic tasks.
We train a single, goal-conditioned policy that can solve many robotic manipulation tasks, including tasks with
previously unseen goals and objects. To do so, we rely on asymmetric self-play for goal discovery, where two agents, Alice
and Bob, play a game. Alice is asked to propose challenging goals and Bob aims to solve them. We show that this method is
able to discover highly diverse and complex goals without any human priors. We further
show that Bob can be trained with only sparse rewards, because the interaction between Alice and Bob results in a natural
curriculum and Bob can learn from Alice’s trajectory when relabeled as a goal-conditioned demonstration. Finally, we show
that our method scales, resulting in a single policy that can transfer to many unseen hold-out tasks such as setting a
table, stacking blocks, and solving simple puzzles. Videos of a learned policy is available at
https://robotics-self-play.github.io.
This paper proposes Omnidirectional Representations from Transformers (OmniNet). In OmniNet, instead of
maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network.
This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field
of the entire width and depth of the network. To this end, the omnidirectional attention is learned via a meta-learner,
which is essentially another self-attention based model. In order to mitigate the computationally expensive costs of full
receptive field attention, we leverage efficient self-attention models such as kernel-based (Choromanski et al), low-rank
attention (Wang et al)
and/or BigBird (Zaheer et
al) as the meta-learner.
Extensive experiments are conducted on autoregressive language modeling (LM1B, C4), Machine Translation, Long Range Arena(LRA), and Image Recognition. The experiments show that OmniNet achieves
considerable improvements across these tasks, including achieving state-of-the-art performance on LM1B, WMT’14 En-De/En-Fr, and Long Range Arena. Moreover, using omnidirectional representation
in Vision
Transformers leads to substantial improvements on image recognition tasks on both few-shot learning and fine-tuning
setups.
Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe
models’ novel capabilities. Using GPT-3 as a case study, we show that 0-shot
prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these
cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking
the role of prompts in controlling and evaluating powerful language models. In this work, we discuss methods of prompt
programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore
techniques for exploiting the capacity of narratives and cultural anchors to encode nuanced intentions and techniques for
encouraging deconstruction of a problem into components before producing a verdict. Informed by this more encompassing
theory of prompt programming, we also introduce the idea of a metaprompt that seeds the model to generate its own natural
language prompts for a range of tasks. Finally, we discuss how these more general methods of interacting with language
models can be incorporated into existing and future benchmarks and practical applications.
The intertwined processes of learning and evolution in complex environmental niches have resulted in a remarkable
diversity of morphological forms. Moreover, many aspects of animal intelligence are deeply embodied in these evolved
morphologies. However, the principles governing relations between environmental complexity, evolved morphology, and the
learnability of intelligent control, remain elusive, partially due to the substantial challenge of performing large-scale
in silico experiments on evolution and learning. We introduce Deep Evolutionary Reinforcement Learning(DERL): a novel
computational framework which can evolve diverse agent morphologies to learn challenging locomotion and manipulation tasks
in complex environments using only low level egocentric sensory information. Leveraging DERL we demonstrate several relations between environmental complexity, morphological
intelligence and the learnability of control. First, environmental complexity fosters the evolution of morphological
intelligence as quantified by the ability of a morphology to facilitate the learning of novel tasks. Second, evolution
rapidly selects morphologies that learn faster, thereby enabling behaviors learned late in the lifetime of early ancestors
to be expressed early in the lifetime of their descendants. In agents that learn and evolve in complex environments, this
result constitutes the first demonstration of a long-conjectured morphological Baldwin effect. Third, our experiments
suggest a mechanistic basis for both the Baldwin effect and the emergence of morphological intelligence through the
evolution of morphologies that are more physically stable and energy efficient, and can therefore facilitate learning and
control.
We consider the problem of efficient blackbox optimization over a large hybrid search space, consisting of a mixture of
a high dimensional continuous space and a complex combinatorial space. Such examples arise commonly in evolutionary
computation, but also more recently, neuroevolution and architecture search for Reinforcement Learning (RL) policies.
In this paper, we introduce ES-ENAS, a simple joint optimization procedure by
combining Evolutionary Strategies (ES) and combinatorial optimization techniques in a highly scalable and intuitive way,
inspired by the one-shot or supernet paradigm introduced in Efficient Neural Architecture Search
(ENAS). Our main insight is noticing that ES is already a highly distributed
algorithm involving hundreds of blackbox evaluations which can not only be used for training neural network weights, but
also for feedback to a combinatorial optimizer. Through this relatively simple marriage between two different lines of
research, we are able to gain the best of both worlds, and empirically demonstrate our approach by optimizing
BBOB functions over hybrid spaces as well as combinatorial neural network
architectures via edge pruning and quantization on popular RL benchmarks. Due to the modularity of the algorithm, we also
are able incorporate a wide variety of popular techniques ranging from use of different continuous and combinatorial
optimizers, as well as constrained optimization.
Learned optimizers are increasingly effective, with performance exceeding that of hand designed optimizers such as
Adam (kingma2014adam) on specific tasks (metz2019understanding). Despite the potential gains
available, in current work the meta-training (or ‘outer-training’) of the learned optimizer is performed by a hand-designed
optimizer, or by an optimizer trained by a hand-designed optimizer (metz2020tasks). We show that a population
of randomly initialized learned optimizers can be used to train themselves from scratch in an online fashion, without
resorting to a hand designed optimizer in any part of the process. A form of population based training is used to
orchestrate this self-training. Although the randomly initialized optimizers initially make slow progress, as they improve
they experience a positive feedback loop, and become rapidly more effective at training themselves. We believe feedback
loops of this type, where an optimizer improves itself, will be important and powerful in the future of machine learning.
These methods not only provide a path towards increased performance, but more importantly relieve research and engineering
effort.
[Blog] We propose a method for meta-learning reinforcement learning
algorithms by searching over the space of computational graphs which compute the loss function for a
value-based model-free RL agent to optimize. The learned algorithms are domain-agnostic and can generalize to new
environments not seen during training.
Our method can both learn from scratch and bootstrap off known existing algorithms, like DQN, enabling interpretable modifications which improve performance. Learning from scratch
on simple classical control and gridworld tasks, our method rediscovers the temporal-difference (TD) algorithm.
Bootstrappedfrom DQN, we highlight two
learned algorithms which obtain good generalization performance over other classical control tasks, gridworld type
tasks, and Atari games.
The analysis of the learned algorithm behavior shows resemblance to recently proposed RL algorithms that address
overestimation in value-based methods.
“Meta Pseudo
Labels”, Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, Quoc V. Le (2021-01-05):
We present Meta Pseudo Labels, a semi-supervised learning method that achieves a new state-of-the-art top-1
accuracy of 90.2% on ImageNet [using JFT-300M], which is 1.6% better than the
existing state-of-the-art. Like Pseudo
Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student
network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted
by the feedback of the student’s performance on the labeled dataset. As a result, the teacher generates better pseudo
labels to teach the student. Our code will be available at this URL.
Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to control fast
weights, hyper networks, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VS-ML) unifies
the above and demonstrates that simple weight-sharing and sparsity in an NN is sufficient to express powerful learning
algorithms (LAs) in a reusable fashion. A simple implementation of VS-ML called VS-ML RNN allows for implementing the backpropagation LA solely by running an RNN in forward-mode. It can even meta-learn new LAs that improve upon
backpropagation and generalize to datasets outside of the meta training distribution
without explicit gradient calculation. Introspection reveals that our meta-learned LAs learn qualitatively different from
gradient descent through fast association.
…Yet in spite of its historical importance, MNIST has three notable shortcomings.
First, it does a poor job of differentiating between linear, nonlinear, and translation-invariant models. For example,
logistic, MLP, andCNN benchmarks obtain 94, 99+, and 99+% accuracy on it. This makes it hard to measure the
contribution of a CNN’s spatial priors or to judge the relative effectiveness of
different regularization schemes. Second, it is somewhat large for a toy dataset. Each input example is a 784-dimensional
vector and thus it takes a non-trivial amount of computation to perform hyperparameter searches or debug a meta-learning
loop. Third, MNIST is hard to hack. The ideal toy dataset should be
procedurally generated so that researchers can smoothly vary parameters such as background noise, translation, and
resolution.
In order to address these shortcomings, we propose the MNIST-1D
dataset. It is a minimalist, low-memory, and low-compute alternative to MNIST, designed for exploratory deep learning research where rapid iteration is a
priority. Training examples are 20× smaller but they are still better at measuring the difference between (1) linear and
nonlinear classifiers and (2) models with and without spatial inductive biases (eg. translation invariance). The dataset is
procedurally generated but still permits analogies to real-world digit classification…Unlike MNIST, each example is a one-dimensional sequence of points. To generate an
example, we begin with a digit template and then randomly pad, translate, and transform it.
Example use cases: In this section we will explore several examples of how MNIST-1D can be used to study core “science of deep learning” phenomena.
Finding lottery tickets…Unlike many follow-up experiments on the lottery ticket, this one took just
two days of researcher time to produce. The curious reader can also reproduce these results in their browser in a few
minutes.
Observing deep double descent…We see the MNIST-1D dataset as a good tool for exploring these properties. In fact, we
were able to reproduce the double descent pattern after a few hours of researcher effort. The figure
below shows our results for a fully-connected network and a convolutional model.
Gradient-based meta-learning…A model does this by having two levels of optimization: the first is a
fast inner loop which corresponds to a traditional learning objective and second is a slow outer loop which updates the
“meta” properties of the learning process…Meta-learning is a promising topic but it is very difficult to scale. First
of all, meta-learning algorithms consume enormous amounts of time and compute. Second of all, implementations tend to
grow complex since there are twice as many hyperparameters (one set for each level of optimization) and most deep
learning frameworks are not set up well for meta-learning. This places an especially high incentive on debugging and
iterating meta-learning algorithms on small-scale datasets such as MNIST-1D.
For example, it took just a few hours to implement and debug the gradient-based hyperparameter optimization of a
learning rate shown below.
Meta-learning an activation function: Having implemented a “minimal working
example” of gradient-based meta-learning, we realized that it permitted a simple and novel extension: meta-learning
an activation function. With a few more hours of researcher time, we were able to parameterize our classifier’s
activation function with a second neural network and then learn the weights using meta-gradients.
Measuring the spatial priors of deep networks: …Principle
among these priors is the translation invariance of convolution. A primary
motivation for this dataset was to construct a toy problem that could effectively quantify a model’s spatial
priors. The second figure in this post illustrates that this is indeed possible
with MNIST-1D.
Benchmarking pooling methods. Our final case study begins with a specific question: What is the
relationship between pooling and sample efficiency? We had not seen evidence that pooling makes models more or less
sample efficient, but this seemed an important relationship to understand. With this in mind, we trained models with
different pooling methods and training set sizes and found that, while pooling tended to be effective in low-data
regimes, it did not make much of a difference in high-data regimes.
…this post argues in favor of small-scale machine learning research. Neural networks do not have problems with
scaling or performance—but they do have problems with interpretability, reproducibility, and iteration speed. We see
carefully-controlled, small-scale experiments as a great way to address these problems…For example, several of the findings
reported in this post are at the point where they should be investigated at scale. We would like to show that large scale
lottery tickets also learn spatial inductive biases, and show evidence that they develop local connectivity. We would also
like to try meta-learning an activation function on a larger model in the hopes of finding an activation that will
outperform ReLU and Swish in generality. We should emphasize that we are only ready to
scale these results now that we have isolated and understood them in a controlled setting. We believe that scaling a system
is only a good idea once the relevant causal mechanisms have been isolated and understood. [cf scaling law papers] …Our work also bears philosophical similarities to the
“Synthetic Petri Dish” by
Rawal et al 2020.
Closing Thoughts: There is a counterintuitive possibility that in order to explore the limits of how
large we can scale neural networks, we may need to explore the limits of how small we can scale them first. Scaling models
and datasets downward in a way that preserves the nuances of their behaviors at scale will allow researchers to iterate
quickly on fundamental and creative ideas. This fast iteration cycle is the best way of obtaining insights about how to
incorporate progressively more complex inductive biases into our models. We can then transfer these inductive biases across
spatial scales in order to dramatically improve the sample efficiency and generalization properties of large-scale models.
We see the humble MNIST-1D dataset as a first step in that
direction.
Learned optimizers are algorithms that can themselves be trained to solve optimization problems. In contrast to baseline
optimizers (such as momentum or Adam) that use simple update rules derived from theoretical principles, learned optimizers
use flexible, high-dimensional, nonlinear parameterizations. Although this can lead to better performance in certain
settings, their inner workings remain a mystery. How is a learned optimizer able to outperform a well tuned baseline? Has
it learned a sophisticated combination of existing optimization techniques, or is it implementing completely new behavior?
In this work, we address these questions by careful analysis and visualization of learned optimizers. We study learned
optimizers trained from scratch on three disparate tasks, and discover that they have learned interpretable mechanisms,
including: momentum, gradient clipping, learning rate schedules, and a new form of learning rate adaptation. Moreover, we
show how the dynamics of learned optimizers enables these behaviors. Our results help elucidate the previously murky
understanding of how learned optimizers work, and establish tools for interpreting future learned optimizers.
Meta-reinforcement learning algorithms can enable autonomous agents, such as
robots, to quickly acquire new behaviors by leveraging prior experience in a set of related training tasks. However, the
onerous data requirements of meta-training compounded with the challenge of learning from sensory inputs such as images
have made meta-RL challenging to apply to real robotic systems. Latent state models,
which learn compact state representations from a sequence of observations, can accelerate representation learning from
visual inputs. In this paper, we leverage the perspective of meta-learning as task inference to show that latent state models can also perform meta-learning given an appropriately defined
observation space. Building on this insight, we develop meta-RL with latent dynamics
(MELD), an algorithm for meta-RL from images that performs inference in a
latent state model to quickly acquire new skills given observations and
rewards. MELD outperforms prior meta-RL methods on several simulated image-based
robotic control problems, and enables a real WidowX robotic arm to insert an Ethernet cable into new locations given a
sparse task completion signal after only 8 hours of real world meta-training. To our knowledge, MELD is the first meta-RL algorithm trained in a real-world robotic control setting from
images.
Memory-based meta-learning is a powerful technique to build agents that adapt fast to any task within a target
distribution. A previous theoretical study has argued that this remarkable performance is because the meta-training
protocol incentivises agents to behave Bayes-optimally. We empirically investigate this claim on a number of prediction and
bandit tasks. Inspired by ideas from theoretical computer science, we show that meta-learned and Bayes-optimal agents not
only behave alike, but they even share a similar computational structure, in the sense that one agent system can
approximately simulate the other. Furthermore, we show that Bayes-optimal agents are fixed points of the meta-learning
dynamics. Our results suggest that memory-based meta-learning might serve as a general technique for numerically
approximating Bayes-optimal agents—that is, even for task distributions for which we currently don’t possess tractable
models.
Animals are equipped with a rich innate repertoire of sensory, behavioral and motor skills, which allows them to
interact with the world immediately after birth. At the same time, many behaviors are highly adaptive and can be tailored
to specific environments by means of learning. In this work, we use mathematical analysis and the framework of
meta-learning (or ‘learning to learn’) to answer when it is beneficial to learn such an adaptive strategy and when to
hard-code a heuristic behavior. We find that the interplay of ecological uncertainty, task complexity and the agents’
lifetime has crucial effects on the meta-learned amortized Bayesian inference
performed by an agent. There exist two regimes: One in which meta-learning yields a learning algorithm that implements
task-dependent information-integration and a second regime in which meta-learning imprints a heuristic or ‘hard-coded’
behavior. Further analysis reveals that non-adaptive behaviors are not only optimal for aspects of the environment that are
stable across individuals, but also in situations where an adaptation to the environment would in fact be highly
beneficial, but could not be done quickly enough to be exploited within the remaining lifetime. Hard-coded behaviors should
hence not only be those that always work, but also those that are too complex to be learned within a reasonable time
frame.
This paper deals with a challenging task of learning from different modalities by tackling the difficulty problem of
jointly face recognition between abstract-like sketches, cartoons, caricatures and real-life photographs. Due to the
substantial variations in the abstract faces, building vision models for recognizing data from these modalities is an
extremely challenging.
We propose a novel framework termed as Meta-Continual Learning with Knowledge Embedding to address the task of
jointly sketch, cartoon, and caricature face recognition. In particular, we firstly present a deep relational network to
capture and memorize the relation among different samples. Secondly, we present the construction of our knowledge graph
that relates image with the label as the guidance of our meta-learner. We then design a knowledge embedding mechanism to
incorporate the knowledge representation into our network. Thirdly, to mitigate catastrophic forgetting, we use a
meta-continual model that updates our ensemble model and improves its prediction accuracy. With this meta-continual model,
our network can learn from its past. The final classification is derived from our network by learning to compare the
features of samples.
Experimental results demonstrate that our approach achieves substantially higher performance compared with other
state-of-the-art approaches.
Much as replacing hand-designed features with learned functions has revolutionized how we solve perceptual tasks, we
believe learned algorithms will transform how we train models. In this work we focus on general-purpose learned optimizers
capable of training a wide variety of problems with no user-specified hyperparameters. We introduce a new, neural network
parameterized, hierarchical optimizer with access to additional features such as validation loss to enable automatic
regularization. Most learned optimizers have been trained on only a single task, or a small number of tasks. We train our
optimizers on thousands of tasks, making use of orders of magnitude more compute, resulting in optimizers that generalize
better to unseen tasks. The learned optimizers not only perform well, but learn behaviors that are distinct from existing
first order optimizers. For instance, they generate update steps that have implicit regularization and adapt as the problem
hyperparameters (e.g. batch size) or architecture (e.g. neural network width) change. Finally, these learned
optimizers show evidence of being useful for out of distribution tasks such as training themselves from scratch.
[Previously: “Deep neuroethology of a virtual rodent”, Merel et al 2020; see also PIGLeT.] Recent work has shown that large text-based neural language models, trained with
conventional supervised learning objectives, acquire a surprising propensity for few-shot and one-shot learning.
Here, we show that an embodied agent situated in a simulated 3D world, and endowed with a novel dual-coding external
memory, can exhibit similar one-shot word learning when trained with conventional reinforcement learning algorithms. After
a single introduction to a novel object via continuous visual perception and a language prompt (“This is a dax”), the agent
can re-identify the object and manipulate it as instructed (“Put the dax on the bed”). In doing so, it seamlessly
integrates short-term, within-episode knowledge of the appropriate referent for the word “dax” with long-term lexical and
motor knowledge acquired across episodes (ie. “bed” and “putting”).
We find that, under certain training conditions and with a particular memory writing mechanism, the agent’s one-shot
word-object binding generalizes to novel exemplars within the same ShapeNet category, and is effective in settings with
unfamiliar numbers of objects. We further show how dual-coding memory can be exploited as a signal for intrinsic
motivation, stimulating the agent to seek names for objects that may be useful for later executing instructions.
Together, the results demonstrate that deep neural networks can exploit meta-learning, episodic memory and an explicitly
multi-modal environment to account for ‘fast-mapping’, a fundamental pillar of human cognitive development and a
potentially transformative capacity for agents that interact with human users.
Matt Botvinick is Director of Neuroscience Research at DeepMind. In
this interview, he discusses results from a 2018 paper which
describe conditions under which reinforcement learning algorithms will spontaneously give rise to separate full-fledged
reinforcement learning algorithms that differ from the original. Here are some notes I gathered from the interview and
paper:
Initial Observation
At some point, a group of DeepMind researchers in Botvinick’s group noticed that when they trained a RNN using RL on a series of related tasks, the RNN itself instantiated a separate reinforcement learning algorithm. These
researchers weren’t trying to design a meta-learning algorithm—apparently, to their surprise, this just spontaneously
happened. As Botvinick describes it, they started “with just one learning algorithm, and then another learning algorithm
kind of… emerges, out of, like out of thin air”:
“What happens… it seemed almost magical to us, when we first started realizing what was going on—the slow learning
algorithm, which was just kind of adjusting the synaptic weights, those slow synaptic changes give rise to a network
dynamics, and the dynamics themselves turn into a learning algorithm.”
Other versions of this basic architecture—eg., using slot-based memory instead of RNNs—seemed to produce the same basic phenomenon, which they termed “meta-RL.” So they
concluded that all that’s needed for a system to give rise to meta-RL are three very general properties: the system must 1.
have memory, 2. whose weights are trained by a RL algorithm, 3. on a sequence of similar input data.
From Botvinick’s description, it sounds to me like he thinks [learning algorithms that find/instantiate other
learning algorithms] is a strong attractor in the space of possible learning algorithms:
“…it’s something that just happens. In a sense, you can’t avoid this happening. If you have a system that has memory,
and the function of that memory is shaped by reinforcement learning, and this
system is trained on a series of interrelated tasks, this is going to happen. You can’t stop it.”
…The account detailed by Botvinick and Wang et al. strikes me as a relatively clear
example of mesa-optimization, and I interpret it as tentative evidence that the attractor toward mesa-optimization is
strong.
Reinforcement learning (RL) algorithms update an agent’s parameters according to one of several possible rules,
discovered manually through years of research. Automating the discovery of update rules from data could lead to more
efficient algorithms, or algorithms that are better adapted to specific environments. Although there have been prior
attempts at addressing this significant scientific challenge, it remains an open question whether it is feasible to
discover alternatives to fundamental concepts of RL such as value functions and temporal-difference learning. This paper
introduces a new meta-learning approach that discovers an entire update rule which includes both ‘what to predict’
(e.g. value functions) and ‘how to learn from it’ (e.g. bootstrapping) by interacting with a set of environments.
The output of this method is an RL algorithm that we call Learned Policy Gradient (LPG). Empirical results show that our method discovers its own alternative to the concept of
value functions. Furthermore it discovers a bootstrapping mechanism to maintain and
use its predictions. Surprisingly, when trained solely on toy environments, LPG
generalises effectively to complex Atari games and achieves non-trivial performance. This shows the potential to
discover general RL algorithms from data.
Lifelong learning and adaptability are two defining aspects of biological agents. Modern reinforcement learning (RL) approaches have shown significant progress in solving complex tasks,
however once training is concluded, the found solutions are typically static and incapable of adapting to new information
or perturbations. While it is still not completely understood how biological brains learn and adapt so efficiently from
experience, it is believed that synaptic plasticity plays a prominent role in this process. Inspired by this biological
mechanism, we propose a search method that, instead of optimizing the weight parameters of neural networks directly, only
searches for synapse-specific Hebbian learning rules that allow the network to continuously self-organize its weights
during the lifetime of the agent. We demonstrate our approach on several reinforcement learning tasks with different
sensory modalities and more than 450K trainable plasticity parameters. We find that starting from completely random
weights, the discovered Hebbian rules enable an agent to navigate a dynamical 2D-pixel environment; likewise they allow a
simulated 3D quadrupedal robot to learn how to walk while adapting to morphological damage not seen during training and in
the absence of any explicit reward or error signal in less than 100×teps. Code is available at
https://github.com/enajx/HebbianMetaLearning.
This paper seeks to establish a framework for directing a society of simple, specialized, self-interested agents to
solve what traditionally are posed as monolithic single-agent sequential decision problems. What makes it challenging to
use a decentralized approach to collectively optimize a central objective is the difficulty in characterizing the
equilibrium strategy profile of non-cooperative games. To overcome this challenge, we design a mechanism for defining the
learning environment of each agent for which we know that the optimal solution for the global objective coincides with a
Nash equilibrium strategy profile of the agents optimizing their own local objectives. The society functions as an economy
of agents that learn the credit assignment process itself by buying and selling to each other the right to operate on the
environment state. We derive a class of decentralized reinforcement learning
algorithms that are broadly applicable not only to standard reinforcement learning
but also for selecting options in semi-MDPs and dynamically composing
computation graphs. Lastly, we demonstrate the potential advantages of a society’s inherent modular structure for
more efficient transfer learning.
Interest in biologically inspired alternatives to backpropagation is driven by
the desire to both advance connections between deep learning and neuroscience and address backpropagation’s shortcomings on
tasks such as online, continual learning. However, local synaptic learning rules like those employed by the brain have so
far failed to match the performance of backpropagation in deep networks. In this
study, we employ meta-learning to discover networks that learn using feedback connections and local, biologically inspired
learning rules. Importantly, the feedback connections are not tied to the feedforward weights, avoiding biologically
implausible weight transport. Our experiments show that meta-trained networks effectively use feedback connections to
perform online credit assignment in multi-layer architectures. Surprisingly, this approach matches or exceeds a
state-of-the-art gradient-based online meta-learning algorithm on regression and classification tasks, excelling in
particular at continual learning. Analysis of the weight updates employed by these models reveals that they differ
qualitatively from gradient descent in a way that reduces interference between updates. Our results suggest the existence
of a class of biologically plausible learning mechanisms that not only match gradient descent-based learning, but also
overcome its limitations.
We propose the challenge of rapid task-solving in novel environments (RTS),
wherein an agent must solve a series of tasks as rapidly aspossible in an unfamiliar environment. An
effective RTS agent must balance between exploring the unfamiliar environment
and solving its current task, all while building a model of the new environment over which it can plan when faced with
later tasks. While modern deep RL agents exhibit some of these abilities in isolation, none are suitable for the full
RTS challenge. To enable progress toward RTS,
weintroduce two challenge domains: (1) a minimal RTS challenge
called the Memory&Planning Game and (2) One-Shot StreetLearn Navigation, which introduces scale and complexity
from real-world data. We demonstrate that state-of-the-art deep RL agents fail at RTS in both domains, and that this failure is due to an inability to plan over
gathered knowledge. We develop Episodic Planning Networks (EPNs) andshow that deep-RL agents with EPNs excel at RTS, outperforming the nearest baseline by factors of 2–3 and learning to navigate held-out
StreetLearn maps within a single episode. We show that EPNs learn to
execute a value iteration-like planning algorithm and that they generalize to situations beyond their training experience.
algorithm and that they generalize to situations beyond their training experience.
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically
task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from
simple instructions—something which current NLP systems still largely struggle
to do.
Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even
reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10× more than any
previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot
demonstrations specified purely via text interaction with the model. GPT-3
achieves strong performanceon many NLP datasets, including translation,
question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain
adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the
same time, we also identify some datasets where GPT-3’s few-shotlearning still struggles, as well as some datasets where GPT-3 faces
methodological issues related to training on large web corpora.
Finally, we find that GPT-3 can generate samples of news articles which
human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of
this finding and of GPT-3 in general.
…The precise architectural parameters for each model are chosen based on computational efficiency and load-balancing in
the layout of models across GPU’s. Previous work[KMH+20] suggests that validation loss is not strongly sensitive to these parameters
within a reasonably broad range.
Neural Architecture Search (NAS) explores a large space of
architectural motifs—a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating
it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired
by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial
Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the
Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned
synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in
the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step
of NAS. Unlike other neural network-based prediction models that parse the
structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the
actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this
paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly
higher accuracy, especially when insufficient ground truth data is available. Our hope is that this work can inspire a new
research direction in studying the performance of extracted components of models in an alternative controlled setting.
When making decisions, people often overlook critical information or are overly swayed by irrelevant information. A
common approach to mitigate these biases is to provide decision-makers, especially professionals such as medical doctors,
with decision aids, such as decision trees and flowcharts. Designing effective decision aids is a difficult problem. We
propose that recently developed reinforcement learning methods for discovering clever heuristics for good decision-making
can be partially leveraged to assist human experts in this design process. One of the biggest remaining obstacles to
leveraging the aforementioned methods is that the policies they learn are opaque to people. To solve this problem, we
introduce AI-Interpret: a general method for transforming idiosyncratic policies into simple and interpretable
descriptions. Our algorithm combines recent advances in imitation learning and program induction with a new clustering
method for identifying a large subset of demonstrations that can be accurately described by a simple, high-performing
decision rule. We evaluate our new algorithm and employ it to translate information-acquisition policies discovered through
metalevel reinforcement learning. The results of large behavioral experiments showed that prividing the decision rules
generated by AI-Interpret as flowcharts significantly improved people’s planning strategies and decisions across three
diferent classes of sequential decision problems. Moreover, another experiment revealed that this approach is
statistically-significantly more effective than training people by giving them performance feedback. Finally, a series of
ablation studies confirmed that AI-Interpret is critical to the discovery of interpretable decision rules. We conclude that
the methods and findings presented herein are an important step towards leveraging automatic strategy discovery to improve
human decision-making.
Robotic insertion tasks are characterized by contact and friction mechanics, making them challenging for conventional
feedback control methods due to unmodeled physical effects. Reinforcement learning
(RL) is a promising approach for learning control policies in such settings. However, RL can be unsafe during exploration
and might require a large amount of real-world training data, which is expensive to collect. In this paper, we study how to
use meta-reinforcement learning to solve the bulk of the problem in simulation by solving a family of simulated industrial
insertion tasks and then adapt policies quickly in the real world. We demonstrate our approach by training an agent to
successfully perform challenging real-world insertion tasks using less than 20 trials of real-world experience. Videos and
other material are available at https://pearl-insertion.github.io/
The field of meta-learning, or learning-to-learn, has seen a dramatic rise in interest in recent years. Contrary to
conventional approaches to AI where tasks are solved from scratch using a fixed learning algorithm, meta-learning aims to
improve the learning algorithm itself, given the experience of multiple learning episodes. This paradigm provides an
opportunity to tackle many conventional challenges of deep learning, including data and computation bottlenecks, as well as
generalization. This survey describes the contemporary meta-learning landscape. We first discuss definitions of
meta-learning and position it with respect to related fields, such as transfer learning and hyperparameter optimization. We
then propose a new taxonomy that provides a more comprehensive breakdown of the space of meta-learning methods today. We
survey promising applications and successes of meta-learning such as few-shot learning and reinforcement learning. Finally,
we discuss outstanding challenges and promising areas for future research.
Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging
learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction
is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and
solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima.
However, the original POET was unable to demonstrate its full creative potential
because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of
a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically
validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its
full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic
innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to
potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents
should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to
enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode
environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended
innovation. Enhanced POET produces a diverse range of sophisticated behaviors
that solve a wide range of environmental challenges, many of which cannot be solved through other means.
Placement Optimization is an important problem in systems and chip design, which consists of mapping the nodes of a
graph onto a limited set of resources to optimize for an objective, subject to constraints. In this paper, we start by
motivating reinforcement learning as a solution to the placement problem. We then
give an overview of what deep reinforcement learning is. We next formulate the
placement problem as a reinforcement learning problem and show how this problem can
be solved with policy gradient optimization. Finally, we describe lessons we have learned from training deep reinforcement
learning policies across a variety of placement optimization problems.
AlphaZero has been very successful in many games. Unfortunately, it still consumes a huge amount of computing resources,
the majority of which is spent in self-play. Hyperparameter tuning exacerbates the training cost since each hyperparameter
configuration requires its own time to train one run, during which it will generate its own self-play records. As a result,
multiple runs are usually needed for different hyperparameter configurations. This paper proposes using population based
training (PBT) to help tune hyperparameters
dynamically and improve strength during training time. Another significant advantage is that this method requires a single
run only, while incurring a small additional time cost, since the time for generating self-play records remains unchanged
though the time for optimization is increased following the AlphaZero training algorithm. In our experiments for 9×9 Go, the PBT method is able to achieve a higher win rate for 9×9 Go than the baselines,
each with its own hyperparameter configuration and trained individually. For 19×19 Go, with PBT, we are able to obtain improvements in playing strength. Specifically,
the PBTagent can obtain up to 74% win rate against ELF OpenGo, an open-source state-of-the-art AlphaZero
program using a neural network of a comparable capacity. This is compared to a saturated non-PBT agent, which achieves a win rate of 47% against ELF OpenGo under the same circumstances.
We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an
agent’s life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We
formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of
curiosity mechanisms that dynamically adapt the agent’s reward signal, and an inner loop will perform standard
reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural
network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to
meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs
combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We
demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or
better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs,
acrobot, lunar lander, ant and hopper.
Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort
to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused
on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks—or
similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically
discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate
this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the
vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on
tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the
top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution
adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe
these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for
the field.
AutoML-Zero aims to automatically discover computer programs that can solve machine learning tasks, starting from empty
or random programs and using only basic math operations. The goal is to simultaneously search for all aspects of an ML
algorithm—including the model structure and the learning strategy—while employing minimal human bias.
GIF for the experiment progress
Despite AutoML-Zero’s challenging search space, evolutionary search shows promising results by discovering
linear regression with gradient descent, 2-layer neural networks with backpropagation, and even algorithms that surpass hand designed baselines of comparable
complexity. The figure above shows an example sequence of discoveries from one of our experiments, evolving algorithms to
solve binary classification tasks. Notably, the evolved algorithms can be interpreted. Below is an analysis of the
best evolved algorithm: the search process “invented” techniques like bilinear interactions, weight averaging, normalized
gradient, and data augmentation (by
adding noise to the inputs).
GIF for the interpretation of the best evolved algorithm
More examples, analysis, and details can be found in the paper.
Covariant.ai has developed a platform that consists of off-the-shelf robot arms equipped with cameras, a special
gripper, and plenty of computer power for figuring out how to grasp objects tossed into warehouse bins. The company,
emerging from stealth Wednesday, announced the first commercial installations of its AI-equipped robots: picking boxes and
bags of products for a German electronics retailer called Obeta.
…The company was founded in 2017 by Pieter Abbeel, a prominent AI professor at UC Berkeley, and several of his students.
Abbeel pioneered the application of machine learning to robotics, and he made a name for himself in academic circles in
2010 by developing a robot capable of folding laundry (albeit very slowly). Covariant uses a range of AI techniques to
teach robots how to grasp unfamiliar objects. These include reinforcement learning,
in which an algorithm trains itself through trial and error, a little like the way animals learn through positive and
negative feedback…Besides reinforcement learning, Abbeel says his company’s robots make use of imitation learning, a way of
learning by observing demonstrations of perception and grasping by another algorithm, and meta-learning, a way of refining
the learning process itself. Abbeel says the system can adapt and improve when a new batch of items arrive. “It’s training
on the fly”, he says. “I don’t think anybody else is doing that in the real world.”
…But reinforcement learning is finicky and needs lots of computer power. “I used
to be skeptical about reinforcement learning, but I’m not anymore”, says Hinton, a
professor at the University of Toronto who also works part time at Google. Hinton says the amount of computer power needed
to make reinforcement learning work has often seemed prohibitive, so it is striking
to see commercial success. He says it is particularly impressive that Covariant’s system has been running in a commercial
setting for a prolonged period.
…Peter Puchwein, vice president of innovation at Knapp, says he is particularly impressed by the way Covariant.ai’s
robots can grasp even products in transparent bags, which can be difficult for cameras to perceive. “Even as a human being,
if you have a box with 20 products in poly bags, it’s really hard to take just one out”, he says…Late last year, the
international robot maker ABB ran a contest. It invited 20 companies to design
software for its robot arms that could sort through bins of random items, from cubes to plastic bags filled with other
objects. Ten of the companies were based in Europe, and the other half were in the United States. Most came nowhere close
to passing the test. A few could handle most tasks but failed on the trickier cases. Covariant was the only company that
could handle every task as swiftly and efficiently as a human. “We were trying to find weaknesses”, said Marc Segura,
managing director of service robotics at ABB. “It is easy to reach a
certain level on these tests, but it is super difficult not to show any weaknesses.”
With the success of modern machine learning, it is becoming increasingly important to understand and control how
learning algorithms interact. Unfortunately, negative results from game theory show there is little hope of understanding
or controlling general n-player games. We therefore introduce smooth markets (SM-games), a class of n-player games with
pairwise zero sum interactions. SM-games codify a common design pattern in machine learning that includes (some) GANs, adversarial training, and other recent algorithms. We show that SM-games are amenable to
analysis and optimization using first-order methods.
The ability to learn new concepts with small amounts of data is a critical aspect of intelligence that has proven
challenging for deep learning methods. Meta-learning has emerged as a promising technique for leveraging data from previous
tasks to enable efficient learning of new tasks. However, most meta-learning algorithms implicitly require that the
meta-training tasks be mutually-exclusive, such that no single model can solve all of the tasks at once. For example, when
creating tasks for few-shot image classification, prior work uses a per-task random assignment of image classes to N-way
classification labels. If this is not done, the meta-learner can ignore the task training data and learn a single model
that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes. This
requirement means that the user must take great care in designing the tasks, for example by shuffling labels or removing
task identifying information from the inputs. In some domains, this makes meta-learning entirely inapplicable. In this
paper, we address this challenge by designing a meta-regularization objective using information theory that places
precedence on data-driven adaptation. This causes the meta-learner to decide what must be learned from the task training
data and what should be inferred from the task testing input. By doing so, our algorithm can successfully use data from
non-mutually-exclusive tasks to efficiently adapt to novel tasks. We demonstrate its applicability to both contextual and
gradient-based meta-learning algorithms, and apply it in practical settings where applying standard meta-learning has been
difficult. Our approach substantially outperforms standard meta-learning algorithms in these settings.
In this report, we introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed
to benchmark both sample efficiency and generalization in reinforcement learning.
We believe that the community will benefit from increased access to high quality training environments, and we provide
detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment
distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of
procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that
larger models substantially improve both sample efficiency and generalization.
…We want the best of both worlds: a benchmark comprised of many diverse environments, each of which fundamentally
requires generalization. To fulfill this need, we have created Procgen Benchmark.
CoinRun [Cobbe et al 2018] now serves as the inaugural
environment in Procgen Benchmark, contributing its diversity to a greater whole.
…We’ve found that all of the Procgen environments require training on 500–1000 different levels before they can
generalize to new levels, which suggests that standard RL benchmarks need much more diversity within each environment.
Procgen Benchmark has become the standard research platform used by the OpenAI RL
team, and we hope that it accelerates the community in creating better RL algorithms. [cf Neural MMO]
Meta-learning, also known as “learning to learn”, intends to design models that can learn new skills or adapt to new
environments rapidly with a few training examples. There are three common approaches: 1. learn an efficient distance metric
(metric-based); 2. use (recurrent) network with external or internal memory (model-based); 3. optimize the model parameters
explicitly for fast learning (optimization-based).
…We expect a good meta-learning model capable of well adapting or generalizing to new tasks and new environments that
have never been encountered during training time. The adaptation process, essentially a mini learning session, happens
during test but with a limited exposure to the new task configurations. Eventually, the adapted model can complete new
tasks. This is why meta-learning is also known as learning to
learn.
Define the Meta-Learning Problem · A Simple View · Training in the Same Way as Testing · Learner and Meta-Learner ·
Common Approaches · Metric-Based · Convolutional Siamese Neural Network · Matching Networks · Simple Embedding · Full
Context Embeddings · Relation
Network · Prototypical Networks · Model-Based · Memory-Augmented Neural Networks · MANN for Meta-Learning · Addressing Mechanism for Meta-Learning · Meta Networks · Fast
Weights · Model Components · Training Process · Optimization-Based · LSTM
Meta-Learner · Why LSTM?· Model Setup · MAML · First-Order MAML · Reptile · The OptimizationAssumption · Reptile vs FOMAML · Reference
[HTML version of Freeman et al 2019, with videos.]
Much of model-based reinforcement learning involves learning a model of an
agent’s world, and training an agent to leverage this model to perform a task more efficiently. While these models are
demonstrably useful for agents, every naturally occurring model of the world of which we are aware—eg., a brain—arose as
the byproduct of competing evolutionary pressures for survival, not minimization of a supervised forward-predictive loss
via gradient descent. That useful models can arise out of the messy and slow optimization process of evolution suggests
that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances. Crucially, this
optimization process need not explicitly be a forward-predictive loss. In this work, we introduce a modification to
traditional reinforcement learning which we call observational dropout, whereby we limit the agents ability to
observe the real environment at each timestep. In doing so, we can coerce an agent into learning a world model to
fill in the observation gaps during reinforcement learning. We show that the emerged
world model, while not explicitly trained to predict the future, can help the agent learn key skills required to perform
well in its environment.
Much of model-based reinforcement learning involves learning a model of an
agent’s world, and training an agent to leverage this model to perform a task more efficiently. While these models are
demonstrably useful for agents, every naturally occurring model of the world of which we are aware—e.g., a brain—arose as
the byproduct of competing evolutionary pressures for survival, not minimization of a supervised forward-predictive loss
via gradient descent. That useful models can arise out of the messy and slow optimization process of evolution suggests
that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances. Crucially, this
optimization process need not explicitly be a forward-predictive loss. In this work, we introduce a modification to
traditional reinforcement learning which we call observational dropout, whereby we limit the agents ability to observe the
real environment at each timestep. In doing so, we can coerce an agent into learning a world model to fill in the
observation gaps during reinforcement learning. We show that the emerged world
model, while not explicitly trained to predict the future, can help the agent learn key skills required to perform well in
its environment. Videos of our results available at https://learningtopredict.github.io/
Meta-reinforcement learning algorithms can enable robots to acquire new skills
much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on
meta-reinforcement learning focuses on task distributions that are very narrow. For
example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as
different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more
quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new
behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new
behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation
tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new,
held-out tasks. We evaluate 7 state-of-the-art meta-reinforcement learning and
multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different
object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the
same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for
future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the
full potential of these methods.
We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented
complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic
domain randomization (ADR) and a robotplatform built for machine
learning. ADR automatically generates a distribution over randomized
environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies,memory-augmented
models trained on an ADR-generated distribution of environments show clear signs
of emergent meta-learning at test time. The combination of ADR with our custom
robot platform allows us to solve a Rubik’s cube with a humanoid robot hand, which involves both control and state
estimation problems. Videos summarizing our results are available:
https://openai.com/blog/solving-rubiks-cube/
We’ve trained a pair of neural networks to solve the Rubik’s Cube with a human-like robot hand. The neural networks are
trained entirely in simulation, using the same reinforcement learning code as
OpenAI Five paired with a new technique called Automatic Domain Randomization
(ADR). The system can handle situations it never saw during training, such
as being prodded by a stuffed giraffe. This shows that reinforcement learning isn’t just a tool for virtual tasks, but can
solve physical-world problems requiring unprecedented dexterity.
…Since May 2017, we’ve been trying to train a human-like robotic hand to solve the Rubik’s Cube. We set this goal
because we believe that successfully training such a robotic hand to do complex manipulation tasks lays the foundation for
general-purpose robots. We solved the Rubik’s Cube in simulation in July 2017. But as of July 2018, we could only
manipulate a block on the robot. Now, we’ve reached our initial goal. Solving a Rubik’s Cube one-handed is a challenging
task even for humans, and it takes children several years to gain the dexterity required to master it. Our robot still
hasn’t perfected its technique though, as it solves the Rubik’s Cube 60% of the time (and only 20% of the time for a
maximally difficult scramble).
“Gradient Descent: The Ultimate Optimizer”, Kartik Chandra, Erik Meijer, Samantha
Andow, Emilio Arroyo-Fang, Irene Dea, Johann George, Melissa Grueter, Basil Hosmer, Steffi Stumpos, Alanna Tempest, Shannon
Yang (2019-09-29):
Working with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer’s
hyperparameters, such as the learning rate. There exist many techniques for automated hyperparameter optimization, but they
typically introduce even more hyperparameters to control the hyperparameter optimization process. We propose to instead
learn the hyperparameters themselves by gradient descent, and furthermore to learn the hyper-hyperparameters by gradient
descent as well, and so on ad infinitum. As these towers of gradient-based optimizers grow, they become
statistically-significantly less sensitive to the choice of top-level hyperparameters, hence decreasing the burden on the
user to search for optimal values.
“Multiplicative Interactions and Where to Find Them”, Siddhant M. Jayakumar,
Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, Razvan
Pascanu (2019-09-25):
TL;DR: We explore the role of multiplicative interaction as a unifying framework to describe a range of
classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic
convolutions amongst others.
We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern
neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst
others.
Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though
this often not emphasized and thus under-appreciated. We begin by showing that such layers strictly enrich the
representable function classes of neural networks. We conjecture that multiplicative interactions offer a particularly
powerful inductive bias when fusing multiple streams of information or when conditional computation is
required. We therefore argue that they should be considered in many situation where multiple compute or information paths
need to be combined, in place of the simple and oft-used concatenation operation. Finally, we back up our claims and
demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling
tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support
of multiplicative interactions playing a more prominent role when designing new neural network architectures.
A core capability of intelligent systems is the ability to quickly learn new tasks by drawing on prior experience.
Gradient (or optimization) based meta-learning has recently emerged as an effective approach for few-shot learning. In this
formulation, meta-parameters are learned in the outer loop, while task-specific models are learned in the inner-loop, by
using only a small amount of data from the current task. A key challenge in scaling these approaches is the need to
differentiate through the inner loop learning process, which can impose considerable computational and memory burdens. By
drawing upon implicit differentiation, we develop the implicit MAML
algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner
loop optimizer. This effectively decouples the meta-gradient computation from the choice of inner loop optimizer. As a
result, our approach is agnostic to the choice of inner loop optimizer and can gracefully handle many gradient steps
without vanishing gradients or memory constraints. Theoretically, we prove that implicit MAML can compute accurate meta-gradients with a memory footprint that is, up to small
constant factors, no more than that which is required to compute a single inner loop gradient and at no overall increase in
the total computational cost. Experimentally, we show that these benefits of implicit MAML translate into empirical gains on few-shot image recognition benchmarks.
Artificial neural networks (ANNs) have undergone a revolution,
catalyzed by better supervised learning algorithms. However, in stark contrast to young animals (including humans),
training such networks requires enormous numbers of labeled examples, leading to the belief that animals must rely instead
mainly on unsupervised learning. Here we argue that most animal behavior is not the result of clever learning
algorithms—supervised or unsupervised—but is encoded in the genome. Specifically, animals are born with highly structured
brain connectivity, which enables them to learn very rapidly. Because the wiring diagram is far too complex to be specified
explicitly in the genome, it must be compressed through a “genomic bottleneck”. The genomic bottleneck suggests a
path toward ANNs capable of rapid learning.
…As the name implies, ANNs were invented in an attempt to build
artificial systems based on computational principles used by the nervous system5. In what follows, we suggest that
additional principles from neuroscience might accelerate the goal of achieving artificial mouse, and eventually human,
intelligence. We argue that in contrast to ANNs, animals rely heavily on a
combination of both learned and innate mechanisms. These innate processes arise through evolution, are encoded in
the genome, and take the form of rules for wiring up the brain6. Specifically, we introduce the notion of the “genomic
bottleneck”—the compression into the genome of whatever innate processes are captured by evolution—as a regularizing
constraint on the rules for wiring up a brain. We discuss the implications of these observations for generating
next-generation machine algorithms.
…In this view, supervised learning in ANNs should not be viewed as the
analog of learning in animals. Instead, since most of the data that contribute an animal’s fitness are encoded by evolution
into the genome, it would perhaps be just as accurate (or inaccurate) to rename it “supervised evolution.” Such a renaming
would emphasize that “supervised learning” in ANNs is really recapitulating the
extraction of statistical regularities that occurs in animals by both evolution and learning. In animals, there are
two nested optimization processes: an outer “evolution” loop acting on a generational timescale, and an inner “learning”
loop, which acts on the lifetime of a single individual. Supervised (artificial) evolution may be much faster than natural
evolution, which succeeds only because it can benefit from the enormous amount of data represented by the life experiences
of quadrillions of individuals over hundreds of millions of years.
Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However,
building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to
more areas. Automated machine learning (AutoML) becomes a promising solution to build a DL system without human assistance,
and a growing number of researchers focus on AutoML. In this paper, we provide a comprehensive and up-to-date review of the
state-of-the-art (SOTA) in AutoML. First, we introduce AutoML methods
according to the pipeline, covering data preparation, feature engineering, hyperparameter optimization, and neural
architecture search (NAS). We focus more on NAS, as it is currently very hot sub-topic of AutoML. We summarize the performance of the
representative NAS algorithms on the CIFAR-10
andImageNet datasets and further discuss several worthy studying
directions of NAS methods:one/two-stage NAS, one-shot NAS, and joint hyperparameter and
architecture optimization. Finally, we discuss some open problems of the existing AutoML methods for future research.
We augment recurrent neural networks with an external memory mechanism that builds upon recent progress in metalearning.
We conceptualize this memory as a rapidly adaptable function that we parameterize as a deep neural network. Reading from
the neural memory function amounts to pushing an input (the key vector) through the function to produce an output (the
value vector). Writing to memory means changing the function; specifically, updating the parameters of the neural network
to encode desired information. We leverage training and algorithmic techniques from metalearning to update the neural
memory function in one shot. The proposed memory-augmented model achieves strong performance on a variety of learning
problems, from supervised question answering to reinforcement learning.
[Review/discussion] Meta-RL is meta-learning on reinforcement learning tasks. After trained over a distribution of
tasks, the agent is able to solve a new task by developing a new RL algorithm with its internal activity dynamics. This
post starts with the origin of meta-RL and then dives into three key components of meta-RL…, a good meta-learning model is expected to generalize to new tasks or new environments
that have never been encountered during training. The adaptation process, essentially a mini learning session,
happens at test with limited exposure to the new configurations. Even without any explicit fine-tuning (no gradient
backpropagation on trainable variables), the meta-learning model autonomously
adjusts internal hidden states to learn.
In unsupervised learning, collecting more data is not always a costly process unlike the training. For example, it is
not hard to enlarge the 40GB WebText used for training GPT-2 by modifying its sampling methodology considering how
many webpages there are in the Internet. On the other hand, given that training on this dataset already costs tens of
thousands of dollars, training on a larger dataset naively is not cost-wise feasible. In this paper, we suggest to train on
a larger dataset for only one epoch unlike the current practice, in which the unsupervised models are trained for from tens
to hundreds of epochs. Furthermore, we suggest to adjust the model size and the number of iterations to be performed
appropriately. We show that the performance of Transformer language model becomes
dramatically improved in this way, especially if the original number of epochs is greater. For example, by replacing the
training for 10 epochs with the one epoch training, this translates to 1.9–3.3× speedup in wall-clock time in our settings
and more if the original number of epochs is greater. Under one epoch training, no overfitting occurs, and regularization
method does nothing but slows down the training. Also, the curve of test loss over iterations follows power-law
extensively. We compare the wall-clock time of the training of models with different parameter budget under one epoch
training, and we show that size/iteration adjustment based on our proposed heuristics leads to 1–2.7× speedup in our
cases. With the two methods combined, we achieve 3.3–5.1× speedup. Finally, we speculate various implications of one epoch
training and size/iteration adjustment. In particular, based on our analysis we believe that we can reduce the cost to
train the state-of-the-art models as BERTand GPT-2
dramatically, maybe even by the factor of 10.
The 2019 ICML edition ofDavid Abel’s famous conference
notes: he goes to as many presentations and talks as possible, jotting down opinionated summaries & equations,
with a particular focus on DRL. Topics covered:
Tutorial: PAC-Bayes Theory (Part II) · PAC-Bayes Theory · PAC-Bayes and Task Awareness ·
Tutorial: Meta-Learning · Two Ways to View Meta-Learning · Meta-Learning Algorithms · Meta-Reinforcement Learning ·
Challenges and Frontiers in Meta Learning · Tuesday June: Main Conference Best Paper Talk: Challenging Assumptions in
Learning Disentangled Representations Contributed Talks: Deep RL · DQN and Time Discretization · Nonlinear Distributional Gradient TD Learning · Composing
Entropic Policies using Divergence Correction · TibGM: A Graphical Model Approach for RL · Multi-Agent Adversarial
IRL · Policy Consolidation for Continual RL · Off-Policy Evaluation Deep RL
w/o Exploration · Random Expert Distillation · Revisiting the Softmax Bellman Operator · Contributed Talks: RL Theory ·
Distributional RL for Efficient Exploration · Optimistic Policy Optimization via Importance Sampling · Neural Logic RL ·
Learning to Collaborate in MDPs · Predictor-Corrector Policy Optimization
·Learning a Prior over Intent via Meta IRL · DeepMDP: Learning Late Space Models for RL · Importance Sampling Policy Evaluation · Learning
from a Learner · Separating Value Functions Across Time-Scales · Learning Action Representations in RL · Bayesian
Counterfactual Risk Minimization · Per-Decision Option Counting · Problem Dependent Regret Bounds in RL · A Theory of
Regularized MDPs · Discovering Options for Exploration by Minimizing Cover Time
· Policy Certificates: Towards Accountable RL · Action Robust RL · The Value Function Polytope · Wednesday June: Main
Conference Contributed Talks: Multitask and Lifelong Learning · Domain Agnostic Learning with Disentangled
Representations · Composing Value Functions in RL · CAVIA: Fast Context
Adaptation via Meta Learning · Gradient Based Meta-Learning · Towards Understanding Knowledge Distillation · Transferable
Adversarial Training · Contributed Talks: RL Theory · Provably Efficient Imitation Learning from Observation Alone · Dead
Ends and Secure Exploration · Statistics and Samples in Distributional RL · Hessian Aided Policy Gradient · Maximum
Entropy Exploration · Combining Multiple Models for Off-Policy Evaluation ·
Sample-Optimal ParametricQ-Learning Using Linear Features · Transfer of Samples in Policy Search · Exploration Conscious RL
Revisited · Kernel Based RL in Robust MDPs · Thursday June: Main Conference
Contributed Talks: RL · Batch Policy learning under Constraints · Quantifying Generalization in RL · Learning
Latent Dynamics for Planning from Pixels · Projections for Approximate Policy
Iteration · Learning Structured Decision Problems with Unawareness · Calibrated Model-Based Deep RL · RL in Configurable
Continuous Environments · Target-Based Temporal-Difference Learning · Linearized Control: Stable Algorithms and Complexity
Guarantees · Contributed Talks: Deep Learning Theory · Why do Larger Models Generalize Better? · On the Spectral Bias of
Neural Nets · Recursive Sketches for Modular Deep Learning · Zero-Shot Knowledge Distillation in Deep Networks ·
Convergence Theory for Deep Learning via Over-Parameterization · Best Paper Award: Rates of Convergence for Sparse Gaussian
Process Regression · Friday June: Workshops Workshop: AI for Climate Change · John Platt on What ML can do to help Climate
Change · Jack Kelly: Why It’s Hard to Mitigate Climate Change, and How to Do Better, Andrew Ng: Tackling Climate Change
with AI through Collaboration · Workshop: RL for Real Life · Panel Discussion · Workshop: Real World Sequential Decision
Making · Emma Brunskill on Efficient RL When Data is Costly · Miro Dudik: Doubly Robust Off-Policy Evaluation via
Shrinkage
Perhaps the most ambitious scientific quest in human history is the creation of general artificial intelligence, which
roughly means AI that is as smart or smarter than humans. The dominant approach in the machine learning community is to
attempt to discover each of the pieces required for intelligence, with the implicit assumption that some future group will
complete the Herculean task of figuring out how to combine all of those pieces into a complex thinking machine. I call this
the “manual AI approach”. This paper describes another exciting path that ultimately may be more successful at producing
general AI. It is based on the clear trend in machine learning that hand-designed solutions eventually are replaced by more
effective, learned solutions. The idea is to create an AI-generating algorithm (AI-GA), which automatically learns how to
produce general AI. Three Pillars are essential for the approach: (1) meta-learning architectures, (2) meta-learning the
learning algorithms themselves, and (3) generating effective learning environments. I argue that either approach could
produce general AI first, and both are scientifically worthwhile irrespective of which is the fastest path. Because both
are promising, yet the ML community is currently committed to the manual approach, I argue that our community should
increase its research investment in the AI-GA approach. To encourage such research, I describe promising work in each of
the Three Pillars. I also discuss AI-GA-specific safety and ethical considerations. Because it it may be the fastest path
to general AI and because it is inherently scientifically interesting to understand the conditions in which a simple
algorithm can produce general AI (as happened on Earth where Darwinian evolution produced human intelligence), I argue that
the pursuit of AI-GAs should be considered a new grand challenge of computer science research.
Model-agnostic meta-learning (MAML) is a meta-learning technique to
train a model on a multitude of learning tasks in a way that primes the model for few-shot learning of new tasks. The
MAML algorithm performs well on few-shot learning problems in classification,
regression, and fine-tuning of policy gradients in reinforcement learning, but comes with the need for costly
hyperparameter tuning for training stability. We address this shortcoming by introducing an extension to MAML, called Alpha MAML, to incorporate an online
hyperparameter adaptation scheme that eliminates the need to tune meta-learning and learning rates. Our results with the
Omniglot database demonstrate a substantial reduction in the need to tune MAML training hyperparameters and improvement to training stability with less sensitivity to
hyperparameter choice.
Recent AI research has given rise to powerful techniques for deep reinforcement learning. In their combination of
representation learning with reward-driven behavior, deep reinforcement learning would appear to have inherent interest for
psychology and neuroscience.
One reservation has been that deep reinforcement learning procedures demand large
amounts of training data, suggesting that these algorithms may differ fundamentally from those underlying human
learning.
While this concern applies to the initial wave of deep RL techniques, subsequent AI work has established methods that
allow deep RL systems to learn more quickly and efficiently. Two particularly interesting and promising techniques center,
respectively, on episodic memory and meta-learning. Alongside their interest as AI techniques, deep RL methods leveraging
episodic memory and meta-learning have direct and interesting implications for psychology and neuroscience. One subtle but
critically important insight which these techniques bring into focus is the fundamental connection between fast and slow
forms of learning.
Deep reinforcement learning (RL) methods have driven impressive advances in
artificial intelligence in recent years, exceeding human performance in domains ranging from Atari to Go to no-limit poker.
This progress has drawn the attention of cognitive scientists interested in understanding human learning. However, the
concern has been raised that deep RL may be too sample-inefficient—that is, it may simply be too slow—to provide a
plausible model of how humans learn. In the present review, we counter this critique by describing recently developed
techniques that allow deep RL to operate more nimbly, solving problems much more quickly than previous methods. Although
these techniques were developed in an AI context, we propose that they may have rich implications for psychology and
neuroscience. A key insight, arising from these AI methods, concerns the fundamental connection between fast RL and slower,
more incremental forms of learning.
Humans achieve efficient learning by relying on prior knowledge about the structure of naturally occurring tasks. There
is considerable interest in designing reinforcement learning (RL) algorithms with
similar properties. This includes proposals to learn the learning algorithm itself, an idea also known as meta learning.
One formal interpretation of this idea is as a partially observable multi-task RL problem in which task information is
hidden from the agent. Such unknown task problems can be reduced to Markov decision processes (MDPs) by augmenting an agent’s observations with an estimate of the belief about the task
based on past experience. However estimating the belief state is intractable in most partially-observed MDPs. We propose a method that separately learns the policy and the task belief by taking
advantage of various kinds of privileged information. Our approach can be very effective at solving standard meta-RL
environments, as well as a complex continuous control environment with sparse rewards and requiring long-term memory.
In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from
past experience to adapt to any task within a target class. Our goal is to equip the reader with the conceptual foundations
of this tool for building new, scalable agents that operate on broad domains. To do so, we present basic algorithmic
templates for building near-optimal predictors and reinforcement learners which behave as if they had a probabilistic model
that allowed them to efficiently exploit task structure. Furthermore, we recast memory-based meta-learning within a
Bayesian framework, showing that the meta-learned strategies are near-optimal because they amortize Bayes-filtered data,
where the adaptation is implemented in the memory dynamics as a state-machine of sufficient statistics. Essentially,
memory-based meta-learning translates the hard problem of probabilistic sequential inference into a regression problem.
Meta-learning is a tool that allows us to build sample-efficient learning systems. Here we show that, once
meta-trained, LSTM Meta-Learners aren’t just faster learners than their
sample-inefficient deep learning (DL) and reinforcement learning (RL) brethren, but
that they actually pursue fundamentally different learning trajectories. We study their learning dynamics on three sets of
structured tasks for which the corresponding learning dynamics of DL and RL systems have been previously described: linear
regression (Saxe et al., 2013), nonlinear regression (Rahaman et al., 2018; Xu et al., 2018), and contextual bandits
(Schaul et al., 2019). In each case, while sample-inefficient DL and RL Learners uncover the task structure in a
staggered manner, meta-trained LSTM Meta-Learners uncover almost all task
structure concurrently, congruent with the patterns expected from Bayes-optimal inference algorithms. This has implications
for research areas wherever the learning behaviour itself is of interest, such as safety, curriculum design, and
human-in-the-loop machine learning.
Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms. We study
the learning dynamics of reinforcement learning (RL), specifically a characteristic
coupling between learning and data generation that arises because RL agents control their future data distribution. In the
presence of function approximation, this coupling can lead to a problematic type of ‘ray interference’, characterized by
learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn
one thing at a time even when learning in parallel is better. We establish the conditions under which ray interference
occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting. We characterize
a number of its properties and discuss possible remedies.
Neural Architecture Search (NAS) has shown great success in automating
the design of neural networks, but the prohibitive amount of computations behind current NAS methods requires further investigations in improving the sample efficiency and the
network evaluation cost to get better results in a shorter time. In this paper, we present a novel scalable Monte
Carlo Tree Search (MCTS)based NAS
agent, named AlphaX, to tackle these two aspects. AlphaX improves the search efficiency by adaptively balancing the
exploration and exploitation at the state level, and by a Meta-Deep Neural Network (DNN) to predict network accuracies for biasing the search toward a promising region. To
amortize the network evaluation cost, AlphaX accelerates MCTS rollouts with a
distributed design and reduces the number of epochs in evaluating a network by transfer learning guided with
the tree structure in MCTS. In 12GPU days and 1000 samples, AlphaX found an architecture that reaches 97.84% top-1
accuracy on CIFAR-10, and 75.5% top-1 accuracy onImageNet, exceeding SOTANAS methods in both the accuracy and sampling efficiency. Particularly, we also
evaluate AlphaX on NASBench-101, a large scale NAS
dataset; AlphaX is 3× and 2.8× more sample efficient than Random Search and Regularized Evolution in finding the
global optimum. Finally, we show the searched architecture improves a variety of vision applications from Neural Style
Transfer, to Image Captioning and Object Detection.
Deep reinforcement learning algorithms require large amounts of experience to
learn an individual task. While in principle meta-reinforcement learning (meta-RL)
algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their
practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack
mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward
problems. In this paper, we address these challenges by developing an off-policy meta-RL algorithm that disentangles task
inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This
probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to
integrate these task variables with off-policy RL algorithms to achieve both meta-training and adaptation efficiency. Our
method outperforms prior algorithms in sample efficiency by 20–100× as well as in asymptotic performance on several meta-RL
benchmarks.
Evolution has produced a multi-scale mosaic of interacting adaptive units. Innovations arise when perturbations push
parts of the system away from stable equilibria into new regimes where previously well-adapted solutions no longer work.
Here we explore the hypothesis that multi-agent systems sometimes display intrinsic dynamics arising from competition and
cooperation that provide a naturally emergent curriculum, which we term an autocurriculum. The solution of one social task
often begets new social tasks, continually generating novel challenges, and thereby promoting innovation. Under certain
conditions these challenges may become increasingly complex over time, demanding that agents accumulate ever more
innovations.
Generative Adversarial Networks (GAN) boast impressive capacity to
generate realistic images. However, like much of the field of deep learning, they require an inordinate amount of data to
produce results, thereby limiting their usefulness in generating novelty. In the same vein, recent advances in
meta-learning have opened the door to many few-shot learning applications. In the present work, we propose Few-shot
Image Generation using Reptile (FIGR), a GAN
meta-trained with Reptile. Our model successfully generates novel images on both MNIST and Omniglot with as little as 4 images from an unseen class. We
further contribute FIGR-8, a new dataset for few-shot image generation, which
contains 1,548,944 icons categorized in over 18,409 classes. Trained on FIGR-8,
initial results show that our model can generalize to more advanced concepts (such as “bird” and “knife”) from as
few as 8 samples from a previously unseen class of images and as little as 10 training steps through those 8 images. This
work demonstrates the potential of training a GAN for few-shot image
generation and aims to set a new benchmark for future work in the domain.
“Malthusian Reinforcement Learning”, Joel Z. Leibo, Julien Perolat, Edward Hughes,
Steven Wheelwright, Adam H. Marblestone, Edgar Duéñez-Guzmán, Peter Sunehag, Iain Dunning, Thore Graepel (2018-12-17):
Here we explore a new algorithmic framework for multi-agent reinforcement learning, called Malthusian reinforcement learning, which extends self-play to include fitness-linked population size
dynamics that drive ongoing innovation. In Malthusian RL, increases in a subpopulation’s average return drive subsequent
increases in its size, just as Thomas Malthus argued in 1798 was the relationship between preindustrial income levels and
population growth. Malthusian reinforcement learning harnesses the competitive pressures arising from growing and shrinking
population size to drive agents to explore regions of state and policy spaces that they could not otherwise reach.
Furthermore, in environments where there are potential gains from specialization and division of labor, we show that
Malthusian reinforcement learning is better positioned to take advantage of such synergies than algorithms based on
self-play.
Deep reinforcement learning is the combination of reinforcement learning (RL) and
deep learning. This field of research has been able to solve a wide range of complex decision-making tasks that were
previously out of reach for a machine. Thus, deep RL opens up many new applications in domains such as healthcare,
robotics, smart grids, finance, and many more. This manuscript provides an introduction to deep reinforcement learning models, algorithms and techniques. Particular focus is on the aspects
related to generalization and how deep RL can be used for practical applications. We assume the reader is familiar with
basic machine learning concepts.
Description of emerging machine learning paradigm identified by commentator starspawn0: discussions of building
artificial brains typically presume either learning a brain architecture & parameters from scratch (AGI) or laboriously ‘scanning’ and reverse-engineering a biological brain in its entirety to
get a functioning artificial brain.
However, the rise of deep learning’s transfer learning & meta-learning shows a wide variety of intermediate approaches,
where ‘side data’ from natural brains can be used as scaffolding to guide & constrain standard deep learning methods. Such
approaches do not seek to ‘upload’ or ‘emulate’ any specific brain, they merely seek to imitate an average brain. A simple
example would be training a CNN to imitate eyetracking saliency data: what a human looks at while
playing a video game or driving is the important part of a scene, and the CNN doesn’t have to learn importance from scratch. A more complex example
would be using EEG as a ‘description’ of music in addition to the music
itself. fMRI data could be used to guide a NN to have a similar modularized
architecture with similar activation patterns given a particular stimulus as a human brain, which presumably is related to
human abilities to zero-shot/few-shot learn and generalize.
While a highly marginal approach at the moment compared to standard approaches like scaling up models & datasets, it is
largely untapped, and progress in VR headsets with eyetracking capabilities (intended for foveated rendering but usable for many other
purposes), brain imaging methods & BCIs has been more rapid than generally appreciated—in part thanks to
breakthroughs using DL itself, suggesting the potential for a positive feedback loop where a BCI breakthrough enables a better NN for BCIs and so
on.
The design of neural network architectures is an important component for achieving state-of-the-art performance with
machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures
automatically through clever construction of a search space paired with simple learning algorithms. Recent progress has
demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification
tasks. An open question is the degree to which such methods may generalize to new domains. In this work we explore the
construction of meta-learning techniques for dense image prediction focused on the tasks of scene parsing, person-part
segmentation, and semantic image segmentation. Constructing viable search spaces in this domain is challenging because of
the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a
survey of techniques in dense image prediction, we construct a recursive search space and demonstrate that even with
efficient random search, we can identify architectures that outperform human-invented architectures and achieve
state-of-the-art performance on three dense prediction tasks including 82.7% on Cityscapes (street scene parsing),
71.3% on PASCAL-Person-Part (person-part segmentation), and 87.9% on PASCALVOC 2012 (semantic image segmentation).
Additionally, the resulting architecture is more computationally efficient, requiring half the parameters and half the
computational cost as previous state of the art systems.
Recent progress in artificial intelligence through reinforcement learning (RL) has shown great success on increasingly
complex single-agent environments and two-player turn-based games. However, the real-world contains multiple agents, each
learning and acting independently to cooperate and compete with other agents, and environments reflecting this degree of
complexity remain an open challenge. In this work, we demonstrate for the first time that an agent can achieve human-level
in a popular 3D multiplayer first-person video game, Quake III Arena Capture the
Flag, using only pixels and game points as input. These results were achieved by a novel two-tier optimisation
process in which a population of independent RL agents are trained concurrently from thousands of parallel matches with
agents playing in teams together and against each other on randomly generated environments. Each agent in the population
learns its own internal reward signal to complement the sparse delayed reward from winning, and selects actions using a
novel temporally hierarchical representation that enables the agent to reason at multiple timescales. During game-play,
these agents display human-like behaviours such as navigating, following, and defending based on a rich learned
representation that is shown to encode high-level game knowledge. In an extensive tournament-style evaluation the trained
agents exceeded the win-rate of strong human players both as teammates and opponents, and proved far stronger than existing
state-of-the-art agents. These results demonstrate a significant jump in the capabilities of artificial agents, bringing us
closer to the goal of human-level intelligence.
Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate
gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is
available instead. This arises when an approximate gradient is easier to compute than the full gradient (e.g. in
meta-learning or unrolled optimization), or when a true gradient is intractable and is replaced with a surrogate
(e.g. in certain reinforcement learning applications, or when using synthetic
gradients). We propose Guided Evolutionary Strategies, a method for optimally using surrogate gradient directions along
with random search. We define a search distribution for evolutionary strategies that is elongated along a guiding subspace
spanned by the surrogate gradients. This allows us to estimate a descent direction which can then be passed to a
first-order optimizer. We analytically and numerically characterize the tradeoffs that result from tuning how strongly the
search distribution is stretched along the guiding subspace, and we use this to derive a setting of the hyperparameters
that works well across problems. Finally, we apply our method to example problems, demonstrating an improvement over both
standard evolutionary strategies and first-order methods (that directly follow the surrogate gradient). We provide a demo
of Guided ES at https://github.com/brain-research/guided-evolutionary-strategies
Active learning (AL) aims to enable training high performance classifiers with low annotation cost by predicting which
subset of unlabelled instances would be most beneficial to label. The importance of AL has motivated extensive research,
proposing a wide variety of manually designed AL algorithms with diverse theoretical and intuitive motivations. In contrast
to this body of research, we propose to treat active learning algorithm design as a meta-learning problem and learn the
best criterion from data. We model an active learning algorithm as a deep neural network that inputs the base learner state
and the unlabelled point set and predicts the best point to annotate next. Training this active query policy network with
reinforcement learning, produces the best non-myopic policy for a given dataset. The
key challenge in achieving a general solution to AL then becomes that of learner generalisation, particularly across
heterogeneous datasets. We propose a multi-task dataset-embedding approach that allows dataset-agnostic active learners to
be trained. Our evaluation shows that AL algorithms trained in this way can directly generalise across diverse
problems.
A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without
access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the
negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as
a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised
learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification
performance, and we meta-learn an algorithm—an unsupervised weight update rule—that produces representations useful for
this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function,
which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the
meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We
further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths,
and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from
image datasets to a text task.
We consider the problem of exploration in meta reinforcement learning. Two new
meta reinforcement learningalgorithms are suggested: E-MAML and E-RL2. Results are presented on a novel environment we call ‘Krazy
World’ and a set of maze environments. We show E-MAML and E-RL2
deliver better performance on tasks where exploration is important.
“Machine
Theory of Mind”, Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S. M. Ali Eslami,
Matthew Botvinick (2018-02-21):
Theory of mind (ToM; Premack & Woodruff, 1978) broadly refers to humans’ ability to represent the mental states of
others, including their desires, beliefs, and intentions. We propose to train a machine to build such models too. We design
a Theory of Mind neural network—a ToMnet—which uses meta-learning to build models of the agents it encounters, from
observations of their behaviour alone. Through this process, it acquires a strong prior model for agents’ behaviour, as
well as the ability to bootstrap to richer predictions about agents’ characteristics
and mental states using only a small number of behavioural observations. We apply the ToMnet to agents behaving in simple
gridworld environments, showing that it learns to model random, algorithmic, and deep reinforcement learning agents from varied populations, and that it passes classic ToM tasks such
as the “Sally-Anne” test (Wimmer & Perner, 1983; Baron-Cohen et al., 1985) of recognising that others can hold false
beliefs about the world. We argue that this system—which autonomously learns how to model other agents in its world—is an
important step forward for developing multi-agent AI systems, for building intermediating technology for machine-human
interaction, and for advancing the progress on interpretable AI.
Humans and animals are capable of learning a new behavior by observing others perform the skill just once. We consider
the problem of allowing a robot to do the same—learning from a raw video pixels of a human, even when there is substantial
domain shift in the perspective, environment, and embodiment between the robot and the observed human. Prior approaches to
this problem have hand-specified how human and robot actions correspond and often relied on explicit human pose detection
systems. In this work, we present an approach for one-shot learning from a video of a human by using human and robot
demonstration data from a variety of previous tasks to build up prior knowledge through meta-learning. Then, combining this
prior knowledge and only a single video demonstration from a human, the robot can perform the task that the human
demonstrated. We show experiments on both a PR2 arm and a Sawyer arm, demonstrating that after meta-learning, the robot can
learn to place, push, and pick-and-place new objects using just one video of a human performing the manipulation.
“Population Based Training of Neural Networks”, Max Jaderberg, Valentin Dalibard,
Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan,
Chrisantha Fernando, Koray Kavukcuoglu (2017-11-27):
Neural networks dominate the modern machine learning landscape, but their training and success still suffer from
sensitivity to empirical choices of hyperparameters such as model architecture, loss
function, and optimisation algorithm. In this work we present Population Based Training (PBT), a simple asynchronous optimisation algorithm which effectively
utilises a fixed computational budget to jointly optimise a population of models and their hyperparameters to maximise
performance. Importantly, PBT discovers a schedule of hyperparameter
settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole
course of training. With just a small modification to a typical distributed hyperparameter training framework, our method
allows robust and reliable training of models. We demonstrate the effectiveness of PBT on deep reinforcement learning problems, showing
faster wall-clock convergence and higher final performance of agents by optimising over a suite of hyperparameters. In
addition, we show the same method can be applied to supervised learning for machine translation, where PBT is used to maximise the BLEU score directly, and also to training
of Generative Adversarial Networks to maximise the Inception score of generated images. In all cases PBT results in the automatic discovery of hyperparameter schedules and model
selection which results in stable training and better final performance.
The promise of learning to learn for robotics rests on the hope that by extracting some information about the learning
process itself we can speed up subsequent similar learning tasks. Here, we introduce a computationally efficient online
meta-learning algorithm that builds and optimizes a memory model of the optimal learning rate landscape from previously
observed gradient behaviors. While performing task specific optimization, this memory of learning rates predicts how to
scale currently observed gradients. After applying the gradient scaling our meta-learner updates its internal memory based
on the observed effect its prediction had. Our meta-learner can be combined with any gradient-based optimizer, learns on
the fly and can be transferred to new optimization tasks. In our evaluations we show that our meta-learning algorithm
speeds up learning of MNIST classification and a variety of learning
control tasks, either in batch or online learning settings.
In order for a robot to be a generalist that can perform a wide range of jobs, it must be able to acquire a wide variety
of skills quickly and efficiently in complex unstructured environments. High-capacity models such as deep neural networks
can enable a robot to represent complex skills, but learning each skill from scratch then becomes infeasible. In this work,
we present a meta-imitation learning method that enables a robot to learn how to learn more efficiently, allowing it to
acquire new skills from just a single demonstration. Unlike prior methods for one-shot imitation, our method can scale to
raw pixel inputs and requires data from significantly fewer prior tasks for effective learning of new skills. Our
experiments on both simulated and real robot platforms demonstrate the ability to learn new tasks, end-to-end, from a
single visual demonstration.
Deep neural networks excel in regimes with large amounts of data, but tend to struggle when data is scarce or when they
need to adapt quickly to changes in the task. In response, recent work in meta-learning proposes training a meta-learner on
a distribution of similar tasks, in the hopes of generalization to novel but related tasks by learning a high-level
strategy that captures the essence of the problem it is asked to solve. However, many recent meta-learning approaches are
extensively hand-designed, either using architectures specialized to a particular application, or hard-coding algorithmic
components that constrain how the meta-learner solves the task. We propose a class of simple and generic meta-learner
architectures that use a novel combination of temporal convolutions and soft attention; the former to aggregate information
from past experience and the latter to pinpoint specific pieces of information. In the most extensive set of meta-learning
experiments to date, we evaluate the resulting Simple Neural AttentIve Learner (or SNAIL) on several heavily-benchmarked tasks. On all tasks, in both supervised and
reinforcement learning, SNAIL attains state-of-the-art performance by
significant margins.
We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model
trained with gradient descent and applicable to a variety of different learning problems, including classification,
regression, and reinforcement learning. The goal of meta-learning is to train a
model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training
samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps
with a small amount of training data from a new task will produce good generalization performance on that task. In effect,
our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art
performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates
fine-tuning for policy gradient reinforcement learning with neural network policies.
Neural networks have been successfully applied in applications with a large amount of labeled data. However, the task of
rapid generalization on new concepts with small training data while preserving performances on previously learned ones
still presents a significant challenge to neural network models. In this work, we introduce a novel meta learning method,
Meta Networks (MetaNet), that learns a meta-level knowledge across tasks and shifts its inductive biases via fast
parameterization for rapid generalization. When evaluated on Omniglot and Mini-ImageNet benchmarks, our MetaNet models achieve a near human-level performance and outperform
the baseline approaches by up to 6% accuracy. We demonstrate several appealing properties of MetaNet relating to
generalization and continual learning.
We propose an LSTM-based meta-learner model to learn the exact
optimization algorithm used to train another learner neural network in the few-shot regime
Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot
learning tasks, where a model has to quickly generalize after seeing very few examples from each class. The general belief
is that gradient-based optimization in high capacity models requires many iterative steps over many examples to
perform well. Here, we propose an LSTM-based meta-learner model to learn the
exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of
our model allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will
be made, while also learning a general initialization of the learner network that allows for quick convergence of training.
We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for few-shot learning.
Our world can be succinctly and compactly described as structured scenes of objects and relations. A typical room, for
example, contains salient objects such as tables, chairs and books, and these objects typically relate to each other by
their underlying causes and semantics. This gives rise to correlated features, such as position, function and shape. Humans
exploit knowledge of objects and their relations for learning a wide spectrum of tasks, and more generally when learning
the structure underlying observed data.
In this work, we introduce relation networks (RNs)—a general purpose neural network architecture for object-relation
reasoning. We show that RNs are capable of learning object relations from scene description data. Furthermore, we show that
RNs can act as a bottleneck that induces the factorization of objects from entangled scene description inputs, and from
distributed deep representations of scene images provided by a variational autoencoder. The model can
also be used in conjunction with differentiable memory mechanisms for implicit relation discovery in one-shot learning
tasks. Our results suggest that relation networks are a potentially powerful architecture for solving a variety of problems
that require object relation reasoning.
“Learning
to reinforcement learn”, Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z.
Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, Matt Botvinick (2016-11-17;
ai):
In recent years deep reinforcement learning (RL) systems have attained superhuman
performance in a number of challenging task domains. However, a major limitation of such applications is their demand for
massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to
new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support
meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is
trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second,
learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is
configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept
experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the
approach, and also point out some potentially important implications for neuroscience.
Deep reinforcement learning (deep RL) has been successful in learning
sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals
can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge
this gap. Rather than designing a “fast” reinforcement learning algorithm, we
propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL2, the
algorithm is encoded in the weights of the RNN, which are learned
slowly through a general-purpose (“slow”) RL algorithm. The RNN
receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination
flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store
the state of the “fast” RL algorithm on the current (previously unseen) MDP. We
evaluate RL2 experimentally on both small-scale and large-scale problems. On the small-scale side, we
train it to solve randomly generated multi-arm bandit problems and finite MDPs.
After RL2is trained, its performance on new MDPs is close
to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL2 on a
vision-based navigation task and show that it scales up to high-dimensional problems.
“HyperNetworks”, David Ha, Andrew Dai, Quoc V. Le (2016-09-27; ai):
This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the
weights for another network. Hypernetworks provide an abstraction that is similar to what is found in nature: the
relationship between a genotype—the hypernetwork—and a phenotype—the main network. Though they are also reminiscent
of HyperNEAT in evolution, our hypernetworks are trained end-to-end with
backpropagation and thus are usually faster. The focus of this work is to make
hypernetworks useful for deep convolutional networks and long recurrent networks, where hypernetworks can be viewed as
relaxed form of weight-sharing across layers. Our main result is that hypernetworks can generate non-shared weights
for LSTM and achieve near state-of-the-art results on a variety of sequence
modelling tasks including character-level language modelling, handwriting generation and neural machine translation,
challenging the weight-sharing paradigm for recurrent networks. Our results also show that hypernetworks applied to
convolutional networks still achieve respectable results for image recognition tasks compared to state-of-the-art baseline
models while requiring fewer learnable parameters.
Training directed neural networks typically requires forward-propagating data through a computation graph, followed by
backpropagating error signal, to produce weight updates. All layers, or more generally, modules, of the network are
therefore locked, in the sense that they must wait for the remainder of the network to execute forwards and propagate error
backwards before they can be updated. In this work we break this constraint by decoupling modules by introducing a model of
the future computation of the network graph. These models predict what the result of the modelled subgraph will produce
using only local information. In particular we focus on modelling error gradients: by using the modelled synthetic gradient
in place of true backpropagated error gradients we decouple subgraphs, and can update them independently and asynchronously
i.e. we realise decoupled neural interfaces. We show results for feed-forward models, where every layer is trained
asynchronously, recurrent neural networks (RNNs) where predicting one’s future
gradient extends the time over which the RNNcan
effectively model, and also a hierarchical RNN system with ticking at different
timescales. Finally, we demonstrate that in addition to predicting gradients, the same framework can be used to predict
inputs, resulting in models which are decoupled in both the forward and backwards pass—amounting to independent networks
which co-learn such that they can be composed into a single functioning corporation.
Despite recent breakthroughs in the applications of deep neural networks, one setting that presents a persistent
challenge is that of “one-shot learning.” Traditional gradient-based networks require a lot of data to learn, often through
extensive iterative training. When new data is encountered, the models must inefficiently relearn their parameters to
adequately incorporate the new information without catastrophic interference. Architectures with augmented memory
capacities, such as Neural Turing Machines (NTMs), offer the ability to
quickly encode and retrieve new information, and hence can potentially obviate the downsides of conventional models. Here,
we demonstrate the ability of a memory-augmented neural network to rapidly assimilate new data, and leverage this data to
make accurate predictions after only a few samples. We also introduce a new method for accessing an external memory that
focuses on memory content, unlike previous methods that additionally use memory location-based focusing mechanisms.
This paper addresses the general problem of reinforcement learning (RL) in partially observable environments.
In 2013, our large RL recurrent neural networks (RNNs) learned from scratch
to drive simulated cars from high-dimensional video input. However, real brains are more powerful in many ways. In
particular, they learn a predictive model of their initially unknown environment, and somehow use it for abstract (e.g.,
hierarchical) planning and reasoning. Guided by algorithmic information theory, we describe RNN-based AIs (RNNAIs)designed to do the same. Such an RNNAI can be trained on never-ending
sequences of tasks, some of them provided by the user, others invented by the RNNAI itself in a curious, playful fashion, to improve itsRNN-based world model. Unlike our previous model-buildingRNN-based RL machines dating back to 1990, the RNNAI learns to actively query its model for abstract reasoning and planning and decision
making, essentially “learning to think.” The basic ideas of this report can be applied to many other cases where one
RNN-like system exploits the algorithmic information content of
another. They are taken from a grant proposal submitted in Fall 2014, and also explain concepts such as “mirror
neurons.” Experimental results will be described in separate papers.
A program of research into weakly supervised learning algorithms led us to ask if learning could occur given only
natural
selection as feedback.
We developed an algorithm that combined evolution and learning, and tested it in an artificial environment populated
with adaptive and non-adaptive organisms. We found that learning and evolution together were more successful than either
alone in producing adaptive populations that survived to the end of our simulation.
In a case study testing long-term stability, we simulated one well-adapted population far beyond the original time
limit. The story of that population’s success and ultimate demise involves both familiar and novel effects in evolutionary
biology and learning algorithms [such as “tree senility”].
