We present a machine learning system that can quantify fine art paintings with a set of visual elements and principles of art. This formal analysis is
fundamental for understanding art, but developing such a system is challenging. Paintings have high visual complexities, but it is also difficult to collect enough
training data with direct labels. To resolve these practical limitations, we introduce a novel mechanism, called proxy learning, which learns visual concepts in
paintings though their general relation to styles. This framework does not require any visual annotation, but only uses style labels and a general relationship
between visual concepts and style. In this paper, we propose a novel proxy model and reformulate four pre-existing methods in the context of proxy learning.
Through quantitative and qualitative comparison, we evaluate these methods and compare their effectiveness in quantifying the artistic visual concepts, where the
general relationship is estimated by language models; GloVe or BERT. The language modeling is a practical and
scalable solution requiring no labeling, but it is inevitably imperfect. We demonstrate how the new proxy model is robust to the imperfection, while the other
models are sensitively affected by it.
Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models.
In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time.
Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training
strategies from summarization pre-training (PEGASUS) into the scalableT5 architecture. The result is a new attention
mechanism we call Transient Global (TGlobal), which mimics ETC’s local/global attention mechanism, but without
requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on
question answering tasks.
We present a very simple algorithm for attention that requires 𝒪(1) memory with respect to sequence length and an extension to self-attention that requires
𝒪(log n) memory. This is in contrast with the frequently stated belief that self-attention requires 𝒪(n2) memory. While the time
complexity is still 𝒪(n2), device memory rather than compute capability is often the limiting factor on modern accelerators. Thus, reducing the
memory requirements of attention allows processing of longer sequences than might otherwise be feasible. We provide a practical implementation for accelerators
that requires 𝒪(√n) memory, is numerically stable, and is within a few percent of the runtime of the standard implementation of attention. We also
demonstrate how to differentiate the function while remaining memory-efficient. For sequence length 16384, the memory overhead of self-attention is reduced by 59×
for inference and by 32× for differentiation.
[Word Golf is a puzzle game similar to Word ladder+Six Degrees of Wikipedia: given a starting word like “dragon” and a grid of ‘similar’ possible words (near by
in the GloVe graph of word embeddings), the player tries to navigate to a target word like “tower” in as few words
as possible.
This requires intuiting similarity of meaning & dimensions of descriptions in English to guess what will make progress towards the target without leaving one
trapped in a semantic niche with no way out.
We then conducted genome-wide association studies in 39,688 individuals, identifying 82 loci associated with ascending
and 47 with descending thoracic aortic diameter, of which 14 loci overlapped. Transcriptome-wide analyses, rare-variant burden tests and human aortic single
nucleus RNA sequencing prioritized genes including SVIL, which was strongly associated with descending aortic diameter. A polygenic score for ascending aortic diameter was associated with thoracic aortic aneurysm in 385,621 UK Biobank participants (hazard ratio = 1.43 per s.d.,
confidence interval = 1.32–1.54, p = 3.3 × 10−20).
Our results illustrate the potential for rapidly defining quantitative traits with deep learning, an approach that can be broadly applied to biomedical
images.
Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. A wealth of effective new methods based on instance matching
rely on data augmentation to drive learning, and these
have reached a rough agreement on an augmentation scheme that optimises popular recognition benchmarks. However, there is strong reason to suspect that different
tasks in computer vision require features to encode different (in)variances, and therefore likely require different augmentation strategies. In this paper, we
measure the invariances learned by contrastive
methods and confirm that they do learn invariance to the augmentations used and further show that this invariance largely transfers to related real-world changes
in pose and lighting. We show that learned invariances strongly affect downstream task performance and confirm that different downstream tasks benefit from polar
opposite (in)variances, leading to performance loss when the standard augmentation strategy is used. Finally, we demonstrate that a simple fusion of
representations with complementary invariances ensures wide transferability to all the diverse downstream tasks considered.
We present techniques for scaling Swin Transformer up to 3 billion parameters and making it capable of training with images of up to 1,536×1,536 resolution. By
scaling up capacity and resolution, Swin Transformer sets new records on four representative vision benchmarks: 84.0% top-1 accuracy on ImageNet-V2 image classification, 63.1/54.4
box/mask mAP on COCOobject detection, 59.9 mIoU on ADE20K semantic segmentation, and
86.8% top-1 accuracy on Kinetics-400 video action classification. Our techniques are generally applicable for scaling up vision models, which has not been
widely explored as that of NLP language models, partly due to the following difficulties in training and applications:
(1) vision models often face instability issues at scale and (2) many downstream vision tasks require high resolution images or windows and it is not clear how to
effectively transfer models pre-trained at low resolutions to higher resolution ones. The GPU memory consumption is also
a problem when the image resolution is high. To address these issues, we present several techniques, which are illustrated by using Swin Transformer as a
case study: (1) a post normalization technique and a scaled cosine attention approach to improve the stability of large vision models; (2) a log-spaced continuous
position bias technique to effectively transfer models pre-trained at low-resolution images and windows to their higher-resolution counterparts. In addition, we
share our crucial implementation details that lead to significant savings of GPU memory consumption and thus make it feasible to train large vision models
with regular GPUs. Using these techniques and self-supervised pre-training, we successfully train a strong 3B Swin
Transformer model and effectively transfer it to various vision tasks involving high-resolution images or windows, achieving the state-of-the-art accuracy on a
variety of benchmarks.
Transformer-based models are widely used in natural language processing (NLP). Central to the transformer model is
the self-attention mechanism, which captures the interactions of token pairs in the input sequences and depends quadratically on the sequence length.
Training such models on longer sequences is expensive. In this paper, we show that a Bernoulli sampling attention mechanism based on Locality Sensitive
Hashing (LSH), decreases the quadratic complexity of such models to linear. We bypass the quadratic cost by considering
self-attention as a sum of individual tokens associated with Bernoulli random variables that can, in principle, be sampled at once by a single hash (although in
practice, this number may be a small constant). This leads to an efficient sampling scheme to estimate self-attention which relies on specific modifications
of LSH (to enable deployment on GPU architectures). We evaluate
our algorithm on the GLUE benchmark with standard 512 sequence length where we see favorable performance relative
to a standard pretrained Transformer. On the Long Range Arena (LRA) benchmark, for evaluating performance on long
sequences, our method achieves results consistent with softmax self-attention but with sizable speed-ups and memory savings and often outperforms other efficient
self-attention methods. Our code is available at https://github.com/mlpen/YOSO
The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual
question with respect to an image. It has been a popular research topic with an increasing number of real-world applications in the last decade. This paper
describes our recent research of AliceMind-MMU (ALIbaba’s Collection of
Encoder-decoders from Machine IntelligeNce lab of Damo academy—MultiMedia Understanding) that obtains similar or even slightly better results than human
being does on VQA. This is achievedby systematically improving the VQA
pipeline including: (1) pre-training with comprehensive visual and textual feature representation; (2) effective cross-modal interaction with learning to
attend; and (3) A novel knowledge mining framework with specialized expert modules for the complex VQA task. Treating
different types of visual questions with corresponding expertise needed plays an important role in boosting the performance of our VQA architecture up to the human level. An extensive set of experiments and analysis are conducted to demonstrate the effectiveness
of the new research work.
A significant obstacle in the development of robust machine learning models is covariate shift, a form of distribution shift that occurs when the input
distributions of the training and test sets differ while the conditional label distributions remain the same. Despite the prevalence of covariate shift in
real-world applications, a theoretical understanding in the context of modern machine learning has remained lacking. In this work, we examine the exact
high-dimensional asymptotics of random feature regression under covariate shift and present a precise characterization of the limiting test error, bias, and
variance in this setting. Our results motivate a natural partial
order over covariate shifts that provides a sufficient condition for determining when the shift will harm (or even help) test performance. We find that
overparameterized models exhibit enhanced robustness to covariate shift, providing one of the first theoretical explanations for this intriguing phenomenon.
Additionally, our analysis reveals an exact linear relationship between in-distribution and out-of-distribution generalization performance, offering an explanation
for this surprising recent empirical observation.
“A Survey of Visual Transformers”, Yang Liu, Yao Zhang, Yixin Wang, Feng
Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongchao Shi, Jianping Fan, et al (2021-11-11; similar):
Transformer, an attention-based encoder-decoder architecture, has revolutionized the field of natural language processing. Inspired by this significant
achievement, some pioneering works have recently been done on adapting Transformerliked architectures to Computer Vision (CV) fields, which have demonstrated their
effectiveness on various CV tasks. Relying on competitive modeling capability, visual Transformers have achieved impressive performance on multiple benchmarks such
as ImageNet, COCO, and ADE20k as compared with modern Convolution NeuralNetworks (CNN). In this paper, we have provided a comprehensive review of over one hundred different visual
Transformers for three fundamental CV tasks (classification, detection, and segmentation), where a taxonomy is proposed to organize these methods according to
their motivations, structures, and usage scenarios. Because of the differences in training settings and oriented tasks, we have also evaluated these methods on
different configurations for easy and intuitive comparison instead of only various benchmarks. Furthermore, we have revealed a series of essential but unexploited
aspects that may empower Transformer to stand out from numerous architectures, eg. slack high-level semantic embeddings to bridge the gap between visual and
sequential Transformers. Finally, three promising future research directions are suggested for further investment.
A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly
on long-range dependencies. Although conventional models including RNNs, CNNs, and
Transformers have specialized variants for capturing long dependencies, they still struggle to scale to very long sequences of 10000 or more steps. A
promising recent approach proposed modeling sequences by simulating the fundamental state space model (SSM)x’(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t), and showed that for appropriate choices of the state matrix
A, this system could handle long-range dependencies mathematically and empirically. However, this method has prohibitive computation and memory
requirements, rendering it infeasible as a general sequence modeling solution. We propose the Structured State Space (S4) sequence model based on a new
parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches while
preserving their theoretical strengths. Our technique involves conditioning A with a low-rank correction, allowing it to be diagonalized stably and
reducing the SSM to the well-studied computation of a Cauchy kernel. S4 achieves strong empirical results across a
diverse range of established benchmarks, including (1) 91% accuracy on sequential CIFAR-10 with nodata augmentation or auxiliary losses, on par with a larger 2-D ResNet, (2) substantially closing the gap to Transformers on image and language modeling tasks, while performing
generation 60× faster (3) SoTA on every task from the Long Range Arena benchmark, including solving the challenging Path-X task of length 16k that all prior work
fails on, while being as efficient as all competitors.
Transformers are expensive to train due to the quadratic time and space complexity in the self-attention mechanism. On the other hand, although kernel machines
suffer from the same computation bottleneck in pairwise dot products, several approximation schemes have been successfully incorporated to considerably reduce
their computational cost without sacrificing too much accuracy. In this work, we leverage the computation methods for kernel machines to alleviate the high
computational cost and introduce Skyformer, which replaces the softmax structure with a Gaussian kernel to stabilize the model training and adapts the Nyström
method to a non-positive semidefinite matrix to accelerate the computation. We further conduct theoretical analysis by showing that the matrix approximation error
of our proposed method is small in the spectral norm. Experiments on Long Range Arena benchmark show that the proposed method is sufficient in getting comparable
or even better performance than the full self-attention while requiring fewer computation resources.
Recurrent neural networks (RNNs), temporal convolutions, and neuraldifferential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power
and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their
shortcomings. The Linear State-Space Layer (LSSL) maps a sequenceu ↦ y by simply simulating a
linear continuous-time state-space representation ͘x = Ax + Bu, y = Cx + Du. Theoretically, we show
that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For
example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a
trainable subset of structured matrices Athat endow LSSLs with long-rangememory. Empirically,
stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks
for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with
length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use
hand-crafted features on 100× shorter sequences.
Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers
can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or
well-structured images produced by DALL-E. These large language models are impressive but also very inefficient and
costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that
efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them
hierarchical. We use the best performing upsampling and downsampling layers to create Hourglass—a hierarchical Transformer language model. Hourglass improves upon
the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new
state-of-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark.
In this work we introduce KERNELIZEDTRANSFORMER, a generic, scalable, data
driven framework for learning the kernel function in Transformers. Our framework approximates the Transformer kernel as a dot product between spectral feature maps
and learns the kernel by learning the spectral distribution. This not only helps in learning a generic kernel end-to-end, but also reduces the time and space
complexity of Transformers from quadratic to linear. We show that KERNELIZEDTRANSFORMERS achieve performance comparable to existing efficient Transformer architectures, both in terms of accuracy as well as
computational efficiency. Our study also demonstrates that the choice of the kernel has a substantial impact on performance, and kernel learning variants are
competitive alternatives to fixed kernel Transformers, both in long as well as short sequence tasks.
Automatic font image synthesis has been an extremely active topic in recent years. Various deep learning-based approaches have been proposed to tackle this font
synthesis task by considering it as an image-to-image translation problem in a supervised setting. However, all such approaches mainly focus on one-to-one font
mapping, i.e., synthesizing a single font style, making it difficult to handle more practical problems such as the font family synthesis, which is a one-to-many
mapping problem. Moreover, this font family synthesis is more challenging because it is an unsupervised image-to-image translation problem, i.e., no paired dataset
is available during training.
To address this font family synthesis problem, we propose a method that utilizes a single generator to conditionally produce various font family styles to form
a font family. To the best of our knowledge, our proposed method is the first to synthesize a font family (multiple font styles belonging to a font), instead of
synthesizing a single font style. More specifically, our method is trained to learn a font family by conditioning on various styles, eg. normal, bold, italic,
bold-italic, etc. After training, given an unobserved single font style (normal style font as an input), our method can successfully synthesize the remaining
styles (eg. bold, italic, bold-italic, etc.) to complete the font family.
Qualitative and quantitative experiments were conducted to demonstrate the effectiveness of our proposed method.
Transformer architectures have achieved state-of-the-art results on a variety of sequence modeling tasks. However, their attention mechanism comes with a
quadratic complexity in sequence lengths, making the computational overhead prohibitive, especially for long sequences. Attention context can be seen as a
random-access memory with each token taking a slot. Under this perspective, the memory size grows linearly with the sequence length, and so does the overhead of
reading from it. One way to improve the efficiency is to bound the memory size. We show that disparate approaches can be subsumed into one abstraction,
attention with bounded-memory control (ABC), and theyvary in their organization of the memory. ABC reveals new, unexplored possibilities. First, it connects several efficient attention variants that would otherwise seem apart.
Second, this abstraction gives new insights—an established approach (Wang et al 2020b) previously thought to be not applicable in causal attention,
actually is. Last, we present a new instance of ABC, which draws inspiration fromexisting ABC approaches, but replaces their heuristic memory-organizing functions with a learned, contextualized one. Our experiments on
language modeling, machine translation, and masked language model finetuning show that our approach outperforms previous efficient attention models; compared to
the strong transformer baselines, it significantly improves the inference time and space efficiency with no or negligible accuracy loss.
A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly
on long dependencies. Although conventional models including RNNs, CNNs, and
Transformers have specialized variants for capturing long-range dependencies, they still struggle to scale to very long sequences of 10,000 or more steps. A
promising recent approach proposed modeling sequences by simulating the fundamental state space model (SSM)x’(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t), and showed that for appropriate choices of the state matrix
A, this system could handle long-range dependencies mathematically and empirically. However, this method has prohibitive computation and memory
requirements, rendering it infeasible as a general sequence modeling solution. We propose the Structured State Space (S3) model based on a new
parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches while
preserving their theoretical strengths. Our technique involves conditioning A with a low-rank correction, allowing it to be diagonalized stably and
reducing the SSM to the well-studied computation of a Cauchy kernel. S3 achieves strong empirical results across a
diverse range of established benchmarks, including (1) 91% accuracy on sequential CIFAR-10 with nodata augmentation or auxiliary losses, on par with a larger 2-D ResNet, (2) substantially closing
the gap to Transformers on image and language modeling tasks, while performing generation 60× faster (3) SoTA on every task from the Long Range Arena benchmark,
including solving the challenging Path-X task of length 16k that all prior work fails on, while being as efficient as all competitors.
[Keywords: sequence models, state space, RNN, CNN, Long Range Arena]
We propose to use an external memory module to allow instant utilization of newly acquired knowledge.
Language models typically need to be trained or finetuned in order to acquire new knowledge, which involves updating their weights. We instead envision language
models that can simply read and memorize new data at inference time, thus acquiring new knowledge immediately. In this work, we extend language models with the
ability to memorize the internal representations of past inputs.
Despite the fact that our implementation of memory is not differentiable, we demonstrate that an approximate k-NN lookup into the memory improves language modeling across
various benchmarks and tasks, including generic webtext (C4), math papers (arXiv), books (PG-19), code (Github), as well as formal theorems (Isabelle). We show that the performance steadily improves when we increase the
size of memory up to 131k tokens. We also find that the model is capable of making use of newly defined functions and theorems during test time.
…We demonstrate that a simple, effective, and scalable way to increase the size of the attention context is to use approximate k-nearest-neighbor
(kNN) search into a large external memory of (key, value) pairs. There are efficient implementations of approximate kNN lookup on
TPU,GPU, and CPU,
including distributed implementations (Guo et al 2020), which opens the door to extremely large external
memories.
In contrast to most other work on sparse, or long-range attention (c.f. Section 2), we treat the external memory as a large “cache”, and gradients are not
back-propagated into the cache. This is a potential limitation, because it means that the network can only learn to query the external memory, and cannot
directly learn what keys and values to put into it. However, we demonstrate empirically that (key, value) pairs which are useful for local (and thus fully
differentiable) self-attention are also useful for long-range attention.
Using a non-differentiable cache is critical
to scalability. The keys and values are a function of model parameters, so attempting to backpropagate gradients into the external memory would necessarily involve
computing all of the keys and values with the current model parameters on every training step. If the external memory is not differentiable, we can instead reuse
keys and values from prior training steps. With our technique, we are easily able to scale external memory up to a sequence lengths of 131k or 262k tokens on a
single device, while maintaining a reasonable step time.
We introduce Dot Product Attention Free Transformer (DAFT), an efficient variant of Transformers that
eliminates the query-key dot product in self attention. The core idea is to construct a decomposable attention map for each dimension of the query, key and value.
This compositionality enables an implementation where the attention tensor does not to be computed or stored explicitly. A DAFT layer has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible with both
large input and model sizes. We also introduce DAFT-conv, a model variant that takes advantage of locality and
spatial weight sharing while maintaining global connectivity. We conduct experiments on ImageNet-1K classification, as
well as CIFAR10 and Enwik8, two autoregressivemodeling tasks. We show that DAFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.
Transformers struggle when attending to long contexts, since the amount of computation grows with the context length, and therefore they cannot model long-term
memories effectively. Several variations have been proposed to alleviate this problem, but they all have a finite memory capacity, being forced to drop old
information. In this paper, we propose the ∞-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space
attention mechanism to attend over the long-term memory, the ∞-former’s attention complexity becomes independent of the context length. Thus, it is able to model
arbitrarily long contexts and maintain “sticky memories” while keeping a fixed computation budget. Experiments on a synthetic sorting task demonstrate the ability
of the ∞-former to retain information from long sequences. We also perform experiments on language modeling, by training a model from scratch and by fine-tuning a
pre-trained language model, which show benefits of unbounded long-term memories.
Convolutional Neural Networks (CNNs) can perform similarly or better than standard genomic prediction methods
when sufficient genetic, environmental, and management data are provided.
Predicting phenotypes from genetic (G), environmental (E), and management (M) conditions is a long-standing challenge with implications to agriculture,
medicine, and conservation. Most methods reduce the factors in a dataset (feature engineering) in a subjective and potentially oversimplified manner. Deep neural
networks such as Multilayer Perceptrons (MPL) and Convolutional Neural Networks (CNN) can overcome this by allowing the data itself to determine which factors are most important. CNN models were developed for predicting agronomic yield from a combination of replicated trials and historical yield survey data.
The results were more accurate than standard methods when tested on held-out G, E, and M data (r = 0.50 vs. r = 0.43), and performed
slightly worse than standard methods when only G was held out (r = 0.74 vs. r = 0.80). Pre-training on historical data increased accuracy
compared to trial data alone. Saliency map analysis indicated the CNN has “learned” to prioritize many factors of
known agricultural importance.
Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are
many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which
is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive
attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In
this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more
efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.
Transformers have improved the state-of-the-art across numerous tasks in sequence modeling. Besides the quadratic computational and memory complexity w.r.t the
sequence length, the self-attention mechanism only processes information at the same scale, i.e., all attention heads are in the same resolution, resulting in the
limited power of the Transformer. To remedy this, we propose a novel and efficient structure named Adaptive Multi-Resolution Attention (AdaMRA for short), which scales linearly to sequence length in terms of time and space. Specifically, we leverage a multi-resolution
multi-head attention mechanism, enabling attention heads to capture long-range contextual information in a coarse-to-fine fashion. Moreover, to capture the
potential relations between query representation and clues of different attention granularities, we leave the decision of which resolution of attention to use to
query, which further improves the model’s capacity compared to vanilla Transformer. In an effort to reduce complexity, we adopt kernel attention without degrading
the performance. Extensive experiments on several benchmarks demonstrate the effectiveness and efficiency of our model by achieving a state-of-the-art
performance-efficiency-memory trade-off. To facilitate AdaMRA utilization by the scientific community, the code
implementation will be made publicly available.
[code; Hugging Face] The
recently-proposed Perceiver model obtains good results on
several domains (images, audio, multimodal, point clouds) while scaling linearly in compute and memory with the input size. While the Perceiver supports many kinds
of inputs, it can only produce very simple outputs such as class scores. Perceiver IO overcomes this limitation without sacrificing the original’s appealing
properties by learning to flexibly query the model’s latent space to produce outputs of arbitrary size and semantics.
Perceiver IO still decouples model depth from data size and still scales linearly with data size, but now with respect to both input and output sizes.
The full Perceiver IO model achieves strong results on tasks with highly structured output spaces, such as natural language and visual understanding, StarCraft II, and multi-task and multi-modal domains. As highlights,
Perceiver IO matches a Transformer-based BERTbaseline on the GLUE language
benchmark without the need for input tokenization, and achieves state-of-the-art performance on Sintel optical flow estimation.
Figure 2: The Perceiver IO architecture. Perceiver IO maps arbitrary input arrays to arbitrary output arrays in a domain
agnostic process. The bulk of the computation happens in a latent space whose size is typically smaller than the inputs and outputs, which makes the process
computationally tractable even for very large inputs & outputs.
…The Perceiver IO architecture relies on the same primitives as Transformers: so why aren’t Transformers all you need? The answer is that Transformers scale very poorly in both
compute and memory.82 A Transformer deploys attention modules homogeneously throughout its architecture, using its full input to generate queries and
keys at every layer. As discussed in,35 this means each layer scales quadratically in compute and memory, which currently makes it impossible to apply
Transformers on high-dimensional data like images without some form of preprocessing. Even on domains like language where Transformers shine, preprocessing (eg. tokenization) is often needed to scale beyond short input sequences. On the other hand, Perceiver
IO uses attention non-homogeneously, first using it to map inputs to a latent space, then using it to process in that latent
space, and finally using it to map to an output space. The resulting architecture has no quadratic dependence on the input or output size: encoder and decoder
attention modules depend linearly on the input and output size (respectively), while latent attention is independent of both
input and output sizes. Because of this structure, and the corresponding reduction in compute and memory requirements, Perceivers scale to much larger inputs and
outputs. While Transformers are typically used in settings with inputs and outputs of at most a few thousand dimensions [9, 63], we show good
results on domains with hundreds of thousands of input and output dimensions.
…Because of this structure, this architecture can be applied to inputs of any shape or spatial layout and even to inputs or outputs which don’t share the same
spatial structure (eg. sound and video). However, in contrast to the latent spaces used elsewhere in vision
(eg.)67 the latent does not explicitly share the structure (spatial or otherwise) of the inputs. To decode this
information, we query for it using cross-attention.
…4.4 StarCraft II: To further demonstrate Perceiver IO’s capabilities on discrete modalities and to serve as a drop-in replacement for
Transformers, we use Perceiver IO to replace the Transformer in AlphaStar, the state-of-the-art system for the complex game
of StarCraft II. At its core, AlphaStar [89] represents the units in the game as a discrete, unordered set of symbols (the “units”). These units are represented by a vector of
properties such as unit type, position, health, etc. At each timestep, the architecture encodes up to 512 units “tokens” with a vanilla Transformer. This representation is used both as a summary of the state (after pooling) and as a rich representation of the 512 units.
This representation is used by a pointer network [90], to
assign a probability to each possible unit selection, effectively parameterizing the agent’s unit selection policy (see 89 and Appendix Section G for
more details). We replaced the Transformer that inputs and outputs 512 units with Perceiver IO with a latent size of 32. Without tuning any additional parameters, we observed that the resulting agent reached the same level of performance
as the original AlphaStar agent, reaching an 87% win-rate versus the Elite bot after behavioral cloning[61]
on human data.
Predictive coding offers a potentially unifying account of cortical function—postulating that the core function of the brain is to minimize prediction errors
with respect to a generative model of the world. The theory is closely related to the Bayesian brain framework and, over the last two decades, has gained
substantial influence in the fields of theoretical and cognitive neuroscience. A large body of research has arisen based on both empirically testing improved and
extended theoretical and mathematical models of predictive coding, as well as in evaluating their potential biological plausibility for implementation in the brain
and the concrete neurophysiological and psychological predictions made by the theory. Despite this enduring popularity, however, no comprehensive review of
predictive coding theory, and especially of recent developments in this field, exists. Here, we provide a comprehensive review both of the core mathematical
structure and logic of predictive coding, thus complementing recent tutorials in the literature. We also review a wide range of classic and recent work within the
framework, ranging from the neurobiologically realistic microcircuits that could implement predictive coding, to the close relationship between predictive coding
and the widely-used backpropagation of error algorithm,
as well as surveying the close relationships between predictive coding and modern machine learning techniques.
[see also “Assessing Human Error Against a Benchmark of
Perfection”, Anderson et al 2016] How does artificial intelligence (AI) improve human decision-making? Answering this question is challenging
because it is difficult to assess the quality of each decision and to disentangle AI’s influence on decisions. We study professional Go games, which provide a
unique opportunity to overcome such challenges.
In 2016 an AI-powered Go program(APG) unexpectedly beat the best human player, surpassing the best human knowledge and skills accumulated over thousands of years. To
investigate the impact of APGs, we compare human moves to AI’s superior solutions, before andafter the
initial public release of an APG [Leela
Zero, KataGo, and NHN’s Handol]. Our
analysis of 750,990 moves in 25,033 games by 1,242 professional players reveals that APGs noticeably improved the
quality of the players’ moves as measured by the changes in winning probability with each move. We also show that the key mechanisms are reductions in the
number of human errors and in the magnitude of the most critical mistake during the game. Interestingly, the improvement is most prominent in the early stage of a
game when uncertainty is higher. Further, young players—who are more open to and better able to utilize APG—benefit more
than senior players, suggesting generational inequality in AI adoption and utilization.
[Keywords: artificial intelligence (AI), technology adoption, decision-making, human capital, professional Go players, AI adoption
inequality]
…The historic Go match (AlphaGo vs. Sedol Lee) was held in 2016; in this game, AI beat the best human professional player for the first time and by a large
margin. Shortly after this event, the first open APG, Leela, became available to players in February 2017.
Our quantitative and qualitative investigation indicates that professional Go players have used APGs heavily in
their training since its release.
The great advantage of this context is that it allows us to observe every single decision of professional Go players before and after the public release
of APGs; a game’s entire move history is well archivedand maintained for all major games. Furthermore, using the
APG’s best solution as a benchmark, we can calculate the probability of winning for every move (ie. 750,990 decisions)
by 1,242 professional Go players in 25,033 major games held from 2015 through 2019; note that this can be done even for the games played before APG’s release. Wethen compare the move-level probability of winning to that of APG’s best solution.
The results show that the quality of moves by professional Go players improved substantially following the release of APG. Before the release, the winning probability of each move by professional Go players averaged 2.47 percentage points lower
than the moves of APG. This gap decreased by about 0.756 percentage points (or 30.5%) after the release of
APG. Additional analyses indicate that the improvement in move quality eventually leads to the final win of the game.
Interestingly, this effect is most prominent in the early stage of a game where higher uncertainty is exhibited and there is more opportunity for players to learn
from AI. Furthermore, quality improvement is more prominent among young players who are open to and capable of utilizing APGs; this has important implications for digital literacy and inequality in AI utilization.
[Example of an absolute human error rate: from the AI’s perspective, each move a human Go pro makes costs them ~1.2% chance of winning!]
We also explore the mechanisms through which professional players achieve a higher probability of winning. Our mediation analysis reveals that improvements in
the quality of moves are driven mainly by reducing the number of errors (moves where the winning probability drops by 10 or more percentage points
compared to the immediately preceding move by a focal player) and by reducing the magnitude of the most critical mistake (the biggest drop in winning
probability during the game). Specifically, the number of errors per game decreased by 0.15–0.50 and the magnitude of the most critical mistake decreased
by 4–7 percentage points.
…3.3.1. Go Games and Players: We collect data on professional Go games held from 2015 through 2019 from the Go4Go database, which has been
widely used in studies of Go (eg. Chao et al 2018, Ramon & Struyf 2003, Wu et al 2018). The data contains detailed information
on the game, its players, Komi (the number of bonus points given to the second mover), the sequence of all moves, and the game outcome. From Go Ratings we gather
additional data on the ages, nationalities (eg. China, Japan, South Korea, Taiwan, and others), genders, and annual rankings of professional players. We multiplied
negative one by the ranking and divide it by 1,000 to ease the interpretation of the result; the higher the value, the better the player. To control for the
difference in players’ capabilities for each game, we create a variable, Rank difference, as the difference between the raw rankings of 2 players; we
divide this difference by 1,000 such that a positive value indicates that the focal player’s ranking is lower than the opponent’s ranking.
…Using 2–8 Nvidia Titan-X GPUs running in parallel, the computational analysis of games took about 3
months.
Figure 2: Effects of APG on average move quality of professional players: Model-free
evidence. Note: This figure illustrates the weekly average Move Quality of players from 2015 through 2019. The black
solid line represents the raw (unprocessed) weekly average value. The blue solid line and the gray area around it show the local smoothed trend and the 95% confidence interval, respectively. The vertical
line on February 2017 represents the first public release of an APG,Leela.
…This analysis is motivated by the norm that, after Go games, players spend substantial time and effort analyzing and evaluating each move—especially if the
move was an error or a mistake. In an interview with news media, Jin-seo Shin
(who was ranked first in the world in 2020) stated:
Before APG, players and their peers replayed the game and discussed which move was an error and which was a
critical mistake. After the public release of APG, this replay and discussion by players becamealmost
meaningless. APG teaches us by showing the accurate winning probability with each move. If the winning probability
drops from 60% to 40% after a move, that is an error. If it drops from 80% to 20%, that is a critical mistake. … I have to admit that the APG-based training provides limitless help in developing my Go skills (Sohn 2021).
Large Neighborhood Search (LNS) is a combinatorial optimization heuristic that starts with an assignment of
values for the variables to be optimized, and iteratively improves it by searching a large neighborhood around the current assignment. In this paper we consider
a learning-based LNS approach for mixed integer programs (MIPs). We train a
Neural Diving model to represent a probability distribution over assignments, which, together with an off-the-shelf MIP
solver, generates an initial assignment. Formulating the subsequent search steps as a Markov Decision Process, we train a Neural Neighborhood Selection policy to select a search neighborhood at each step,
which is searched using a MIP solver to find the next assignment. The policy network is trained using imitation
learning. We propose a target policy for imitation that, given enough compute resources, is guaranteed to select the neighborhood containing the optimal next
assignment amongst all possible choices for the neighborhood of a specified size. Our approach matches or outperforms all the baselines on five real-world
MIP datasets with large-scale instances from diverse applications, including two production applications at Google. It
achieves 2× to 37.8× better average primal gap than the best baseline on three of the datasets at large running times.
Background: Technology to restore the ability to communicate in paralyzed persons who cannot speak has the potential to improve autonomy and
quality of life. An approach that decodes words and sentences directly from the cerebral cortical activity of such patients may represent an advancement over
existing methods for assisted communication.
Methods: We implanted a subdural, high-density, multielectrode array over the area of the sensorimotor cortex that controls speech in a person
with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis caused by a brain-stem stroke. Over the course of 48 sessions, we recorded
22 hours of cortical activity while the participant attempted to say individual words from a vocabulary set of 50 words. We used deep-learning algorithms to create
computational models for the detection and classification of words from patterns in the recorded cortical activity. We applied these computational models, as well
as a natural-language model that yielded next-word probabilities given the preceding words in a sequence, to decode full sentences as the participant attempted to
say them.
Results: We decoded sentences from the participant’s cortical activity in real time at a median rate of 15.2 words per minute, with a median
word error rate of 25.6%. In post hoc analyses, we detected 98% of the attempts by the participant to produce individual words, and we classified words with 47.1%
accuracy using cortical signals that were stable throughout the 81-week study period.
Conclusions: In a person with anarthria and spastic quadriparesis caused by a brain-stem stroke, words and sentences were decoded directly from
cortical activity during attempted speech with the use of deep-learning models and a natural-language model. (Funded by Facebook and others;
ClinicalTrials.gov number, NCT03698149.)
Inferotemporal cortex (IT) in humans and other primates is topo-graphically organized, containing multiple hierarchically-organized areas selective for
particular domains, such as faces and scenes. This organization is commonly viewed in terms of evolved domain-specific visual mechanisms. Here, we develop an
alternative, domain-general and developmental account of IT cortical organization. The account is instantiated as an Interactive Topographic Network
(ITN), a form of computational model in which a hierarchy of model IT areas, subject to connectivity-based constraints,
learns high-level visual representations optimized for multiple domains. We find that minimizing a wiring cost on spatially organized feedforward and lateral
connections within IT, combined with constraining the feedforward processing to be strictly excitatory, results in a hierarchical, topographic organization. This
organization replicates a number of key properties of primate IT cortex, including the presence of domain-selective spatial clusters preferentially involved in the
representation of faces, objects, and scenes, columnar responses across separate excitatory and inhibitory units, and generic spatial organization whereby the
response correlation of pairs of units falls off with their distance. We thus argue that domain-selectivity is an emergent property of a visual system optimized to
maximize behavioral performance while minimizing wiring costs.
Significance Statement
We introduce the Interactive Topographic Network, a framework for modeling high-level vision, to demonstrate in computational simulations that the spatial
clustering of domains in late stages of the primate visual system may arise from the demands of visual recognition under the constraints of minimal wiring costs
and excitatory between-area neuronal communication. The learned organization of the model is highly specialized but not fully modular, capturing many of the
properties of organization in primates. Our work is significant for cognitive neuroscience, by providing a domain-general developmental account of topo-graphic
functional specialization, and for computational neuroscience, by demonstrating how well-known biological details can be successfully incorporated into neural
network models in order to account for critical empirical findings.
Transformers provide a class of expressive architectures that are extremely effective for sequence modeling. However, the key limitation of transformers is
their quadratic memory and time complexity 𝑂(L2) with respect to the sequence length in attention layers, which restricts application in
extremely long sequences. Most existing approaches leverage sparsity or low-rank
assumptions in the attention matrix to reduce cost, but sacrifice expressiveness.
Instead, we propose Combiner, which provides full attention capability in each attention head while maintaining low computation and memory
complexity. The key idea is to treat the self-attention mechanism as a conditional expectation over embeddings at each location, and approximate the conditional
distribution with a structured factorization. Each location can attend to all other locations, either via direct attention, or through indirect attention to
abstractions, which are again conditional expectations of embeddings from corresponding local regions.
We show that most sparse attention patterns used in existing sparse transformers are able to inspire the design of such factorization for full attention,
resulting in the same sub-quadratic cost (𝑂(log(L)) or 𝑂(√L)). Combiner is a drop-in replacement for attention layers in existing transformers
and can be easily implemented in common frameworks.
An experimental evaluation on both autoregressive and
bidirectional sequence tasks demonstrates the effectiveness of this approach, yielding state-of-the-art results on several image and text modeling tasks.
In standard neural networks the amount of computation used grows with the size of the inputs, but not with the complexity of the problem being learnt. To
overcome this limitation we introduce PonderNet, a new algorithm that learns to adapt the amount of computation based on the complexity of the problem at hand.
PonderNet learns end-to-end the number of computational steps to achieve an effective compromise between training prediction accuracy, computational cost and
generalization. On a complex synthetic problem, PonderNet dramatically improves performance over previous adaptive computation methods and additionally succeeds at
extrapolation tests where traditional neural networks fail. Also, our method matched the current state of the art results on a real world question and answering
dataset, but using less compute. Finally, PonderNet reached state of the art results on a complex task designed to test the reasoning capabilities of neural
networks.1
Transformers have achieved success in both language and vision domains. However, it is prohibitively expensive to scale them to long sequences such as long
documents or high-resolution images, because self-attention mechanism has quadratic time and memory complexities with respect to the input sequence length. In this
paper, we propose Long-Short Transformer (Transformer-LS), an efficient self-attention mechanism for modeling long sequences with linear complexity for both
language and vision tasks. It aggregates a novel long-range attention with dynamic projection to model distant correlations and a short-term attention to capture
fine-grained local correlations. We propose a dual normalization strategy to account for the scale mismatch between the two attention mechanisms. Transformer-LS
can be applied to both autoregressive and bidirectional models without additional complexity. Our method outperforms the state-of-the-art models on multiple tasks
in language and vision domains, including the Long Range Arena benchmark, autoregressive language modeling, and ImageNet classification. For instance,
Transformer-LS achieves 0.97 test BPC on enwik8 using half the number of parameters than previous method, while being
faster and is able to handle 3× as long sequences compared to its full-attention version on the same hardware. On ImageNet,
it can obtain the state-of-the-art results (eg. Top-1 accuracy 84.1% trained on 224×224 ImageNet-1K only), while being
more scalable on high-resolution images. The models and source code will be released soon.
The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity. Many works focus
on approximating the dot-then-exponentiate softmax function in the original attention, leading to sub-quadratic or even linear-complexity Transformer
architectures. However, we show that these methods cannot be applied to more powerful attention modules that go beyond the dot-then-exponentiate style, eg.
Transformers with relative positional encoding (RPE). Since in many state-of-the-art models, relative positional
encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
In this paper, we propose a novel way to accelerate attention calculation for Transformers with RPE on top of the
kernelized attention. Based upon the observation that relative positional encoding forms a Toeplitz matrix, we mathematically show that kernelized attention with RPE can be calculated
efficiently using Fast Fourier Transform(FFT). With FFT, our method achieves 𝒪(n log n) time
complexity. Interestingly, we further demonstrate that properly using relative positional encoding can mitigate the training instability problem of vanilla
kernelized attention.
On a wide range of tasks, we empirically show that our models can be trained from scratch without any optimization issues. The learned model performs better
than many efficient Transformer variants and is faster than standard Transformer in the long-sequence regime.
The quest for determinism in machine learning has disproportionately focused on characterizing the impact of noise introduced by algorithmic design choices. In
this work, we address a less well understood and studied question: how does our choice of tooling introduce randomness to deep neural network training. We conduct
large scale experiments across different types of hardware, accelerators, state of art networks, and open-source datasets, to characterize how tooling choices
contribute to the level of non-determinism in a system, the impact of said non-determinism, and the cost of eliminating different sources of noise.
Our findings are surprising, and suggest that the impact of non-determinism in nuanced. While top-line metrics such as top-1 accuracy are not noticeably
impacted, model performance on certain parts of the data distribution is far more sensitive to the introduction of randomness. Our results suggest that
deterministic tooling is critical for AI safety. However, we also find that the cost of ensuring determinism varies dramatically between neural network
architectures and hardware types, eg. with overhead up to 746%, 241%, and 196% on a spectrum of widely used GPU accelerator architectures, relative to non-deterministic training. The source code used in this paper is available at
Github.
Google deployed several TPU generations since 2015, teaching us lessons that changed our views: semiconductor
technology advances unequally; compiler compatibility trumps binary compatibility, especially for VLIW domain-specific
architectures (DSAs); target total cost of ownership vs initial cost; support multi-tenancy; deep neural networks
(DNN) grow 1.5× annually; DNN advances evolve workloads; someinference tasks
require floating point; inference DSAs need air-cooling; apps limit latency, not batch size; and backwards ML
compatibilityhelps deploy DNNs quickly. These lessons molded TPUv4i,an inference DSA deployed since 2020.
Document the unequal improvement in logic, wires, SRAM, and DRAM
from 45 nm to 7 nm—including an update of Horowitz’s operation energy table16 from 45 nm to 7 nm—and show how these changes led to 4 systolic floating
point matrix units for TPUv4i in 2020 versus one systolicinteger matrix unit for TPUv1 in 2015.
Explain the difference between designing for performance per TCO vs per CapEx, leading to
HBM and a low TDP for TPUv4i, and showhow TPUv1’s headroom led to application scaleup after the 2017 paper21.
Explain backwards ML compatibility, including why inference can need floating point and how it spurred the TPUv4i and TPUv4designs (§3). Backwards ML compatible training also tailors
DNNsto TPUv4i (§2).
Measure production inference applications to show that DSAsnormally run multiple DNNs concurrently, requiring Google inference DSAs to support
multi-tenancy.
Discuss how DNN advances change the production inference workload.The 2020 workload keeps
MLP and CNN from 2017 but addsBERT,
andRNN succeeds LSTM.
Document the growth of production DNNs in memory size and computation by ~1.5× annually since 2016,
which encourages designing DSAs with headroom.
Show that Google’s TCO and TDP for DNN
DSAs are strongly correlated (r = 0.99), likely due to the end of Dennard scaling. TDP offers a good proxy for DSA TCO.
Document that the SLO limit is P99 time for inference applications, list typical batch sizes, and
show how large on-chip SRAM helps P99 performance.
Explain why TPUv4i architects chose compiler compatibility over binary compatibilityfor its
VLIWISA.
Describe Google’s latest inference accelerator in production since March 2020and evaluate its performance/TDP vs. TPUv3 andNVIDIA’s T4 inferenceGPUusing production apps and MLPerf Inference benchmarks
0.5–0.7.
…TPUv1 required quantization—since it supported only integer arithmetic—which proved a problem for some
datacenter applications. Early in TPUv1 development, application developers said a 1% drop in quality was
acceptable, but they changed their minds by the time the hardware arrived, perhaps because DNN overall quality improved
so that 1% added to a 40% error was relatively small but 1% added to a 12% error was relatively large.
…Alas, DNN DSA designers often ignore multi-tenancy. Indeed,multi-tenancy is not mentioned in the
TPUv1 paper21. (It was lucky that the smallest available DDR3DRAM held 8GB, allowing TPUv1 software to add multi-tenancy.)
…BERT appeared in 2018, yet it’s already 28% of the workload.
Transformer is important for text modeling. However, it has difficulty in handling long documents due to the quadratic complexity with input text length. In
order to handle this problem, we propose a hierarchical interactive Transformer (Hi-Transformer) for efficient and effective long document modeling. Hi-Transformer
models documents in a hierarchical way, i.e., first learns sentence representations and then learns document representations. It can effectively reduce the
complexity and meanwhile capture global document context in the modeling of each sentence. More specifically, we first use a sentence Transformer to learn the
representations of each sentence. Then we use a document Transformer to model the global document context from these sentence representations. Next, we use another
sentence Transformer to enhance sentence modeling using the global document context. Finally, we use hierarchical pooling method to obtain document embedding.
Extensive experiments on three benchmark datasets validate the efficiency and effectiveness of Hi-Transformer in long document modeling.
Modern law enforcement agencies strive to identify current trends and developments in Darknet markets. Extracting information from such markets requires
knowledge about the contained entities, which can be extracted via Named
Entity Recognition(NER).
Modern NER models are trained via supervised learning, which requires an annotated dataset, but such datasets
for specific application domains, e.g. drug detection in Darknet markets, are rarely available. In this work, we created a NER dataset focused on drugs in Darknet markets and evaluated resources and techniques for domain and task adaptation of our
NER models. The dataset, with about 3,500 item listings, was created via crowd-Sourcing and refined via a manual review.
It is approximately 3× the size of the only other available NER dataset for Darknet markets, we were aware of at this
time.
We found that we were able to improve our NER prediction performance by ‘domain adaptation’ via fine-tuning our
language models on Darknet item descriptions and reduced versions of Wikipedia texts about illicit drugs. Our models were able to predict drug entities with a
F1-Score of up to 84.04 points according to the CoNLL2003NER evaluation
metric.
[Keywords: NER, Named Entity Recognition, noisy user-generated text, darknet, drug detection,
crowd-sourcing, Mechanical Turk]
…The Darknet data is loaded from 2 primary sources, the Darknet Market Archives [BCDH+15] and AZSecure-data [DZE+18].
The Darknet Market Archives contain multiple datasets
about Darknet Market platforms and forums. We only used the “grams” dataset. This dataset contains nearly daily scrapes of multiple market platforms
(e.g. “Agora”). We chose to use the last date where these markets were scraped “2015-07-12” and only a subset of these markets, namely: “Abraxas”, “Agora”,
“Alpha”, “ME”, and “Oxygen”. This dataset was only used for adjusting our language models to the target domain, called domain adaptation (see section 2.1). For the
dataset creation we used a dataset from AZSecure-data, which was scraped from a platform called “Dream Market”. At
this time it was the largest Darknet market platform according to [DZE+18]. The data was collected from 2013 to
2017 and contained 91,463 listings of which 61,420 were found in a category associated with drugs. The dataset contains a variety of product and vendor
information.
In scope of this work, we were only interested in the product name and description. The item description was used for the annotation of named entities and the
product name, was used to provide context to the annotators. However, other types of information were used during the pre-processing for pseudonymization purposes.
The pseudonymization included removing all vendor names from the item listings, removing email addresses and telephone numbers and all links found in the dataset
(those might also identify a vendor profile). A recent example for a drug item listing, which was online at the time of our project, can be seen in Figure
3.1.
Our experiment design required further datasets as representatives for standard NER corpora and text corpora with
noisy user-generated data.Our standard NER text corpus is the well-known CoNLL2003NER dataset[TKSDM03], which is based on
newswire texts annotated with Person, Location, Organization and Miscellaneous entities. As representatives for the noisy user-generated text datasets we
chose the Broad Twitter Corpus [DBR16] and the WNUT 2017 dataset [DNEL17]. The Broad Twitter Corpus contains 9,551 Tweets with annotations for entities of type Person, Location and
Organization. The WNUT 2017 dataset contains 2,295 text from various sources (Reddit, Twitter, YouTube, and
StackExchange comments) with annotations for Person, Location, Corporation, Product, Creative-Work and Group as named entity types. Furthermore, we used the
extension from Al-Nabki [NFAFR20] of theWNUT 2017 dataset called
“NuToT”. This dataset version is extended by Darknet market listings, which advertise illicit goods.
We study objective robustness failures, a type of out-of-distribution robustness failure in reinforcement learning (RL). Objective robustness failures occur when an RL agent retains its capabilities
out-of-distribution yet pursues the wrong objective. This kind of failure presents different risks than the robustness problems usually considered in the
literature, since it involves agents that leverage their capabilities to pursue the wrong objective rather than simply failing to do anything useful. We provide
the first explicit empirical demonstrations of objective robustness failures and present a partial characterization of its causes.
The rapidly-changing ML model landscape presents a unique opportunity for building hardware accelerators optimized for specific datacenter-scale workloads. We
propose Full-stack Accelerator Search Technique (FAST), a hardware accelerator search framework that defines a
broad optimization environment covering key design decisions within the hardware-software stack, including hardware datapath, software scheduling, and compiler
passes such as operation fusion and tensor padding. Although FAST can be used on any number and type of deep
learning workload, in this paper we focus on optimizing for a single or small set of vision models, resulting in significantly faster and more power-efficient
designs relative to a general purpose ML accelerator. When evaluated on EfficientNet, ResNet50v2, and OCR
inference performance relative to a TPU-v3, designs generated by FASToptimized for single workloads can improve Perf/TDP (peak power) by over 6×
in the best case and 4× on average. On a limited workload subset, FAST improves Perf/TDP 2.85× on average, with a reduction to 2.35× for a single design optimized over the set of workloads. In addition, we demonstrate
a potential 1.8× speedup opportunity for TPU-v3 with improved scheduling.
The People’s Liberation Army (PLA) seeks not only to equal but also to overtake the US military through seizing
the initiative in the ongoing Revolution in Military Affairs (RMA). Chinese military leaders believe the form of
warfare is changing from today’s ‘informatised’ (信息化) warfare to future ‘intelligentised’ (智能化) warfare.
The PLA’s approach to leveraging emerging technologies is likely to differ from parallel American initiatives
because of its distinct strategic culture, organisational characteristics, and operational requirements. This research examines the evolution of the
PLA’s strategic thinking and concepts of operations, seeking to contribute to the military innovation literature by
evaluating major theoretical frameworks for the case of China.
The widespread use of experimental benchmarks in AI research has created competition and collaboration dynamics that are still poorly understood. Here we
provide an innovative methodology to explore these dynamics and analyse the way different entrants in these challenges, from academia to tech giants, behave and
react depending on their own or others’ achievements.
We perform an analysis of 25 popular benchmarks in AI from Papers With Code[HMDB-51 · UCF101 ·Montezuma’s Revenge · Space Invaders· CIFAR-100 ·ImageNet · CIFAR-10 · Set5 · Enwik8 ·
Penn Treebank · WN18RR · WMT2014English-French · WMT2014 English-German ·
CoNLL 2003 · Ontonotes v5 ·COCO Minival · COCO test-dev · MPII Human Pose · SQuAD1.1 · WikiQA · Cityscapes test · PASCALVOC2012 test · IMD · SST-2 Binary
classification · LibriSpeech], with around 2,000 result entries overall, connected with their underlying research papers. We identify links between researchers and
institutions (that is, communities) beyond the standard co-authorship relations, and we explore a series of hypotheses about their behaviour as well as some
aggregated results in terms of activity, performance jumps and efficiency. We characterize the dynamics of research communities at different levels of abstraction,
including organization, affiliation, trajectories, results and activity.
We find that hybrid, multi-institution and persevering communities are more likely to improve state-of-the-art performance, which becomes a watershed for many
community members.
Although the results cannot be extrapolated beyond our selection of popular machine learning benchmarks, the methodology can be extended to other areas of
artificial intelligence or robotics, and combined with bibliometric studies.
Table 1: list of AI benchmarks used in the analysis by task
Figure 1: Progress in accuracy over time for ImageNet. Coloured shapes show the different
communities (with one or more institutions in the legend). Dashed lines show the global SOTA front (ingrey) for all the entries (results) and local SOTA front per community (incolour). The blue dotted line shows
the smoothed means (all results) with 95% confidence level intervals. Different shapes indicate the types of institution (companies, universities or hybrid).
…Figure 1 represents the results for the ImageNet dataset, which consists of 1.2 million images in
1,000 classes. The results of the different communities show that several long-term collaborative ‘hybrid’ groups, formed mostly by American universities (Johns
Hopkins, University of California, Los Angeles, Cornell, Stanford, Toronto and so on) in collaboration with tech giants (Microsoft and Google) are those that
have dominated the SOTA front from early on (communities numbered as #1 and #2). Although hybrid communities
dominate the SOTA front, there are also some isolated company players, possibly representing different divisions,
departments and research groups from companies such as Google, Xiaomi, Facebook and Microsoft. However, only a single non-hybrid community, Google, is able to
achieve a score on the SOTA front.
…While all benchmark plots can be found in the Supplementary Information (Supplementary Figs. 4–6), we include another example here, in
Figure 2. This is the Stanford Question Answering Dataset (SQuAD1.1), a reading comprehension benchmark with more than 100,000
question-answer pairs from more than 500 articles. Questions derive from Wikipedia articles where the answer may be a segment of text from the corresponding
reading passage, or may be unanswerable (for example, written adversarially to look similar to answerable ones). Like ImageNet, the SOTA front is dominated by hybrid long-term collaboration groups
(communities numbered #2 and #3) formed by American universities (Stanford, Carnegie Mellon, Washington and so on) in collaboration with tech giants (Facebook and
Google), but also by large hybrid communities formed by Asian universities (Beihang, Fudan, Peking and so on) jointly with Microsoft (community #1). We also
observe that the participation of European universities initiatives is very low. Unlike ImageNet, most entries correspond to
the period between 2016 and 2018, with a clear decline in activity from 2018 to 2020. This is probably due to the introduction of the new (and much more difficult)
version of the benchmark (SQuAD2.0), with attention moving to the new challenge. However, SQuAD1.1 is still being addressed by communities 2 and 3, which have
participated since 2016 and have led the SOTA from 2018 to 2020. Again, we see that long-term collaborative groups
obtain better results than isolated communities
…From the above results, we reach a number of conclusions about the dynamics of communities engaging with AI benchmarks. We find that (1) SOTA jumps are mainly obtained by multi-institution communities, compared with the number of jumps obtained by single-institution
communities; (2) multi-attempt communities are more likely to achieve SOTA jumps (compared with one-shot efforts); (3)
jumps are mainly obtained by hybrid communities involving both universities and companies, meaning that heterogeneous communities achieve more success
through collaborative efforts compared with ‘pure’ communities (only universities or companies); and, finally, (4) the presence of companies in a community, such
as Google, Microsoft and Facebook, increases the odds of achieving a jump in an AI benchmark. All the above reinforces the usefulness of the increasing tendency of
collaboration between universities and industry in AI research.
Figure 3: Most prolific institutions (at least 10 entries) in terms of total SOTA jumps entries
and activity. Both axes are logarithmic. Point size represents the efficiency (ratio between number of
SOTA jumps and attempts). Note that ‘Academic’ represents both higher education and independent research
institutions. We refer the reader to Supplementary Table 1 for further detail about the abbreviations of the institutions.
…While institutions from the United States represent about 56.7% of all jumps, China only represents about 18%. However, the gap becomes smaller if we only
consider the recent years. For instance, Table 3 (orange) shows the same results for year 2019 only. Here the institutions from Asia come at the
forefront in terms of activity compared with those from America. At the country level, activities from the United States and China are much more similar (41%
versus 37%) and although the United States keeps leading the chart with respect to to the number of SOTA jumps compared
with China (54% versus 26%), the difference has narrowed. This country-level concentration is also reflected when we compute the (country-wise) HHIto analyse concentration
and competitiveness. In this case, the HHI is 0.33, showing a much higher concentration level per country compared with
the analysis per institution.
These results are loosely consistent with analyses framing AI research progress as a ‘race’ being led by the United States and China…The data we analyse here
represent only a small snapshot of all AI progress, but it still suggests that the United States has had a relevant lead if we look at the whole period, but the
gap is being reduced by other countries such as China (Table 3). In the whole period, as Figure 3 shows, 6 out of the top 10
institutions are from the United States (the top 3 being tech giants).
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the
computational cost of preserving and storing memories. However, not all content in the past is equally important to remember. We propose Expire-Span, a method that
learns to retain the most important information and expire the irrelevant information. This forgetting of memories enables Transformers to scale to attend over
tens of thousands of previous timesteps efficiently, as not all states from previous timesteps are preserved. We demonstrate that Expire-Span can help models
identify and retain critical information and show it can achieve strong performance on reinforcement learning tasks
specifically designed to challenge this functionality. Next, we show that Expire-Span can scale to memories that are tens of thousands in size, setting a new state
of the art on incredibly long context tasks such as character-level language modeling and a frame-by-frame moving objects task. Finally, we analyze the efficiency
of Expire-Span compared to existing approaches and demonstrate that it trains faster and uses less memory.
[Poster; paper; official code (reimplementation); Yannic Kilcher video] In this paper we propose to
study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization,
generalization, and speed of learning can be studied in great detail.
In some situations we show that neural networks learn through a process of grokking a pattern in the data, improving generalization performance
from random-chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting.
We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We
argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks
beyond memorization of the finite training dataset.
…We find that adding weight decay has a very
large effect on data efficiency, more than halving the amount of samples needed compared to most other interventions. We found that weight decay towards the initialization of the network is also effective, but not quite as effective as weight decay towards the origin. This makes us believe that the prior, that approximately zero weights are suitable for small
algorithmic tasks, explains part, but not all of the superior performance of weight decay. Adding some noise to the
optimization process (eg. gradient noise from using minibatches, Gaussian noise applied to weights before or after computing the gradients) is beneficial for
generalization, consistent with the idea that such noise might induce the optimization to find flatter minima that generalize better. We found that learning rate
had to be tuned in a relatively narrow window for the generalization to happen (within 1 order of magnitude).
Figure 1: Left. ‘Grokking’: A dramatic example of generalization far after overfitting on an algorithmic dataset. We train on
the binary operation of division mod 97 with 50% of the data in the training set. Each of the 97 residues is presented to the network as a separate symbol,
similar to the representation in the figure to the right. The red curves show training accuracy and the green ones show validation accuracy. Training accuracy
becomes close to perfect at <103 optimization steps, but it takes close to 106 steps for validation accuracy to reach that level, and
we see very little evidence of any generalization until 105 steps. Center: Training time required to reach 99% validation accuracy
increases rapidly as the training data fraction decreases. Right: An example of a small binary operation table. We invite the reader to make their
guesses as to which elements are missing.
Biological cognition is based on self-generated learning objectives. However, the mechanism by which this epistemic autonomy is realized by the neuronal
substrate is not understood.
Artificial neural networks based on error backpropagation lack epistemic autonomy because they are mostly trained in a
supervised fashion. In this respect, they face the symbol grounding problem of artificial intelligence.
We propose that the entorhinal-hippocampal
complex, a brain structure located in the medial temporal lobe and central to memory, combines epistemic autonomy with intrinsically generated error gradients
akin to error backpropagation.
We present evidence supporting the hypothesis that the counter-current inhibitory projections of the entorhinal-hippocampal complex implement a continuous
self-supervised error minimization between network input and output.
Biological cognition is based on the ability to autonomously acquire knowledge, or epistemic autonomy.
Such self-supervision is largely absent in artificial neural networks (ANN) because they depend on externally set learning criteria. Yet training
ANN using error backpropagation has created the current revolution in
artificial intelligence, raising the question of whether the epistemic autonomy displayed in biological cognition can be achieved with error backpropagation-based learning.
We present evidence suggesting that the entorhinal-hippocampal complex combines epistemic autonomy with error
backpropagation. Specifically, we propose that the hippocampus minimizes the error between its input and output signals
through a modulatory counter-current inhibitory network. We further discuss the computational emulation of this principle and analyze it in the context of
autonomous cognitive systems.
Figure 1: Forward Excitatory and Counter-Current Inhibitory Circuitry of the Entorhinal-Hippocampal Complex (EHC) Supporting Self-Supervision and Epistemic Autonomy.
(A) Forward and backward hippocampal circuits. Left. The feed-forward information flow of the hippocampal trisynaptic pathway and its
constituents19. The pathway (grey arrow) comprises projections from layer II entorhinal cortex (EC) stellate cells to the dentate gyrus (DG)
and CA3 via the medial (MPP, light green) andlateral (LPP, yellow)
perforant path (PP), mossy fiber projections of DG granule cells to CA3 pyramidal neurons (dark green), and CA3 projections to CA1 pyramidal neurons
(the Schaffer collaterals, pink). The feedforward input is completed with direct projections from layer III EC
neurons projecting to CA119. The output of the hippocampus (HPC) originates in CA1 and passes via
the subiculum (not shown) to the EC LV/VI (purple). Right. Cortical input and hippocampal output coincide in EC, allowing the EHC comparator to compute the mismatch between the 2 signals. (B) Left. Counter-current inhibitory circuit
complementing the forward excitatory loop (Box 3). Right. We hypothesize that this counter-current circuit carries error signals (yellow) that define a
gradient that shapes synaptic plasticity along the forward excitatory loop, therefore implementing a biological version of error backpropagation. (C) Top. The synergy between the forward and feedback circuits shapes the continuous synaptic update in
the forward loop such that it increasingly minimizes the error between the HPC input and output signals of the EC
comparator. Middle. In this self-supervised learning scenario, environmental change (pink line) is reflected in the error amplitude, where error
magnitude activates distinct physiological and behavioral responses. Bottom. Small amplitude errors perturbate the firing rate of principal cells, for
instance, expressed as firing rate modulation in spatial navigation tasks. In contrast, large magnitude errors signal novelty and drive relearning supporting
the reconstruction of this novel signal, leading to global remapping (see20 for a possible threshold-triggered synaptic mechanism based on neuronal
depolarization levels). (D) Left. The interplay between excitatory and inhibitory cells in the EHC
comparator. Cortical signals coded by input neurons (yellow) are propagated throughout the trisynaptic circuit (green arrow) to neurons reflecting the
HPC reconstruction of the input signal (blue). Comparator neurons (brown) receive both the reconstruction and an
inhibitory copy of the input activity (grey), which in turn modulate the firing level of counter-current GABAergic
interneurons that backpropagate the error signal (orange line). At this stage, recurrent GABAergic projections
within the HPC modulate network-level synaptic distributions, leading to a convergence of cortical and hippocampal
signals and thus performing self-supervision. Right. Dependent on its magnitude, mismatch error activates a range of molecular, physiological, and behavioral
phenomena observed during spatial navigation. See [36,42,91,92,93].
Meta-overfitting Error(MOE) of a model is the difference between its average error on the
test data and its expected error on the full distribution. (It is closely related to false discovery rate in statistics.)
This blog post concerns our new paper, which gives meaningful upper bounds on this
sort of trouble for popular deep net architectures, whereas prior ideas from adaptive data analysis gave no nontrivial estimates. We call our estimate Rip van
Winkle’s Razor which combines references to Occam’s Razor and the mythical person who fell asleep for 20 years.
…MOE bounds anddescription length: The starting point of our work is the following classical concentration bounds:
Folklore Theorem: With high probability over the choice of a test set of size N, the MOE
of all models with description length at most k bits is 𝒪(√k⁄N).
At first sight this doesn’t seem to help us because one cannot imagine modern deep nets having a short description. The most obvious description involves
reporting values of the net parameters, which requires millions or even hundreds of millions of bits, resulting in a vacuous upper bound on MOE. Another obvious description would be the computer program used to produce the model using the (publicly available) training and
validation sets. However, these programs usually rely on imported libraries through layers of encapsulation and so the effective program size is pretty large as
well.
Rip van Winkle’s Razor: Our new upper bound involves a more careful definition of Description Length: it is the smallest description that
allows a referee to reproduce a model of similar performance using the (universally available) training and validation datasets.
…Estimating MOE ofResNet-152: As an illustration, here
we provide a suitable description allowing Rip van Winkle to reproduce a mainstream ImageNet model, ResNet-152, which achieves 4.49% top-5 test error.
The description consists of 3 types of expressions: English phrases, Math equations, and directed graphs. In the paper, we describe in detail how to encode each
of them into binary strings and count their lengths. The allowed vocabulary includes primitive concepts that were known before 2012, such as CONV, MaxPool, ReLU, SGD etc., as well as a graph-theoretic
notation/shorthand for describing net architecture. The newly introduced concepts including Batch-Norm, Layer, Block are defined
precisely using Math, English, and other primitive concepts.
Description for reproducing ResNet-152
According to our estimate, the length of the above description is 1032 bits, which translates into a upper bound on meta-overfitting error of merely 5%! This
suggests the real top-5 error of the model on full distribution is at most 9.49%. In the paper we also provide a 980-bit long description for reproducing
DenseNet-264, which leads to 5.06% upper bound on its meta-overfitting error.
…But the important point is that unlike existing bounds in Adaptive Data Analysis, there is no dependence on t, the number of models
that have been tested before, and the bound is non-vacuous. Our estimates indicate that the issue of meta-overfitting on ImageNet for these mainstream models is mild. The reason is that despite the vast number of parameters and hyper-parameters in today’s
deep nets, the information content of these models is not high given knowledge circa 2012.
Many real-world applications such as robotics provide hard constraints on power and compute that limit the viable model complexity of Reinforcement Learning
(RL) agents. Similarly, in many distributed RL settings, acting is done on un-accelerated hardware such as CPUs,
which likewise restricts model size to prevent intractable experiment run times. These “actor-latency” constrained settings present a major obstruction to the
scaling up of model complexity that has recently been extremely successful in supervised learning. To be able to utilize large model capacity while still operating
within the limits imposed by the system during acting, we develop an “Actor-Learner Distillation” (ALD) procedure that
leverages a continual form of distillation that transfers learning progress from a large capacity learner model to a small capacity actor model. As a case
study, we develop this procedure in the context of partially-observable environments, where transformer models have had large improvements over LSTMs recently, at the cost of significantly higher computationalcomplexity. With transformer models as the learner and
LSTMs as the actor, we demonstrate in several challenging memory environments that using Actor-Learner Distillation
recovers the clear sample-efficiency gains of the transformer learner model while maintaining the fast inference and reduced total training time of the
LSTM actor model.
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions
between the inputs at every timestep. While attention is powerful, it does not scale efficiently to long sequences due to its quadratic time and space complexity
in the sequence length. We propose RFA, a linear time and space attention that uses random feature methods to
approximate the softmax function, and explore its application in transformers. RFA can be used as a drop-in
replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism.
Experiments on language modeling and machine translation demonstrate that RFA achieves similar or better performance
compared to strongtransformer baselines. In the machine translation experiment, RFA decodes twice as fast
as a vanilla transformer. Compared to existing efficient transformer variants, RFA is competitive in terms of both
accuracy and efficiency on three long text classification datasets. Our analysis shows that RFA’s efficiency gains are
especially notableon long sequences, suggesting that RFA will be particularly useful in tasks that require
working with large inputs, fast decoding speed, or low memory footprints.
[Keywords: Attention, transformers, machine translation, language modeling]
This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in
adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high
resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with
shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing
for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image
size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (86.4 top-1 accuracy on
ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation(53.5 mIoU on ADE20K val). Its performance
surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU
on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The code and models will be made
publicly available at https://github.com/microsoft/Swin-Transformer.
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. This comes with
a significant computational overhead, as the attention mechanism scales with a quadratic complexity in sequence length. Efficient transformer variants have
received increasing interest from recent works. Among them, a linear-complexity recurrent variant has proven well suited for autoregressive generation. It
approximates the softmax attention with randomized or heuristic feature maps, but can be difficult to train or yield suboptimal accuracy. This work aims to convert
a pretrained transformer into its efficient recurrent counterpart, improving the efficiency while retaining the accuracy. Specifically, we propose a
swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and
then finetune. With a learned feature map, our approach provides an improved tradeoff between efficiency and accuracy over the standard transformer and other
recurrent variants. We also show that the finetuning process needs lower training cost than training these recurrent variants from scratch. As many recent models
for natural language tasks are increasingly dependent on large-scale pretrained transformers, this work presents a viable approach to improving inference
efficiency without repeating the expensive pretraining process.
Recent progress in self-supervised learning has demonstrated promising results in multiple visual tasks. An important
ingredient in high-performing self-supervised methods is the use of data augmentation by training models to place different augmented views of the same image
nearby in embedding space. However, commonly used augmentation pipelines treat images holistically, ignoring the semantic relevance of parts of an
image-e.g. a subject vs. a background-which can lead to the learning of spurious correlations. Our work addresses this problem by investigating a class
of simple, yet highly effective “background augmentations”, which encourage models to focus on semantically-relevant content by discouraging them from focusing on
image backgrounds. Through a systematic investigation, we show that background augmentations lead to substantial improvements in performance across a spectrum of
state-of-the-art self-supervised methods (MoCo-v2, BYOL, SwAV) on a variety of tasks, eg. ~+1–2% gains on ImageNet, enabling
performance on par with the supervised baseline. Further, we find the improvement in limited-labels settings is even larger (up to 4.2%). Background augmentations
also improve robustness to a number of distribution shifts, including natural adversarial examples, ImageNet-9, adversarial
attacks, ImageNet-Renditions. We also make progress in completely unsupervised saliency detection, in the process of
generating saliency masks used for background augmentations.
[video, code (including Perceiver IO); cf Universal Transformers] Biological systems understand the world by simultaneously processing
high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand
are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision
models. These priors introduce helpful inductive
biases, but also lock models to individual modalities.
In this paper we introduce the Perceiver—a model that builds upon Transformers and hence makes few architectural assumptions about the
relationship between its inputs, but that also scales to hundreds of thousands of inputs, like Convnets. The model leverages an asymmetric attention mechanism to
iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs.
We show that this architecture performs competitively or beyond strong, specialized models on classification tasks across various modalities: images, point
clouds, audio, video and video+audio. The Perceiver obtains performance comparable to ResNet-50 on ImageNet without
convolutions and by directly attending to 50,000 pixels. It also surpasses state-of-the-art results for all modalities in AudioSet. [See also “Set Transformer” for efficient attention.]
Figure 1: The Perceiver is an architecture based on attentional principles that scales to high-dimensional inputs such as images, videos,
audio, point-clouds (and multimodal combinations) without making any domain-specific assumptions. The Perceiver uses a cross-attention module to project an
input high-dimensional byte array to a fixed-dimensional latent bottleneck (M ≫ N) before processing it using a stack of transformers in the
low-d latent space. The Perceiver iteratively attends to the input byte array by alternating cross-attention and latent transformer blocks.
Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers
segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are
modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions; object-centric architectures make use
of graph neural networks to model interactions among entities. However, pairwise interactions may not achieve global coordination or a coherent, integrated
representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally specialized
components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep
learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist
modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational
basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.
This paper proposes Omnidirectional Representations from Transformers (OmniNet). In OmniNet, instead of maintaining a strictly horizontal
receptive field, each token is allowed to attend to all tokens in the entire network.
This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field of the entire width and depth of the
network. To this end, the omnidirectional attention is learned via a meta-learner, which is essentially another self-attention based model. In order to mitigate
the computationally expensive costs of full receptive field attention, we leverage efficient self-attention models such as kernel-based (Choromanski et al), low-rank attention (Wang et al) and/or BigBird (Zaheer et al) as the meta-learner.
Extensive experiments are conducted on autoregressive language modeling (LM1B, C4), Machine Translation, Long Range Arena(LRA),
and Image Recognition. The experiments show that OmniNet achieves considerable improvements across these tasks, including achieving state-of-the-art
performance on LM1B, WMT’14 En-De/En-Fr, and Long Range Arena. Moreover, using omnidirectional representation in
Vision Transformers leads to substantial improvements
on image recognition tasks on both few-shot learning and fine-tuning setups.
Improving the efficiency of Transformer-based language pre-training is an important task in NLP, especially for the
self-attention module, which is computationally expensive. In this paper, we propose a simple but effective solution, called LazyFormer, which
computes the self-attention distribution infrequently. LazyFormer composes of multiple lazy blocks, each of which contains multiple Transformer layers. In each
lazy block, the self-attention distribution is only computed once in the first layer and then is reused in all upper layers. In this way, the cost of computation
could be largely saved. We also provide several training tricks for LazyFormer. Extensive experiments demonstrate the effectiveness of the proposed method.
Large language models have become increasingly difficult to train because of the required computation time and cost. In this work, we present SRU++, a recurrent unit with optional built-in attention that exhibits state-of-the-art modeling capacity and training efficiency.
On standard language modeling benchmarks such as enwik8, Wiki-103 and Billion Word datasets, our model obtains better perplexity and bits-per-character (bpc) while
using 3×-10× less training time and cost compared to top-performing Transformer models. Our results reaffirm that attention is not all we need and can be
complementary to other sequential modeling modules. Moreover, fast recurrence with little attention can be a leading model architecture.
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which
have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the
common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore,
most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that
performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental
results.
We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early ’90s, where a “slow” neural net learns by
gradient descent to program the “fast weights” of another net through sequences of elementary programming instructions which are additive outer products of
self-invented activation patterns (today called keys and values). Such Fast Weight Programmers (FWPs) learn to
manipulate the contents of a finite memory and dynamically interact with it. We infer a memory capacity limitation of recent linearised softmax attention variants,
and replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more
easily learn to correct the current mapping from keysto values. The FWP also learns to compute dynamically
changing learning rates. We also propose a new kernel function to linearise attention which balances simplicity and effectiveness. We conduct experiments on
synthetic retrieval problems as well as standard machine translation and language modelling tasks which demonstrate the benefits of our methods.
Predictive coding represents a promising framework for understanding brain function. It postulates that the brain continuously inhibits predictable sensory
input, ensuring a preferential processing of surprising elements. A central aspect of this view is its hierarchical connectivity, involving recurrent message
passing between excitatory bottom-up signals and inhibitory top-down feedback. Here we use computational modelling to demonstrate that such architectural
hard-wiring is not necessary. Rather, predictive coding is shown to emerge as a consequence of energy efficiency. When training recurrent neural networks to
minimise their energy consumption while operating in predictive environments, the networks self-organise into prediction and error units with appropriate
inhibitory and excitatory interconnections, and learn to inhibit predictable sensory input. Moving beyond the view of purely top-down driven predictions, we
demonstrate via virtual lesioning experiments that networks perform predictions on two timescales: fast lateral predictions among sensory units, and slower
prediction cycles that integrate evidence over time.
Artwork is increasingly being created by machines through algorithms with little or no input from humans. Yet, very little is known about people’s attitudes and
evaluations of artwork generated by machines. The current study investigates (a) whether individuals are able to accurately differentiate human-made artwork from
AI-generated artwork and (b) the role of attribution knowledge (ie. information about who created the content) in their evaluation and reception of artwork. Data
was collected using an Amazon Turk sample from two survey experiments designed on Qualtrics. Findings suggest that individuals are unable to accurately identify
AI-generated artwork and they are likely to associate representational art to humans and abstract art to machines. There is also an interaction effect between
attribution knowledge and the type of artwork (representational vs. abstract) on purchase intentions and evaluations of artworks.
There has recently been significant interest in training reinforcement learning (RL) agents in vision-based environments. This poses many challenges, such as
high dimensionality and potential for observational overfitting through spurious correlations. A promising approach to solve both of these problems is a
self-attention bottleneck, which provides a simple and effective framework for learning high performing policies, even in the presence of distractions. However,
due to poor scalability of attention architectures, these methods do not scale beyond low resolution visual inputs, using large patches (thus small attention
matrices). In this paper we make use of new efficient attention algorithms, recently shown to be highly effective for Transformers, and demonstrate that these new
techniques can be applied in the RL setting. This allows our attention-based controllers to scale to larger visual inputs, and facilitate the use of smaller
patches, even individual pixels, improving generalization. In addition, we propose a new efficient algorithm approximating softmax attention with what we call
hybrid random features, leveraging the theory of angular kernels. We show theoretically and empirically that hybrid random features is a promising approach when
using attention for vision-based RL.
Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component that drives the impressive performance of
Transformers is the self-attention mechanism that encodes the influence or dependence of other tokens on each specific token. While beneficial, the quadratic
complexity of self-attention on the input sequence length has limited its application to longer sequences—a topic being actively studied in the community.
To address this limitation, we propose Nyströmformer—a model that exhibits favorable scalability as a function of sequence length. Our idea is
based on adapting the Nyström method to approximate standard self-attention
with 𝒪(n) complexity. The scalability of Nyströmformer enables application to longer sequences with thousands of tokens.
We perform evaluations on multiple downstream tasks on the GLUEbenchmark and IMDB reviews with standard sequence length, and find that our
Nyströmformer performs comparably, or in a few cases, even slightly better, than standard self-attention. On longer sequence tasks in the Long Range Arena(LRA) benchmark, Nyströmformer performs favorably relative to other efficient self-attention methods. Our code is available.
Ubiquitous facial recognition technology can expose individuals’ political orientation, as faces of liberals and conservatives consistently differ. A facial
recognition algorithm was applied to naturalistic images of 1,085,795 individuals to predict their political orientation by comparing their similarity to faces of
liberal and conservative others. Political orientation was correctly classified in 72% of liberal-conservative face pairs, remarkably better than chance (50%),
human accuracy (55%), or one afforded by a 100-item personality questionnaire (66%). Accuracy was similar across countries (the U.S., Canada, and the UK),
environments (Facebook and dating websites), and when comparing faces across samples. Accuracy remained high (69%) even when controlling for age, gender, and
ethnicity. Given the widespread use of facial recognition, our findings have critical implications for the protection of privacy and civil liberties.
[Stats
evaluation by Andrew Gelman et al, plus copy of behind-the-scenes letter lobbying to censor such research in the future.]
We introduce a new pretraining approach for language models that are geared to support multi-document NLP tasks. Our
cross-document language model (CD-LM) improves masked language modeling for these tasks with two key ideas. First, we pretrain with multiple related
documents in a single input, via cross-document masking, which encourages the model to learn cross-document and long-range relationships. Second, extending the
recent Longformer model, we pretrain with long contexts of several thousand tokens and introduce a new attention pattern that uses sequence-level global attention
to predict masked tokens, while retaining the familiar local attention elsewhere. We show that our CD-LM sets new state-of-the-art results for several multi-text
tasks, including cross-document event and entity coreference resolution, paper citation recommendation, and documents plagiarism detection, while using a
significantly reduced number of training parameters relative to prior works.
Five years ago, few would have predicted that a software company like Google would build its own computers. Nevertheless, Google has been deploying computers
for machine learning (ML) training since 2017, powering key Google services. These Tensor Processing Units (TPUs)
are composed of chips, systems, and software, all co-designed in-house. In this paper, we detail the circumstances that led to this outcome, the challenges
and opportunities observed, the approach taken for the chips, a quick review of performance, and finally a retrospective on the results. A companion paper
describes the supercomputers built from these chips, the compiler, and a detailed performance analysis [Jou20].
Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful,
and achieve perplexity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short
subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to
improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens when generating sequences that exceed the
maximal length the transformer can handle at once. Existing methods require computationally expensive relative position embeddings; we introduce a simple
alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. We show that
these recurrent models also benefit from short input lengths. Combining these techniques speeds up training by a factor of 1.65, reduces memory usage, and
substantially improves perplexity on WikiText-103, without adding any parameters.
[Followup: Sonnerat et al 2021] Mixed Integer Programming(MIP) solvers rely on an array of sophisticated heuristics developedwith decades of research to solve large-scale MIP instances encountered in practice. Machine learning offers to automatically construct better heuristics from data by exploiting
shared structure among instances in the data.
This paper applies learning to the two key sub-tasks of a MIP solver, generating a high-quality joint variable
assignment, and bounding the gap in objective value between that assignment and an optimal one. Our approach constructs two corresponding neural network-based
components, Neural Diving and Neural Branching, to use in a base MIPsolver such as SCIP. Neural Diving learns a deep neural network to generate multiple partial assignments for its
integer variables, and the resulting smaller MIPs for un-assigned variables are solved withSCIP to construct high quality joint assignments. Neural Branching learns a deep neural network to make variable selection decisions
in branch-and-bound to bound the objective value gap with a small tree. This is
done by imitating a new variant of Full Strong Branching we propose that scales to large instances using GPUs.
We evaluate our approach on six diverse real-world datasets, including two Google production datasets and MIPLIB, by
training separate neural networks on each. Most instances in all the datasets combined have 103–106 variables and constraints after
presolve, which is substantially larger than previous learning approaches. Comparing solvers with respect to primal-dual gap averaged over a held-out set of
instances, the learning-augmented SCIP is 2× to 10× better on all datasets except one on which it is 105×
better, at large time limits. To the best of our knowledge, ours is the first learning approach to demonstrate such large improvements over SCIP on bothlarge-scale real-world application datasets and MIPLIB.
The Transformer architecture has revolutionized deep learning on sequential data, becoming ubiquitous in state-of-the-art solutions for a wide variety of
applications. Yet vanilla Transformers are notoriously resource-expensive, requiring 𝒪(L2) in serial time and memory as functions of input
length L. Recent works proposed various linear self-attention mechanisms, scaling only as 𝒪(L) for serial computation.
We perform a thorough analysis of recent Transformer mechanisms with linear self-attention, Performers, in terms of overall computational complexity. We observe a
remarkable computational flexibility: forward and backward propagation can be performed with no approximations using sublinear memory as a function of L
(in addition to negligible storage for the input sequence), at a cost of greater time complexity in the parallel setting. In the extreme case, a Performer consumes
only 𝒪(1) memory during training, and still requires 𝒪(L) time.
This discovered time-memory tradeoff can be used for training or, due to complete backward-compatibility, for fine-tuning on a low-memory device, eg. a
smartphone or an earlier-generation GPU, thus contributing towards decentralized and democratized deep
learning.
Understanding the degree to which human facial expressions co-vary with specific social contexts across cultures is central to the theory that emotions enable
adaptive responses to important challenges and opportunities. Concrete evidence linking social context to specific facial expressions is sparse and is largely
based on survey-based approaches, which are often constrained by language and small sample sizes. Here, by applying machine-learning methods to real-world, dynamic
behaviour, we ascertain whether naturalistic social contexts (for example, weddings or sporting competitions) are associated with specific facial expressions
across different cultures. In two experiments using deep neural networks, we examined the extent to which 16 types of facial expression occurred systematically in
thousands of contexts in 6 million videos from 144 countries. We found that each kind of facial expression had distinct associations with a set of contexts that
were 70% preserved across 12 world regions. Consistent with these associations, regions varied in how frequently different facial expressions were produced as a
function of which contexts were most salient. Our results reveal fine-grained patterns in human facial expressions that are preserved across the modern world.
Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series
forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range
dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However,
there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including
quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture.
To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three
distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves 𝒪(L log L) in time complexity and memory
usage, and has comparable performance on sequences’ dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading
layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series
sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions.
Extensive experiments on four large-scale datasets demonstrate that Informer substantially outperforms existing methods and provides a new solution to the
LSTF problem.
Online misinformation has become a constant; only the way actors create and distribute that information is changing. Advances in artificial intelligence (AI)
such as GPT-2 mean that actors can now
synthetically generate text in ways that mimic the style and substance of human-created news stories. We carried out three original experiments to study whether
these AI-generated texts are credible and can influence opinions on foreign policy. The first evaluated human perceptions of AI-generated text relative to an
original story. The second investigated the interaction between partisanship and AI-generated news. The third examined the distributions of perceived credibility
across different AI model sizes. We find that individuals are largely incapable of distinguishing between AI-generated and human-generated text; partisanship
affects the perceived credibility of the story; and exposure to the text does little to change individuals’ policy views. The findings have important implications
in understanding AI in online misinformation campaigns.
[Keywords: misinformation, disinformation, foreign policy, public opinion, media]
Interest in deciphering the fundamental mechanisms and processes of the human mind represents a central driving force in modern neuroscience research.
Activities in support of this goal rely on advanced methodologies and engineering systems that are capable of interrogating and stimulating neural pathways, from
single cells in small networks to interconnections that span the entire brain. Recent research establishes the foundations for a broad range of creative
neurotechnologies that enable unique modes of operation in this context. This review focuses on those systems with proven utility in animal model studies and with
levels of technical maturity that suggest a potential for broad deployment to the neuroscience community in the relatively near future. We include a brief summary
of existing and emerging neuroscience techniques, as background for a primary focus on device technologies that address associated opportunities in electrical,
optical and microfluidic neural interfaces, some with multimodal capabilities. Examples of the use of these technologies in recent neuroscience studies illustrate
their practical value. The vibrancy of the engineering science associated with these platforms, the interdisciplinary nature of this field of research and its
relevance to grand challenges in the treatment of neurological disorders motivate continued growth of this area of study.
For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this
challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large
repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding
ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and
lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations and complete camera information for every image; and (4) factors
every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects.
We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, computation time, and annotation effort. Remarkably,
we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a popular open-source natural language processing
model. We also evaluate sim-to-real transfer performance on two real-world scene understanding tasks—semantic segmentation and 3D shape prediction—where we find
that pre-training on our dataset significantly improves performance on both tasks, and achieves state-of-the-art performance on the most challenging Pix3D test
set. All of our rendered image data, as well as all the code we used to generate our dataset and perform our experiments, is available online.
In the last 20 years the Turing test has been left further behind by new developments in artificial intelligence. At the same time, however, these developments
have revived some key elements of the Turing test: imitation and adversarialness. On the one hand, many generative models, such as generative adversarial networks
(GAN),
build imitators under an adversarial setting that strongly resembles the Turing test (with the judge being a learnt discriminative model). The term “Turing
learning” has been used for this kind of setting. On the other hand, AI benchmarks are suffering an adversarial situation too, with a ‘challenge-solve-and-replace’
evaluation dynamics whenever human performance is ‘imitated’. The particular AI community rushes to replace the old benchmark by a more challenging benchmark, one
for which human performance would still be beyond AI. These two phenomena related to the Turing test are sufficiently distinctive, important and general for a
detailed analysis. This is the main goal of this paper. After recognising the abyss that appears beyond superhuman performance, we build on Turing learning to
identify two different evaluation schemas: Turing testing and adversarial testing. We revisit some of the key questions surrounding the Turing test, such as
‘understanding’, commonsense reasoning and extracting meaning from the world, and explore how the new testing paradigms should work to unmask the limitations of
current and future AI. Finally, we discuss how behavioural similarity metrics could be used to create taxonomies for artificial and natural intelligence. Both
testing schemas should complete a transition in which humans should give way to machines—not only as references to be imitated but also as judges—when pursuing and
measuring machine intelligence.
Despite being the workhorse of deep learning, the backpropagation algorithm is no panacea. It enforces sequential layer updates, thus preventing efficient
parallelization of the training process. Furthermore, its biological plausibility is being challenged. Alternative schemes have been devised; yet, under the
constraint of synaptic asymmetry, none have scaled to modern deep learning tasks and architectures. Here, we challenge this perspective, and study the
applicability of Direct Feedback Alignment (DFA) to neural view synthesis, recommender systems,
geometric learning, and natural language processing. In contrast with previous studies limited
to computer vision tasks, our findings show that it successfully trains a large range of state-of-the-art deep learning architectures, with performance close to
fine-tuned backpropagation. When a larger gap between DFA and
backpropagation exists, like in Transformers, we attribute this to a need to rethink common practices for large and complex architectures. At variance with common beliefs, our work supports that challenging tasks can be tackled in the absence of weight transport.
We introduce Mem2Mem, a memory-to-memory mechanism for hierarchical recurrent neural network based encoder decoder architectures and we explore its use for
abstractive document summarization. Mem2Mem transfers “memories” via readable/writable external memory modules that augment both the encoder and decoder. Our
memory regularization compresses an encoded input article into a more compact set of sentence representations. Most importantly, the memory compression step
performs implicit extraction without labels, sidestepping issues with suboptimal ground-truth data and exposure bias of hybrid extractive-abstractive summarization
techniques. By allowing the decoder to read/write over the encoded input memory, the model learns to read salient information about the input article while
keeping track of what has been generated. Our Mem2Mem approach yields results that are competitive with state of the art transformer based summarization methods,
but with 16 times fewer parameters
We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech
utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models
pre-trained using wav2vec 2.0
pre-training. By doing so, we are able to achieve word-error-rates (WERs) of 1.4%/2.6% on the LibriSpeechtest/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.
This chapter explores the creators and potential consumers of sex robots.
With Realbotix as our case study, we take a closer look at the language and sentiments of those developing the technology and those who are testing, consuming,
or showing an interest in it. We do this by means of website and chat forum analysis, and via interviews with those involved.
From this, we can see the motivation for developing a sexual companion robot places the emphasis firmly on the companionship aspect, and that those involved in
creating and consuming the products share an ideology of intimacy and affection, with sexual gratification only playing a minor role.
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only
linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness.
To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalablekernel methods. FAVOR+ can be
also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with
other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear
architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix,
uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from
pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention
methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.
Scalable framework for capturing long-range interactions between input and structured contextual information, which leads to strong improvements in vision
tasks.
We present a general framework for capturing long-range interactions between an input and structured contextual information (eg. a pixel surrounded by other
pixels). Our method, called the lambda layer, captures such interactions by transforming available contexts into linear functions, termed lambdas, and applying
these linear functions to each input separately. Lambda layers are versatile and may be implemented to model content and position-based interactions in global,
local or masked contexts. As they bypass the need for expensive attention maps, lambda layers can routinely be applied to inputs of length in the thousands,
enabling their applications to long sequences or high-resolution images. The resulting neural network architectures, LambdaNetworks, are computationally efficient
and simple to implement using direct calls to operations available in modern neural network libraries. Experiments on ImageNet classification and MS-COCO object detection and instance
segmentation demonstrate that LambdaNetworks substantially outperform their convolutional and attentional counterparts while being more computationally efficient.
Finally, we introduce LambdaResNets, a family of LambdaNetworks, that considerably improve the speed-accuracy tradeoff of image classification models.
LambdaResNets reach state-of-the-art accuracies on ImageNet while being ~4.5× faster than the popular EfficientNets on
modern machine learning accelerators.
[Keywords: deep learning, neural networks, attention, transformer, vision, image classification]
“Figure 3: Performance (y-axis), speed (x-axis), and memory footprint (size of the circles) of different models.”
Better benchmarking for X-formers. Transformers do not scale very well to long sequence lengths largely because of quadratic
self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not
claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of
models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This
paper proposes a systematic and unified benchmark, Long Range Arena, specifically focused on evaluating model quality under long-context scenarios. Our benchmark
is a suite of tasks consisting of sequences ranging from 1k to 16k tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic
images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate 10 well-established long-range
Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers,
Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. Long Range Arena paves the way towards better understanding this
class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle.
In this paper, we propose a novel transformer architecture in which the context length (the number of past tokens on which the output states depend) can grow at
best quadratically with the memory and computational usage.
Transformer networks have shown outstanding performance on many natural language processing tasks. However the context length (the number of previous tokens on
which the output states depend) of a Transformer network grows at best linearly with the memory and computational power used. This limitation prevents a
transformer network to have very long context in a resource limited application.
In this work, we propose a class of transformer networks, namely Transformer-QL (Quadratically Large), in which, the context
length can grow at best quadratically with the memory and computational power used. We have empirically evaluated a Transformer-QL model in 3 long range language
modeling datasets. The results show that Transformer-QL can provide substantial improvements over other state of the art networks.
[Keywords: deep learning, language model, Transformer network, multi-scale Transformer network, natural language processing,
Transformer-XL]
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and
reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable
staple in the modern deep learning stack. Recently, a dizzying number of X-former models have been proposed—Reformer, Linformer, Performer, Longformer, to name a few—which improve
upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid
researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized
and comprehensive overview of existing work and models across multiple domains.
“Figure 2: Taxonomy of Efficient Transformer Architectures.” (Tay et al 2020)
“Table 1:
Summary of Efficient Transformer Models presented in chronological order of their first public disclosure”
Transformer has become ubiquitous in the deep learning field. One of the key ingredients that destined its success is the self-attention mechanism, which allows
fully-connected contextual encoding over input tokens. However, despite its effectiveness in modeling short sequences, self-attention suffers when handling inputs
with extreme long-range dependencies, as its complexity grows quadratically with respect to the sequence length. Therefore, long sequences are often encoded by
Transformer in chunks using a sliding window. In this paper, we propose Cluster-Former, a novel clustering-based sparse Transformer to perform attention across
chunked sequences. The proposed framework is pivoted on two unique types of Transformer layer: Sliding-Window Layer and Cluster-Former Layer, which encode local
sequence information and global context jointly and iteratively. This new design allows information integration beyond local windows, which is especially
beneficial for question answering (QA) tasks that rely on long-range dependencies. Experiments show that Cluster-Former achieves state-of-the-art performance on
several major QA benchmarks.
However, despite the effectiveness of attention modules to capture long term dependencies, in practice, their application to long sequence input is limited by
compute and memory requirements of the attention computation that grow quadratically, 𝒪(n2), with the sequence length n. To address
this limitation, DeepSpeed offers a suite of sparse attention kernels—an
instrumental technology that can reduce the compute and memory requirement of attention computation by orders-of-magnitude via block-sparse computation. The suite
not only alleviates the memory bottleneck of attention calculation, but also performs sparse computation efficiently.
To address this limitation, DeepSpeed offers a suite of sparse attention kernels –an instrumental technology that can
reduce the compute and memory requirement of attention computation by orders-of-magnitude via block-sparse computation. The suite not only alleviates the memory
bottleneck of attention calculation, but also performs sparse computation efficiently. Its APIs allow convenient
integration with any transformer-based models. Along with providing a wide spectrum of sparsity structures, it has the flexibility of handling any user-defined
block-sparse structures. More specifically, sparse attention (SA) can be designed to compute local attention between nearby tokens, or global attention via summary
tokens computed with local attention. Moreover, SA can also allow random attention, or any combination of local, global, and random attention as shown in the
following figure with blue, orange, and green blocks, respectively. As a result, SA decreases the memory footprint to 𝒪(w × n), in which 1 <
w < n is a parameter, whose value depends on the attention structure.
…Performance results:
Power over 10× longer sequences
up to 6.3× faster computation
higher accuracy
comparison with state of the art, Longformer: 1.47× faster execution pre-training MLM on Wikitext103 / 3.13× faster
execution inference on BERT-Base
In this paper, we present GrokNet, a deployed image recognition system for commerce applications. GrokNet leverages a multi-task learning approach to train a
single computer vision trunk. We achieve a 2.1× improvement in exact product match accuracy when compared to the previous state-of-the-art Facebook product
recognition system. We achieve this by training on 7 datasets across several commerce verticals, using 80 categorical loss functions and 3 embedding losses. We
share our experience of combining diverse sources with wide-ranging label semantics and image statistics, including learning from human annotations, user-generated
tags, and noisy search engine interaction data. GrokNet has demonstrated gains in production applications and operates at Facebook scale.
The global burden of diabetes is rapidly increasing, from 451 million people in 2019 to 693 million by 2045. The insidious onset of type 2 diabetes delays
diagnosis and increases morbidity. Given the multifactorial vascular effects of diabetes, we hypothesized that smartphone-based photoplethysmography could provide a widely accessible digital biomarker for diabetes.
Here we developed a deep neural network (DNN) to detect prevalent diabetes using smartphone-based
photoplethysmography from an initial cohort of 53,870 individuals (the ‘primary cohort’), which we then validated in a separate cohort of 7,806 individuals (the
‘contemporary cohort’) and a cohort of 181 prospectively enrolled individuals from three clinics (the ‘clinic cohort’). The DNN achieved an area under the curve for prevalent diabetes of 0.766 in the primary cohort (95% confidence interval: 0.750–0.782;
sensitivity 75%, specificity 65%) and 0.740 in the contemporary cohort (95% confidence interval: 0.723–0.758; sensitivity 81%, specificity 54%). When the
output of the DNN, calledthe DNN score, was included in a regression analysis
alongside age, gender, race/ethnicity and body mass
index, the area under the curve was 0.830 and the DNN score remained independently predictive ofdiabetes. The performance of the DNN in the clinic cohort was similar to that in other validation datasets. There
was a statistically-significant and positive association between the continuous DNN score and hemoglobin A1c
(p ≤ 0.001) among those with hemoglobin A1c data. These findings demonstrate that smartphone-based photoplethysmography provides a readily
attainable, non-invasive digital biomarker of prevalent diabetes.
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest
architectures. We present a novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference
time. An additional benefit is that our method removes the fixed context size constraint that most transformer models have, allowing for more flexible use. When
applied to the GPT-2 language model, we find that our method attains betterperplexity than an unmodified
GPT-2 model on the PG-19 and WikiText-103 corpora, for a given amount of computation or memory.
If the structure of language vocabularies mirrors the structure of natural divisions that are universally perceived, then the meanings of words in different
languages should closely align. By contrast, if shared word meanings are a product of shared culture, history and geography, they may differ between languages in
substantial but predictable ways. Here, we analysed the semantic neighbourhoods of 1,010 meanings in 41 languages. The most-aligned words were from semantic
domains with high internal structure (number, quantity and kinship). Words denoting natural kinds, common actions and artefacts aligned much less well. Languages
that are more geographically proximate, more historically related and/or spoken by more-similar cultures had more aligned word meanings. These results provide
evidence that the meanings of common words vary in ways that reflect the culture, history and geography of their users.
We compare the impact of hardware advancement and algorithm advancement for SAT solving over the last two decades. In particular, we compare 20-year-old SAT-solvers on new computer hardware withmodern SAT-solvers on 20-year-old hardware. Our
findings show that the progress on the algorithmic side has at least as much impact as the progress on the hardware side.
A scan of the history of gross world product (GWP) over millennia raises fundamental questions about the human
past and prospect. What is the distribution of shocks ranging from recession to pandemic? Were the agricultural and industrial revolutions one-offs or did they
manifest ongoing dynamics? Is growth exponential, if with occasional step changes in the rate, or is it superexponential? If the latter, how do we interpret the
implication that output will become infinite in finite time? This paper introduces the first coherent statistical model of GWP history. It casts a GWP series as a sample path in astochastic diffusion, one
whose specification is novel yet rooted in neoclassical growth theory. After maximum likelihoodfitting to GWPback to 10,000 BCE, most observations fall between the 40th and 60th percentiles of predicted distributions. The fit
implies that GWP explosion is all but inevitable, in a median year of 2047. This projection cuts against the
steadiness of growth in income per person seen in the last two centuries in countries at the economic frontier. And it essentially contra-dicts the laws of
physics. But neither tension justifies immediate dismissal of the explosive projection. Accelerating economic growth is better explained by theory than constant
growth. And if physical limits are articulated in a neoclassical-type model by endogenizing natural resources, explosion leads to implosion, formally avoiding
infinities. The quality of the superexponential fit to the past suggests not so much that growth is destined to ascend as that the human system is unstable.
Transformers-based models, such as BERT, have been one of the mostsuccessful deep learning models for
NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the
sequence length due to their full attention mechanism.
To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a
universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our
theoretical analysis reveals some of the benefits of having 𝒪(1) global tokens (such as CLS), that attend to the
entire sequence as part of the sparse attention mechanism.
The proposed sparse attention can handle sequences of length up to 8× of what was previously possible using similar hardware. As a consequence of the capability
to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering
and summarization. We also propose novel applications to genomics data.
Deep neural networks have been revolutionizing the field of machine learning for the past several years. They have been applied with great success in many
domains of the biomedical data sciences and are outperforming extant methods by a large margin. The ability of deep neural networks to pick up local image features
and model the interactions between them makes them highly applicable to regulatory genomics. Instead of an image, the networks analyze DNA and RNA sequences and additional epigenomic data. In this review, we survey the successes of
deep learning in the field of regulatory genomics. We first describe the fundamental building blocks of deep neural networks, popular architectures used in
regulatory genomics, and their training process on molecular sequence data. We then review several key methods in different gene regulation domains. We start with
the pioneering method DeepBind and its successors, which were developed to predict protein-DNA binding. We then review
methods developed to predict and model epigenetic information, such as histone marks and nucleosome occupancy. Following epigenomics, we review methods to
predict protein-RNA binding with its unique challenge of incorporatingRNA structure information. Finally, we provide our overall view of the strengths and weaknesses of deep neural networks and
prospects for future developments.
Transformers are one of the most important machine learning workloads today. Training one is a very compute-intensive task, often taking days or weeks, and
significant attention has been given to optimizing transformers. Despite this, existing implementations do not efficiently utilize GPUs. We find that data movement is the key bottleneck when training. Due to Amdahl’s Law and massive improvements in compute
performance, training has now become memory-bound. Further, existing frameworks use suboptimal data layouts. Using these insights, we present a recipe for globally
optimizing data movement in transformers. We reduce data movement by up to 22.91% and overall achieve a 1.30× performance improvement over state-of-the-art
frameworks when training a BERT encoder layer and 1.19× for the entireBERT. Our approach is applicable more broadly to optimizing deep neural networks, and offers insight into how to tackle emerging
performance bottlenecks.
Inspired by progress in unsupervised representation learning for natural language, we examine whether similar models can learn useful representations for
images. We train a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure. Despite training on
low-resolution ImageNet without labels, we find that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification. On
CIFAR-10, we achieve 96.3% accuracy with a linear probe, outperforming a supervised Wide ResNet, and 99.0% accuracy with
full fine-tuning, matching the top supervised pre-trained models. An even larger model trained on a mixture of ImageNet and
web images is competitive with self-supervised benchmarks on ImageNet, achieving 72.0% top-1 accuracy on a linear probe of
our features.
Attention and self-attention mechanisms, are now central to state-of-the-art deep learning on sequential tasks. However, most recent progress hinges on
heuristic approaches with limited understanding of attention’s role in model optimization and computation, and rely on considerable memory and computational
resources that scale poorly. In this work, we present a formal analysis of how self-attention affects gradient propagation in recurrent networks, and prove that it
mitigates the problem of vanishing gradients when trying to capture long-term dependencies by establishing concrete bounds for gradient norms. Building on these
results, we propose a relevancy screening mechanism, inspired by the cognitive process of memory consolidation, that allows for a scalable use of sparse
self-attention with recurrence. While providing guarantees to avoid vanishing gradients, we use simple numerical experiments to demonstrate the tradeoffs in
performance and computational resources by efficiently balancing attention and recurrence. Based on our results, we propose a concrete direction of research to
improve scalability of attentive networks.
One strand of analysis that has caught our attention is about the pattern of growth of human society over many millennia, as measured by number of people or
value of economic production. Perhaps the mathematical shape of the past tells us about the shape of the future. I dug into that subject. A draft of my technical
paper is here. (Comments welcome.) In this post, I’ll explain in less
technical language what I learned.
It’s extraordinary that the larger the human economy has become—the more people and the more goods and services they produce—the faster it has grown on average.
Now, especially if you’re reading quickly, you might think you know what I mean. And you might be wrong, because I’m not referring to exponential growth. That
happens when, for example, the number of people carrying a virus doubles every week. Then the growth rate (100% increase per week) holds fixed. The human
economy has grown super-exponentially. The bigger it has gotten, the faster it has doubled, on average. The global economy churned out $74 trillion in
goods and services in 2019, twice as much as in 2000.1 Such a quick doubling was unthinkable in the Middle Ages and ancient times. Perhaps our earliest
doublings took millennia.
If global economic growth keeps accelerating, the future will differ from the present to a mind-boggling degree. The question is whether there might be some
plausibility in such a prospect. That is what motivated my exploration of the mathematical patterns in the human past and how they could carry forward. Having now
labored long on the task, I doubt I’ve gained much perspicacity. I did come to appreciate that any system whose rate of growth rises with its size is inherently
unstable. The human future might be one of explosion, perhaps an economic upwelling that eclipses the industrial revolution as thoroughly as it eclipsed the
agricultural revolution. Or the future could be one of implosion, in which environmental thresholds are crossed or the creative process that drives growth runs
amok, as in an AI dystopia. More likely, these impulses will mix. I now understand more fully a view that shapes the work of Open Philanthropy. The range of
possible futures is wide. So it is our task as citizens and funders, at this moment of potential leverage, to lower the odds of bad paths and raise the odds of
good ones.
The human past, coarsely quantified
Capturing the randomness of history
Land, labor, capital, and more
Interpreting infinity
Conclusion
…Conclusion: I do not know whether most of the history of technological advance on Earth lies behind or ahead of us. I do know that it is far
easier to imagine what has happened than what hasn’t. I think it would be a mistake to laugh off or dismiss the predictions of infinity emerging from good models
of the past. Better to take them as stimulants to our imaginations. I believe the predictions of infinity tell us two key things. First, if the patterns of history
continue, then some sort of economic explosion will take place again, the most plausible channel being AI. It wouldn’t reach infinity, but it could be big. Second,
and more generally, I take the propensity for explosion as a sign of instability in the human trajectory. Gross world product, as a rough proxy for the scale of
the human enterprise, might someday spike or plunge or follow a complicated path in between. The projections of explosion should be taken as indicators of the
long-run tendency of the human system to diverge. They are hinting that realistic models of long-term development are unstable, and stable models of long-term
development unrealistic. The credible range of future paths is indeed wide.
Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism
to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention
matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these
assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Performer, based on Fast Attention
Via Orthogonal Random features (FAVOR). Our mechanism scales linearly rather than quadratically in the number of tokens
in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors.
Furthermore, it provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence. It is also backwards-compatible with
pre-trained regular Transformers. We demonstrate its effectiveness on the challenging task of protein sequence modeling and provide detailed theoretical
analysis.
With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant
unlabeled data at a lower cost. To improve the efficiency, we examine the much-overlooked redundancy in maintaining a full-length token-level presentation,
especially for tasks that only require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which gradually compresses
the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further improve the model capacity. In addition, to perform
token-level predictions as required by common pretraining objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced
hidden sequence via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the
standard Transformer on a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading comprehension. The
code and pretrained checkpoints are available at https://github.com/laiguokun/Funnel-Transformer.
From bottom to top: The doubling of the number of transistors on a chip every 2 years, a seemly inevitable trend that has been called Moore’s
law, has contributed immensely to improvements in computer performance. However, silicon-based transistors cannot get much smaller than they are today, and other
approaches should be explored to keep performance growing. Leiserson et al review recent examples and argue that the most promising place to look is at the top of
the computing stack, where improvements in software, algorithms, and hardware architecture can bring the much-needed boost…The miniaturization of semiconductor
transistors has driven the growth in computer performance for more than 50 years. As miniaturization approaches its limits, bringing an end to Moore’s law,
performance gains will need to come from software, algorithms, and hardware. We refer to these technologies as the “Top” of the computing stack to distinguish them
from the traditional technologies at the “Bottom”: semiconductor physics and silicon-fabrication technology. In the post-Moore era, the Top will provide
substantial performance gains, but these gains will be opportunistic, uneven, and sporadic, and they will suffer from the law of diminishing returns. Big system components offer a promising
context for tackling the challenges of working at the Top.
…Unfortunately, semiconductor miniaturization is running out of steam as a viable way to grow computer performance—there isn’t much more room at the “Bottom.”
If growth in computing power stalls, practically all industries will face challenges to their productivity. Nevertheless, opportunities for growth in computing
performance will still be available, especially at the “Top” of the computing-technology stack: software, algorithms, and hardware architecture.
Advances: Software can be made more efficient by performance engineering: restructuring software to make it run faster. Performance engineering
can remove inefficiencies in programs, known as software bloat, arising from traditional software-development strategies that aim to minimize an application’s
development time rather than the time it takes to run. Performance engineering can also tailor software to the hardware on which it runs, for example, to take
advantage of parallel processors and vector units.
Algorithms offer more-efficient ways to solve problems. Indeed, since the late 1970s, the time to solve the maximum-flow problem improved nearly as much from
algorithmic advances as from hardware speedups. But progress on a given algorithmic problem occurs unevenly and sporadically and must ultimately face diminishing returns. As such, we see the biggest benefits coming from algorithms for new problem domains (eg. machine learning) and from
developing new theoretical machine models that better reflect emerging hardware.
Hardware architectures can be streamlined—for instance, through processor simplification, where a complex processing core is replaced with a simpler core that
requires fewer transistors. The freed-up transistor budget can then be redeployed in other ways—for example, by increasing the number of processor cores running in
parallel, which can lead to large efficiency gains for problems that can exploit parallelism. Another form of streamlining is domain specialization, where hardware
is customized for a particular application domain. This type of specialization jettisons processor functionality that is not needed for the domain. It can also
allow more customization to the specific characteristics of the domain, for instance, by decreasing floating-point precision for machine-learning applications.
In the post-Moore era, performance improvements from software, algorithms, and hardware architecture will increasingly require concurrent changes across other
levels of the stack. These changes will be easier to implement, from engineering-management and economic points of view, if they occur within big system
components: reusable software with typically more than a million lines of code or hardware of comparable complexity. When a single organization or company controls
a big component, modularity can be more easily re-engineered to obtain performance gains. Moreover, costs and benefits can be pooled so that important but costly
changes in one part of the big component can be justified by benefits elsewhere in the same component.
OUTLOOK: As miniaturization wanes, the silicon-fabrication improvements at the Bottom will no longer
provide the predictable, broad-based gains in computer performance that society has enjoyed for more than 50 years. Software performance engineering, development
of algorithms, and hardware streamlining at the Top can continue to make computer applications faster in the post-Moore era. Unlike the historical gains at the
Bottom, however, gains at the Top will be opportunistic, uneven, and sporadic. Moreover, they will be subject to diminishing returns as specific computations
become better explored.
The inferotemporal (IT) cortex is responsible for object recognition,
but it is unclear how the representation of visual objects is organized in this part of the brain. Areas that are selective for categories such as faces, bodies,
and scenes have been found1,2,3,4,5, but large parts of IT cortex lack any known specialization, raising the question of what general principle
governs IT organization. Here we used functional MRI, microstimulation, electrophysiology, and deep networks to
investigate the organization of the macaque IT cortex. We built a low-dimensional object space to describe general objects using a feedforward deep neural network
trained on object classification6. Responses of IT cells to a large set of objects revealed that single IT cells project incoming objects onto specific
axes of this space. Anatomically, cells were clustered into four networks according to the first two components of their preferred axes, forming a map of object
space. This map was repeated across three hierarchical stages of increasing view invariance, and cells that comprised these maps collectively harboured sufficient
coding capacity to approximately reconstruct objects. These results provide a unified picture of IT organization in which category-selective regions are part of a
coarse map of object space whose dimensions can be extracted from a deep network.
Companies use about 300,000× more computation training the best AI systems today than they did in 2012 and algorithmic innovations have also made them 25 times
more efficient at the same tasks.
These are the headline results of two recent papers—“AI
and Compute” and
“AI and Efficiency”—from the Foresight Team at OpenAI. In
today’s episode I spoke with one of the authors, Danny Hernandez, who joined OpenAI after helping develop better forecasting
methods at Twitch and Open Philanthropy. Danny and I talk about how to understand his team’s results and what they mean (and don’t mean) for how we should think
about progress in AI going forward.
Debates around the future of AI can sometimes be pretty abstract and theoretical. Danny hopes that providing rigorous measurements of some of the inputs to AI
progress so far can help us better understand what causes that progress, as well as ground debates about the future of AI in a better shared understanding of the
field…In the interview, Danny and I also discuss a range of other topics, including:
The question of which experts to believe
Danny’s journey to working at OpenAI
The usefulness of “decision boundaries”
The importance of Moore’s law for people who care about the long-term future
What OpenAI’s Foresight Team’s findings might imply for policy
The question whether progress in the performance of AI systems is linear
The safety teams at OpenAI and who they’re looking to hire
One idea for finding someone to guide your learning
The importance of hardware expertise for making a positive impact
If you believe AI progress is fast, what would progress look like that would convince you it’s slow? Paint a picture of that five years from now. What does
slow progress look like to you? And now you’re like, “Oh yeah, progress is actually slow”. And what could have happened that would convince you that it’s
actually fast. But you can make what would update you clear to yourself and others and that for big decisions, this is generally worthwhile.
[See also “Universal Paralinguistic Speech Representations Using
Self-Supervised Conformers”, Shor et al 2021] Recently Transformer and Convolution neural network (CNN) based models have shown promisingresults in Automatic Speech Recognition (ASR),
outperforming Recurrentneural networks (RNNs). Transformer models are good at capturingcontent-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the
best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a
parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer substantially
outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely
used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9%
with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M
parameters.
Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally
been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and
interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a
classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44× between 2012 and 2019. This corresponds
to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore’s Law would only have yielded an 11× cost improvement. We observe
that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress
should integrate measures from both.
Compared to 2012, it now takes 44× less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law3 would yield an 11× cost
improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than
classical hardware efficiency.
…For our analysis, we primarily leveraged open-source re-implementations19,20,21 to measure progress on AlexNet level performance over a long
horizon. We saw a similar rate of training efficiency improvement for ResNet-50 level performance on ImageNet (17-month
doubling time).7,16 We saw faster rates of improvement over shorter timescales in Translation, Go, and DoTA 2:
Within translation, the Transformer22 surpassed seq2seq23performance on English to French translation on WMT’14 with 61× less training compute 3 years later.
We estimate AlphaZero24 took 8× less compute to get to AlphaGo
Zero25 level performance 1 year later.
OpenAI Five Rerun required 5× less training compute to surpass OpenAI Five26 (which beat the world champions, OG) 3 months later.
It can be helpful to think of compute in 2012 not being equal to compute in 2019 in a similar way that dollars need to be inflation-adjusted over time. A fixed
amount of compute could accomplish more in 2019 than in 2012. One way to think about this is that some types of AI research progress in two stages, similar to the
“tick tock” model of development seen in semiconductors; new capabilities (the “tick”) typically require a substantial amount of compute expenditure to obtain,
then refined versions of those capabilities (the “tock”) become much more efficient to deploy due to process improvements. Increases in algorithmic efficiency
allow researchers to do more experiments of interest in a given amount of time and money. In addition to being a measure of overall progress, algorithmic
efficiency gains speed up future AI research in a way that’s somewhat analogous to having more compute.
…We also find increases in inference efficiency in terms of GPU time32, parameters16, and
flops meaningful, but mostly as a result of their economic implications [ Inference costs dominate total costs for successful deployed systems. Inference costs
scale with usage of the system, whereas training costs only need to be paid once.] rather than their effect on future research progress. ShuffleNet13
achieved AlexNet-level performance with an 18× inference efficiency increase in 5 years (15-month doubling time), which suggests that training efficiency and
inference efficiency might improve at similar rates.
…For all these reasons, we’re going to start tracking efficiency SOTAs publicly. We’ll start with vision and
translation efficiency benchmarks (ImageNetand WMT14), and we’ll consider
adding morebenchmarks over time. We believe there are efficiency SOTAs on these benchmarks we’re unaware
of and encourage the research community to submit them here (we’ll give credit to original authors and collaborators).
We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that
have an inductive bias to handle the hierarchical nature of language. Experiments on large-scale language modeling benchmarks empirically demonstrate favorable
likelihood vs memory footprint trade-offs, eg. we show that it is possible to train a hierarchical variant with 30 layers that has 23% smaller memory footprint and
better perplexity, compared to a vanilla transformer with less than half the number of layers, on the Toronto BookCorpus. We analyze the advantages of learned
representations at multiple scales in terms of memory footprint, compute time, and perplexity, which are particularly appealing given the quadratic scaling of
transformers’ run time and memory usage with respect to sequence length.
Many natural language processing and information retrieval problems can be formalized as the task of semantic matching. Existing work in this area has been
largely focused on matching between short texts (eg. question answering), or between a short and a long text (eg. ad-hoc retrieval). Semantic matching between
long-form documents, which has many important applications like news recommendation, related article recommendation and document clustering, is relatively less
explored and needs more research effort. In recent years, self-attention based models like Transformers and BERT have
achieved state-of-the-art performance in the task of text matching. These models, however, are still limited to short text like a few sentences or one
paragraph due to the quadratic computational complexity of self-attention with respect to input text length. In this paper,
we address the issue by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for long-form
document matching. Our model contains several innovations to adapt self-attention models for longer text input. In order to better capture sentence level
semantic relations within a document, we pre-train the model with a novel masked sentence block language modeling task in addition to the masked word language
modeling task used by BERT. Our experimental results on several benchmark datasetsfor long-form document
matching show that our proposed SMITH model outperforms the previous state-of-the-art models including hierarchical
attention, multi-depth attention-based hierarchical recurrent neural network, and BERT. Comparing to BERT based baselines, our model is able to increase maximum input text length from 512 to 2048. We will open source a Wikipedia
based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching.
Transformer has become ubiquitous in natural language processing (eg. machine translation, question answering); however, it requires enormous amount of
computations to achieve high performance, which makes it not suitable for mobile applications that are tightly constrained by the hardware resources and battery.
In this paper, we present an efficient mobile NLP architecture, Lite Transformer tofacilitate deploying
mobile NLP applications on edge devices. The keyprimitive is the Long-Short Range Attention (LSRA), where one group of heads specializes in the local context modeling (by convolution) while another group specializes in the
long-distance relationship modeling (by attention). Such specialization brings consistent improvement over the vanilla transformer on three well-established
language tasks: machine translation, abstractive summarization, and language modeling. Under constrained resources (500M/100M MACs), Lite Transformeroutperforms transformer on WMT’14 English-French by
1.2/1.7BLEU,
respectively. Lite Transformer reduces the computation of transformer base model by 2.5× with 0.3 BLEU score
degradation. Combining with pruning and quantization, we further compressed the model size of Lite Transformer by 18.2×. For language modeling, Lite Transformer
achieves 1.8 lower perplexity than the transformer at around 500M MACs. Notably, Lite Transformer outperforms the
AutoML-based Evolved Transformer by 0.5 higher BLEUfor the mobile NLP setting without the costly architecture search that requires more than 250 GPU years. Code has been made available at https://github.com/mit-han-lab/lite-transformer.
In this work, we present a learning-based approach to chip placement, one of the most complex and time-consuming stages of the chip design process. Unlike prior
methods, our approach has the ability to learn from past experience and improve over time. In particular, as we train over a greater number of chip blocks, our
method becomes better at rapidly generating optimized placements for previously unseen chip blocks. To achieve these results, we pose placement as a Reinforcement
Learning (RL) problem and train an agent to place the nodes of a chip netlist onto a chip canvas. To enable our RL policy to generalize to unseen blocks, we ground
representation learning in the supervised task of predicting placement quality. By designing a neural architecture that can accurately predict reward across a wide
variety of netlists and their placements, we are able to generate rich feature embeddings of the input netlists. We then use this architecture as the encoder of
our policy and value networks to enable transfer learning. Our objective is to minimize PPA (power, performance, and
area), and we show that, in under 6 hours, our method can generate placements that are superhuman or comparable on modern accelerator netlists, whereas
existing baselines require human experts in the loop and take several weeks.
Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper,
we present a newTransformer architecture, Extended Transformer Construction (ETC), that addresses two key
challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a
novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position
encodings and a ContrastivePredictive Coding (CPC)pre-training
objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets
requiring long and/or structured inputs.
2020-lillicrap.pdf: “Backpropagation and the brain”, Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, Geoffrey
Hinton (2020-04-17; similar):
During learning, the brain modifies synapses to improve behaviour. In the cortex, synapses are embedded within multilayered networks, making it difficult to
determine the effect of an individual synaptic modification on the behaviour of the system. The backpropagation algorithm solves this problem in deep artificial
neural networks, but historically it has been viewed as biologically problematic. Nonetheless, recent developments in neuroscience and the successes of artificial
neural networks have reinvigorated interest in whether backpropagation offers insights for understanding learning in the cortex. The backpropagation algorithm learns quickly by computing synaptic updates using feedback connections to deliver error signals. Although
feedback connections are ubiquitous in the cortex, it is difficult to see how they could deliver the error signals required by strict formulations of backpropagation. Here we build on past and recent developments to argue that feedback connections may instead induce neural activities
whose differences can be used to locally approximate these signals and hence drive effective learning in deep networks in the brain.
Human sketches can be expressive and abstract at the same time. Generating anime avatars from simple or even bad face drawing is an interesting area. Lots of
related work has been done such as auto-coloring sketches to anime or transforming real photos to anime. However, there aren’t many interesting works yet to show
how to generate anime avatars from just some simple drawing input. In this project, we propose using GAN to
generate anime avatars from sketches.
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To
address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of
thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention
with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve
state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art
results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant
for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.
Placement Optimization is an important problem in systems and chip design, which consists of mapping the nodes of a graph onto a limited set of resources to
optimize for an objective, subject to constraints. In this paper, we start by motivating reinforcement learning as a
solution to the placement problem. We then give an overview of what deep reinforcement learning is. We next formulate the
placement problem as a reinforcement learning problem and show how this problem can be solved with policy gradient
optimization. Finally, we describe lessons we have learned from training deep reinforcement learning policies across a variety of placement optimization
problems.
Convolution exploits locality for efficiency at a cost of missing long range context. Self-attention has been adopted to augment CNNs with non-local interactions. Recent works prove it possible to stack self-attention layers to obtain a fully attentional
network by restricting the attention to a local region. In this paper, we attempt to remove this constraint by factorizing 2D self-attention into two 1D
self-attentions. This reduces computation complexity and allows performing attention within a larger or even global region. In companion, we also propose a
position-sensitive self-attention design. Combining both yields our position-sensitive axial-attention layer, a novel building block that one could stack to form
axial-attention models for image classification and dense prediction. We demonstrate the effectiveness of our model on four large-scale datasets. In particular,
our model outperforms all existing stand-alone self-attention models on ImageNet. Our Axial-DeepLab improves 2.8% PQ
over bottom-up state-of-the-art on COCO test-dev. This previous state-of-the-art is attained by our small variant
that is 3.8× parameter-efficient and 27× computation-efficient. Axial-DeepLab also achieves state-of-the-art results on Mapillary Vistas and Cityscapes.
Modern C++ servers have memory footprints that vary widely over time, causing persistent heap fragmentation of up to 2× from long-lived objects allocated during
peak memory usage. This fragmentation is exacerbated by the use of huge (2MB) pages, a requirement for high performance on large heap sizes. Reducing fragmentation
automatically is challenging because C++ memory managers cannot move objects.
This paper presents a new approach to huge page fragmentation. It combines modern machine learning techniques with a novel memory manager (LLAMA) that manages the heap based on object lifetimes and huge pages (divided into blocks and lines). A neural network-based
language model predicts lifetime classes using symbolized calling contexts. The model learns context-sensitive per-allocation site lifetimes from previous runs,
generalizes over different binary versions, and extrapolates from samples to unobserved calling contexts. Instead of size classes, LLAMA’s heap is organized bylifetime classes that are dynamically adjusted based on observed behavior at a block
granularity
LLAMA reduces memory fragmentation by up to 78% while only using huge pages on several production servers. We
address ML-specific questions such as tolerating mispredictions and amortizing expensive predictions across application execution. Although our results focus on
memory allocation, the questions we identify apply to other system-level problems with strict latency and resource requirements where machine learning could be
applied.
Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic
compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a
small set of locations independent of content.
Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of
interest. This work builds upon 2 lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains
from approaches based on local, temporal sparse attention.
Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the
overall complexity of attention to 𝑂(n1.5d) from 𝑂(n2d) for sequence length n and hidden
dimension d.
We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity) as well as on image
generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new
state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22-layer Routing Transformer model trained on sequences of length
8192.
We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend. Our method is based on differentiable sorting of internal
representations. Concretely, we introduce a meta sorting network that learns to generate latent permutations over sequences. Given sorted sequences, we are then
able to compute quasi-global attention with only local windows, improving the memory efficiency of the attention module. To this end, we propose new algorithmic
innovations such as Causal Sinkhorn Balancing and SortCut, a dynamic sequence truncation method for tailoring Sinkhorn Attention for encoding and/or decoding
purposes. Via extensive experiments on algorithmic seq2seq sorting, language modeling, pixel-wise image generation, document classification and natural language
inference, we demonstrate that our memory efficient Sinkhorn Attention method is competitive with vanilla attention and consistently outperforms recently proposed
efficient Transformer models such as Sparse Transformers.
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks,
Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally
efficient, it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations
from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all
previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract
representation of the past. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased
representation capacity can create small, shallow models with much stronger performance than comparable Transformers.
Deep neural networks are widely used in image classification problems. However, little work addresses how features from different deep neural networks affect
the domain adaptation problem. Existing methods often extract deep features from one ImageNet model, without exploring other
neural networks. In this paper, we investigate how different ImageNet models affect transfer accuracy on domain adaptation problems. We extract features from
sixteen distinct pre-trained ImageNet models and examine the performance of twelve benchmarking methods when using the
features. Extensive experimental results show that a higher accuracy ImageNet model produces better features, and leads to higher accuracy on domain adaptation
problems (with a correlation coefficient of up to 0.95). We also examine the architecture of each neural network to find the best layer for feature extraction.
Together, performance from our features exceeds that of the state-of-the-art in three benchmark datasets.
Evolution is a blind fitting process by which organisms become adapted to their environment. Does the brain use similar brute-force fitting processes to learn
how to perceive and act upon the world? Recent advances in artificial neural networks have exposed the power of optimizing millions of synaptic weights over
millions of observations to operate robustly in real-world contexts. These models do not learn simple, human-interpretable rules or representations of the world;
rather, they use local computations to interpolate over task-relevant manifolds in a high-dimensional parameter space. Counterintuitively, similar to evolutionary
processes, over-parameterized models can be simple and parsimonious, as they provide a versatile, robust solution for learning a diverse set of functions. This new
family of direct-fit models present a radical challenge to many of the theoretical assumptions in psychology and neuroscience. At the same time, this shift in
perspective establishes unexpected links with developmental and ecological psychology.
Covariant.ai has developed a platform that consists of off-the-shelf robot arms equipped with cameras, a special gripper, and plenty of computer power for
figuring out how to grasp objects tossed into warehouse bins. The company, emerging from stealth Wednesday, announced the first commercial installations of its
AI-equipped robots: picking boxes and bags of products for a German electronics retailer called Obeta.
…The company was founded in 2017 by Pieter Abbeel, a prominent AI professor at UC Berkeley, and several of his students. Abbeel pioneered the application of
machine learning to robotics, and he made a name for himself in academic circles in 2010 by developing a robot capable of folding laundry (albeit very slowly).
Covariant uses a range of AI techniques to teach robots how to grasp unfamiliar objects. These include reinforcement
learning, in which an algorithm trains itself through trial and error, a little like the way animals learn through positive and negative feedback…Besides
reinforcement learning, Abbeel says his company’s robots make use of imitation learning, a way of learning by observing demonstrations of perception and grasping
by another algorithm, and meta-learning, a way of refining the learning process itself. Abbeel says the system can adapt and improve when a new batch of items
arrive. “It’s training on the fly”, he says. “I don’t think anybody else is doing that in the real world.”
…But reinforcement learning is finicky and needs lots of computer power. “I used to be skeptical about reinforcement learning, but I’m not anymore”, says Hinton, a professor at the University of Toronto who also works part time at Google.
Hinton says the amount of computer power needed to make reinforcement learning work has often seemed prohibitive, so it is
striking to see commercial success. He says it is particularly impressive that Covariant’s system has been running in a commercial setting for a prolonged
period.
…Peter Puchwein, vice president of innovation at Knapp, says he is particularly impressed by the way Covariant.ai’s robots can grasp even products in
transparent bags, which can be difficult for cameras to perceive. “Even as a human being, if you have a box with 20 products in poly bags, it’s really hard to take
just one out”, he says…Late last year, the international robot maker ABB ran a contest. It invited 20 companies to
design software for its robot arms that could sort through bins of random items, from cubes to plastic bags filled with other objects. Ten of the companies were
based in Europe, and the other half were in the United States. Most came nowhere close to passing the test. A few could handle most tasks but failed on the
trickier cases. Covariant was the only company that could handle every task as swiftly and efficiently as a human. “We were trying to find weaknesses”, said Marc
Segura, managing director of service robotics at ABB. “It is easy to reach a certain level on these tests, but it
is super difficult not to show any weaknesses.”
Obtaining venous access for blood sampling or intravenous (IV) fluid delivery is an essential first step in patient care. However, success rates rely heavily on
clinician experience and patient physiology. Difficulties in obtaining venous access result in missed sticks and injury to patients, and typically require
alternative access pathways and additional personnel that lengthen procedure times, thereby creating unnecessary costs to healthcare facilities.
Here, we present the first-in-human assessment of an automated robotic venipuncture device designed to safely perform blood draws on peripheral forearm veins.
The device combines ultrasound imaging and miniaturized robotics to identify suitable vessels for cannulation and robotically guide an attached needle toward the
lumen center. The device demonstrated results comparable to or exceeding that of clinical standards, with a success rate of 87% on all participants (n =
31), a 97% success rate on non-difficult venous access participants (n = 25), and an average procedure time of 93 ± 30 s (n = 31).
In the future, this device can be extended to other areas of vascular access such as IV catheterization, central venous access, dialysis, and arterial line
placement.
Over the following year, Woody came to believe that the most promising path to automated facial recognition was one that reduced a face to a set of
relationships between its major landmarks: eyes, ears, nose, eyebrows, lips. The system that he imagined was similar to one that Alphonse Bertillon, the French
criminologist who invented the modern mug shot, had pioneered in 1879. Bertillon described people on the basis of 11 physical measurements, including the length of
the left foot and the length from the elbow to the end of the middle finger. The idea was that, if you took enough measurements, every person was unique. Although
the system was labor-intensive, it worked: In 1897, years before fingerprinting became widespread, French gendarmes used it to identify the serial killer
Joseph Vacher. Throughout 1965, Panoramic attempted to create a fully automated Bertillon system for the face. The team tried to devise a program that could
locate noses, lips, and the like by parsing patterns of lightness and darkness in a photograph, but the effort was mostly a flop.
…Even with this larger sample size, though, Woody’s team struggled to overcome all the usual obstacles. The computer still had trouble with smiles, for
instance, which “distort the face and drastically change inter-facial measurements.” Aging remained a problem too, as Woody’s own face proved. When asked to
cross-match a photo of Woody from 1945 with one from 1965, the computer was flummoxed. It saw little resemblance between the younger man, with his toothy smile and
dark widow’s peak, and the older one, with his grim expression and thinning hair. It was as if the decades had created a different person.
…In 1967, more than a year after his move to Austin, Woody took on one last assignment that involved recognizing patterns in the human face. The purpose of
the experiment was to help law enforcement agencies quickly sift through databases of mug shots and portraits, looking for matches…Woody’s main collaborator on the
project was Peter Hart, a research engineer in the Applied Physics Laboratory at the Stanford Research Institute. (Now known as SRI International, the institute split from Stanford University in 1970 because its heavy reliance on military funding had become so
controversial on campus.) Woody and Hart began with a database of around 800 images—two newsprint-quality photos each of about “400 adult male caucasians”, varying
in age and head rotation. (I did not see images of women or people of color, or references to them, in any of Woody’s facial-recognition studies.) Using the
RAND tablet, they
recorded 46 coordinates per photo, including five on each ear, seven on the nose, and four on each eyebrow. Building on Woody’s earlier experience at normalizing
variations in images, they used a mathematical equation to rotate each head into a forward-looking position. Then, to account for differences in scale, they
enlarged or reduced each image to a standard size, with the distance between the pupils as their anchor metric. The computer’s task was to memorize one version of
each face and use it to identify the other. Woody and Hart offered the machine one of two shortcuts. With the first, known as group matching, the computer would
divide the face into features—left eyebrow, right ear, and so on—and compare the relative distances between them. The second approach relied on Bayesian decision theory; it used 22
measurements to make an educated guess about the whole.
In the end, the two programs handled the task about equally well. More important, they blew their human competitors out of the water. When Woody and Hart asked
three people to cross-match subsets of 100 faces, even the fastest one took six hours to finish. The CDC 3800
computer completed a similar task in about three minutes, reaching a hundredfold reduction in time. The humans were better at coping with head rotation and
poor photographic quality, Woody and Hart acknowledged, but the computer was “vastly superior” at tolerating the differences caused by aging. Overall, they
concluded, the machine “dominates” or “very nearly dominates” the humans.
This was the greatest success Woody ever had with his facial-recognition research. It was also the last paper he would write on the subject. The paper was never
made public—for “government reasons”, Hart says—which both men lamented. In 1970, two years after the collaboration with Hart ended, a roboticist named
Michael Kassler alerted Woody to a facial-recognition study that Leon Harmon at Bell Labs was planning. “I’m irked that this second rate study will now be
published and appear to be the best man-machine system available”, Woody replied. “It sounds to me like Leon, if he works hard, will be almost 10 years behind us
by 1975.” He must have been frustrated when Harmon’s research made the cover of Scientific American a few years later, while his own, more advanced work
was essentially kept in a vault.
We propose Axial Transformers, a self-attention-based autoregressive model for images and other data organized as high dimensional tensors. Existing
autoregressive models either suffer from excessively large computational resource requirements for high dimensional data, or make compromises in terms of
distribution expressiveness or ease of implementation in order to decrease resource requirements. Our architecture, by contrast, maintains both full expressiveness
over joint distributions over data and ease of implementation with standard deep learning frameworks, while requiring reasonable memory and computation and
achieving state-of-the-art results on standard generative modeling benchmarks. Our models are based on axial attention, a simple generalization of self-attention
that naturally aligns with the multiple dimensions of the tensors in both the encoding and the decoding settings. Notably the proposed structure of the layers
allows for the vast majority of the context to be computed in parallel during decoding without introducing any independence assumptions. This semi-parallel
structure goes a long way to making decoding from even a very large Axial Transformer broadly applicable. We demonstrate state-of-the-art results for the Axial
Transformer on the ImageNet-32 and ImageNet-64 image benchmarks as well as on the
BAIR Robotic Pushing video benchmark. We open source the implementation of Axial Transformers.
We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive
Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also
find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of
long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.
The Transformer model is widely successful on many natural language processing tasks. However, the quadratic complexity of self-attention limit its application
on long text.
In this paper, adopting a fine-to-coarse attention mechanism on multi-scale spans via binary partitioning (BP), we propose BP-Transformer (BPT for short). BPT yields 𝑂(k × log(n⁄k)) connections where
k is a hyperparameter to control the density of attention. BPT has a good balance between computation complexity
and model capacity.
A series of experiments on text classification, machine translation and language modeling shows BPT has a superior
performance for long text than previous self-attention models. Our code, hyperparameters and CUDA kernels for
sparse attention are available in PyTorch.
We present BlockBERT, a lightweight and efficientBERT model for better modeling long-distance dependencies.
Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory
consumption and training/inference time, which also enables attention heads to capture either short-range or long-range contextual information. We conduct
experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7–36.1% less memory and 12.0–25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced
BERT-based model,RoBERTa.
Predicting the relationship between a molecule’s structure and its odor remains a difficult, decades-old task. This problem, termed quantitative
structure-odor relationship (QSOR) modeling, is an important challenge in chemistry, impacting human nutrition,
manufacture of synthetic fragrance, the environment, and sensory neuroscience. We propose the use of graph neural networks for QSOR, and show they significantly out-perform prior methods on a novel data set labeled by olfactory experts. Additional analysis
shows that the learned embeddings from graph neural networks capture a meaningful odor space representation of the underlying relationship between structure and
odor, as demonstrated by strong performance on two challenging transfer learning tasks. Machine learning has already had a large impact on the senses of sight and
sound. Based on these early results with graph neural networks for molecular properties, we hope machine learning can eventually do for olfaction what it has
already done for vision and hearing.
A plethora of problems in AI, engineering and the sciences are naturally formalized as inference in discrete probabilistic models. Exact inference is often
prohibitively expensive, as it may require evaluating the (unnormalized) target density on its entire domain. Here we consider the setting where only a limited
budget of calls to the unnormalized density oracle is available, raising the challenge of where in the domain to allocate these function calls in order to
construct a good approximate solution. We formulate this problem as an instance of sequential decision-making under uncertainty and leverage methods from
reinforcement learning for probabilistic inference with budget constraints. In particular, we propose the TreeSample
algorithm, an adaptation of Monte Carlo Tree
Search to approximate inference. This algorithm caches all previous queries to the density oracle in an explicit search tree, and dynamically allocates new
queries based on a “best-first” heuristic for exploration, using existing upper confidence bound methods. Our non-parametric inference method can be effectively
combined with neural networks that compile approximate conditionals of the target, which are then used to guide the inference search and enable generalization
across multiple target distributions. We show empirically that TreeSample outperforms standard approximate inference methods on synthetic factor graphs.
Understanding the inductive bias of neural networks is critical to explaining their ability to generalize. Here, for one of the simplest neural networks—a
single-layer perceptron with n input neurons, one output neuron, and no threshold bias term—we prove that upon random initialization of weights, the a priori
probability P(t) that it represents a Boolean function that classifies t points in 0,1^n as 1 has a remarkably simple form: P(t) = 2^-n for 0 t < 2^n.
Since a perceptron can express far fewer Boolean functions with small or large values of t (low entropy) than with intermediate values of t (high
entropy) there is, on average, a strong intrinsic a-priori bias towards individual functions with low entropy. Furthermore, within a class of functions with fixed t, we often observe a further intrinsic bias towards functions of lower
complexity. Finally, we prove that, regardless of the distribution of inputs, the bias towards low entropy becomes monotonically stronger upon adding ReLU layers, and empirically show that increasing the
variance of the bias term has a similar effect.
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human
judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values
is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper,
we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment
or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic
continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole
sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be
exploiting the fact that labelers rely on simple heuristics.
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human
judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values
is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper,
we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment
or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic
continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole
sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good
performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.
Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at
scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool
use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing
team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome
obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that
centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer
and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random
initialization baselines in a suite of domain-specific intelligence tests.
Gravitational wave (GW) detection is now commonplace and as the
sensitivity of the global network of GW detectors improves, we will observe 𝒪(100)s of transient GW events per year. The current methods used to estimate their
source parameters employ optimally sensitive but computationally costly Bayesian inference approaches where typical analyses have taken between 6 hours and 5 days. For binary neutron star and
neutron star black hole systems prompt counterpart electromagnetic (EM) signatures are expected on timescales of 1s–1min and the current fastest method for
alerting EM follow-up observers, can provide estimates in 𝒪(1) minutes, on a limited range of key source parameters.
Here we show that a conditional variational autoencoder pre-trained on binary black hole signals can return Bayesian posterior probability estimates.
The training procedure need only be performed once for a given prior parameter space and the resulting trained machine can then generate samples describing the
posterior distribution ~6 orders of magnitude faster than existing techniques.
[previously: R2D2] This paper introduces
R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly
variable initial conditions. We also introduce a suite of 8 tasks that combine these 3 properties, and show that R2D3 can solve several of the tasks where other
state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.
…Wall Sensor Stack: The original Wall Sensor Stack environment had a bug that the R2D3 agent was able to exploit. We fixed the bug and verified the
agent can learn the proper stacking behavior.
…Another desirable property of our approach is that our agents are able to learn to outperform the demonstrators, and in some cases even to discover strategies
that the demonstrators were not aware of. In one of our tasks the agent is able to discover and exploit a bug in the environment in spite of all the demonstrators
completing the task in the intended way…R2D3 performed better than our average human demonstrator on Baseball, Drawbridge, Navigate Cubes and the Wall Sensor
tasks. The behavior on Wall Sensor Stack in particular is quite interesting. On this task R2D3 found a completely different strategy than the human demonstrators
by exploiting a bug in the implementation of the environment. The intended strategy for this task is to stack 2 blocks on top of each other so that one of them can
remain in contact with a wall mounted sensor, and this is the strategy employed by the demonstrators. However, due to a bug in the environment the strategy learned
by R2D3 was to trick the sensor into remaining active even when it is not in contact with the key by pressing the key against it in a precise way.
We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation
that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ
approach that directly finds these equilibrium points via root-finding. Such a method is equivalent to running an infinite depth (weight-tied) feedforward
network, but has the notable advantage that we can analytically backpropagate through the equilibrium point using implicit differentiation. Using this approach,
training and prediction in these networks require only constant memory, regardless of the effective “depth” of the network. We demonstrate how DEQs can be applied to two state-of-the-art deep sequence models: self-attention transformers and trellis networks. On large-scale
language modeling tasks, such as the WikiText-103 benchmark, we show that DEQs (1) often improve performance over
these state-of-the-art models (for similar parameter counts); (2) have similar computational requirements to existing models; and (3) vastly reduce memory
consumption (often the bottleneck for training large sequence models), demonstrating an up-to 88% memory reduction in our experiments. The code is available at
https://github.com/locuslab/deq .
Reinforcement Learning (RL) based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, because they directly use ROUGE as the rewards during
training. However, summaries with high ROUGE scores often receive low human judgement. To find a better
reward function that can guide RL to generate human-appealing summaries, we learn a reward function from human ratings on 2,500 summaries. Our reward function only
takes the document and system summary as input. Hence, once trained, it can be used to train RL-based summarisation systems without using any reference summaries.
We show that our learned rewards have significantly higher correlation with human ratings than previous approaches. Human evaluation experiments show that,
compared to the state-of-the-art supervised-learning systems and ROUGE-as-rewards RL summarisation systems,
the RL systems using our learned rewards during training generate summarieswith higher human ratings. The learned reward function and our source code are available
at https://github.com/yg211/summary-reward-no-reference.
Artificial intelligence (AI) is surpassing human performance in a growing number of domains. However, there is limited evidence of its economic effects. Using
data from a digital platform, we study a key application of AI: machine translation. We find that the introduction of a new machine translation system has
substantially increased international trade on this platform, increasing exports by 10.9%. Furthermore, heterogeneous treatment effects are consistent with a
substantial reduction in translation costs. Our results provide causal evidence that language barriers substantially hinder trade and that AI has already begun to
improve economic efficiency in at least one domain.
Physical limits do not preclude simultaneous recordings of all spikes in neocortex
Future electrodes need nontraditional materials and fabrication procedures
Challenges for dense recording include heat dissipation from interface electronics
The classic approach to measure the spiking response of neurons involves the use of metal electrodes to record extracellular potentials. Starting over 60 years
ago with a single recording site, this technology now extends to ever larger numbers and densities of sites.
We argue, based on the mechanical and electrical properties of existing materials, estimates of signal-to-noise ratios, assumptions regarding extracellular
space in the brain, and estimates of heat generation by the electronic interface, that it should be possible to fabricate rigid electrodes to concurrently record
from essentially every neuron in the cortical mantle. This will involve fabrication with existing yet nontraditional materials and procedures. We further emphasize
the need to advance materials for improved flexible electrodes as an essential advance to record from neurons in brainstem and spinal cord in moving animals.
Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade
Learning Environment(ALE), small changes in environment parameters such as stochasticity or the maximum
allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step furthertowards reproducible and comparable DRL, we introduce
SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous
recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that
previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we proposeRainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading
to new state-of-the-art performance. Source code is available for reproducibility.
Many real-world systems problems require reasoning about the long term consequences of actions taken to configure and manage the system. These problems with
delayed and often sequentially aggregated reward, are often inherently reinforcement learning problems and present the
opportunity to leverage the recent substantial advances in deep reinforcement learning. However, in some cases, it is not clear why deep reinforcement learning is a good fit for the problem. Sometimes, it does not perform better than the state-of-the-art solutions. And in
other cases, random search or greedy algorithms could outperform deep reinforcement learning. In this paper, we review, discuss, and evaluate the recent trends of
using deep reinforcement learning in system optimization. We propose a set of essential metrics to guide future works in
evaluating the efficacy of using deep reinforcement learning in system optimization. Our evaluation includes challenges, the types of problems, their formulation
in the deep reinforcement learning setting, embedding, the model used, efficiency, and robustness. We conclude with a discussion on open challenges and potential
directions for pushing further the integration of reinforcement learning in system optimization.
Deep convolutional neural network based image super-resolution (SR) models have shown superior performance in recovering the underlying high resolution (HR)
images from low resolution (LR) images obtained from the predefined downscaling methods. In this paper we propose a learned image downscaling method based on
content adaptive resampler (CAR) with consideration on the upscaling process. The proposed resampler network
generates content adaptive image resampling kernels that are applied to the original HR input to generate pixels on the downscaled image. Moreover, a differentiable upscaling (SR) module is employed to upscale the LR result into its underlying HR counterpart. By back-propagating the
reconstruction error down to the original HR input across the entire framework to adjust model parameters, the proposed framework achieves a new state-of-the-art
SR performance through upscaling guided image resamplers which adaptively preserve detailed information that is essential to the upscaling. Experimental results
indicate that the quality of the generated LR image is comparable to that of the traditional interpolation based method, but the significant SR performance gain is
achieved by deep SR models trained jointly with the CAR model. The code is publicly available on:URL https://github.com/sunwj/CAR.
Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic
jam situation. In this paper, we propose to tackle such forecasting problem with Transformers.
Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product
self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory
bottleneck: space complexity of canonical Transformer grows quadratically with sequence length L, making directly modeling long time series
infeasible.
In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context
can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only 𝑂(L(log L)2) memory cost,
improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget.
Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.
Recently, deep convolutional neural networks (CNNs) have been widelyexplored in single image
super-resolution (SISR) and obtainedremarkable performance. However, most of the existing CNN-based SISR methods mainly focus on wider or deeper architecture design, neglecting to
explore the feature correlations of intermediate layers, hence hindering the representational power of CNNs. To address
this issue, in this paper, we propose a second-order attention network (SAN) for more powerful feature expression
and feature correlation learning. Specifically, a novel trainable second-order channel attention (SOCA) module is
developed to adaptively rescale the channel-wise features by using second-order feature statistics for more discriminative representations. Furthermore, we
present a non-locally enhanced residual group (NLRG) structure, which not only incorporates non-local operations
to capture long-distance spatial contextual information, but also contains repeated local-source residual attention groups (LSRAG) to learn increasingly abstract feature representations. Experimental results demonstrate the superiority of our
SAN network over state-of-the-art SISR methods in terms of both quantitative metrics
and visual quality.
Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a
challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural
network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall
short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple
autoregressive video generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics on popular
benchmark datasets, for which they produce continuations of high fidelity and realism. We also present results from training our models on Kinetics, a large scale
action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While
modeling these phenomena consistently remains elusive, we hope that our results, which include occasional realistic continuations encourage further research on
comparatively complex, large scale datasets such as Kinetics.
Artificial teamwork: Artificially intelligent agents are getting better and better at 2-player games, but most real-world endeavors require
teamwork. Jaderberg et al designed a computer program that excels at playing the video game Quake III
Arena in Capture the Flag mode, where 2 multiplayer teams compete in capturing the flags of the opposing team. The agents were trained by playing
thousands of games, gradually learning successful strategies not unlike those favored by their human counterparts. Computer agents competed successfully against
humans even when their reaction times were slowed to match those of humans.
Reinforcement learning (RL) has shown great success in increasingly complex single-agent environments and 2-player turn-based games. However, the real world
contains multiple agents, each learning and acting independently to cooperate and compete with other agents. We used a tournament-style evaluation to demonstrate
that an agent can achieve human-level performance in a 3-dimensional multiplayer first-person video game, Quake III
Arena in Capture the Flag mode, using only pixels and game points scored as input. We used a 2-tier optimization process in which a population of
independent RL agents are trained concurrently from thousands of parallel matches on randomly generated environments. Each agent learns its own internal reward
signal and rich representation of the world. These results indicate the great potential of multiagent reinforcement learning
for artificial intelligence research.
Figure 1: CTF task and computational training framework. (A and
B) 2 example maps that have been sampled from the distribution of (A) outdoor maps and (B) indoor maps. Each agent in the game sees
only its own first-person pixel view of the environment. (C) Training data are generated by playing thousands of CTF games in parallel on a diverse distribution of procedurally generated maps and (D) used to train the agents that
played in each game with RL. (E) We trained a population of 30 different agents together, which provided a diverse set of teammates and opponents to
play with and was also used to evolve the internal rewards and hyperparameters of agents and learning process. Each circle represents an agent in the
population, with the size of the inner circle representing strength. Agents undergo computational evolution (represented as splitting) with descendants
inheriting and mutating hyperparameters (represented as color). Gameplay footage and further exposition of the environment variability can be found in movie
S1.
Figure 2: Agent architecture and benchmarking. (A) How the agent processes a temporal sequence of observations
xt from the environment. The model operates at 2 different time scales, faster at the bottom and slower by a factor of τ at the top. A
stochastic vector-valued latent variable is sampled at the fast time scale from distribution ℚt on the basis of observations
xt. The action distribution πt is sampled conditional on the latent variable at each time step t. The latent variable is
regularized by the slow moving prior ℙt, which helps capture long-range temporal correlations and promotes memory. The network parameters
are updated by using RL according to the agent’s own internal reward signal rt, which is obtained from a learned transformation w
of game points ρt. wis optimized for winning probability through PBT, another level
of training performed at yet a slower time scale than that of RL. Detailed network architectures are described in figure S11.
(B) (Top) The Elo skill ratings of the FTW agent population throughout
training (blue) together with those of the best baseline agents by using hand-tuned reward shaping (RS) (red) and game-winning reward signal only (black), compared with human and random agent reference points
(violet, shaded region shows strength between 10th and 90thpercentile). The FTW agent achieves a skill level considerably beyond strong human subjects, whereas the baseline agent’s skill plateaus below
and does not learn anything without reward shaping [evaluation procedure is provided in (28 [supplements])]. (Bottom) The
evolution of 3 hyperparameters of the FTW agent population: learning rate, Kullback-Leibler divergence (KL)
weighting, and internal time scale τ, plotted as mean and standard deviation across the population.
Figure 3: Knowledge representation and behavioral analysis. (A) The 2D t-SNEembedding of an FTW agent’s internal states during gameplay. Each point
represents the internal state (hp, hq) at a particular point in the game and is
colored according to the high-level game state at this time—the conjunction of (B) 4 basic CTF situations,
each state of which is colored distinctly. Color clusters form, showing that nearby regions in the internal representation of the agent correspond to
the same high-level game state. (C) A visualization of the expected internal state arranged in a similarity-preserving topological embedding and
colored according to activation (figure S5). (D) Distributions of situation conditional activations (each conditional distribution is
colored gray and green) for particular single neurons that are distinctly selective for
these CTF situations and show the predictive accuracy of this neuron. (E) The true return of the agent’s
internal reward signal and (F) the agent’s prediction, its value function (orange denotes high value, and purple denotes low value). (G) Regions where the agent’s internal 2-time scale representation diverges (red), the agent’s surprise, measured as the KL between the agent’s slow-time and fast-time scale representations (28). (H) The
4-step temporal sequence of the high-level strategy “opponent base camping.” (I) 3 automatically discovered high-level behaviors of agents and
corresponding regions in the t-SNE embedding. (Right)
Average occurrence per game of each behavior for the FTW agent, the FTW agent
without temporal hierarchy (TH), self-play with reward shaping agent, and human subjects (figure S9).
Figure 4: Progression of agent during training. Shown is the development of knowledge representation and behaviors of the
FTW agent over the training period of 450,000 games, segmented into 3 phases (movie S2).
“Knowledge” indicates the percentage of game knowledge that is linearly decodable from the agent’s representation, measured by average scaled
AUC/ROC across 200 features of game state. Some knowledge is compressed to single-neuron responses (Figure
3A), whose emergence in training is shown at the top. “Relative internal reward magnitude” indicates the relative magnitude of the agent’s internal
reward weights of 3 of the 13 events corresponding to game points ρ. Early in training, the agent puts large reward weight on picking up the opponent’s flag,
whereas later, this weight is reduced, and reward for tagging an opponent and penalty when opponents capture a flag are increased by a factor of 2. “Behavior
probability” indicates the frequencies of occurrence for 3 of the 32 automatically discovered behavior clusters through training. Opponent base camping
(red) is discovered early on, whereas teammate following (blue) becomes very prominent midway
through training before mostly disappearing. The “home base defense” behavior (green) resurges in occurrence toward the end of
training, which is in line with the agent’s increased internal penalty for more opponent flag captures. “Memory usage” comprises heat maps of visitation
frequencies for (left) locations in a particular map and (right) locations of the agent at which
the top-10 most frequently read memories were written to memory, normalized by random reads from memory, indicating which locations the agent learned to
recall. Recalled locations change considerably throughout training, eventually showing the agent recalling the entrances to both bases, presumably in order to
perform more efficient navigation in unseen maps (figure S7).
In this paper, we develop a neural summarization model which can effectively process multiple input documents and distill Transformer architecture with the
ability to encode documents in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as
opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among textual units, but can also take
advantage of explicit graph representations focusing on similarity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed
architecture brings substantial improvements over several strong baselines.
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are
available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better
performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet
highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.
To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which
achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on
ImageNet, while being 8.4× smaller and 6.1× faster on inference than the best existing ConvNet. Our EfficientNets also
transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer
learning datasets, with an order of magnitude fewer parameters. Source code is at
https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet.
Although the popular MNIST dataset [LeCun et al 1994] is derived from the NIST database
[Grother & Hanaoka 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough
to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy.
We trace each MNIST digitto its NIST source
and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed,
they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing
performances. Our results unambiguously confirm the trends observed by Recht et al [2018, 2019]: although the misclassification rates are slightly
off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same
digits.
We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in
Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character level
language modeling, where we achieve state-of-the-art performances on text8 and enwiki8 by using a maximum context of 8k characters.
Technology that translates neural activity into speech would be transformative for people who are unable to communicate as a result of neurological impairments.
Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators.
Here we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech.
Recurrent neural networks first decoded directly recorded cortical activity into representations of articulatory movement, and then transformed these
representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity.
Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers,
enabling a component of the decoder to be transferable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed
sentences.
These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication.
Emerging reinforcement learning techniques using deep neural networks have shown great promise in control optimization.
They harness non-local regularities of noisy control trajectories and facilitate transfer learning between tasks. To leverage these powerful capabilities for
quantum control optimization, we propose a new control framework to simultaneously optimize the speed and fidelity of quantum computation against both leakage and
stochastic control errors. For a broad family of two-qubit unitary gates that are important for quantum simulation of many-electron systems, we improve the control
robustness by adding control noise into training environments for reinforcement learning agents trained with
trusted-region-policy-optimization. The agent control solutions demonstrate a two-order-of-magnitude reduction in average-gate-error over baseline
stochastic-gradient-descent solutions and up to a one-order-of-magnitude reduction in gate time from optimal gate synthesis counterparts. These substantial
improvements in both fidelity and runtime are achieved by combining new physical understandings and state-of-the-art machine learning techniques. Our results open
a venue for wider applications in quantum simulation, quantum chemistry and quantum supremacy tests using near-term quantum devices.
One existing challenge in AI research is modeling long-range, subtle interdependencies in complex data like images, videos, or sounds. The Sparse Transformer
incorporates an 𝒪(N ⋅ √N) reformulation of the 𝒪(N2) Transformer self-attention mechanism, along with several other improvements, to apply it directly
to these rich data types. Previously, models used on these data were specifically crafted for one domain or difficult to scale to sequences more than a few
thousand elements long. In contrast, our model can model sequences with tens of thousands of elements using hundreds of layers, achieving state-of-the-art
performance across multiple domains. At OpenAI, we’re using it to help us build AI systems that possess a greater ability to
understand the world…Even computing a single attention matrix, however, can become impractical for very large inputs. We instead use sparse attention patterns,
where each output position only computes weightings from a subset of input positions.
Future work and limitations:
The sparse attention patterns we introduced are only preliminary steps in the direction of efficient modeling of long sequences. We think exploring different
patterns and combinations of sparsity is useful, and that learning sparse patterns is a particularly promising avenue of research for the next generation of
neural network architectures.
Even with the improvements we described above, autoregressive sequence generation still seems impractical for very high resolution images or video. The
optimized attention operations we have introduced, however, may be useful primitives to combine with other approaches to modeling high dimensional data, like
multi-scale approaches.
Current state-of-the-art convolutional architectures for object detection are manually designed. Here we aim to learn a better architecture of feature pyramid
network for object detection. We adopt Neural Architecture Search and discover a new feature pyramid architecture in a novel
scalable search space covering all cross-scale connections. The discovered architecture, named NAS-FPN, consists of
a combination of top-down and bottom-up connections to fuse features across scales. NAS-FPN, combined with various
backbone models in the RetinaNet framework, achieves better accuracy and latency tradeoff compared to state-of-the-art object detectionmodels. NAS-FPN improvesmobile detection accuracy by 2 AP
compared to state-of-the-art SSDLite with MobileNetV2 model in [32] and achieves 48.3 AP which surpasses Mask
R-CNN [10] detection accuracy with less computation time.
We propose an efficient algorithm to embed a given image into the latent space of StyleGAN. This embedding enables semantic image editing
operations that can be applied to existing photographs. Taking the StyleGAN trained on
the FFHQ dataset as an example, we show results for image morphing, style transfer, and expression transfer. Studying the
results of the embedding algorithm provides valuable insights into the structure of the StyleGAN latent space. We propose a set of experiments to test what class of images can be embedded, how they are embedded, what latent
space is suitable for embedding, and if the embedding is semantically meaningful.
…Going beyond faces, interestingly, we find that although the FFHQStyleGAN generator is trained on a human face dataset, the embedding algorithm is capable to go far beyond human faces. As
Figure 1 shows, although slightly worse than those of human faces, we can obtain reasonable and relatively high-quality embeddings of
cats, dogs and even paintings and cars. This reveals the effective
embedding capability of the algorithm and the generality of the learned filters of the generator.
Figure 1: Top row: input images. Bottom row: results of embedding the images into the StyleGAN latent space.
While reinforcement learning (RL) has achieved impressive advances in games and robotics, it has not been widely adopted
in recommender systems. Framing recommendation as an RL problem offers new perspectives, but also faces substantial challenges in practice. Industrial recommender
systems deal with extremely large action spaces—many millions of items to recommend and complex user state spaces—billions of users, who are unique at any point in
time. In this talk, I will discuss our work on scaling up a policy-gradient-based algorithm, ie. REINFORCE
to a production recommender system at Youtube. We proposed algorithms to address data biases when deriving policy updates from logged implicit feedback. I will
also discuss some follow up work and outstanding research questions in applying RL, in particular off-policy optimization in recommender systems. [33m:16s; with
slides]
The use of Artificial Intelligence (AI) machines using deep learning neural networks to create material that facially looks like it should be protected by
copyright is growing exponentially. From articles in national news media to music, film, poetry and painting, AI machines create material that has economic value
and that competes with productions of human authors. The Article reviews both normative and doctrinal arguments for and against the protection by copyright of
literary and artistic productions made by AI machines.
The Article finds that the arguments in favor of protection are flawed and unconvincing and that a proper analysis of the history, purpose, and major doctrines
of copyright law all lead to the conclusion that productions that do not result from human creative choices belong to the public domain.
The Article proposes a test to determine which productions should be protected, including in case of collaboration between human and machine. Finally, the
Article applies the proposed test to three specific fact patterns to illustrate its application.
Graph partitioning is the problem of dividing the nodes of a graph into balanced partitions while minimizing the edge cut across the partitions. Due to its
combinatorial nature, many approximate solutions have been developed, including variants of multi-level methods and spectral clustering. We propose
GAP, a Generalizable Approximate Partitioning framework that takes a deep learning approach to graph partitioning. We
define a differentiable loss function that represents the partitioning objective and use backpropagation to optimize the network parameters. Unlike baselines that redo the optimization per graph, GAP is capable of generalization, allowing us to train models that produce performant partitions at inference time, even on unseen
graphs. Furthermore, because we learn the representation of the graph while jointly optimizing for the partitioning loss function, GAP can
be easily tuned for a variety ofgraph structures. We evaluate the performance of GAP on graphs of varying
sizes and structures, including graphs of widely used machine learning models (eg. ResNet, VGG, and Inception-V3), scale-freegraphs, and random graphs. We show that GAP achieves
competitive partitions while being up to 100 times faster than the baseline and generalizes to unseen graphs.
Recent research used machine learning methods to predict a person’s sexual orientation from their photograph (Wang & Kosinski 2017). To verify this result,
two of these models are replicated, one based on a deep neural network (DNN) and one on facial morphology (FM). Using
a new dataset of 20,910 photographs from dating websites, the ability to predict sexual orientation is confirmed (DNN accuracy male 68%, female 77%, FM male 62%, female 72%). To investigate whether facial features such as brightness or
predominant colours are predictive of sexual orientation, a new model based on highly blurred facial images was created. This model was also able to predict sexual
orientation (male 63%, female 72%). The tested models are invariant to intentional changes to a subject’s makeup, eyewear, facial hair and head pose (angle that
the photograph is taken at). It is shown that the head pose is not correlated with sexual orientation. While demonstrating that dating profile images carry rich
information about sexual orientation these results leave open the question of how much is determined by facial morphology and how much by differences in grooming,
presentation and lifestyle. The advent of new technology that is able to detect sexual orientation in this way may have serious implications for the privacy and
safety of gay men and women.
Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly “intelligent” behavior. Here we
apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This
showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance
evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance
Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a
learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement
about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.
Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected
attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful
sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are
connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving capacity to capture both local composition and
long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer
for the modestly sized datasets.
We build new test sets for the CIFAR-10 andImageNet datasets. Both
benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following
the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find
accuracy drops of 3%—15% on CIFAR-10 and 11%—14% onImageNet. However,
accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity,
but by the models’ inability to generalize to slightly “harder” images than those found in the original test sets.
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search
(NAS) has begun to outperform human-designedmodels. Our goal is to apply NAS
to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence
models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the
computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles
method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments—the Evolved
Transformer—demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014
English-German, WMT2014 English-French, WMT 2014 English-Czech and LM1B. At a
big modelsize, the Evolved Transformer establishes a new state-of-the-art BLEUscore of 29.8 on
WMT’14 English-German; at smaller sizes, it achieves the same quality as the original “big” Transformer with 37.6% less
parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search
(NAS) has begun to outperform human-designedmodels. Our goal is to apply NAS
to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence
models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the
computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles
method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments—the Evolved
Transformer—demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014
English-German, WMT2014 English-French, WMT 2014 English-Czech and LM1B. At a
big modelsize, the Evolved Transformer establishes a new state-of-the-art BLEUscore of 29.8 on
WMT’14 English-German; at smaller sizes, it achieves the same quality as the original “big” Transformer with 37.6% less
parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.
Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each
element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results.
Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the
current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length,
whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions
improve over strong self-attention models. On the WMT’14 English-German test set dynamic convolutions achieve a
new state of the art of 29.7 BLEU.
The performance of the code generated by a compiler depends on the order in which the optimization passes are applied. In high-level synthesis, the quality of
the generated circuit relates directly to the code generated by the front-end compiler. Choosing a good order—often referred to as the phase-ordering problem—is an
NP-hard problem. In this paper, we evaluate a new technique to address the phase-ordering problem: deep reinforcement
learning. We implement a framework in the context of the LLVM compiler to optimize theordering for
HLS programs and compare the performance of deep reinforcement learning to state-of-the-art algorithms that address the
phase-ordering problem. Overall, our framework runs one to two orders of magnitude faster than these algorithms, and achieves a 16% improvement in circuit
performance over the -O3 compiler flag.
Existing personal assistants and agents are by design limited in their ability to form or encourage close personal bonds.
The Harmony system is designed to be a customizable personal companion agent capable of close personal interaction via the user’s phone, virtual reality
headset, as well as through a physical interactive android body. In this chapter, we will describe the history that led to Harmony’s creation, the unique
challenges and the overall system design.
We will also look at user reactions to the system and anticipated future developments.
Particular deep artificial neural networks (ANNs) are today’s most accurate models of the primate brain’s
ventral visual stream. Using an ANN-driven image synthesis method, we found that luminous power patterns (ie.,
images) can be applied to primate retinae to predictably push the spiking activity of targeted V4 neural sites beyond naturally occurring levels. This method,
although not yet perfect, achieves unprecedented independent control of the activity state of entire populations of V4 neural sites, even those with overlapping
receptive fields. These results show how the knowledge embedded in today’s ANN models might be used to
noninvasively set desired internal brain states at neuron-level resolution, and suggest that more accurate ANN models would produce even more accurate control.
We propose a class of Evolutionary-Neural hybrid agents, that retain the best qualities of the two approaches.
Neural Architecture Search has recently shown potential to automate the design of Neural Networks. The use of Neural Network agents trained with Reinforcement Learning can offer the possibility to learn complex patterns, as well as the ability to explore a vast and compositional
search space. On the other hand, evolutionary algorithms offer the greediness and sample efficiency needed for such an application, as each sample requires a
considerable amount of resources. We propose a class of Evolutionary-Neural hybrid agents (Evo-NAS), that retain the best
qualities of the two approaches. Weshow that the Evo-NAS agent can outperform both Neural and Evolutionary
agents, both on a synthetic task, and on architecture search for a suite of text classification datasets.
In this paper, we propose an inverse reinforcement learning method for architecture search (IRLAS), which trains an agent to learn to search network structures that are topologically inspired by human-designed network. Most
existing architecture search approaches totally neglect the topological characteristics of architectures, which results in complicated architecture with a high
inference latency. Motivated by the fact that human-designed networks are elegant in topology with a fast inference speed, we propose a mirror stimuli function
inspired by biological cognition theory to extract the abstract topological knowledge of an expert human-design network (ResNeXt). To avoid raising a too strong prior over the search space, we introduce inverse
reinforcement learning to train the mirror stimuli function and exploit it as a heuristic guidance for architecture search, easily generalized to different
architecture search algorithms. On CIFAR-10,the best architecture searched by our proposed IRLAS achieves 2.60% error rate. For ImageNet mobile setting, our model achieves a
state-of-the-art top-1 accuracy 75.28%, while being 2 4× faster than most auto-generated architectures. A fast version of this model achieves 10% faster than
MobileNetV2, while maintaining a higher accuracy.
Industrial recommender systems deal with extremely large action spaces—many millions of items to recommend. Moreover, they need to serve billions of users, who
are unique at any point in time, making a complex user state space. Luckily, huge quantities of logged implicit feedback (eg. user clicks, dwell time) are
available for learning. Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the
previous versions of the recommender. In this work, we present a general recipe of addressing such biases in a production top-K recommender system at
Youtube, built with a policy-gradient-based algorithm, i.e. REINFORCE.The contributions of the paper
are: (1) scaling REINFORCE to a production recommender system with an action space on the orders of millions; (2)
applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies; (3) proposing a novel top-K
off-policy correction to account for our policy recommending multiple items at a time; (4) showcasing the value of exploration. We demonstrate the efficacy of our
approaches through a series of simulations and multiple live experiments on Youtube.
Dot-product attention has wide applications in computer vision and natural language processing. However, its memory and computational costs grow quadratically
with the input size. Such growth prohibits its application on high-resolution inputs. To remedy this drawback, this paper proposes a novel efficient attention
mechanism equivalent to dot-product attention but with substantially less memory and computational costs. Its resource efficiency allows more widespread and
flexible integration of attention modules into a network, which leads to better accuracies. Empirical evaluations demonstrated the effectiveness of its advantages.
Efficient attention modules brought significant performance boosts to object detectors and instance segmenters on MS-COCO
2017. Further, the resource efficiency democratizes attention to complex models, where high costs prohibit the use of dot-product attention. As an exemplar,
a model with efficient attention achieved state-of-the-art accuracies for stereo depth estimation on the Scene Flow dataset. Code is available at
https://github.com/cmsflash/efficient-attention.
Neural architecture search (NAS) has a great impact by automatically designing effective neural network
architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. 104GPU hours) makes it difficult to directly search the architectures on large-scale tasks
(e.g. ImageNet). DifferentiableNAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to
utilize proxy tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These
architectures optimized on proxy tasks are not guaranteed to be optimal on the target task. In this paper, we present ProxylessNAS that can directly learn the architectures for large-scale target tasks and target hardware platforms. We address
the high memory consumption issue of differentiableNAS and reduce the
computational cost (GPU hours and GPU memory) to
the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and
ImageNet demonstrate the effectiveness of directness and specialization. On CIFAR-10, our model achieves 2.08% test error with only 5.7M parameters, better than the previous state-of-the-art architecture
AmoebaNet-B, while using 6× fewer parameters. On ImageNet, our model achieves 3.1% better top-1 accuracy than MobileNetV2, while being 1.2× faster with measured
GPU latency. We also apply ProxylessNAS to specialize neural
architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN
architecture design.
Contextual information is vital in visual understanding problems, such as semantic segmentation and object detection. We
propose a Criss-Cross Network (CCNet) for obtaining full-image contextual information in a very effective and
efficient way. Concretely, for each pixel, a novel criss-cross attention module harvests the contextual information of all the pixels on its criss-cross path. By
taking a further recurrent operation, each pixel can finally capture the full-image dependencies. Besides, a category consistent loss is proposed to enforce the
criss-cross attention module to produce more discriminative features. Overall, CCNet is with the following merits:
(1) GPU memory friendly. Compared with the non-local block, the proposed recurrent criss-cross
attention module requires 11× less GPU memory usage. (2) High computational efficiency. The recurrent criss-cross
attention significantly reduces FLOPs by about 85% of the non-local block. (3) The state-of-the-art performance. We
conduct extensive experiments on semantic segmentation benchmarks including Cityscapes, ADE20K, human parsing benchmark
LIP, instance segmentationbenchmark COCO, video segmentation benchmark
CamVid. In particular,our CCNet achieves the mIoU scores of 81.9%, 45.76% and 55.47% on theCityscapes test set, the ADE20K validation set and the LIP validation set
respectively, which are the new state-of-the-art results. The source codes are available at https: / / github.com / speedinghzl / CCNet.
We present a new method for finding video CNN architectures that capture rich spatio-temporal information in
videos. Previous work, taking advantage of 3D convolutions, obtained promising results by manually designing video CNN
architectures. We here develop a novel evolutionary search algorithm that automatically explores models with different types and combinations of layers to
jointly learn interactions between spatial and temporal aspects of video representations. We demonstrate the generality of this algorithm by applying it to two
meta-architectures, obtaining new architectures superior to manually designed architectures. Further, we propose a new component, the iTGM layer, which more efficiently utilizes its parameters to allow learning of space-time interactions over longer time
horizons. The iTGM layer is often preferred by the evolutionary algorithm and allows building cost-efficient networks.
The proposed approach discovers new and diverse video architectures that were previously unknown. More importantly they are both more accurate and faster than
prior models, and outperform the state-of-the-art results on multiple datasets we test, including HMDB, Kinetics, and
Moments in Time. We will open source the code and models, to encourage future model development.
Conventional Neural Architecture Search (NAS) aims at finding a single architecture that achieves the best
performance, which usually optimizes task related learning objectives such as accuracy. However, a single architecture may not be representative enough for the
whole dataset with high diversity and variety. Intuitively, electing domain-expert architectures that are proficient in domain-specific features can further
benefit architecture related objectives such as latency. In this paper, we propose InstaNAS—an instance-aware
NAS framework—that employs a controller trained to search for a “distribution of architectures” instead of a single
architecture; This allows the model to use sophisticated architectures for the difficult samples, which usually comes with large architecture related cost, and
shallow architectures for those easy samples. During the inference phase, the controller assigns each of the unseen input samples with a domain expert architecture
that can achieve high accuracy with customized inference costs. Experiments within a search space inspired by MobileNetV2 show InstaNAS can achieve up to 48.8% latency reduction without compromising accuracy on a series of datasets against MobileNetV2.
One can infer from the broken window theory that the perception of a city street’s safety level relies importantly on the visual appearance of the street.
Previous works have addressed the feasibility of using computer vision algorithms to classify urban scenes. Most of the existing urban perception predictions focus
on binary outcomes such as safe or dangerous, wealthy or poor. However, binary predictions are not representative and cannot provide informative inferences such as
the potential crime types in certain areas.
In this paper, we explore the connection between urban perception and crime inferences. We propose a convolutional neural network(CNN)—StreetNet—to learn crime rankings from street view images. The learning process is formulated on the basis of
preference learning and label ranking settings. We design a street view images retrieval algorithm to improve the representation of urban perception. A
data-driven, spatiotemporal algorithm is proposed to find unbiased label mappings between the street view images and the crime ranking records.
Extensive evaluations conducted on images from different cities and comparisons with baselines demonstrate the effectiveness of our proposed method.
[Keywords: preference learning, street view, convolutional neural networks, spatial analysis]
Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned
optimizers may similarly outperform current hand-designed optimizers, especially for specific problems. However, learned optimizers are notoriously difficult to
train and have yet to demonstrate wall-clock speedups over hand-designed optimizers, and thus are rarely used in practice. Typically, learned optimizers are
trained by truncated backpropagation through an unrolled optimization process resulting in gradients that are either
strongly biased (for short truncations) or have exploding norm (for long truncations). In this work we propose a training scheme which overcomes both of these
difficulties, by dynamically weighting two unbiased gradient estimators for a variational loss on optimizer performance, allowing us to train neural networks to
perform optimization of a specific task faster than tuned first-order methods. We demonstrate these results on problems where our learned optimizer trains
convolutional networks faster in wall-clock time compared to tuned first-order methods and with an improvement in test loss.
We present Optimal Completion Distillation(OCD), a training procedure for optimizing sequence
to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not
require pretraining or joint optimization with conditional log-likelihood.
Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient
dynamic programming algorithm. Then, for each
position of the generated sequence, we use a target distribution that puts equal probability on the first token of all the optimal suffixes.
OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and
LibriSpeech datasets, achieving 9.3% WER and 4.5% WER respectively.
Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since
solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant.
We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the
input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature.
It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set.
We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent
methods for set-structured data.
Search-based methods for hard combinatorial optimization are often guided by heuristics. Tuning heuristics in various conditions and situations is often
time-consuming. In this paper, we propose NeuRewriter that learns a policy to pick heuristics and rewrite the local components of the current solution to
iteratively improve it until convergence. The policy factorizes into a region-picking and a rule-picking component, each parameterized by a neural network trained
with actor-critic methods in reinforcement learning. NeuRewriter captures the general structure of combinatorial problems
and shows strong performance in three versatile tasks: expression simplification, online job scheduling and vehicle routing problems. NeuRewriter outperforms the
expression simplification component in Z3; outperforms DeepRM and Google OR-tools in online job scheduling; and outperforms recent neural baselines and Google
OR-tools in vehicle routing problems.
We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question
answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to
produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method
makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as an ensemble of agents trained
on the full data. We show that the improved performance is due to the increased diversity of reformulation strategies.
The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad
array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning
algorithms. Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks. An
open question is the degree to which such methods may generalize to new domains. In this work we explore the construction of meta-learning techniques for dense
image prediction focused on the tasks of scene parsing, person-part segmentation, and semantic image segmentation. Constructing viable search spaces in this domain
is challenging because of the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a survey of
techniques in dense image prediction, we construct a recursive search space and demonstrate that even with efficient random search, we can identify architectures
that outperform human-invented architectures and achieve state-of-the-art performance on three dense prediction tasks including 82.7% on Cityscapes (street
scene parsing), 71.3% on PASCAL-Person-Part (person-part segmentation), and 87.9% on PASCALVOC 2012 (semantic image segmentation). Additionally, the
resulting architecture is more computationally efficient, requiring half the parameters and half the computational cost as previous state of the art systems.
A major obstacle in reinforcement learning-based sentence generation is the large action space whose size is equal to the
vocabulary size of the target-side language. To improve the efficiency of reinforcement learning, we present a novel approach for reducing the action space based
on dynamic vocabulary prediction.
Our method first predicts a fixed-size small vocabulary for each input to generate its target sentence. The input-specific vocabularies are then used at
supervised and reinforcement learning steps, and also at test time.
In our experiments on 6 machine translation and 2 image captioning datasets, our method achieves faster reinforcement
learning (~2.7× faster) with less GPU memory (~2.3× less) than the full-vocabulary counterpart.
The reinforcement learning with our method consistently leads to substantial improvement of BLEU scores, and the scores are equal to or better than those of baselines using the full vocabularies, with faster
decoding time (~3× faster) on CPUs.
Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performance in many
tasks such as image classification and language understanding. However, most existing works only optimize for model accuracy and largely ignore other important
factors imposed by the underlying hardware and devices, such as latency and energy, when making inference. In this paper, we first introduce the problem of
NAS and provide a survey on recent works.Then we deep dive into two recent advancements on extending NAS intomultiple-objective frameworks: MONAS and DPP-Net. Both MONAS andDPP-Net are capable of
optimizing accuracy and other objectives imposed by devices, searching for neural architectures that can be best deployed on a wide spectrum of devices:
from embedded systems and mobile devices to workstations. Experimental results are poised to show that architectures found by MONAS and DPP-Net achieves Pareto optimality w.r.t the given objectives for various devices.
Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of
neural machine translation (NMT) system. However, due to its instability, successfully RL training is challenging,
especially in real-world systems where deep models and large datasets are leveraged. In this paper, taking several large-scale translation tasks as testbeds, we
conduct a systematic study on how to train better NMT models using reinforcement learning.
We provide a comprehensive comparison of several important factors (eg. baseline reward, reward shaping) in RL training. Furthermore, to fill in the gap that it
remains unclear whether RL is still beneficial when monolingual data is used, we propose a new method to leverage RL to further boost the performance of
NMT systems trained with source/target monolingual data.
By integrating all our findings, we obtain competitive results on WMT14 English-German, WMT17 English-Chinese, and WMT17 Chinese-English translation tasks, especially setting a
state-of-the-art performance on WMT17 Chinese-English translation task.
Abstractive text summarization aims to shorten long text documents into a human readable form that contains the most important facts from the original document.
However, the level of actual abstraction as measured by novel phrases that do not appear in the source document remains low in existing approaches. We propose two
techniques to improve the level of abstraction of generated summaries. First, we decompose the decoder into a contextual network that retrieves relevant parts of
the source document, and a pretrained language model that incorporates prior knowledge about language generation. Second, we propose a novelty metric that is
optimized directly through policy learning to encourage the generation of novel phrases. Our model achieves results comparable to state-of-the-art models, as
determined by ROUGE scores and human evaluations, while achieving a significantly higher level of abstraction
as measured by n-gram overlap with the source document.
Exhaustive enumeration of all possible join orders is often avoided, and most optimizers leverage heuristics to prune the search space. The design and
implementation of heuristics are well-understood when the cost model is roughly linear, and we find that these heuristics can be significantly suboptimal when
there are non-linearities in cost. Ideally, instead of a fixed heuristic, we would want a strategy to guide the search space in a more data-driven way—tailoring
the search to a specific dataset and query workload. Recognizing the link between classical Dynamic Programming enumeration methods and recent results in
Reinforcement Learning (RL), we propose a new method for learning optimized join search strategies. We present our RL-based
DQ optimizer, which currently optimizes select-project-join blocks. We implement three versions of DQ to illustrate the ease of integration into existing
DBMSes: (1) A version built on top of Apache Calcite,(2) a version integrated into PostgreSQL, and (3) a version integratedinto SparkSQL. Our extensive evaluation shows that DQ
achieves plans with optimization costs and query execution times competitive with the native query optimizer in each system, but can execute significantly
faster after learning (often by orders of magnitude).
“Backprop Evolution”, Maximilian
Alber, Irwan Bello, Barret Zoph, Pieter-Jan Kindermans, Prajit Ramachandran, Quoc Le (2018-08-08; backlinks; similar):
The back-propagation algorithm is the cornerstone of deep learning. Despite its importance, few variations of the algorithm have been attempted. This work
presents an approach to discover new variations of the back-propagation equation. We use a domain specific language to describe update equations as a list of
primitive functions. An evolution-based method is used to discover new propagation rules that maximize the generalization performance after a few epochs of
training. We find several update equations that can train faster with short training times than standard back-propagation, and perform similar as standard
back-propagation at convergence.
Designing convolutional neural networks (CNN) for mobile devices is challenging because mobile models need to
be small and fast, yet still accurate. Although significant efforts have been dedicated to design and improve mobile CNNs
on all dimensions, it is very difficult to manually balance these trade-offs when there are so many architectural possibilities to consider. In this paper,
we propose an automated mobile neural architecture search (MNAS) approach, which explicitly incorporate model
latency into the main objective so that the search can identify a model that achieves a good trade-off between accuracy and latency. Unlike previous work, where
latency is considered via another, often inaccurate proxy (eg. FLOPS), our approach directly measures real-world
inference latency by executing the model on mobile phones. To further strike the right balance between flexibility and search space size, we propose a novel
factorized hierarchical search space that encourages layer diversity throughout the network. Experimental results show that our approach consistently outperforms
state-of-the-art mobile CNN models across multiple vision tasks. On the ImageNet classification task, our MnasNet achieves 75.2% top-1 accuracy with 78ms latency on a Pixel phone, which is 1.8× faster than
MobileNetV2 [29] with 0.5% higher accuracy and 2.3× faster than NASNet [36] with 1.2% higher accuracy. Our MnasNet also
achievesbetter mAP quality than MobileNets for COCOobject
detection. Code is at https://github.com/tensorflow/tpu/tree/master/models/official/mnasnet
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing
step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we
demonstrate that the common practice of using very few epochs during the main NAS and much larger numbers of epochs
during a post-processing step is inefficient due to little correlation in the relative rankings for these two training regimes. To combat both of these
problems, we propose to use a recent combination of Bayesian optimization and Hyperband for efficient joint neural architecture and hyperparameter search.
A generally intelligent learner should generalize to more complex tasks than it has previously encountered, but the two common paradigms in machine
learning—either training a separate learner per task or training a single learner for all tasks—both have difficulty with such generalization because they do not
leverage the compositional structure of the task distribution. This paper introduces the compositional problem graph as a broadly applicable formalism to relate
tasks of different complexity in terms of problems with shared subproblems. We propose the compositional generalization problem for measuring how readily old
knowledge can be reused and hence built upon. As a first step for tackling compositional generalization, we introduce the compositional recursive learner, a
domain-general framework for learning algorithmic procedures for composing representation transformations, producing a learner that reasons about what computation
to execute by making analogies to previously seen problems. We show on a symbolic and a high-dimensional domain that our compositional approach can generalize to
more complex problems than the learner has previously encountered, whereas baselines that are not explicitly compositional do not.
Early results in using convolutional neural networks (CNNs) on x-rays to diagnose disease have been promising,
but it has not yet been shown that models trained on x-rays from one hospital or one group of hospitals will work equally well at different hospitals. Before these
tools are used for computer-aided diagnosis in real-world clinical settings, we must verify their ability to generalize across a variety of hospital systems. A
cross-sectional design was used to train and evaluate pneumonia screening CNNs on 158,323 chest x-rays from NIH (n = 112,120 from 30,805 patients), Mount Sinai (42,396 from 12,904 patients), and Indiana (n = 3,807 from
3,683 patients). In 3 / 5 natural comparisons, performance on chest x-rays from outside hospitals was significantly lower than on held-out x-rays from the
original hospital systems. CNNs were able to detect where an x-ray was acquired (hospital system, hospital
department) with extremely high accuracy and calibrate predictions accordingly. The performance of CNNs in diagnosing
diseases on x-rays may reflect not only their ability to identify disease-specific imaging findings on x-rays, but also their ability to exploit confounding information. Estimates of CNN performance based on test data from hospital systems used for model training may overstate their likely real-world
performance.
We present a new dataset of image caption annotations, Conceptual Captions, which contains an order of magnitude more images [3.3m
training] than the MS-COCO dataset (Lin et al 2014) and represents a wider variety of both images and image caption
styles. We achieve this by extracting and filtering image caption annotations from billions of webpages.
We also present quantitative evaluations of a number of image captioning models and show that a model architecture based on Inception-ResNetv2 (Szegedy et al 2016) for image-feature extraction and
Transformer (Vaswani et al 2017) for sequence modeling achieves the best performance when trained on the Conceptual Captions
dataset.
We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization
layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. SN
switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network
architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented
(e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization
that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as
ImageNet, COCO, CityScapes, ADE20K, and
Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of
SN has been made available in https://github.com/switchablenorms/.
Neural Architecture Search (NAS) is a laborious process. Prior work onautomated NAS targets mainly on improving accuracy, but lacks consideration of computational resource use. We propose the
Resource-Efficient Neural Architect (RENA), an efficientresource-constrained NAS usingreinforcement learning with network embedding. RENA uses a policy network to process the networkembeddings to generate new configurations. We demonstrate RENA onimage recognition and keyword spotting (KWS) problems. RENA can find novel architectures that achieve high performance even with tight resource constraints. For CIFAR10, it achieves 2.95% test error whencompute intensity is greater than 100 FLOPs/byte, and 3.87% test error when model size is less than 3M parameters. For Google Speech Commands Dataset,
RENA achieves the state-of-the-art accuracy without resource constraints, and it outperforms the optimized architectures
with tight resource constraints.
Active learning (AL) aims to enable training high performance classifiers with low annotation cost by predicting which subset of unlabeled instances would be
most beneficial to label. The importance of AL has motivated extensive research, proposing a wide variety of manually designed AL algorithms with diverse
theoretical and intuitive motivations. In contrast to this body of research, we propose to treat active learning algorithm design as a meta-learning problem and
learn the best criterion from data. We model an active learning algorithm as a deep neural network that inputs the base learner state and the unlabeled point set
and predicts the best point to annotate next. Training this active query policy network with reinforcement learning,
produces the best non-myopic policy for a given dataset. The key challenge in achieving a general solution to AL then becomes that of learner generalisation,
particularly across heterogeneous datasets. We propose a multi-task dataset-embedding approach that allows dataset-agnostic active learners to be trained. Our
evaluation shows that AL algorithms trained in this way can directly generalize across diverse problems.
Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the
best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of
overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images.
Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range
of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is
likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even
minute natural variations in the data distribution.
Neural Machine Translation (NMT) systems rely on large amounts of parallel data. This is a major challenge for
low-resource languages. Building on recent work on unsupervised and semi-supervised methods, we present an approach that combines zero-shot and dual learning. The
latter relies on reinforcement learning, to exploit the duality of the machine translation task, and requires only
monolingual data for the target language pair. Experiments show that a zero-shot dual system, trained on English-French and English-Spanish, outperforms by large
margins a standard NMT system in zero-shot translation performance on Spanish-French (both directions). The
zero-shot dual method approaches the performance, within 2.2 BLEU points, of a comparable supervised setting.
Our method can obtain improvements also on the setting where a small amount of parallel data for the zero-shot language pair is available. Adding Russian, to
extend our experiments to jointly modeling 6 zero-shot translation directions, all directions improve between 4 and 15 BLEU points, again, reaching performance near that of the supervised setting.
Data augmentation is an effective technique for improving the accuracy of modern image classifiers. However, current data augmentation implementations are
manually designed. In this paper, we describe a simple procedure called AutoAugment to automatically search for improved data
augmentation policies. In our implementation, we have designed a search space where a policy consists of many sub-policies, one of which is randomly chosen
for each image in each mini-batch. A sub-policy consists of two operations, each operation being an image processing function such as translation, rotation, or
shearing, and the probabilities and magnitudes with which the functions are applied. We use a search algorithm to find the best policy such that the neural network
yields the highest validation accuracy on a target dataset. Our method achieves state-of-the-art accuracy on CIFAR-10,
CIFAR-100,SVHN, andImageNet
(without additional data). On ImageNet, we attain a Top-1 accuracy of 83.5% which is 0.4% better than the previous record
of 83.1%. On CIFAR-10, we achieve an error rate of 1.5%, which is 0.6% better than the previous state-of-the-art.
Augmentation policies we find are transferable between datasets. The policy learned on ImageNet transfers well to achieve significant improvements on other
datasets, such as Oxford Flowers, Caltech-101, Oxford-IIT Pets, FGVC Aircraft,
and Stanford Cars.
Transfer learning is a cornerstone of computer vision, yet little work has been done to evaluate the relationship between architecture and transfer. An implicit
hypothesis in modern computer vision research is that models that perform better on ImageNet necessarily perform better on
other vision tasks. However, this hypothesis has never been systematically tested.
Here, we compare the performance of 16 classification networks on 12 image classification datasets. We find that, when networks are used as fixed feature
extractors or fine-tuned, there is a strong correlation between ImageNet accuracy and transfer accuracy (r = 0.99
and 0.96, respectively).
In the former setting, we find that this relationship is very sensitive to the way in which networks are trained on ImageNet; many common forms of regularization slightly improve ImageNet accuracy but yield
penultimate layer features that are much worse for transfer learning. Additionally, we find that, on two small fine-grained image classification datasets,
pretraining on ImageNet provides minimal benefits, indicating the learned features from ImageNet do not transfer well to fine-grained tasks.
Together, our results show that ImageNet architectures generalize well across datasets, but ImageNet features are less general than previously suggested.
We introduce a learning-based framework to optimize tensor programs for deep learning workloads. Efficient implementations of tensor operators, such as matrix
multiplication and high dimensional convolution, are key enablers of effective deep learning systems. However, existing systems rely on manually optimized
libraries such as cuDNN where only a narrow range of server class GPUs are
well-supported. The reliance on hardware-specific operator libraries limits the applicability of high-level graph optimizations and incurs significant engineering
costs when deploying to new hardware targets. We use learning to remove this engineering burden. We learn domain-specific statistical cost models to guide the
search of tensor operator implementations over billions of possible program variants. We further accelerate the search by effective model transfer across
workloads. Experimental results show that our framework delivers performance competitive with state-of-the-art hand-tuned libraries for low-power
CPU, mobileGPU, and server-class GPU.
Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent
approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1)the
neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption, (2) most architecture search
methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an
evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto-front of architectures under multiple objectives, such
as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance
mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of
their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions
allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.
Existing applications include a huge amount of knowledge that is out of reach for deep neural networks. This paper presents a novel approach for integrating
calls to existing applications into deep learning architectures. Using this approach, we estimate each application’s functionality with an estimator, which is
implemented as a deep neural network (DNN). The estimator is then embedded into a base network that we direct into
complying with the application’s interface during an end-to-end optimization process. At inference time, we replace each estimator with its existing application
counterpart and let the base network solve the task by interacting with the existing application. Using this ‘Estimate and Replace’ method, we were able to
train a DNN end-to-end with less data andoutperformed a matching DNN that did
not interact with the external application.
In web search, typically a candidate generation step selects a small set of documents—from collections containing as many as billions of web pages—that are
subsequently ranked and pruned before being presented to the user. In Bing, the candidate generation involves scanning the index using statically designed match
plans that prescribe sequences of different match criteria and stopping conditions. In this work, we pose match planning as a reinforcement learning task and
observe up to 20% reduction in index blocks accessed, with small or no degradation in the quality of the candidate sets.
The explosion in workload complexity and the recent slow-down in Moore’s law scaling call for new approaches towards efficient computing. Researchers are now
beginning to use recent advances in machine learning in software optimizations, augmenting or replacing traditional heuristics and data structures. However, the
space of machine learning for computer hardware architecture is only lightly explored. In this paper, we demonstrate the potential of deep learning to address the
von Neumann bottleneck of memory performance. We focus
on the critical problem of learning memory access patterns, with the goal of constructing accurate and efficient memory prefetchers. We relate contemporary
prefetching strategies to n-gram models in natural language processing, and show how recurrent neural networks can serve as
a drop-in replacement. On a suite of challenging benchmark datasets, we find that neural networks consistently demonstrate superior performance in terms of
precision and recall. This work represents the first step towards practical neural-network based prefetching, and opens a wide range of exciting directions for
machine learning in computer architecture research.
Evolution Strategies (ES) have recently been demonstrated to be a viable
alternative to reinforcement learning (RL) algorithms on a set of challenging deep RL problems, including Atari games and MuJoCo humanoid locomotion benchmarks.
While the ES algorithms in that work belonged to the specialized class of natural evolution strategies (which resemble approximate gradient RL algorithms, such as
REINFORCE), we demonstrate that even a very basic canonical ES algorithm can achieve the same or even better performance. This success
of a basic ES algorithm suggests that the state-of-the-art can be advanced further by integrating the many advances made in the field of ES in the last
decades.
We also demonstrate qualitatively that ES algorithms have very different performance characteristics than traditional RL algorithms: on some games, they learn
to exploit the environment and perform much better while on others they can get stuck in suboptimal local minima. Combining their strengths with those of
traditional RL algorithms is therefore likely to lead to new advances in the state of the art.
Adversarial error has similar power-law form for all
datasets and models studied, and architecture matters.
It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples. In attempting to explain the origin of
adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too
linear. Here we show that distributions of logit differences have a universal functional form. This functional form is independent of architecture, dataset, and
training protocol; nor does it change during training. This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation. We show that this universality holds for a broad range of datasets
(MNIST, CIFAR10, ImageNet, and random data), models
(including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks
(FGSM, step l.l.,PGD). Motivated by these results, we study the effects of
reducing prediction entropy on adversarial robustness. Finally, we study the effect of network architectures on
adversarial sensitivity. To do this, we use neural architecture search with reinforcement learning to find
adversarially robust architectures on CIFAR10. Our resulting architecture is more robust to white and black box
attacks compared to previous attempts.
“Image Transformer”, Niki
Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran (2018-02-15; backlinks; similar):
Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an
effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a
sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we
significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical
convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on
ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also present results on image super-resolution with a large
magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we find that images generated by our
super-resolution model fool human observers three times more often than the previous state of the art.
Deep learning models with convolutional and recurrent networks are now ubiquitous and analyze massive amounts of audio, image, video, text and graph data, with
applications in automatic translation, speech-to-text, scene understanding, ranking user preferences, ad placement, etc. Competing frameworks for building these
networks such as TensorFlow, Chainer, CNTK, Torch/PyTorch, Caffe1/2, MXNet
and Theano, explore different tradeoffs between usability and expressiveness, research or production orientation and supported hardware. They operate
on a DAG of computational operators, wrappinghigh-performance libraries such as CUDNN (for NVIDIAGPUs) or NNPACK(for various CPUs), and automate memory allocation, synchronization,
distribution. Custom operators are needed where the computation does not fit existing high-performance library calls, usually at a high engineering cost. This is
frequently required when new operators are invented by researchers: such operators suffer a severe performance penalty, which limits the pace of innovation.
Furthermore, even if there is an existing runtime call these frameworks can use, it often doesn’t offer optimal performance for a user’s particular network
architecture and dataset, missing optimizations between operators as well as optimizations that can be done knowing the size and shape of data. Our contributions
include (1) a language close to the mathematics of deep learning called Tensor Comprehensions, (2) a polyhedral Just-In-Time compiler to convert a mathematical
description of a deep learning DAG into a CUDA kernel with delegated memory
management and synchronization, also providing optimizations such as operator fusion and specialization for specific sizes, (3) a compilation cache populated by an
autotuner. [Abstract cutoff]
The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically. Although
evolutionary algorithms have been repeatedly applied to neural network topologies, the image classifiers thus discovered have remained inferior to human-crafted
ones. Here, we evolve an image classifier—AmoebaNet-A—that surpasses hand-designs for the first time. To do this, we modify the tournament selection evolutionary
algorithm by introducing an age property to favor the younger genotypes. Matching size, AmoebaNet-A has comparable accuracy to current state-of-the-art
ImageNet models discovered with more complex architecture-search methods. Scaled to larger size, AmoebaNet-A sets a new
state-of-the-art 83.9% / 96.6% top-5 ImageNet accuracy. In a controlled comparison against a well known reinforcement learning algorithm, we give evidence that
evolution can obtain results faster with the same hardware, especially at the earlier stages of the search. This is relevant when fewer compute resources are
available. Evolution is, thus, a simple method to effectively discover high-quality architectures.
We show that generating English Wikipedia articles can be approached as a multi-document summarization of source documents. We use extractive summarization to
coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture
that can scalably attend to very long sequences, much longer than typical encoder-decoder architectures used in sequence transduction. We show that this model can
generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual
information as reflected in perplexity, ROUGE scores and human evaluations.
Learning to optimize—the idea that we can learn from data algorithms that optimize a numerical criterion—has recently been at the heart of a growing number of
research efforts. One of the most challenging issues within this approach is to learn a policy that is able to optimize over classes of functions that are fairly
different from the ones that it was trained on. We propose a novel way of framing learning to optimize as a problem of learning a good navigation policy on a
partially observable loss surface. To this end, we develop Rover Descent, a solution that allows us to learn a fairly broad optimization policy from training on a
small set of prototypical two-dimensional surfaces that encompasses the classically hard cases such as valleys, plateaus, cliffs and saddles and by using strictly
zero-order information. We show that, without having access to gradient or curvature information, we achieve state-of-the-art convergence speed on optimization
problems not presented at training time such as the Rosenbrock function and other hard cases in two dimensions. We extend our framework to optimize over high
dimensional landscapes, while still handling only two-dimensional local landscape information and show good preliminary results.
Loihi is a 60-mm2 chip fabricated in
Intel’s 14-nm process that advances the state-of-the-art modeling of spiking neural networks in silicon. It integrates a wide range of novel features for the field, such as hierarchical connectivity,
dendritic compartments, synaptic delays, and, most importantly, programmable synaptic learning rules. Running a spiking convolutional form of the Locally
Competitive Algorithm, Loihi can solve LASSO optimization problems with over 3 orders of magnitude superior energy-delay-product compared to conventional solvers
running on a CPU iso-process/voltage/area. This provides an unambiguous example of spike-based computation,
outperforming all known conventional solutions.
…Spiking Neural Networks: We consider an SNN a model of computation with neurons as the basic
processing elements. Different from ANNs, SNNs incorporate time as an explicit
dependency in their computations. At some instant in time, one or more neurons might send out single-bit impulses, the spike, to neighbors through directed
connections known as synapses, with a potentially non-zero traveling time. Neurons have local state variables with rules governing their evolution and timing of
spike generation. Hence, the network is a dynamical system where individual neurons interact through spikes
…Chip Overview: Loihi features a many-core mesh comprising 128 neuromorphic cores, 3 embedded x86 processor cores, and off-chip communication
interfaces that hierarchically extend the mesh in 4 planar directions to other chips. An asynchronous network-on-chip (NoC) transports all communication between
cores in the form of packetized messages. The NoC supports write, read request, and read response messages for core management and x86-to-x86 messaging, spike
messages for SNN computation, and barrier messages for time synchronization between cores. All message types may
be sourced externally by a host CPU or on-chip by the x86 cores, and these may be directed to any on-chip core.
Messages may be hierarchically encapsulated for off-chip communication over a second-level network. The mesh protocol supports scaling to 4096 on-chip cores and,
through hierarchical addressing, up to 16,384 chips.
Each neuromorphic core implements 1,024 primitive spiking neural units (compartments) grouped into sets of trees constituting neurons. The compartments, along
with their fan-in and fan-out connectivity, share configuration and state variables in 10 architectural memories. Their state variables are updated in a
time-multiplexed, pipelined manner every algorithmic time-step. When a neuron’s activation exceeds some threshold level, it generates a spike message that is
routed to a set of fan-out compartments contained in some number of destination cores.
We propose to learn a curriculum or a syllabus for supervised learning and deep reinforcement learning with deep neural
networks by an attachable deep neural network, called ScreenerNet. Specifically, we learn a weight for each sample by jointly training the ScreenerNet and the main
network in an end-to-end self-paced fashion. The ScreenerNet neither has sampling bias nor requires to remember the past learning history. We show the networks
augmented with the ScreenerNet achieve early convergence with better accuracy than the state-of-the-art curricular learning methods in extensive experiments using
three popular vision datasets such as MNIST, CIFAR10 and Pascal
VOC2012, and a Cart-pole task using Deep Q-learning. Moreover, the ScreenerNet can extend other curriculum learning methods such as Prioritized
Experience Replay (PER) for further accuracy improvement.
Traditionally, medical discoveries are made by observing associations, making hypotheses from them and then designing and running experiments to test the
hypotheses. However, with medical images, observing and quantifying associations can often be difficult because of the wide variety of features, patterns, colours,
values and shapes that are present in real data. Here, we show that deep learning can extract new knowledge from retinal fundus images. Using deep-learning models
trained on data from 284,335 patients and validated on 2 independent datasets of 12,026 and 999 patients, we predicted cardiovascular risk factors not previously
thought to be present or quantifiable in retinal images, such as age (mean absolute error within 3.26 years), gender (area under the receiver operating
characteristic curve (AUC) =0.97), smoking status (AUC = 0.71), systolic
blood pressure (mean absolute error within 11.23 mmHg) and major adverse cardiac events (AUC = 0.70). We also show
that the trained deep-learning models used anatomical features, such as the optic disc or blood vessels, to generate each prediction. [Sex detection
replicated in Korot et al 2021.]
The design of neural network architectures for a new data set is a laborious task which requires human deep learning expertise. In order to make deep learning
available for a broader audience, automated methods for finding a neural network architecture are vital. Recently proposed methods can already achieve human expert
level performances. However, these methods have run times of months or even years of GPU computing time, ignoring
hardware constraints as faced by many researchers and companies. We propose the use of Monte Carlo planning in combination with two different UCT (upper confidence bound applied to trees) derivations to search for network architectures. We adapt the UCT algorithm to the needs of network architecture search by proposing two ways of sharing information between different branches of
the search tree. In an empirical study we are able to demonstrate that this method is able to find competitive networks for MNIST, SVHNand CIFAR-10 in just a
singleGPU day. Extending the search time to five GPU days, we are able to outperform human architectures and our competitors which consider the same types of layers.
The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these
requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in
both flexibility and components. We propose a domain-specific language (DSL) for use in automated architecture search
which can produce novelRNNs of arbitrary depth and width. The DSL is flexible
enough to define standard architectures such as the Gated Recurrent Unit and Long Short Term Memory and allows the introduction of non-standard
RNN components such as trigonometric curves and layer normalization. Using two different candidate generation
techniques, random search with a ranking function and reinforcement learning, we explore the novel architectures produced by
the RNNDSL for language modeling and machine translation
domains. The resulting architectures do not follow human intuition yet perform well on their targeted tasks, suggesting the space of usable RNN architectures is far larger than previously assumed.
Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key
to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not. In this exploratory research paper,
we start from this premise and posit that all existing index structures can be replaced with other types of models, including deep-learning models, which we term
learned indexes. The key idea is that a model can learn the sort order or structure of lookup keys and use this signal to effectively predict the position or
existence of records. We theoretically analyze under which conditions learned indexes outperform traditional index structures and describe the main challenges in
designing learned index structures. Our initial results show, that by using neural nets we are able to outperform cache-optimized B-Trees by up to 70% in speed
while saving an order-of-magnitude in memory over several real-world data sets. More importantly though, we believe that the idea of replacing core components of a
data management system through learned models has far reaching implications for future systems designs and that this work just provides a glimpse of what might be
possible.
During President Trump’s visit to Beijing, he appeared on screen for a special address at a tech conference. First he spoke in English. Then he switched to
Mandarin Chinese. Mr. Trump doesn’t speak Chinese. The video was a publicity stunt, designed to show off the voice capabilities of iFlytek, a Chinese artificial intelligence company with both innovative
technology and troubling ties to Chinese state security. IFlyTek has said its technology can monitor a car full of people or a crowded room, identify a targeted
individual’s voice and record everything that person says.
“IFlyTek”, the image of Mr. Trump said in Chinese, “is really fantastic.”
As China tests the frontiers of artificial intelligence, iFlyTek serves as a compelling example of both the country’s sci-fi ambitions and the technology’s
darker dystopian possibilities.
The Chinese company uses sophisticated A.I. to power image and voice recognition systems that can help doctors with their diagnoses, aid teachers in grading
tests and let drivers control their cars with their voices. Even some global companies are impressed: Delphi, a major American auto supplier, offers iFlyTek’s technology to carmakers in China, while Volkswagen plans to build the Chinese company’s speech recognition technology
into many of its cars in China next year. At the same time, iFlyTek hosts a laboratory to develop voice surveillance capabilities for China’s domestic security
forces. In an October report, a human rights group said the company was helping the authorities compile a biometric
voice database of Chinese citizens that could be used to track activists and others…“The Chinese government has been collecting the voice patterns of tens of
thousands of people with little transparency about the program or laws regulating who can be targeted or how that information is going to be used”, Sophie
Richardson, Human Rights Watch’s China director, wrote in a report in October. In its home province of Anhui, iFlyTek has assembled a database of 70,000 voice
patterns, according to the report, which also said that the police had begun collecting records of voice patterns as they would fingerprints. The report cited as
one example 3 women suspected of being sex workers whose voices were registered in a database, it said, in part because they had been arrested in Anhui. The local
Chinese media has also reported about a new plan in Anhui to scan voice calls automatically for the voice-prints of wanted criminals, and alert the police if they
are detected. IFlyTek did not respond to requests for comment on the Human Rights Watch report but has said its data-gathering efforts will not stop, particularly
as it participates in China’s push to develop self-driving cars. “We are always talking about big data—the vehicle produces many images every day”, said
Mr. Liu, the iFlyTek automotive executive.
Deep CNNs are known to exhibit the following peculiarity: on the one hand they generalize extremely well to a
test set, while on the other hand they are extremely sensitive to so-called adversarial perturbations. The extreme sensitivity of high performance
CNNs to adversarial examples casts serious doubt that these networks are learning high level abstractions in the
dataset. We are concerned with the following question: How can a deep CNN that does not learn any high level
semantics of the dataset manage to generalize so well? The goal of this article is to measure the tendency of CNNs to
learn surface statistical regularities of the dataset. To this end, we use Fourier filtering to construct datasets which share the exact same high level
abstractions but exhibit qualitatively different surface statistical regularities. For the SVHN and CIFAR-10 datasets, we present two Fourier filtered variants: a low frequency variant and a randomly filtered variant. Each of the
Fourier filtering schemes is tuned to preserve the recognizability of the objects. Our main finding is that CNNs exhibit
a tendency to latch onto the Fourier image statistics of the training dataset, sometimes exhibiting up to a 28% generalization gap across the various test
sets. Moreover, we observe that significantly increasing the depth of a network has a very marginal impact on closing the aforementioned generalization gap. Thus
we provide quantitative evidence supporting the hypothesis that deep CNNs tend to learn surface statistical regularities
in the dataset rather than higher-level abstract concepts.
Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world
applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce
total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer
dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual
reward of utilizing a minimal number of blocks while preserving recognition accuracy. We conduct extensive experiments on CIFAR andImageNet. The results provide strong quantitative and qualitative evidence that
these learned policies not only accelerate inference but also encode meaningful visual information. Built upon a ResNet-101
model, our method achieves a speedup of 20% on average, going as high as 36% for some images, while maintaining the same 76.4% top-1 accuracy on ImageNet.
We study a classification problem where each feature can be acquired for a cost and the goal is to optimize a trade-off between the expected classification
error and the feature cost. We revisit a former approach that has framed the problem as a sequential decision-making problem and solved it by Q-learning with a linear approximation, where individual actions are either requests for feature values or terminate the episode by
providing a classification decision. On a set of eight problems, we demonstrate that by replacing the linear approximation with neural networks the approach
becomes comparable to the state-of-the-art algorithms developed specifically for this problem. The approach is flexible, as it can be improved with any new
reinforcement learning enhancement, it allows inclusion of pre-trained high-performance classifier, and unlike prior art,
its performance is robust across all evaluated datasets.
The efficient use of limited computational resources is an essential ingredient of intelligence. Selecting computations optimally according to rational
metareasoning would achieve this, but this is computationally intractable. Inspired by psychology and neuroscience, we propose the first concrete and
domain-general learning algorithm for approximating the optimal selection of computations: Bayesian metalevel policy search (BMPS). We derive this general, sample-efficient search algorithm for a computation-selecting metalevel policy based on the insight
that the value of information lies between the myopic value of information and the value of perfect information. We evaluate BMPS
on three increasingly difficult metareasoning problems: when to terminate computation, how to allocate computation between competing options, and planning.
Across all three domains, BMPS achieved near-optimal performance and compared favorably to previously proposed
metareasoning heuristics. Finally, we demonstrate the practical utility of BMPS in an emergency management
scenario, even accounting for the overhead of metareasoning.
Supervised approaches for text summarisation suffer from the problem of mismatch between the target labels/scores of individual sentences and the evaluation
score of the final summary. Reinforcement learning can solve this problem by providing a learning mechanism that uses the score of the final summary as a guide to
determine the decisions made at the time of selection of each sentence. In this paper we present a proof-of-concept approach that applies a policy-gradient
algorithm to learn a stochastic policy using an undiscounted reward. The method has been applied to a policy consisting of a simple neural network and simple
features. The resulting deep reinforcement learning system is able to learn a global policy and obtain encouraging
results.
Speech recognition systems have achieved high recognition performance for several tasks. However, the performance of such systems is dependent on the
tremendously costly development work of preparing vast amounts of task-matched transcribed speech data for supervised training. The key problem here is the cost of
transcribing speech data. The cost is repeatedly required to support new languages and new tasks. Assuming broad network services for transcribing speech data for
many users, a system would become more self-sufficient and more useful if it possessed the ability to learn from very light feedback from the users without
annoying them. In this paper, we propose a general reinforcement learning framework for speech recognition systems based on
the policy gradient method. As a particular instance of the framework, we also propose a hypothesis selection-based reinforcement learning method. The proposed
framework provides a new view for several existing training and adaptation methods. The experimental results show that the proposed method improves the recognition
performance compared to unsupervised adaptation.
Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network
(RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant
input tokens. Skim-RNN gives computational advantage over an RNN that always updates the entire hidden state. Skim-RNN uses the
same input and output interfaces as a standard RNN and can be easily used instead of RNNs in existing models. In our experiments, we show that Skim-RNN can achieve
significantly reduced computational cost without losing accuracy compared to standard RNNs across five different
natural language tasks. In addition, we demonstrate that the trade-off between accuracy and speed of Skim-RNN can be dynamically controlled during inference time in a stable manner. Our analysis also shows that
Skim-RNNrunning on a single CPU offers lower latency compared
tostandard RNNs on GPUs.
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel
corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this
work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a
model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both
languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used
datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and
WMT English-French datasets, without using even a single parallel sentence at training time.
A long-standing goal of artificial intelligence is an algorithm that learns, tabula rasa, superhuman proficiency in challenging domains. Recently, AlphaGo
became the first program to defeat a world champion in the game of Go. The tree search in AlphaGo evaluated positions and selected moves using deep neural
networks. These neural networks were trained by supervised learning from human expert moves, and by reinforcement learning from self-play. Here we introduce an
algorithm based solely on reinforcement learning, without human data, guidance or domain knowledge beyond game rules.
AlphaGo becomes its own teacher: a neural network is trained to predict AlphaGo’s own move selections and also the winner of AlphaGo’s games. This neural network
improves the strength of the tree search, resulting in higher quality move selection and stronger self-play in the next iteration. Starting tabula rasa, our new
program AlphaGo Zero achieved superhuman performance, winning 100–0 against the previously published, champion-defeating AlphaGo.
We develop a method for policy architecture search and adaptation via gradient-free optimization which can learn to perform autonomous driving tasks. By
learning from both demonstration and environmental reward we develop a model that can learn with relatively few early catastrophic failures. We first learn an
architecture of appropriate complexity to perceive aspects of world state relevant to the expert demonstration, and then mitigate the effect of domain-shift during
deployment by adapting a policy demonstrated in a source domain to rewards obtained in a target environment. We show that our approach allows safer learning than
baseline methods, offering a reduced cumulative crash metric over the agent’s lifetime as it learns to drive in a realistic simulated environment.
The choice of activation functions in deep networks has a statistically-significant effect on the training dynamics and task performance. Currently, the most
successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed
alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains.
In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by
conducting an empirical evaluation with the best discovered activation function.
Our experiments show that the best discovered activation function, f(x) = x × sigmoid(β x), which we name Swish,
tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs
with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2.
The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in
any neural network.
Feature representations from pre-trained deep neural networks have been known to exhibit excellent generalization and utility across a variety of related tasks.
Fine-tuning is by far the simplest and most widely used approach that seeks to exploit and adapt these feature representations to novel tasks with limited data.
Despite the effectiveness of fine-tuning, itis often sub-optimal and requires very careful optimization to prevent severe over-fitting to small datasets. The
problem of sub-optimality and over-fitting, is due in part to the large number of parameters used in a typical deep convolutional neural network. To address these
problems, we propose a simple yet effective regularization method for fine-tuning pre-trained deep networks for the task of k-shot learning. To prevent
overfitting, our key strategy is to cluster the model parameters while ensuring intra-cluster similarity and inter-cluster diversity of the parameters, effectively
regularizing the dimensionality of the parameter search space. In particular, we identify groups of neurons within each layer of a deep network that shares similar
activation patterns. When the network is to be fine-tuned for a classification task using only k examples, we propagate a single gradient to all of the neuron
parameters that belong to the same group. The grouping of neurons is non-trivial as neuron activations depend on the distribution of the input data. To efficiently
search for optimal groupings conditioned on the input data, we propose a reinforcement learning search strategy using
recurrent networks to learn the optimal group assignments for each network layer. Experimental results show that our method can be easily applied to several
popular convolutional neural networks and improve upon other state-of-the-art fine-tuning based k-shot learning strategies by more than10%
The promise of learning to learn for robotics rests on the hope that by extracting some information about the learning process itself we can speed up subsequent
similar learning tasks. Here, we introduce a computationally efficient online meta-learning algorithm that builds and optimizes a memory model of the optimal
learning rate landscape from previously observed gradient behaviors. While performing task specific optimization, this memory of learning rates predicts how to
scale currently observed gradients. After applying the gradient scaling our meta-learner updates its internal memory based on the observed effect its prediction
had. Our meta-learner can be combined with any gradient-based optimizer, learns on the fly and can be transferred to new optimization tasks. In our evaluations we
show that our meta-learning algorithm speeds up learning of MNIST classification and a variety of learning
control tasks, either in batch or online learning settings.
We consider the problem of active feature acquisition, where we sequentially select the subset of features in order to achieve the maximum prediction
performance in the most cost-effective way. In this work, we formulate this active feature acquisition problem as a reinforcement learning problem, and provide a
novel framework for jointly learning both the RL agent and the classifier (environment). We also introduce a more systematic way of encoding subsets of features
that can properly handle innate challenge with missing entries in active feature acquisition problems, that uses the orderless LSTM-based set encoding mechanism that readily fits in the joint learning framework. We evaluate our model on a carefully
designed synthetic dataset for the active feature acquisition as well as several real datasets such as electric health record (EHR) datasets, on
which it outperforms all baselines in terms of prediction performance as well feature acquisition cost.
Automatically learned quality assessment for images has recently become a hot topic due to its usefulness in a wide variety of applications such as evaluating
image capture pipelines, storage techniques and sharing media. Despite the subjective nature of this problem, most existing methods only predict the mean opinion
score provided by datasets such as AVA [1] and TID2013 [2].
Our approach differs from others in that we predict the distribution of human opinion scores using a convolutional neural network. Our architecture also has the
advantage of being substantially simpler than other methods with comparable performance. Our proposed approach relies on the success (and retraining) of proven,
state-of-the-art deep object recognition networks.
Our resulting network can be used to not only score images reliably and with high correlation to human perception, but also to assist with adaptation and
optimization of photo editing/enhancement algorithms in a photographic pipeline. All this is done without need for a “golden” reference image, consequently
allowing for single-image, semantic-aware and perceptually-aware, no-reference quality assessment.
Recurrent Neural Networks (RNNs) continue to show outstandingperformance in sequence modeling tasks.
However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty
in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph
resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip
state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget
constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at https://imatge-upc.github.io/skiprnn-2017-telecombcn/ .
Convolutional neural networks have gained a remarkable success in computer vision. However, most usable network architectures are hand-crafted and usually
require expertise and elaborate design. In this paper, we provide a block-wise network generation pipeline called BlockQNN which automatically builds high-performance networks using the Q-Learning paradigm with epsilon-greedy exploration strategy. The
optimal network block is constructed by the learning agent which is trained sequentially to choose component layers. We stack the block to construct the whole
auto-generated network. To accelerate the generation process, we also propose a distributed asynchronous framework and an early stop strategy. The block-wise
generation brings unique advantages: (1) it performs competitive results in comparison to the hand-crafted state-of-the-art networks on image classification,
additionally, the best network generated by BlockQNN achieves 3.54%top-1 error rate on CIFAR-10 which beats all existing auto-generate networks. (2) in the meanwhile, it offers tremendous reduction of the search space
in designing networks which only spends 3 days with 32 GPUs, and (3) moreover, it has strong generalizability that
thenetwork built on CIFAR also performs well on a larger-scale ImageNet dataset.
Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture
selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model’s architecture. By comparing the relative
validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training
run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns,
with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 andCIFAR-100,STL-10, ModelNet10, and Imagenet32×32, achieving competitive performance with similarly-sized hand-designed networks. Our code
is available at https://github.com/ajbrock/SMASH
Recent years have witnessed the great success of convolutional neural network (CNN) based models in the field of
computer vision. CNN is able to learn hierarchically abstracted features from images in an end-to-end training
manner. However, most of the existing CNN models only learn features through a feedforward structure and no feedback
information from top to bottom layers is exploited to enable the networks to refine themselves. In this paper, we propose a “Learning with Rethinking” algorithm.
By adding a feedback layer and producing the emphasis vector, the model is able to recurrently boost the performance based on previous prediction. Particularly, it
can be employed to boost any pre-trained models. This algorithm is tested on four object classification benchmark datasets: CIFAR-100, CIFAR-10,MNIST-background-image andILSVRC-2012 dataset. These results have demonstrated the advantage of training CNN models with the proposed feedback mechanism.
Visual object tracking is a fundamental and time-critical vision task. Recent years have seen many shallow tracking methods based on real-time pixel-based
correlation filters, as well as deep methods that have top performance but need a high-end GPU. In this
paper, we learn to improve the speed of deep trackers without losing accuracy. Our fundamental insight is to take an adaptive approach, where easy frames are
processed with cheap features (such as pixel values), while challenging frames are processed with invariant but expensive deep features. We formulate the adaptive
tracking problem as a decision-making process, and learn an agent to decide whether to locate objects with high confidence on an early layer, or continue
processing subsequent layers of a network. This significantly reduces the feed-forward cost for easy frames with distinct or slow-moving objects. We train the
agent offline in a reinforcement learning fashion, and further demonstrate that learning all deep layers (so as to provide good features for adaptive tracking) can
lead to near real-time average tracking speed of 23 fps on a single CPU while achieving state-of-the-art
performance. Perhaps most tellingly, our approach provides a 100× speedup for almost 50% of the time, indicating the power of an adaptive approach.
Many stochastic optimization algorithms work by estimating the gradient of the cost function on the fly by sampling datapoints uniformly at random from a
training set. However, the estimator might have a large variance, which inadvertently slows down the convergence rate of the
algorithms. One way to reduce this variance is to sample the datapoints from a carefully selected non-uniform distribution.
In this work, we propose a novel non-uniform sampling approach that uses the multi-armed bandit framework. Theoretically, we show that our algorithm asymptotically approximates the optimal
variance within a factor of 3. Empirically, we show that using this datapoint-selection technique results in a significant
reduction in the convergence time and variance of several stochastic optimization algorithms such as SGD, SVRG and SAGA. This approach for sampling datapoints is
general, and can be used in conjunction with any algorithm that uses an unbiased gradient estimation—we expect it to have broad applicability beyond the
specific examples explored in this work.
Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement
learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic
metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that
corrects phrase-matchingbased metrics (such as CIDEr) to only allow for logically-implied partial matches
and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, ourCIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT
dataset.
Convolutional neural networks (CNNs) have been successfully applied to many recognition and learning tasks
using a universal recipe; training a deep model on a very large dataset of supervised examples. However, this approach is rather restrictive in practice since
collecting a large set of labeled images is very expensive. One way to ease this problem is coming up with smart ways for choosing images to be labelled from a
very large collection (ie. active learning).
Our empirical study suggests that many of the active learning heuristics in the literature are not effective when applied to CNNs in batch setting. Inspired by these limitations, we define the problem of active learning as core-set selection, ie. choosing
set of points such that a model learned over the selected subset is competitive for the remaining data points. We further present a theoretical result
characterizing the performance of any selected subset using the geometry of the datapoints. As an active learning algorithm, we choose the subset which is expected
to yield best result according to our characterization. Our experiments show that the proposed method significantly outperforms existing approaches in image
classification experiments by a large margin.
Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model
architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building
block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the
“NASNet search space”) which enables transferability. In our experiments, we search for the best convolutional
layer (or “cell”) on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture,
named “NASNet architecture”. We also introduce a new regularization technique called ScheduledDropPath that
significantly improves generalization in the NASNet models. OnCIFAR-10
itself, NASNet achieves 2.4% error rate, which is state-of-the-art. On ImageNet,
NASNet achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet.
Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS—a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of
computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For
instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than
equivalently-sized, state-of-the-art models for mobile platforms. Finally, the learned features by NASNet used with the
Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.
AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they
haven’t yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or
otherwise causes serious damage? For model-free reinforcement learning, having a human “in the loop” and ready to intervene
is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a
supervised learner to imitate the human’s intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four
hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent’s learning (whereas an RL baseline fails due
to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the
supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not
scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a
single catastrophe.
We formulate tracking as an online decision-making process, where a tracking agent must follow an object despite ambiguous image frames and a limited
computational budget. Crucially, the agent must decide where to look in the upcoming frames, when to reinitialize because it believes the target has been lost, and
when to update its appearance model for the tracked object. Such decisions are typically made heuristically. Instead, we propose to learn an optimal
decision-making policy by formulating tracking as a partially observable decision-making process (POMDP). We learn policies with deep reinforcement learning algorithms that need supervision (a reward signal) only when the track has
gone awry. We demonstrate that sparse rewards allow us to quickly train on massive datasets, several orders of magnitude more than past work. Interestingly, by
treating the data source of Internet videos as unlimited streams, we both learn and evaluate our trackers in a single, unified computational stream.
Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches
have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from
scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture
search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller,
whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for
further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural
networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restrictedcomputational resources (5 GPUs). Our method can design
highly competitive networks that outperform existing networks using the same design scheme. On CIFAR-10, our model
without skip-connections achieves 4.23% test error rate, exceeding a vast majority of modern architectures and approaching DenseNet. Furthermore, by
applying our method to explore the DenseNet architecture space, we are able to achieve more accurate networks with fewer parameters.
We propose a neural encoder-decoder model with reinforcement learning (NRL) for grammatical error correction
(GEC). Unlike conventionalmaximum likelihood estimation (MLE), the model
directly optimizes towards an objective that considers a sentence-level, task-specific evaluation metric, avoiding the exposure bias issue in
MLE. Wedemonstrate that NRL outperforms MLE both in human and automated evaluation metrics, achieving the state-of-the-art on a fluency-oriented GEC corpus.
The past few years have witnessed a growth in size and computational requirements for training and inference with neural networks. Currently, a common approach
to address these requirements is to use a heterogeneous distributed environment with a mixture of hardware devices such as CPUs and GPUs. Importantly, the decision of placing parts of the neural models on devices is
often made by human experts based on simple heuristics and intuitions. In this paper, we propose a method which learns to optimize device placement for TensorFlow
computational graphs. Key to our method is the use of a sequence-to-sequence model to predict which subsets of operations in a TensorFlow graph should run on which
of the available devices. The execution time of the predicted placements is then used as the reward signal to optimize the parameters of the sequence-to-sequence
model. Our main result is that on Inception-V3 for ImageNet classification, and on RNNLSTM, for language modeling and neural machine translation, our
model finds non-trivial device placements that outperform hand-crafted heuristics and traditional algorithmic methods.
Understanding the simultaneously very diverse and intricately fine-grained set of possible human actions is a critical open problem in computer vision. Manually
labeling training videos is feasible for some action classes but doesn’t scale to the full long-tailed distribution of actions. A promising way to address this is
to leverage noisy data from web queries to learn new actions, using semi-supervised or “webly-supervised” approaches. However, these methods typically do not learn
domain-specific knowledge, or rely on iterative hand-tuned data labeling policies. In this work, we instead propose a reinforcement
learning-based formulation for selecting the right examples for training a classifier from noisy web search results. Our method uses Q-learning to learn a data labeling policy on a small labeled training dataset, and then uses this to automatically label noisy web data
for new visual concepts. Experiments on the challenging Sports-1M action recognition benchmark as well as on additional fine-grained and newly emerging action
classes demonstrate that our method is able to learn good labeling policies for noisy data and use this to learn accurate visual concept classifiers.
Advances in deep learning have led to substantial increases in prediction accuracy but have been accompanied by increases in the cost of rendering predictions.
We conjecture that fora majority of real-world inputs, the recent advances in deep learning have created models that effectively “overthink” on simple inputs. In
this paper, we revisit the classic question of building model cascades that primarily leverage class asymmetry to reduce cost. We introduce the “I Don’t
Know”(IDK) prediction cascades framework, a general framework to systematically compose a set of pre-trained
models to accelerate inference without a loss in prediction accuracy. We propose two search based methods for constructing cascades as well as a new cost-aware
objective within this framework. The proposed IDK cascade framework can be easily adopted in the existing model
serving systems without additional model re-training. We evaluate the proposed techniques on a range of benchmarks to demonstrate the effectiveness of the proposed
framework.
Robots will eventually be part of every household. It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we
bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. We argue that a descriptive sentence
can provide a much stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the
problem of image captioning in which the quality of the output can easily be judged by non-experts. We propose a hierarchical phrase-based captioning model trained
with policy gradients, and design a feedback network that provides reward to the learner by conditioning on the human-provided feedback. We show that by exploiting
descriptive feedback our model learns to perform better than when given independently written human captions.
We propose to focus on the problem of discovering neural network architectures efficient in terms of both prediction quality and cost. For instance, our
approach is able to solve the following tasks: learn a neural network able to predict well in less than 100 milliseconds or learn an efficient model that fits in a
50 Mb memory. Our contribution is a novel family of models called Budgeted Super Networks (BSN). They are learned
using gradient descent techniques applied on a budgeted learning objective function which integrates a maximum authorized cost, while making no assumption on the
nature of this cost. We present a set of experiments on computer vision problems and analyze the ability of our technique to deal with three different costs: the
computation cost, the memory consumption cost and a distributed computation cost. We particularly show that our model can discover neural network architectures
that have a better accuracy than the ResNet and Convolutional Neural Fabrics architectures on CIFAR-10 andCIFAR-100, at a lower cost.
Importance sampling has been successfully used to accelerate stochastic optimization in many convex problems. However, the lack of an efficient way to calculate
the importance still hinders its application to Deep Learning.
In this paper, we show that the loss value can be used as an alternative importance metric, and propose a way to efficiently approximate it for a deep model,
using a small model trained for that purpose in parallel.
This method allows in particular to utilize a biased gradient estimate that implicitly optimizes a soft max-loss, and leads to better generalization
performance. While such method suffers from a prohibitively high variance of the gradient estimate when using a standard
stochastic optimizer, we show that when it is combined with our sampling mechanism, it results in a reliable procedure.
We showcase the generality of our method by testing it on both image classification and language modeling tasks using deep convolutional and recurrent neural
networks. In particular, our method results in 30% faster training of a CNN for CIFAR10 than when using uniform sampling.
Stochastic gradient descent (SGD), which updates the model parameters by adding a local gradient times a learning rate at each step, is widely used in model
training of machine learning algorithms such as neural networks. It is observed that the models trained by SGD are sensitive to learning rates and good learning rates are problem specific. We propose an algorithm to
automatically learn learning rates using neural network based actor-critic methods from deep reinforcement learning (RL).In particular, we train a policy network
called actor to decide the learning rate at each step during training, and a value network called critic to give feedback about quality of the decision (eg. the
goodness of the learning rate outputted by the actor) that the actor made. The introduction of auxiliary actor and critic networks helps the main network achieve
better performance. Experiments on different datasets and network architectures show that our approach leads to better convergence of SGD than human-designed competitors.
Using deep learning and difference-in-difference analyses on an Airbnb panel dataset spanning 7,423 properties over 16 months, we find that properties with
verified images had 8.98% higher occupancy than properties without verified images (images taken by the host).
To explore what constitutes a good image for an Airbnb property, we quantify 12 human-interpretable image attributes that pertain to 3 artistic
aspects—composition, color, and the figure-ground relationship—and we find systematic differences between the verified and unverified images. We also predict the
relationship between each of the 12 attributes and property demand, and we find that most of the correlations are statistically-significant and in the theorized direction.
Our results provide actionable insights for both Airbnb photographers and amateur host photographers who wish to optimize their images. Our findings contribute
to and bridge the literature on photography and marketing (eg. staging), which often either ignores the demand side (photography) or does not systematically
characterize the images (marketing).
…One of our key objectives is to determine what makes a good image for an Airbnb property. Our CNN model is highly accurate at predicting
image quality, but the CNN-extracted features are uninterpretable. To provide better
guidance for managers, we use the photography literature to identify 12 human-interpretable image attributes that are relevant to image quality in the real estate
context. We theorize the relationship between each of the 12 image attributes and property demand. The 12 attributes fall under 3 key artistic aspects: composition, color, and the figure-ground
relationship. Composition is the arrangement of visual elements in the photograph; ideally, the composition leads the viewer’s eyes to the center of focus
(Freeman 2007). We capture composition with 4 attributes: diagonal dominance, the rule of thirds, visual balance of color, and visual balance of intensity. Color can affect the viewer’s emotional arousal. The
marketing literature has studied the impact of color on consumer behavior particularly in the context of web design, product packaging design, and advertisement
design (Gorn et al 1997, Gorn et al 2004; Miller & Kahn 2005). We include 5 aspects related to color: warm hue, saturation,
brightness, contrast of brightness, and image clarity. The
principle of the figure-ground relationship is one of the most
basic laws of perception and is used extensively by expert photographers to plan their photographs. In visual art, the figure refers to the key region
(ie. foreground), and the ground refers to the background; photographs in which the figure is inseparable from the ground do not retain the viewer’s
attention. We include 3 attributes: the area difference, texture difference, and color difference between the figure and ground.
…Of the 12 image attributes, the visual balance of color is most strongly related to property demand, followed by image clarity and the contrast of brightness.
The visual balance of color refers to color symmetry, which can be affected by both the property itself and the position from which the image is captured. Image
clarity refers to the extent to which the image conveys visual information. The unverified low-quality images scored poorly on image clarity; the verified photos
scored almost twice as high. Even without employing a professional photographer, hosts can improve image clarity through the effective use of lighting and access
to a good camera. Finally, the contrast of brightness captures the difference in illumination between the brightest and dimmest points in the image; a low contrast
of brightness indicates that illumination is relatively even across the image. The verified photos have a substantially lower contrast of brightness than
unverified high-quality images. Interestingly, several hosts on the Airbnb community forums complained that the contrast of brightness is so low in the verified
photos that they appear washed out, but we find the predicted negative relationship between the contrast of brightness and property demand. In other words,
consumers seem to prefer the low contrast of brightness that appears in verified photos.
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We
propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the
system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The
reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from
Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the
agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language
paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting
(tf-idf) and stemming.
Picasso is a free open-source (Eclipse Public License) web application written in Python for rendering standard visualizations useful for analyzing
convolutional neural networks. Picasso ships with occlusion maps and saliency maps, two visualizations which help reveal issues that evaluation metrics like loss
and accuracy might hide: for example, learning a proxy classification task. Picasso works with the Tensorflow deep learning framework, and Keras (when the model
can be loaded into the Tensorflow backend). Picasso can be used with minimal configuration by deep learning researchers and engineers alike across various neural
network architectures. Adding new visualizations is simple: the user can specify their visualization code and HTML
template separately from the application code.
Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance
on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural
network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines
standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit “exposure bias”—they assume ground
truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the resulting
summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our
model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail
dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries.
Existing methods for visual reasoning attempt to directly map inputs to outputs using black-box architectures without explicitly modeling the underlying
reasoning processes. As a result, these black-box models often learn to exploit biases in the data rather than learning to perform visual reasoning. Inspired by
module networks, this paper proposes a model for visual reasoning that consists of a program generator that constructs an explicit representation of the reasoning
process to be performed, and an execution engine that executes the resulting program to produce an answer. Both the program generator and the execution engine are
implemented by neural networks, and are trained using a combination of backpropagation and REINFORCE. Using the CLEVR benchmark for visual reasoning, we show that our model substantially outperforms strong baselines and generalizes better in a
variety of settings.
Many machine learning systems are built to solve the hardest examples of a particular task, which often makes them large and expensive to run—especially with
respect to the easier examples, which might require much less computation. For an agent with a limited computational budget, this “one-size-fits-all” approach may
result in the agent wasting valuable computation on easy examples, while not spending enough on hard examples. Rather than learning a single, fixed policy for
solving all instances of a task, we introduce a metacontroller which learns to optimize a sequence of “imagined” internal simulations over predictive models of the
world in order to construct a more informed, and more economical, solution. The metacontroller component is a model-free reinforcement learning agent, which decides both how many iterations of the optimization procedure to run, as well as which model to
consult on each iteration. The models (which we call “experts”) can be state transition models, action-value functions, or any other mechanism that provides
information useful for solving the task, and can be learned on-policy or off-policy in parallel with the metacontroller. When the metacontroller, controller, and
experts were trained with “interaction networks” (Battaglia et al 2016) as expert models, our approach was able to solve a challenging
decision-making problem under complex non-linear dynamics. The metacontroller learned to adapt the amount of computation it performed to the difficulty of the
task, and learned how to choose which experts to consult by factoring in both their reliability and individual computational resource costs. This allowed the
metacontroller to achieve a lower overall cost (task loss plus computational cost) than more traditional fixed policy approaches. These results demonstrate that
our approach is a powerful framework for using…
In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic
hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of
architecture is done manually by the human expert through a slow trial and error process guided mainly by intuition. In this paper we describe a framework for
automatically designing and training deep models. We propose an extensible and modular language that allows the human expert to compactly represent complex search
spaces over architectures and their hyperparameters. The resulting search spaces are tree-structured and therefore easy to traverse. Models can be automatically
compiled to computational graphs once values for all hyperparameters have been chosen. We can leverage the structure of the search space to introduce different
model search algorithms, such as random search, Monte Carlo tree search (MCTS), and sequential
model-based optimization (SMBO). Wepresent experiments comparing the different algorithms on CIFAR-10 and show that MCTSand SMBO
outperform random search. In addition, these experiments show that our framework can be used effectively for model discovery, as it is possible to describe
expressive search spaces and discover competitive models without much effort from the human expert. Code for our framework and experiments has been made publicly
available.
Stochastic Constraint Programming (SCP) is an extension of Constraint Programming (CP) used for modelling and
solving problems involving constraints and uncertainty. SCP inherits excellent modelling abilities and filtering
algorithms from CP, but so far it has not been applied to large problems. Reinforcement Learning (RL) extends Dynamic
Programming to large stochastic problems, but is problem-specific and has no generic solvers. We propose a hybrid combining the scalability of RL with the
modelling and constraint filtering methods of CP. We implement a prototype in a CP system and demonstrate its usefulness on SCP problems.
In this paper, we study a new learning paradigm for Neural Machine Translation (NMT). Instead of maximizing the
likelihood of the human translation as in previous works, we minimize the distinction between human translation and the translation given by an
NMT model.
To achieve this goal, inspired by the recent success of generative adversarial networks (GANs), we employ
an adversarial training architecture and name it as Adversarial-NMT. In
Adversarial-NMT,the training of the NMT model is assisted by an adversary,
which is anelaborately designed Convolutional Neural Network (CNN). The goal of the adversary is to
differentiate the translation result generated by the NMT model from that by human. The goal of the NMT model is to produce high quality translations so as to cheat the adversary. A policy gradient method is leveraged to
co-train the NMT model and the adversary.
Experimental results on English→French and German→English translation tasks show that Adversarial-NMT can achieve
substantially better translation quality than several strong baselines.
Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For
example, to answer “is there an equal number of balls and boxes?” we can look for balls, look for boxes, count them, and compare the results. The recentlyproposedNeural Module Network (NMN) architecture implements this approach to question answering by parsing questions into linguistic substructures and assembling
question-specific deep networks from smaller modules that each solve one subtask. However, existing NMN
implementations rely on brittle off-the-shelf parsers, and are restricted to the module configurations proposed by these parsers rather than learning them
from data. In this paper, we propose End-to-End Module Networks (N2NMNs), which learn to reason by directly
predicting instance-specific network layouts without the aid of a parser. Our model learns to generate network structures (by imitating
expert demonstrations) while simultaneously learning network parameters (using the downstream task loss). Experimental results on the new CLEVRdataset targeted at compositional question answering show that N2NMNs achieve an
error reduction of nearly 50 attentional approaches, while discovering interpretable network architectures specialized for each question.
Deep learning and reinforcement learning methods have recently been used to solve a variety of problems in continuous
control domains. An obvious application of these techniques is dexterous manipulation tasks in robotics which are difficult to solve using traditional control
theory or hand-engineered approaches. One example of such a task is to grasp an object and precisely stack it on another. Solving this difficult and practically
relevant problem in the real world is an important long-term goal for the field of robotics. Here we take a step towards this goal by examining the problem in
simulation and providing models and techniques aimed at solving it. We introduce two extensions to the Deep Deterministic Policy Gradient algorithm
(DDPG), a model-free Q-learning based method, which make it
statistically-significantly more data-efficient and scalable. Our results show that by making extensive use of off-policy data and replay, it is possible to find
control policies that robustly grasp objects and stack them. Further, our results hint that it may soon be feasible to train successful stacking policies by
collecting interactions on real robots.
We propose a dynamic computational time model to accelerate the average processing time for recurrent visual attention (RAM). Rather than attention with a fixed number of steps for each input image, the model learns to decide when to stop on the fly.
To achieve this, we add an additional continue/stop action per time step to RAM and use reinforcement learning
to learn both the optimal attention policy and stopping policy. The modification is simple but could dramatically save the average computational time while keeping
the same recognition performance as RAM. Experimental results on CUB-200-2011 and
Stanford Cars dataset demonstrate the dynamic computational model can work effectively for fine-grained image recognition.The source code of this
paper can be obtained from https://github.com/baidu-research/DT-RAM
In this paper we investigate image classification with computational resource limits at test time. Two such settings are: 1. anytime classification, where the
network’s prediction for a test example is progressively updated, facilitating the output of a prediction at any time; and 2. budgeted batch classification, where
a fixed amount of computation is available to classify a set of examples that can be spent unevenly across “easier” and “harder” inputs. In contrast to most prior
work, such as the popular Viola and Jones algorithm, our approach is based on convolutional neural networks. We train multiple classifiers with varying resource
demands, which we adaptively apply during test time. To maximally re-use computation between the classifiers, we incorporate them as early-exits into a single deep
convolutional neural network and inter-connect them with dense connectivity. To facilitate high quality classification early on, we use a two-dimensional
multi-scale network architecture that maintains coarse and fine level features all-throughout the network. Experiments on three image-classification tasks
demonstrate that our framework substantially improves the existing state-of-the-art in both settings.
We propose and systematically evaluate three strategies for training dynamically-routed artificial neural networks: graphs of learned transformations through
which different input signals may take different paths. Though some approaches have advantages over others, the resulting networks are often qualitatively similar.
We find that, in dynamically-routed networks trained to classify images, layers and branches become specialized to process distinct categories of images.
Additionally, given a fixed computational budget, dynamically-routed networks tend to perform better than comparable statically-routed networks.
End-to-end design of dialogue systems has recently become a popular research topic thanks to powerful tools such as encoder-decoder architectures for
sequence-to-sequence learning. Yet, most current approaches cast human-machine dialogue management as a supervised learning problem, aiming at predicting the next
utterance of a participant given the full history of the dialogue. This vision is too simplistic to render the intrinsic planning problem inherent to dialogue as
well as its grounded nature, making the context of a dialogue larger than the sole history. This is why only chit-chat and question answering tasks have been
addressed so far using end-to-end architectures. In this paper, we introduce a Deep Reinforcement Learning method to optimize visually grounded task-oriented
dialogues, based on the policy gradient algorithm. This approach is tested on a dataset of 120k dialogues collected through Mechanical Turk and provides
encouraging results at solving both the problem of generating natural dialogues and the task of discovering a specific object in a complex picture.
Learning to learn has emerged as an important direction for achieving artificial intelligence. Two of the primary barriers to its adoption are an inability to
scale to larger problems and a limited ability to generalize to new tasks. We introduce a learned gradient descent optimizer that generalizes well to new tasks,
and which has significantly reduced memory and computation overhead. We achieve this by introducing a novel hierarchical RNN architecture, with minimal per-parameter overhead, augmented with additional architectural features that mirror the
known structure of optimization tasks. We also develop a meta-training ensemble of small, diverse optimization tasks capturing common properties of loss
landscapes. The optimizer learns to outperform RMSProp/ADAM on problems in this
corpus. More importantly, it performs comparably or better when applied to small convolutional neural networks, despite seeing no neural networks in its
meta-training set. Finally, it generalizes to train Inception V3 and ResNet V2 architectures on the ImageNet dataset for thousands of steps, optimization problems that are of a vastly different scale than those it was trained on. We
release an open source implementation of the meta-training algorithm.
Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems
alone. Our goal is to minimize human participation, so we employ evolutionary algorithms to discover such networks automatically. Despite significant computational
requirements, we show that it is now possible to evolve models with accuracies within the range of those published in the last year. Specifically, we employ simple
evolutionary techniques at unprecedented scales to discover models for the CIFAR-10 and CIFAR-100 datasets, starting from trivial initial conditions and reaching accuracies of 94.6% (95.6% for ensemble) and 77.0%,
respectively. To do this, we use novel and intuitive mutation operators that navigate large search spaces; we stress that no human participation is required once
evolution starts and that the output is a fully-trained model. Throughout this work, we place special emphasis on the repeatability of results, the variability in
the outcomes and the computational requirements.
Neural networks have been successfully applied in applications with a large amount of labeled data. However, the task of rapid generalization on new concepts
with small training data while preserving performances on previously learned ones still presents a significant challenge to neural network models. In this work, we
introduce a novel meta learning method, Meta Networks (MetaNet), that learns a meta-level knowledge across tasks and shifts its inductive biases via fast
parameterization for rapid generalization. When evaluated on Omniglot and Mini-ImageNet benchmarks, our MetaNet models
achieve a near human-level performance and outperform the baseline approaches by up to 6% accuracy. We demonstrate several appealing properties of MetaNet relating
to generalization and continual learning.
We propose an LSTM-based meta-learner model to learn the exact optimization algorithm used to train
another learner neural network in the few-shot regime
Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot learning tasks, where a model has to
quickly generalize after seeing very few examples from each class. The general belief is that gradient-based optimization in high capacity models requires many
iterative steps over many examples to perform well. Here, we propose an LSTM-based meta-learner model to
learn the exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of our model allows it to learn
appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of the learner
network that allows for quick convergence of training. We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for
few-shot learning.
The success of deep learning depends on finding an architecture to fit the task. As deep learning has scaled up to more challenging tasks, the architectures
have become difficult to design by hand. This paper proposes an automated method, CoDeepNEAT, for optimizing deep
learning architectures through evolution. By extending existing neuroevolution methods to topology, components, and hyperparameters, this method achieves
results comparable to best human designs in standard benchmarks in object recognition and language modeling. It also supports building a real-world application of
automated image captioning on a magazine website. Given the anticipated increases in available computing power, evolution of deep networks is promising approach to
constructing deep learning applications in the future.
Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning.
In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present
interesting challenges for existing reinforcement learning algorithms. We develop an extension that is suited to learning optimization algorithms in this setting
and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes
in stochasticity of gradients and the neural net architecture. More specifically, we show that an optimization algorithm trained with the proposed method on
the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto
Faces Dataset, CIFAR-10 and CIFAR-100.
We present an approach to adaptively utilize deep neural networks in order to reduce the evaluation time on new examples without loss of accuracy. Rather than
attempting to redesign or approximate existing networks, we propose two schemes that adaptively utilize networks. We first pose an adaptive network evaluation
scheme, where we learn a system to adaptively choose the components of a deep network to be evaluated for each example. By allowing examples correctly classified
using early layers of the system to exit, we avoid the computational time associated with full evaluation of the network. We extend this to learn a network
selection system that adaptively selects the network to be evaluated for each example. We show that computational time can be dramatically reduced by exploiting
the fact that many examples can be correctly classified using relatively efficient networks and that complex, computationally costly networks are only necessary
for a small fraction of examples. We pose a global objective for learning an adaptive early exit or network selection policy and solve it by reducing the policy
learning problem to a layer-by-layer weighted binary classification problem. Empirically, these approaches yield dramatic reductions in computational cost, with up
to a 2.8× speedup on state-of-the-art networks from the ImageNet image recognition challenge with minimal (<1%) loss of
top-5 accuracy.
We present a framework to tackle combinatorial optimization problems using neural networks and reinforcement learning. We
focus on the traveling salesman problem (TSP) and train a recurrent neural network that, given a set of city
coordinates, predicts a distribution over different city permutations. Using negative tour length as the reward signal, we optimize the parameters of the
recurrent neural network using a policy gradient method. Without much engineering and heuristic designing, Neural Combinatorial Optimization achieves close to
optimal results on 2D Euclidean graphs with up to 100 nodes. These results, albeit still quite far from state-of-the-art, give insights into how neural networks
can be used as a general tool for tackling combinatorial optimization problems.
![Figure 1: Forward Excitatory and Counter-Current Inhibitory Circuitry of the Entorhinal-Hippocampal Complex (EHC) Supporting Self-Supervision and Epistemic Autonomy. (A) Forward and backward hippocampal circuits. Left. The feed-forward information flow of the hippocampal trisynaptic pathway and its constituents19. The pathway (grey arrow) comprises projections from layer II entorhinal cortex (EC) stellate cells to the dentate gyrus (DG) and CA3 via the medial (MPP, light green) and lateral (LPP, yellow) perforant path (PP), mossy fiber projections of DG granule cells to CA3 pyramidal neurons (dark green), and CA3 projections to CA1 pyramidal neurons (the Schaffer collaterals, pink). The feedforward input is completed with direct projections from layer III EC neurons projecting to CA119. The output of the hippocampus (HPC) originates in CA1 and passes via the subiculum (not shown) to the EC LV/VI (purple). Right. Cortical input and hippocampal output coincide in EC, allowing the EHC comparator to compute the mismatch between the 2 signals. (B) Left. Counter-current inhibitory circuit complementing the forward excitatory loop (Box 3). Right. We hypothesize that this counter-current circuit carries error signals (yellow) that define a gradient that shapes synaptic plasticity along the forward excitatory loop, therefore implementing a biological version of error backpropagation. (C) Top. The synergy between the forward and feedback circuits shapes the continuous synaptic update in the forward loop such that it increasingly minimizes the error between the HPC input and output signals of the EC comparator. Middle. In this self-supervised learning scenario, environmental change (pink line) is reflected in the error amplitude, where error magnitude activates distinct physiological and behavioral responses. Bottom. Small amplitude errors perturbate the firing rate of principal cells, for instance, expressed as firing rate modulation in spatial navigation tasks. In contrast, large magnitude errors signal novelty and drive relearning supporting the reconstruction of this novel signal, leading to global remapping (see20 for a possible threshold-triggered synaptic mechanism based on neuronal depolarization levels). (D) Left. The interplay between excitatory and inhibitory cells in the EHC comparator. Cortical signals coded by input neurons (yellow) are propagated throughout the trisynaptic circuit (green arrow) to neurons reflecting the HPC reconstruction of the input signal (blue). Comparator neurons (brown) receive both the reconstruction and an inhibitory copy of the input activity (grey), which in turn modulate the firing level of counter-current GABAergic interneurons that backpropagate the error signal (orange line). At this stage, recurrent GABAergic projections within the HPC modulate network-level synaptic distributions, leading to a convergence of cortical and hippocampal signals and thus performing self-supervision. Right. Dependent on its magnitude, mismatch error activates a range of molecular, physiological, and behavioral phenomena observed during spatial navigation. See [36,42,91,92,93].](/images/ai/2021-santospata-figure1-hippocampusselfsupervisionlearning.jpg)
![Figure 2: Agent architecture and benchmarking. (A) How the agent processes a temporal sequence of observations xt from the environment. The model operates at 2 different time scales, faster at the bottom and slower by a factor of τ at the top. A stochastic vector-valued latent variable is sampled at the fast time scale from distribution ℚt on the basis of observations xt. The action distribution πt is sampled conditional on the latent variable at each time step t. The latent variable is regularized by the slow moving prior ℙt, which helps capture long-range temporal correlations and promotes memory. The network parameters are updated by using RL according to the agent’s own internal reward signal rt, which is obtained from a learned transformation w of game points ρt. w is optimized for winning probability through PBT, another level of training performed at yet a slower time scale than that of RL. Detailed network architectures are described in figure S11. (B) (Top) The Elo skill ratings of the FTW agent population throughout training (blue) together with those of the best baseline agents by using hand-tuned reward shaping (RS) (red) and game-winning reward signal only (black), compared with human and random agent reference points (violet, shaded region shows strength between 10th and 90th percentile). The FTW agent achieves a skill level considerably beyond strong human subjects, whereas the baseline agent’s skill plateaus below and does not learn anything without reward shaping [evaluation procedure is provided in (28 [supplements])]. (Bottom) The evolution of 3 hyperparameters of the FTW agent population: learning rate, Kullback-Leibler divergence (KL) weighting, and internal time scale τ, plotted as mean and standard deviation across the population.](/images/rl/2019-jaderberg-figure2-agentarchitectureandbenchmarking.jpg)
