AI

• “Vector Quantized Models for Planning”⁠, Ozair et al 2021 (MCTS on VQVAE to generalize MuZero to stochastic/​​hidden-info environments—towards planning over world models…)

• “A graph placement methodology for fast chip design”⁠, Mirhoseini et al 2021 (media⁠; optimizing TPU ‘chip floor planning’ circuit placement)

• “The Robot Household Marathon Experiment”⁠, Kazhoyan et al 2020 (media⁠; benchmarking PR2 robot on making & cleaning up breakfast: successful setup, but many failures in cleanup—bad planning & control software continues to bottleneck robotics. “Intelligent matter” doesn’t matter if you lack enough ‘intelligence’ to make the matter, matter.)

• “GANs N’ Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too!)”⁠, Chong & Forsyth 2021 (code  /​ ​​ ​video  /​ ​​ ​Colab)

• “Alias-Free GAN (Generative Adversarial Networks)”⁠, Karras et al 2021 (successor to StyleGAN2-ADA introduces a major architectural revamp to enable ultra-smooth interpolations; this should be much more useful for video: AFHQ example⁠, more samples)

• “What A Long, Strange Trip It’s Been: EleutherAI One Year Retrospective” (AmA)

  • “Alien Dreams: An Emerging Art Scene”⁠, Charlie Snell (catching up on hobbyist exploration of CLIP-generated images); “Tour of the Sacred Library”⁠, Ryan Moulton (curated Dinotopia-style images)

• PurpleSmart.ai beta opens (feedback to the Discord)

Matters Of Scale:

• “CPM-2: Large-scale Cost-effective Pre-trained Language Models”⁠, Zhang et al 2021 (11b-dense/​​198b MoE Zh+En; models have been released—now the largest publicly-available autoregressive English models!)

• “Scaling Vision Transformers”⁠, Zhai et al 2021 (testing w/​​n = 0.03–3b & parameters 0.05–2b using an upgraded JFT-300M dataset, ‘JFT-3B’, confirms the scaling laws: larger models continue to perform much better, more sample-efficient, incredible few-shot performance, & weight decay still useful; ImageNet top-1 classification is approaching saturation, like top-5 before it)

• “Scaling Laws for Acoustic Models”⁠, Droppo & Elibol 2021 (smooth scaling of Transformer vs LSTM audio models for log-Mel → next-word prediction; Transformers scale better: 33× per halving vs 63×)

• Tabular DL: “Regularization is all you Need: Simple Neural Nets can Excel on Tabular Data”⁠, Kadra et al 2021 (or if you prefer, Gorishniy et al 2021⁠, Somepalli et al 2021⁠, Liu et al 2021—the last ML holdout, tabular, falls to the Bitter Lesson of Transformers & carefully-regularized MLPs—although those heavily invested in tree approaches will doubtless fight to the bitter end, saying this or that hyperparameter optimization or hand-feature-engineering would reestablish some slight superiority, between this and earlier tabular NNs, the writing is on the wall, and reportedly many industry ML users have moved to DL already for its other benefits)

• “GitHub Copilot: Your AI pair programmer” (media); “Codex: Evaluating Large Language Models Trained on Code”⁠, Chen et al 2021 (on small versions)

  New GPT-3-based code completion for GitHub⁠; like TabNine or IntelliCode⁠, but more so, and a tasty lollipop indeed; puzzlingly, OA/​​GH appear to do no checks like n-grams for possible copying, but copying is rare anyway⁠.

  It is darkly hilarious to see programmers react little better than artists did in peddling misinformation about Copilot like unlabeled “humor”, instantly turning into substance dualists insisting that “computers can never truly understand code or be creative unlike us humans with souls^Wminds”, Internet IP lawyers who have never heard of the word “transformative”⁠, and infosec experts engaging in histrionics about it “leaking secrets”—from public Github repos, y’know, made up of public commits to public repos you really should not be uploading any passwords or keys to because attackers have been actively monitoring in realtime for credentials to steal for over a decade now? In a year, who will remember any of this BS? A few months later (also like TFDNE), it looked like the world hadn’t ended and people moved on⁠. Machiavelli had it right: “…there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things. Because the innovator has for enemies all those who have done well under the old conditions, and lukewarm defenders in those who may do well under the new. This coolness arises partly from…the incredulity of men, who do not readily believe in new things until they have had a long experience of them.” (What if there was a revolution and no one cared?)

  Regardless, “attacks only get better”, and Copilot surely will. I’m a little surprised that Copilot/​​Codex appear to be trained only on entire source code files, when patches are the perfect training data for making a promptable edit-capable LM: a patch is a human-readable summary/​​explanation of the following changes, provides an immediately promptable description of quality before/​​after, and a compact word-diff format is an ideal output format for a LM to bugfix/​​update the context without the overhead of generating the entire module. (Plus, who has more Git repos or expertise in parsing Git patches than Github?) I look forward to seeing what better Codex models can do—Sam Altman mentioned in the 2021-09-05 SSC Q&A that the next version will be much better, and that one underappreciated benefit of the OA/​​MS partnership is the sheer volume of human feedback that OA can train models on.

  Incidentally, Perlis also remarked that “In a 5 year period we get one superb programming language. Only we can’t control when the 5 year period will begin.” A fancy tab-complete can do wonders in making a enormously-overcomplicated ecosystem ‘discoverable’ and getting one unstuck, but it is still just a tab-complete. If you were designing a language/​​IDE/​​ecosystem on the presumption of a GPT-3-level model, would you really settle on “Visual Studio IDE for an OO-glazed ALGOL language but with a fancier tab-complete”?

• “Multimodal Few-Shot Learning with Frozen Language Models”⁠, Tsimpoukelli et al 2021; “AudioCLIP: Extending CLIP to Image, Text and Audio”⁠, Guzhov et al 2021

• “Google demonstrates leading performance in latest MLPerf Benchmarks” using TPUv4s

• “Time-Aware Language Models as Temporal Knowledge Bases”⁠, Dhingra et al 2021 (a nice use of the inline metadata trick to greatly improve T5 temporal knowledge/​​reasoning—just include more metadata! No fancy new symbolic architectures or training methods. Just provide more useful data.)

Genetics

Everything Is Heritable:

• “Genetics of substance use disorders in the era of big data”⁠, Gelernter & Polimanti 2021
• “Whole-exome imputation within UK Biobank powers rare coding variant association and fine-mapping analyses”⁠, Barton et al 2020

Engineering:

• Aurea Smigrodzki: “The first baby in history to be conceived with the help of polygenic testing” (born mid-2020; father’s panel⁠; discussion⁠, Hsu⁠, media⁠; see also Conley)

  I was surprised how little discussion the announcement provoked, after all the Sturm und drang the prospect of embryo selection provoked. But when it happened, there wasn’t a single media article for 3 months, and that article seemed to focus more on her father’s distaste for journalists. (Such a focus seems unlikely to improve Dr. Smigrodzki’s opinion of them.) What if there was a revolution and no one cared?

• “CRISPR-Cas9 In Vivo Gene Editing for Transthyretin Amyloidosis”⁠, Gillmore et al 2021

• Gametogenesis (catching up):

  • “Artificially produced gametes in mice, humans and other species”⁠, Hayashi et al 2021
  • “Gametes from stem cells: Status and applications in animal reproduction”⁠, Goszczynski et al 2019 (previously: “In vitro breeding: application of embryonic stem cells to animal production”⁠, Goszczynski et al 2018)
  • “Danish dairy farmers’ acceptance of and willingness to use semen from bulls produced by means of in vitro embryo production and genomic selection”⁠, Lund et al 2021

Statistics/Meta-Science

• “What We Learned Doing Fast Grants”⁠, Patrick Collison, Tyler Cowen⁠, & Patrick Hsu (media)
• “Shifting the impossible to the inevitable: A Private ARPA (PARPA) user manual”, Ben Reinhardt (how to try to clone the ARPA model outside government)
• “Test & Roll: Profit-Maximizing A  /​ ​​ ​B Tests”⁠, Feit & Berman 2017
• “A visual introduction to Gaussian Belief Propagation: A Framework for Distributed Inference with Emerging Hardware”⁠, Ortiz et al 2021 (explorable ‘loopy’ belief propagation for computing Bayesian networks)
• “Statistical Inquiries into the Efficacy of Prayer”⁠, Galton 1872

Politics/Religion

• “Get lucky: Happy 245^th, America!”⁠, Razib Khan
• The 1989 California medfly attack (a successful bioterrorism extortion of the California government; cf “Dark Harvest”)

Psychology/Biology

• “Air Pollution and Adult Cognition: Evidence from Brain Training”, La Nauze & Severnini 2021 (using 4.6m scores from Lumosity brain-game players to measure PM2.5 cognitive effects: large but heterogeneous by task/​​age/​​experience—this may explain the inconsistencies of previous air-pollution/​​cognition studies)
• “Efficacy of Wolbachia-Infected Mosquito Deployments for the Control of Dengue”⁠, Utarini et al 2021
• “Senescent cell turnover slows with age providing an explanation for the Gompertz law”⁠, Karin et al 2019
• “Human click-based echolocation: Effects of blindness and age, and real-life implications in a 10-week training program”⁠, Norman et al 2021
• Hypospray
• “The Scavenging Patterns of Feral Cats on Human Remains in an Outdoor Setting”⁠, Garcia et al 2019

Technology

• “5 Years of Graduate CS Education Online and at Scale”⁠, Joyner et al 2019 (why are there so many interesting grad students/​​researchers at Georgia Tech eg in EleutherAI? perhaps their master’s MOOC⁠, which admits everyone; an experience report)
• Meteor burst communications (“AMBCS was able to greatly improve the data rates, averaging 4 kilobits per second”)
• Kapitza’s pendulum (surprisingly, you can stabilize an inverted pendulum without any feedback or control by brute force—simply wiggling it fast)

Economics

• Another reminder for my tech readers: ask for more money! As the USA comes out of coronavirus, there may well have never been a better time in history for programmers (especially for in-demand specialties) to negotiate more pay. Demand money, equity, or true remote (not ‘come in one day less’, that doesn’t let you escape the Bay Area)—nothing less. Don’t be fobbed off with trinkets like “free kitchenette snacks”. If you don’t get it, leave and get the raise elsewhere. (Only rolling stones gather moss these days.)

Miscellaneous

• “Harris’s List of Covent Garden Ladies⁠, a directory of London prostitutes published annually from 1757 to 1795, containing entries describing the physical appearance and sexual specialties of the women…”
• Cummingtonite