2018 News

Annual summary of 2018 gwern.net newsletters, selecting my best writings, the best 2018 links by topic, and the best books/movies/anime I saw in 2018, with some general discussion of the year.
newsletter
2018-12-082020-11-25 finished certainty: log importance: 0


This is the of the Gwern.net newslet­ter (), sum­ma­riz­ing the best of the monthly 2018 newslet­ters:

  1. end of year sum­mary

Pre­vi­ous annual newslet­ters: , , .

Writings

2018 went well, with much inter­est­ing news and sev­eral stim­u­lat­ing trips. My 2018 writ­ings includ­ed:

  1. Embryo selec­tion: Overview of major cur­rent approaches for com­plex-trait genetic engi­neer­ing, FAQ, mul­ti­-stage selec­tion, chromosome/gamete selec­tion, opti­mal search of batches, & robust­ness to error in util­ity weights

  2. reviews:

Site traffic (more detailed break­down) was again up as com­pared with the year before: 2018 saw 736,486 pageviews by 332,993 unique users (vs 551,635 by 265,836 in 2017).

Media

Overview

Over­all, 2018 was much like 2017 was, but more so. In all of AI, genet­ics, VR, Bit­coin, and gen­eral culture/politics, the trends of 2017 con­tin­ued through 2018 or even accel­er­at­ed.

AI: In 2018, the DL rev­o­lu­tion came for NLP. Con­vo­lu­tions, atten­tion, and big­ger com­pute cre­ated ever larger NNs which could then kick bench­mark ass and take names. Addi­tional seeds were planted for logical/relational/numerical rea­son­ing (was­n’t logic another one of those things deep learn­ing would never be able to do…?).

Else­where, rein­force­ment learn­ing was hot (eg the RL sub­red­dit traffic stats increased sev­er­al­fold over 2017, which itself had increased sev­er­al­fold), with Go fol­lowed by human-level DoTA 2. OA5 was an amaz­ing achieve­ment given how com­plex DoTA is, with fog of war, team tac­tics, and far larger state space, inte­grat­ing the full spec­trum of strat­egy from twitch tac­tics up to long-term strat­egy and pre-s­e­lec­tion of units. (Given OA5’s pro­gress, I was dis­ap­pointed to see min­i­mal DM progress on the Star­craft II front in 2018, but it turned out I just needed more patience.) DRL of course enjoyed addi­tional pro­gress, notably robot­ics: sam­ple-effi­cient robotic con­trol and learn­ing from observations/imitation are closer than ever.

Think­ing a lit­tle more broadly about where DL/DRL has seen suc­cess­es, the rise of DL has been the fall of .

No one is now sur­prised in the least bit when a com­puter mas­ters some com­plex sym­bolic task like chess or Go these days; we are sur­prised by the details like it hap­pen­ing about 10 years before many would’ve pre­dict­ed, or that the Go player can be trained overnight in wall­clock time, or that the same archi­tec­ture can be applied with min­i­mal mod­i­fi­ca­tion to give a top chess engine. For all the fuss over AlphaGo, no one pay­ing atten­tion was really sur­prised. If you went back 10 years ago and told some­one, ‘by the way, by 2030, both Go and Ari­maa can be played at a human level by an AI’, they’d shrug.

Peo­ple are much more sur­prised to see DoTA 2 agents, or Google Waymo cars dri­ving around entire met­ro­pol­i­tan areas, or gen­er­a­tion of pho­to­re­al­is­tic faces or totally real­is­tic voic­es. The progress in robot­ics has also been excit­ing to any­one pay­ing atten­tion to the space: the DRL approaches are get­ting ever bet­ter and sam­ple-effi­cient and good at imi­ta­tion. I don’t know how many blue-col­lar work­ers they will put out of work—even if soft­ware is solved, the robotic hard­ware is still expen­sive! But fac­to­ries will be sali­vat­ing over them, I’m sure. (The future of self­-driv­ing cars is in con­sid­er­ably more doubt.)

A stan­dard­-is­sue min­i­mum-wage Homo sapi­ens work­er-u­nit has a lot of advan­tages. I expect there will be a lot of blue-col­lar jobs for a long time to come, for those who want them. But they’ll be increas­ingly crummy jobs. This will make a lot of peo­ple unhap­py. I think of Turch­in’s ‘elite over­pro­duc­tion’ con­cep­t—how much of polit­i­cal strife now is sim­ply that we’ve overe­d­u­cated so many peo­ple in degrees that were almost entirely sig­nal­ing-based and not of intrin­sic value in the real world and there were no slots avail­able for them and now their expec­ta­tions & lack of use­ful skills are col­lid­ing with real­i­ty? In polit­i­cal sci­ence, they say rev­o­lu­tions hap­pen not when things are going bad­ly, but when things are going not as well as every­one expect­ed.

We’re at an inter­est­ing point—as LeCun put it, I think, ‘any­thing a human can do with <1s of thought, deep learn­ing can do now’, while older sym­bolic meth­ods can out­per­form humans in a num­ber of domains where they use >>1s of thought. As NNs get big­ger and the train­ing meth­ods and archi­tec­tures and datasets are refined, the ‘<1s’ will grad­u­ally expand. So there’s a pin­cer move­ment going on, and some­times hybrid approaches can crack a human redoubt (eg AlphaGo com­bined the hoary tree search for long-term >>1s thought with CNNs for the intu­itive instan­ta­neous gut-re­ac­tion eval­u­a­tion of a board <1s, and together they could learn to be super­hu­man). As long as what humans do with <1s of thought was out of reach, as long as the ‘sim­ple’ prim­i­tives of vision and move­ment could­n’t be han­dled, the sym­bol ground­ing and frame prob­lems were hope­less. “How does your design turn a photo of a cat into the sym­bol CAT which is use­ful for inference/planning/learning, exact­ly?” But now we have a way to reli­ably go from chaotic real-world data to rich seman­tic numeric encod­ings like vec­tor embed­dings. That’s why peo­ple are so excited about the future of DL.

The biggest dis­ap­point­ment, by far, in AI was self­-driv­ing cars.

2018 was going to be the year of self­-driv­ing cars, as Waymo promised all & sundry a full pub­lic launch and the start of scal­ing out, and every report of expen­sive deals & invest­ments bade fair to launch, but the launch kept not hap­pen­ing—and then the Uber pedes­trian fatal­ity hap­pened. This fatal­ity was the result of a cas­cade of inter­nal deci­sions & pres­sure to put an unsta­ble, errat­ic, known dan­ger­ous self­-driv­ing car on the road, then delib­er­ately dis­able its emer­gency brak­ing, delib­er­ately dis­able the car’s emer­gency brak­ing, not pro­vide any alerts to the safety dri­vers, and then remove half the safety dri­vers, result­ing in a fatal­ity hap­pen­ing under what should have been near-ideal cir­cum­stances, and indeed the soft­ware detected the pedes­trian long in advance and would have braked if it had been allowed (“Pre­lim­i­nary NTSB Report: High­way HWY18MH010”); par­tic­u­larly egre­gious given Uber’s past inci­dents (like cov­er­ing up run­ning a red light). Com­par­isons to Chal­lenger come to mind.

The inci­dent should not have affected per­cep­tion of self­-driv­ing cars—the fact that a far-below-SOTA sys­tem is unsafe when its brakes are delib­er­ately dis­abled so it can­not avoid a fore­seen acci­dent tells us noth­ing about the safety of the best self­-driv­ing cars. That self­-driv­ing cars are dan­ger­ous when done badly should not come as news to any­one or change any beliefs, but it black­ened per­cep­tions of self­-driv­ing cars nev­er­the­less. Per­haps because of it, the promised Waymo launch was delayed all the way to Decem­ber and then was purely a ‘paper launch’, with no dis­cernible differ­ence from its pre­vi­ous smal­l­-s­cale oper­a­tions.

Which leads me to ques­tion why the cred­i­ble buildup before­hand of vehi­cles & per­son­nel & deals if the paper launch was what was always intend­ed; did the Uber inci­dent trig­ger an inter­nal review and a major re-e­val­u­a­tion of how capa­ble & safe their sys­tem really is and a resort to a paper launch to save face? What went wrong, not at Uber but at Waymo? As Waymo goes, so the sec­tor goes.

2018 for genet­ics saw many of the fruits of 2017 begin to mature: the usual large-s­cale GWASes con­tin­ued to come out, includ­ing both SSGAC3 (Lee et al 2018) and an imme­di­ate boost by bet­ter analy­sis in Alle­grini et al 2018 (as I pre­dicted last year); uses of PGSes in other stud­ies, such as the for­bid­den exam­i­na­tion of life out­come differ­ences pre­dicted by IQ/EDU PGSes, are increas­ingly rou­tine. In par­tic­u­lar, med­ical PGSes are now reach­ing lev­els of clin­i­cal util­ity that even doc­tors can see their val­ue.

This trend need not peter out, as the oncom­ing datasets keep get­ting more enor­mous; con­sumer DTC extrap­o­lat­ing from announced sales num­bers has reached stag­ger­ing num­bers and poten­tially into the hun­dreds of mil­lions, and there are var­i­ous announce­ments like the UKBB aim­ing for 5 mil­lion whole-genomes, which would’ve been bonkers even a few years ago. (Why now? Prices have fallen enough. Per­haps an enter­pris­ing jour­nal­ist could dig into why Illu­mina could keep WGS prices so high for so long…) The promised land is being reached.

The drum­beat of CRISPR suc­cesses reached a peak in the case of He Jiankui, who—­com­pletely out of the blue—p­re­sented the world with the fait accom­pli of CRISPR babies. The most strik­ing aspect is the tremen­dous back­lash: not just from West­ern­ers (which is to be expect­ed, and is rather hyp­o­crit­i­cal of many of the geneti­cists involved, who talked pre­vi­ously of being wor­ried about poten­tial back­lash from pre­ma­ture CRISPR use and then, when that hap­pened, did their level best to make the back­lash hap­pen by com­pet­ing for the most hyper­bolic con­dem­na­tion), but also from Chi­na. Almost as strik­ing was how quickly com­men­ta­tors set­tled on a Nar­ra­tive, inter­pret­ing every­thing as neg­a­tively as pos­si­ble even where that required flatly ignor­ing report­ing (claim­ing he launched a PR blitz, when the AP scooped him) or strains credulity (how can we believe the hos­pi­tal’s face-sav­ing claims that Jiankui ‘forged’ every­thing, when they were so effu­sive before the back­lash began? Or any gov­ern­ment state­ments com­ing out of Chi­na, of all places, about an indefi­nitely impris­oned sci­en­tist?), or cit­ing the most dubi­ous pos­si­ble research (like can­di­date-gene or ani­mal model research on CCR5).

Regard­less, the taboo has been bro­ken. Only time will tell if this will spur more rig­or­ous­ly-con­ducted CRISPR research to do it right, or will set back the field for decades & be an exam­ple of the . I am cau­tiously opti­mistic that it will be the for­mer.

Genome syn­the­sis work appears to con­tinue to roll along, although noth­ing of major note occurred in 2018. Prob­a­bly the most inter­est­ing area in terms of fun­da­men­tal work was the progress on both mice & human game­to­ge­n­e­sis and stem cell con­trol. This is the key enabling tech­nol­ogy for both mas­sive embryo selec­tion (break­ing the egg bot­tle­neck by allow­ing gen­er­a­tion of hun­dreds or thou­sands of embryos and thus mul­ti­ple-SD gains from selec­tion) and then IES (It­er­ated Embryo Selec­tion).

VR con­tin­ued steady grad­ual growth; with no major new hard­ware releases (Ocu­lus Go does­n’t coun­t), there was not much to tell beyond the Steam sta­tis­tics or Sony announc­ing PSVR sales >3m. (I did have an oppor­tu­nity to play the pop­u­lar Beat Saber with my mother & sis­ter; all of us enjoyed it.) More inter­est­ing will be the 2019 launch of which comes close to the hypo­thet­i­cal mass-con­sumer break­through VR head­set: mobile/no wires, with a res­o­lu­tion boost and full hand/position track­ing, in a rea­son­ably priced pack­age, with a promised large library of estab­lished VR games ported to it. It lacks foveated ren­der­ing or retina res­o­lu­tion, but oth­er­wise seems like a major upgrade in terms of mass appeal; if it con­tin­ues to eke out mod­est sales, that will be con­sis­tent with the nar­ra­tive that VR is on the long slow slog adop­tion path sim­i­lar to early PCs or the Inter­net (in­stantly appeal­ing & clearly the future to the early adopters who try it, but still tak­ing decades to achieve any mass pen­e­tra­tion) rather than post-i­Phone smart­phones.

Bit­coin: the long slide from the bub­ble con­tin­ued, to my con­sid­er­able schaden­freude (2017 but more so…). The most inter­est­ing story of the year for me was the rea­son­ably suc­cess­ful launch of the long-awaited Augur pre­dic­tion mar­ket, which had no ‘DAO moment’ and the over­all mech­a­nism appears to be work­ing. Oth­er­wise, not much to remark on.

A short note on pol­i­tics: I main­tain my 2017 com­ments (but more so…). For all the emo­tion invested in the ‘great awok­en­ing’ and the con­tin­ued Girar­dian scapegoating/backlash, now 3 years in, it is increas­ingly clear that Don­ald Trump’s pres­i­dency has been absurdly over­rated in impor­tance. Despite his abil­ity to do sub­stan­tial dam­age like launch­ing trade wars or dis­tor­tionary tax cuts, that is hardly unprece­dented as most pres­i­dents do severe eco­nomic dam­age of some form or anoth­er; while other blun­ders like his ineffec­tual North Korea pol­icy merely con­tin­ues a long his­tory of ineffec­tive pol­icy (and was inevitable once the South Korean pop­u­la­tion chose to elect Moon Jae-in). Every minute you spend obsess­ing over stuff like the Mueller Report has been wast­ed: Trump remains what New York­ers have always known him to be—an incom­pe­tent nar­cis­sist.

Let’s try to focus more on long-term issues such as global eco­nomic growth or genetic engi­neer­ing.

Books

Non­fic­tion:

  1. McNa­ma­ra’s Fol­ly: The Use of Low-IQ Troops in the Viet­nam War, Gre­gory 2015
  2. Bad Blood: Secrets and Lies in a Sil­i­con Val­ley Startup, 2018
  3. The Vac­ci­na­tors: Small­pox, Med­ical Knowl­edge, and the ‘Open­ing’ of Japan, Jan­netta 2007 (review)
  4. Like Engen­dr’ing Like: Hered­ity and Ani­mal Breed­ing in Early Mod­ern Eng­land, Rus­sell 1986 ()
  5. Cat Sense: How the New Feline Sci­ence Can Make You a Bet­ter Friend to Your Pet, Brad­shaw 2013 ()
  6. , Roland & Shi­man 2002 ()
  7. The Oper­a­tions Eval­u­a­tion Group: A His­tory of Naval Oper­a­tions Analy­sis, Tid­man 1984

Fic­tion:

  1. Fuji­wara Teika’s Hun­dred-Poem Sequence of the Shōji Era, 1200: A Com­plete Trans­la­tion, with Intro­duc­tion and Com­men­tary, Brower 1978

TV/movies

Non­fic­tion movies:

  1. , 2016 (review)

Fic­tion:

  1. (review)
  2. (2000)
  3. (review)
  4. (review)

Ani­me:

  1. , sea­sons 1–8 ()
  2. (review)