Technology Holy Wars are Coordination Problems

Flamewars over platforms & upgrades are so bitter not because people are jerks but because the choice will influence entire ecosystems, benefiting one platform through network effects & avoiding “bitrot” while subtly sabotaging the rest through “bitcreep”.
technology, sociology, Python, insight-porn
2020-06-152020-07-09 finished certainty: likely importance: 3

The endur­ing phe­nom­e­non of ‘holy wars’ in com­put­ing, such as the bit­ter­ness around the pro­longed Python 2 to Python 3 migra­tion, are not due to mere pet­ti­ness or love of con­flict, but because they are a coor­di­na­tion prob­lem: dom­i­nant plat­forms enjoy strong net­work effects, such as reduced ‘bitrot’ as it is reg­u­larly used & main­tained by many users, and can inflict a mir­ror-im­age ‘bit­creep’ on other plat­forms which grad­u­ally are neglected and begin to bitrot because of the dom­i­nant plat­form.

The out­right neg­a­tive effect of bit­creep mean that hold­outs do not just cost early adopters the pos­si­ble net­work effects, they also greatly reduce the value of a given thing, and may cause the early adopters to be actu­ally worse off and more mis­er­able on a daily basis. Given the extent to which hold­outs have ben­e­fited from the com­mu­ni­ty, hold­out behav­ior is per­ceived as par­a­sitic and immoral behav­ior by adopters, while hold­outs in turn deny any moral oblig­a­tion and resent the meth­ods that adopters use to increase adop­tion (such as, in the absence of for­mal con­trols, infor­mal ones like bul­ly­ing).

This des­per­ate need for there to be a vic­tor, and the large tech­ni­cal benefits/costs to those who choose the winning/losing side, explain the (only appar­ent­ly) dis­pro­por­tion­ate ener­gy, ven­om, and intractabil­ity of holy wars.

Per­haps if we explic­itly under­stand holy wars as coor­di­na­tion prob­lems, we can avoid the worst excesses and tap into knowl­edge about the topic to bet­ter man­age things like lan­guage migra­tions.

If there is a com­mu­ni­ca­tion plat­form for nerds, whether it’s , , forums, Twit­ter, Github, Hacker News, Red­dit or what have you, there will surely be holy wars—those intractable peren­nial argu­ments over whether tech­nol­ogy A or B sucks and why users of the other tech­nol­ogy are not just sim­ple-­mind­ed, but socio­pathic, sin­ning against all that is good and pure. The recur­rent ‘holy war’ phe­nom­e­non in com­put­ing is defined in the :

flame wars…The paper by Danny Cohen that pop­u­lar­ized the terms in con­nec­tion with the LSB-first/MSB-first con­tro­versy was enti­tled …Great holy wars of the past have included vs.: , Unix vs.: , vs.: , vs.: , C vs.: , etc. In the year 2003, pop­u­lar favorites of the day are vs., vs., vs. [||]BSD. Hardy peren­ni­als include vs.: …The char­ac­ter­is­tic that dis­tin­guishes holy wars from nor­mal tech­ni­cal dis­putes is that in a holy war most of the par­tic­i­pants spend their time try­ing to pass off per­sonal value choices and cul­tural attach­ments as objec­tive tech­ni­cal eval­u­a­tions. This hap­pens pre­cisely because in a true holy war, the actual sub­stan­tive dif­fer­ences between the sides are rel­a­tively minor.

The silent dog: where holy wars are not. One can prob­a­bly name fur­ther ones. For exam­ple, the great con­sole wars of the 1990s were accom­pa­nied by equally end­less and often acri­mo­nious debates over the mer­its of the SNES vs the Sega Gen­e­sis (then the N64 vs the PS1, the PS2 vs XBox…). Or mov­ing 2 decades ahead, cryp­tocur­ren­cies such as Bit­coin vs Ethereum. We should also note what things aren’t holy wars: the exact choice of string search algo­rithm in a tool like grep, one’s choice of FPS vs RPG video game gen­re, how large a hard drive or how many CPU cores1 to buy… There may be the occa­sional debate over the­se, but they will tend to be lev­el-­headed and pro­duc­tive in a way holy wars aren’t. It’s not mere “brand loy­alty” either—there are count­less brands out there with cults of fol­low­ers but no holy wars, and to the extent they are debat­ed, most shrug and con­clude De gustibus non est dis­putan­dum.

Choices mat­ter. Are holy wars just nerds being nerds? To some extent, sure, argu­ing is often a recre­ational activ­i­ty; but this can’t pos­si­bly be the whole expla­na­tion, because they descend into highly acri­mo­nious flame­wars that no one enjoys and from which every­one comes away embit­tered. And con­tra ESR, the tech­ni­cal dif­fer­ences are often quite sub­stan­tial. (Even using his own exam­ples, ITS vs Unix or C vs Pas­cal or Emacs vs vi, they embody dra­mat­i­cally dif­fer­ent tech­ni­cal philoso­phies.) , com­plain­ing about the “emo­tional dis­tress” used to encour­age upgrades like Python 2→3 which appear to “under­-de­liver” direct ben­e­fits com­men­su­rate with all the Sturm und Drang, is per­plexed by the whole mat­ter, and can come up with no bet­ter expla­na­tion than holy wars com­ing from peo­ple enjoy­ing “suf­fer­ing together” & being “on the moral right side”, which are rad­i­cally inad­e­quate (why suf­fer in the first place?) and he does not enquire any fur­ther, but reveal­ingly notes in pass­ing that “very few projects actu­ally are large enough that a fork by some third party would actu­ally sur­vive”.

Net­work effects. What’s the dif­fer­ence? The dif­fer­ence is that holy wars (much like polit­i­cal debates) are always about plat­forms with net­work effects, and the indi­rect sec­ond-order effects are much more impor­tant than the first-order changes. What OS or pro­gram­ming lan­guage or binary num­ber encod­ing you use mat­ters to other peo­ple. Peo­ple flame over their favorite MMO, but not their favorite Soli­taire; even two neigh­bor­hood kids argu­ing over video game con­soles ben­e­fit if they can agree on which con­sole is best, because then they can eas­ily share games & save files & video game mag­a­zines.

In con­trast, what you do with them mat­ters a lot less. Once you have set­tled on Microsoft Excel rather than Visi­Calc or Lotus-1-2-3 for doing your spread­sheets on, the con­tents of said spread­sheets don’t mat­ter nearly as much to other peo­ple as the fact that you are an Excel user.

(Inter)Network Effects

“Should array indices start at 0 or 1? My com­pro­mise of 0.5 was rejected with­out, I thought, proper con­sid­er­a­tion.”


Grow the net­work, even with dirty tricks. If I use Python 3 and can con­vince every­one else to use Python 3 rather than 2, regard­less of how much the Python 3 improve­ments them­selves actu­ally mat­ter to them or how dis­hon­est my argu­ments are, I am much bet­ter off. The path of a pro­gram­mer who uses the most com­mon lan­guage is a blessed path: libraries abound, bugs pro­gram­mers of lesser lan­guages rou­tinely encounter daily will mys­te­ri­ously fail to occur in rel­a­tively bul­let­proof code­bas­es, doc­u­men­ta­tion always addresses his use-­case with help­ful snip­pets to copy­-­paste, APIs mirac­u­lously return data for­mat­ted in a way con­ve­nient for his default data struc­tures, which their rich fea­ture­ful IDE will tab-­com­plete the code for them, Stack Over­flow will over­flow with tips & tricks for any glitches (which come almost as fun puz­zles, bagatelles between the real work), for all is for the best in this best of all pos­si­ble worlds… There is a large per­sonal and col­lec­tive invest­ment in explicit & (one must even learn to think dif­fer­ently for spe­cific tool­s), but once the price is paid, life is good. While those who ven­ture away from the beaten path quickly dis­cover that how many of their own libraries they will have to write, how under­-spec­i­fied many stan­dards are, how many assump­tions their OS or API makes (the bet­ter to stab them in the back), how few sign­posts there are or fel­lows to ask advice, how quickly the yak shav­ing kills the beast by a thou­sand cuts (and the more dili­gent they are about fil­ing bugs or patch­es, the deeper they are sucked into the mire), and in their dark­est moments, their faith waver as they won­der if per­haps their favorite lan­guage is not the best pro­gram­ming lan­guage in the world.

Not just zero, but neg­a­tive-­sum. Those pos­i­tive net­work effects are clear enough. But one could acknowl­edge them and think they are not ade­quate to drive the fer­vor of a holy war. So let’s con­sider a sub­tler prob­lem, which is how the suc­cess of one plat­form indi­rectly harms other plat­forms by wrest­ing away shared resources.


Always in motion. The phrase bitrot encap­su­lates the prob­lem of net­work effects for such plat­forms. Pro­grams do not bitrot because they changed on disk, or because the mere pas­sage of time ren­dered them buggy (with occa­sional excep­tions like Y2K). Bitrot hap­pens when the rest of the world changes around the pro­gram. APIs change a field name; or the name remains the same but the mean­ing changes; or an obscure func­tion is bro­ken by a change some­where else and no one noticed it until your pro­gram tried to use it as always and it broke; or a ver­sion num­ber changed and some­thing else will no longer work because it assumed the old ver­sion num­ber and is unsure if the new one is safe; or you need a par­tic­u­lar pro­gram which can’t be installed with­out upgrad­ing exist­ing pro­grams (thereby poten­tially break­ing thou­sands of oth­er­s); or a sys­tem backup uses keys which qui­etly expired or ran out of space or was never updated to backup a new server as well. The implic­it­ness of depen­den­cies and sys­tem inter­ac­tions means that there is no way to avoid tak­ing ‘the path of least resis­tance’ because one does not real­ize there is even a path or that one is being locked into any­thing. (“What do you mean, ‘a isn’t always 8 bits’‽”) Bitrot must be actively fought: if you are seri­ous about not build­ing in too many depen­den­cies, you need tricks like , or delib­er­ately crash­ing ser­vices which exceed their expected reli­a­bil­i­ty.

No sil­ver bul­let. Tech­ni­cal fixes like type sys­tems can help reduce bitrot by iden­ti­fy­ing pre­cisely where things have changed dan­ger­ous­ly, but there is always a sys­tem out­side the type sys­tem, and the only truly end-­to-end test is the end-user.3

What gets used, works. What does­n’t… There is no cheap way to avoid bitrot: what gets used gets main­tained. Pro­grams which are in wide use avoid bitrot less because of any intrin­si­cally supe­rior design but because of the ecosys­tem around them: because break­ing changes are imme­di­ately detected by a cut­ting-edge user (spar­ing all the oth­er­s), or the break­ing changes are never made in the first place (be­cause they tested & detected it, or were well-aware of the con­se­quences, or sim­ply because it broke their own sys­tem first!), and because every­one adapts their things to fit the pop­u­lar pro­gram rather than vice ver­sa. Thus con­stantly refreshed and updat­ed, the pro­gram+e­cosys­tem avoids bitrot—­like liv­ing flesh, which avoids rot­ting (de­com­po­si­tion by bac­te­ria etc) by con­stant renew­al.


“We all have strength enough to endure the mis­for­tunes of oth­ers.”

, Maxim #19, Reflec­tions; or Sen­tences and Moral Max­ims

Dying sys­tems ‘bitrot’, grow­ing ones ‘bit­creep’. The flip side of bitrot is what we might call bit­creep: because there is only so much time and energy to go around, a sys­tem which avoids bitrot will also expe­ri­ence ‘bit­creep’, where other pro­grams begin to ‘bitrot’ in that they increas­ingly assume, depend, and are tested only with that sys­tem, and that grad­u­ally creeps through the ecosys­tem. In a sys­tem with heavy bit­creep, get­ting things done in any way other than the dom­i­nant way means thrash­ing around in what is not so much a ‘Tur­ing tarpit’ as a La Brea tarpit, divert­ing new pro­grams towards it. It becomes a black hole, bend­ing space around it; once a pro­gram has built in enough implicit and explicit depen­den­cies, it has passed the event hori­zon, and it can no longer escape (be­cause it would be eas­ier to rewrite the pro­gram from scratch). And because the alter­na­tives are not being used, they are not get­ting main­tained, which means that as bit­creep spreads, any­one inter­act­ing with older pro­grams in a way that used to work will dis­cover their pro­gram has suf­fered bitrot, with­out hav­ing changed at all; all past invest­ments are jeop­ar­dized, rot­ting away invis­i­bly. (This sug­gests there are two evo­lu­tion­ary strate­gies for sys­tems: to be so sim­ple that no one would replace them, and to be so com­plex no one could replace them.)

“Embrace, extend, extin­guish.” An exam­ple might be how has metas­ta­sized in Lin­ux: a boot script alter­na­tive has become involved in every­thing from desk­top envi­ron­ments to audio dae­mons; increas­ing­ly, the only way to do some­thing is via sys­temd. Things which used to be com­part­men­tal­ized and usable with alter­na­tive tools now assume the user uses only sys­temd and tai­lor their capa­bil­i­ties to sys­temd. Increas­ing­ly, root­ing out sys­temd ceases to be pos­si­ble: it sim­ply breaks too many things, even things which osten­si­bly have noth­ing to do with the orig­i­nal pur­pose of boot­ing up Lin­ux. Some­one who dis­cov­ers the hard way that a key script of theirs has been bro­ken by sys­temd, through no fault of their own, and that sys­temd can­not be removed with­out brick­ing their sys­tem and the last OS ver­sion with­out sys­temd was many years ago and would break a dozen other pro­grams and be extremely inse­cure, and who dis­likes sys­temd for other rea­sons, could be for­given for being upset. And for a par­ti­san of sys­temd, it is not enough that sys­temd suc­ceed—others must fail.

One man’s bit­creep is another man’s stan­dard­iza­tion. Bitrot/bitcreep are them­selves qua­si­-neu­tral, nei­ther clearly good nor bad. The rest of the world is con­stantly chang­ing for good rea­sons, and bitrot is an unavoid­able con­se­quence of that; it also changes for bad rea­sons, and that bitrot is bad. Fight­ing bitrot requires a con­stant invest­ment in run­ning a pro­gram, and it may be bet­ter to just let bitrot hap­pen and move on; too much invest­ment in back­wards com­pat­i­bil­ity & legacy sys­tems can be a major mis­take (con­sider Make­files remain­ing tab-sen­si­tive after this was clearly a mis­take because the cre­ator was too wor­ried about hurt­ing his lit­er­ally dozens of users, or the bur­den on Microsoft Win­dows to try to run every­thing back to DOS no mat­ter how bug­gy). Bit­creep is bad when it is forc­ing through an infe­rior tech­nol­o­gy, but it also is just what con­ver­gence and stan­dard­iza­tion looks like. Fight­ing bit­creep requires a con­stant com­mu­ni­ty-wide invest­ment in explor­ing the alter­na­tives and enforc­ing, and it may be bet­ter to just let it hap­pen and move on to more impor­tant things; it is not a bad thing if, say, the def­i­n­i­tion of a byte as 8 bits grad­u­ally bit­creeps through an ecosys­tem, because how use­ful is it really to make the soft­ware ecosys­tem byte-ag­nos­tic?

Platform Life and Death

“All cru­elty springs from weak­ness.”


Total war: best never to begin, but once begun—win! The con­se­quence of the power of bitrot/bitcreep, how­ev­er, is to raise the stakes of choices dra­mat­i­cal­ly, and turn it into a . The more encom­pass­ing a plat­form, the more impor­tant it is to con­vert oth­ers, to not just ben­e­fit from pos­i­tive exter­nal­i­ties, but to avoid bitrot, and a com­peti­tor caus­ing bit­creep. Thus the holy war hier­ar­chy: endi­an-­ness > OS > lan­guage > edi­tor > desk­top… ter­mi­nat­ing in mock­-se­ri­ous debates like tabs vs spaces. (And also why some holy wars end. Peo­ple have ceased to debate Emacs vs vim not because any con­sen­sus was reached or either one dis­ap­peared, but because both are now such exceed­ingly niche edi­tors that they are far more threat­ened by the giants like or or .5) What mat­ters is less the mer­its of the vic­tor than that there is a vic­tor: “Worse Is Bet­ter”—enough JavaScript pro­gram­mers can push a noodle, or Node­JS, around the equa­tor a dozen times.

A bad com­pro­mise makes every­one unhap­py. In the case of Python 2 vs Python 3, both are large com­mu­ni­ties, and could poten­tially go on indef­i­nite­ly. The per­sis­tence of 2 Pythons is a per­pet­ual headache for all Python pro­gram­mers, as it does not just for­feit the ben­e­fits of scale (2 half-­com­mu­ni­ties add up to less than 1 whole-­com­mu­ni­ty), but inflicts com­plex­ity and error on every­one deal­ing with Python, who must make sure they have mul­ti­ple ver­sions safely installed and always use the right one etc. And because of that, Python 2 will enjoy the ben­e­fits of being able to cause bit­creep in other sys­tems, which must work around and cope with Python 2 (rather than vice ver­sa). This is a pas­sive­ly-en­joyed sub­sidy, a ‘rent’, as it were, on the intel­lec­tual prop­erty built by past Python pro­gram­mers (iron­i­cal­ly, many of whom are behind Python 3). Why should the Python 2 users invest in learn­ing all the new idio­syn­crasies of Python 3? The users of Python 2 enjoy the ben­e­fits of a quiet life as their pro­grams do not bitrot—but that comes at the steep triple cost of the oppor­tu­nity cost of the Python 3 changes (so easy to for­get in all this dis­cus­sion of net­work effect­s), of bit­creep, and of divid­ing the com­mu­ni­ty. And almost all of this triple cost is borne by oth­ers rather than the hold­outs! (It would be hard to cal­cu­late, but con­sid­er­ing the imbal­ance between library/tool main­tain­ers and their poten­tially mil­lions of down­stream users, it seems likely that the ratio of cost:ben­e­fits is prob­a­bly imbal­anced by mul­ti­ple orders of mag­ni­tude.) A Python 3 devel­oper could be for­given for being upset.6

War, by other means. Given all this, it’s no won­der that argu­ments might become less than polite or some resort to bul­ly­ing: the stakes can in fact be quite high, and even if not a mat­ter of life & death, make a con­sid­er­able dif­fer­ence to one’s qual­ity of life. Still, such decen­tral­ized unco­or­di­nated flame­wars & bul­ly­ing is a bad idea; if it worked, per­haps it could be jus­ti­fied, but the his­tory of free­lance flame­wars shows that it almost never works and it is merely destruc­tive. Peo­ple should “be nice, at least until you can coor­di­nate mean­ness”.

Explaining Holy Wars

This explains why holy wars exist for some tech­ni­cal things but not oth­ers, why they per­sist, why they are often the most vicious between the most sim­i­lar groups (it is not but prag­matic because they are com­pet­ing for the same scarce resources)7, why they are so intel­lec­tu­ally dis­hon­est and intractable, why par­tic­i­pants feel forced to resort to bul­ly­ing and emo­tional abuse, why they do in fact mat­ter a lot, and how they tend.

This per­spec­tive also sug­gests some ways to reduce the dam­age.

Coordinating Migrations

Such coor­di­na­tion prob­lems are not unique to com­put­ing. No one wants to be the first pen­guin off the ice­berg—but those fish won’t catch them­selves. Instead of ad hoc upgrades, and resort­ing to flame­wars, pro­pa­gan­da, and bul­ly­ing of peo­ple who have dif­fer­ent needs or under­stand­ably sim­ply do the self­ish thing, could there be bet­ter ways?

Appeal to pub­lic-spirit­ed­ness. One bet­ter way would be to just raise aware­ness of the costs. Is an upgrade or alter­na­tive really nec­es­sary? If it really is, then hold­outs like Armin Ronacher should be made aware of the costs they are cre­at­ing in a rea­soned man­ner, instead of mak­ing vague out­raged noises at them.

Eco­nomic incen­tives? Given the need to do an upgrade, another option is to use eco­nom­ics approach­es: cre­ate a fund to pay main­tain­ers to do the tran­si­tion, or spon­sor a fork if need be (which can be returned to the orig­i­nal main­tain­ers once the tran­si­tion is a fait accom­pli and the sta­tus quo now cre­ates bit­creep in the other direc­tion). An might be a use­ful social tech­nol­ogy here (pay­ing main­tain­ers for their upgrade costs is smaller than the total gains and so is ), or per­haps sim­ply a pub­lic com­mit­ment to do the tran­si­tion at a spe­cific date, sim­i­lar to Python 2 being EOLed.

Delib­er­ately cen­tral­ize to solve coor­di­na­tion prob­lems. From the per­spec­tive of bit­creep, attempts at mak­ing peace by bend­ing over back­wards to improve mod­u­lar­iza­tion and try to sup­port mul­ti­ple ecosys­tems in par­al­lel are noth­ing but self­-s­ab­o­tag­ing fol­ly: find­ing a ‘com­mand­ing height’ to do the tran­si­tion may be the sin­gle most effec­tive strat­e­gy, and it self­-ex­e­cutes by default. (Such a strat­egy may seem under­hand­ed, but in coor­di­na­tion prob­lems, kind­ness is cru­el­ty.) Indeed, per­haps we should recon­ceive the real role of a “” or main­tainer as not being ordi­nary every­day project main­te­nance or orga­niz­ing, but sim­ply serv­ing as a final author­ity to force the com­mu­nity into a new bet­ter equi­lib­ri­um—a role the more valu­able the more rarely exer­cised.

See Also

  1. As opposed to which CPU brand or archi­tec­ture.↩︎

  2. As quoted in Expert C Pro­gram­ming: Deep C Secrets, Peter Van der Lin­den 1994 (while this is the ear­li­est I found, Van der Lin­den says he does not know where it came from, sug­gest­ing it was already folk­lore by 1994). ↩︎

  3. A corol­lary here is that a sys­tem which is also cre­ated end-­to-end, like a deep learn­ing sys­tem, may be able to reduce bitrot. In “Soft­ware 2.0”, con­ven­tional code like Ten­sor­flow (it­self noto­ri­ously bitrot prone) is min­i­mized as much as pos­si­ble in favor of learn­ing all func­tion­al­i­ty. Con­sider a NN like when used for code com­ple­tion or spread­sheets: even if the text for­mat­ting changes, which would imme­di­ately bitrot a con­ven­tional soft­ware sys­tem reliant on brit­tle reg­exps or parsers, they can be given exam­ples to do few-shot learn­ing () & smoothly adapt on the fly to “do what I mean”, and can retrain on future data with­out human inter­ven­tion. As the ulti­mate input/output pairs change, it can retrain itself end-­to-end.↩︎

  4. Omnis enim ex infir­mi­tate fer­i­tas est.; from , ↩︎

  5. Indeed, vi/vim users are now nat­ural allies of Emacs users sim­ply because they both are threat­ened by creep­ing mono­lithic IDEs ren­der­ing it impos­si­ble for third-­party edi­tors to effec­tively inte­grate with com­pil­ers etc, and if Emacs can still inter­op­er­ate with some­thing suc­cess­ful­ly, then so too can vim (see eg the ).↩︎

  6. Ear­ly-adopters & com­put­ing pio­neers are our bridge to the future—be­cause we knife them in the back by tak­ing their work with­out giv­ing, and then step across the muddy ditch on their bod­ies.↩︎

  7. Eg a Scheme Lisp pro­gram­mer vs a Com­mon Lisp pro­gram­mer: if one Lisp pro­gram­mer decides to change his pre­ferred lan­guage, he’s prob­a­bly not going to switch to Java!↩︎