Lizardman Constant in Surveys

A small fraction of human responses will always be garbage because we are lazy, bored, trolling, or crazy.
bibliography⁠, psychology⁠, sociology⁠, survey
2013-04-122021-06-11 finished certainty: possible importance: 3 backlinks

Researchers have demonstrated repeatedly in human surveys the stylized fact that, far from being an oracule or gold standard, a certain small percentage of human responses will reliably be bullshit: “jokester” or “mischievous responders”, or more memorably, “lizardman constant” responders—respondents who give the wrong answer to simple questions.

Below a certain percentage of responses, for sufficiently rare responses, much or all of responding humans may be lying, lazy, crazy, or maliciously responding and the responses are false. This systematic error seriously undermines attempts to study rare beliefs such as conspiracy theories, and puts bounds on how accurate any single survey can hope to be.

Non-zero base rate of nonsense human answers. To briefly review Scott Alexander’s “lizardman constant”: human survey-takers will, with >0% probability, endorse the most absurd items on a survey, for a mix of reasons like laziness, boredom, humor, sabotage, ignorance, and stupidity. For example, 4% of respondents may endorse the claim ‘lizard-people rule the earth’, 5% of atheists believe in God, and so on. This cautions us against taking survey results about extremely unusual people or traits too literally, or expecting perfectly accurate results, as given the lizardman constant and other crud factors⁠, it is entirely possible that some or all of the outliers may just be the lizardman constant at work.

Why? ‘All of the above’. What drives the lizardman constant, such that a small percentage of people will report in response to survey questions believing that run the world? These people are some unanalyzable mix of lazy, stupid1⁠, ignorant, trolling, ‘joking’, or just ⁠, possibly causing extreme anecdotes which are re-amplified by clickbait global media⁠. ( in falsely reporting being amputees, adoptees, LGBT, binge-drinkers etc, and ⁠.)

Human failures of logic & common sense, examples. One interesting example: 75–96% of the almost-million-strong and thought they had registered as just “independents”. Science examples are well-known to demonstrate severe problems with both knowledge and responses, as and so on; similarly, in arithmetic, ⁠, with <20% of Americans able to identify the even numbers in a list of 6, and ~10% of Americans able to identify which of 6 are prime (equivalent to random guessing—forget about multiplying 4-digit numbers!); & & hold concerning beliefs about the origins of (and children have ⁠, although we should also note that adult naturalist knowledge is nothing to boast about, as 6–8% of Americans think they could kill a grizzly bear, lion, elephant, or gorilla with their bare hands); Pew’s finds 5% of atheists are “absolutely” or “fairly certain” that they believe in God (I’ll charitably pass over meat-eating rates in vegans/vegetarians as a case of “the spirit is willing but the flesh is sweet”); in the ⁠, 14% of undecided voters said Hillary Clinton might be a demon, but they might vote for her; 2% of Clinton supporters said she was & they would; Scott Alexander mentions the eponymous 4% of responders who say lizardmen rule the earth, but, to continue the infernal theme, notes also that 13% say Barack Obama is the Antichrist (5% voted Obama); while (and—the monsters—5% approve of using cell phones in movie theaters); and perhaps trafficking with the ruinous powers explains how and yet are still answering surveys, with satanic deals surely helping the (not that they actually use bleach, ). Such examples could surely be multiplied (literally ad nauseam, even?). Certainly, I (and Scott Alexander) see many bizarre responses any time we are unlucky enough to run a survey which contains a free response field!

Sampling proves presence but not absence in humans too… If challenged on their absurd responses, they will dig their heels in and ⁠. Unfortunately, there is no foolproof remedy against lizardman responses (one can use ‘attention checks’ and tests for overclaiming, like letting them endorse lists of items with fakes thrown in to catch the bullshitters), because humans gonna human. The reality is that humans don’t answer questions reliably, accurately, or honestly even close to 100% of the time, and shamelessly fail ‘common sense’ or ‘logic’ or ‘arithmetic’ questions all the time, requiring extensive precautions, careful survey design, and just throwing out a lot of data as garbage.

  1. And regular people are indeed often fairly incompetent⁠, struggling with basic reading comprehension tasks etc.↩︎