Researchers have demonstrated repeatedly in human surveys the stylized fact that, far from being an oracule or gold standard, a certain small percentage of human responses will reliably be bullshit: “jokester” or “mischievous responders”, or more memorably, “lizardman constant” responders—respondents who give the wrong answer to simple questions.
Below a certain percentage of responses, for sufficiently rare responses, much or all of responding humans may be lying, lazy, crazy, or maliciously responding and the responses are false. This systematic error seriously undermines attempts to study rare beliefs such as conspiracy theories, and puts bounds on how accurate any single survey can hope to be.
Non-zero base rate of nonsense human answers. To briefly review Scott Alexander’s “lizardman constant”: human survey-takers will, with >0% probability, endorse the most absurd items on a survey, for a mix of reasons like laziness, boredom, humor, sabotage, ignorance, and stupidity. For example, 4% of respondents may endorse the claim ‘lizard-people rule the earth’, 5% of atheists believe in God, and so on. This cautions us against taking survey results about extremely unusual people or traits too literally, or expecting perfectly accurate results, as given the lizardman constant and other crud factors, it is entirely possible that some or all of the outliers may just be the lizardman constant at work.
Why? ‘All of the above’. What drives the lizardman constant, such that a small percentage of people will report in response to survey questions believing that reptilians run the world? These people are some unanalyzable mix of lazy, stupid1, ignorant, trolling, ‘joking’, or just mentally deviant, possibly causing extreme anecdotes which are re-amplified by clickbait global media. (Teens delight in falsely reporting being amputees, adoptees, LGBT, binge-drinkers etc, and “joke through the entire thing”.)
Human failures of logic & common sense, examples. One interesting example: 75–96% of the almost-million-strong American Independent Party apparently have no idea they joined it and thought they had registered as just “independents”. Science examples are well-known to demonstrate severe problems with both knowledge and responses, as ~25% say geocentrism is true, and 60% that the Big Bang didn’t happen and so on; similarly, in arithmetic, performance can be quite low, with <20% of Americans able to identify the even numbers in a list of 6, and ~10% of Americans able to identify which of 6 are prime (equivalent to random guessing—forget about multiplying 4-digit numbers!); 7% say chocolate milk comes from brown cows & 19% that hamburgers come from pigs & 20% hold concerning beliefs about the origins of fish fingers (and children have more peculiar beliefs, although we should also note that adult naturalist knowledge is nothing to boast about, as 6–8% of Americans think they could kill a grizzly bear, lion, elephant, or gorilla with their bare hands); Pew’s Religious Landscape Study finds 5% of atheists are “absolutely” or “fairly certain” that they believe in God (I’ll charitably pass over meat-eating rates in vegans/
Sampling proves presence but not absence in humans too… If challenged on their absurd responses, they will dig their heels in and give ridiculous justifications. Unfortunately, there is no foolproof remedy against lizardman responses (one can use ‘attention checks’ and tests for overclaiming, like letting them endorse lists of items with fakes thrown in to catch the bullshitters), because humans gonna human. The reality is that humans don’t answer questions reliably, accurately, or honestly even close to 100% of the time, and shamelessly fail ‘common sense’ or ‘logic’ or ‘arithmetic’ questions all the time, requiring extensive precautions, careful survey design, and just throwing out a lot of data as garbage.