Big Data and Hidden Feelings

In a classical survey or poll, people are simply asked a question and researchers make assumptions based on their answers, but in Seth Stephens-Davidowitz's Big Data speech, he avers that such a system is no longer necessary. Whether intentionally or not, people leave troves of information about themselves online through things like their search histories alone, and Stephens-Davidowitz, a trained data scientist, has made a career from mining these lodes.

One of the big problems with surveys is that people can easily lie. An example Stephens-Davidowitz gives is when people are asked if they'll vote in an upcoming election: beforehand, and overwhelming majority of respondents say they will, but on average only 55 percent of Americans vote.

Data collected from search histories is far less noisy, however. People typically search for things that have meaning to them, and they conduct these searches without fear of judgement (whereas, in the example above, a respondent might lie to make themselves seem more responsible to the pollsters.) This means that pointed analysis of Google search data can lead to insights on human psychology and behavior.