Dowsing Alpha is a blog about the effects of critical thinking on our lives. Critical thinking comes in many forms. From p-hacking¹ to a replication crisis² in the sciences to the under-performance of mutual funds³, critical thinking today is more relevant than ever (and sorely needed). For every generation save the last three, it has been possible to get away with an unseemly amount of fudging, deceit, and editorializing without the awareness of the public. These measures served their purpose in a bygone era, but they now fail to hold up to the scrutiny that characterizes the modern age of over-information.
Blogs, like the one you’re reading, are one symptom of the overarching theme of history’s progress toward openness of information. If knowledge was once considered power, it is now considered a mixed bag. How did this come to be?
The age of over-information
It wasn’t until April of 1993 that the “switch was flipped¹” and the internet as we know it was born. It was not as though the populace were in the dark, of course. However, like the printing press and the atomic bomb, we now associate our newfound creation with an age of its own – the “information age.” This has created numerous issues, of which many have waxed poetic in the time since.
One of the major problems that has arisen as a consequence is the division between “big data” and “small data.” Big data, for all the glamour that has accompanied its rise, has fundamental flaws at its roots that cannot be corrected. The first is the problem of causality. Small data gathering, through cause and effect and anecdotal intuition, allows for specific evidence gathering. If I eat, I get full. If I leave a small tip, the ungrateful worker spits in my food. Observation begets information when it comes to small data. Big data doesn’t allow for this. On the contrary, laymen often roll their eyes at justifications for absurdly simplistic reasoning spelled out in volumes because academics fail to let the power of big data yield to common sense observation. Statistically, LeBron James may never get on a “hot streak,” but anyone who watched Game 6 of the Heat vs. Celtics on June 7, 2012 would beg to differ.
The paradox of data doesn’t end there, though. We now live in an age where the past few years have spawned more data than was previously produced in the entirety of mankind. We are heavy on noise, and light on signal. Often, small data is the signal. Small data has its limitations, of course. This much is obvious. Like our ancestors who looked out the window to get the week’s weather forecast, those who deny the power of data are doomed to hopeless ineptitude. However, data is too often used as a justification for intentional blindness to failing ideas or movement in the face of countervailing winds, as the examples below point out.
Misadventures of markets