It could be easy for a man who catapulted to fame with a 50-for-50 election forecast to become overconfident. But statistical savant Nate Silver, who spoke at a packed Statler Auditorium Friday, impressed that modesty goes a long way in the field of big data.
Stepping out from behind the podium, Silver — named one of the 100 most influential people in the world by Time in 2009 — introduced himself as a simple man.
“I’m from the Midwest. We don’t deal with that much praise or technology,” he said, gesturing at the screen behind him, which briefly showed a panned-in shot of his face.
Despite his self-described humble beginnings, Silver has seen praise heaped upon him since he accurately predicted the winner of all 50 states in the 2012 U.S. presidential election. While some skeptics have questioned Silver’s methodology, the 35-year-old writer-statistician has been described by media outlets as “America’s Chief Wizard,” a “mathematical wunderkind” and a pollster genius — something that has not been lost on him.
“It seemed really out of whack to me that a Google search for my name resulted in more hits than searches for Vice President Joe Biden — humble-brag! I had to wonder, why is a data geek getting this much love?” Silver said. “Fortunately for America, Justin Bieber still has 100 times more traffic than Vice President Biden and I,” he added.
Silver said the election forecast model on his New York Times blog, FiveThirtyEight, is relatively simple: its methodology is comprised of “averaging polls, counting to 270 and account for the margin of error.”
“It’s not as complicated as things used in other branches of economics,” he said. “But the success of [my book,] The Signal and the Noise, spoke to how many problems we have in the field of big data. Expectations are not matching reality.”
For instance, the explosion of data available for analysis has led some journalists to believe that the scientific method has been rendered obsolete, Silver said. The housing bubble bursting, the Tohoku earthquake in 2011, flu epidemics — all of these events, he said, are examples of catastrophes scientists and economists failed to predict, despite having masses of data available to analyze.
“This is more the rule than the exception,” Silver said, adding that statisticians, economists and scientists have made faulty decisions throughout the past decade because they have made bad assumptions about data. “It hasn’t exactly been the best decade for [big data].”
Even in rigorous fields like medicine, some studies have suggested that most published research findings are false, Silver said.
“We have a crisis of science, ironically, in this era of big data,” Silver said. “Why isn’t big data producing big progress?”
The answer, Silver said, lies in the unequal growth of information and analysis. By some measures, 90 percent of all data in the world has been created in the past two years, he added.
“People sometimes have difficulty picking out the reality, or signal, from the noise,” Silver said, referencing the title of his book. “As you have more and more data, you have an exponential increase in the number of two-way relationships you have to test.”
The result?
“There is a widening gap between what we really know and what we think we know,” Silver said.
In one study, a Princeton University professor showed six graphs of the Dow Jones Industrial Average’s performance to participants. Wall Street analysts were unable to recognize that some of the graphs were fake — an example of how people can be led astray and think they see patterns in random streams of data, according to Silver.
“By the way, those ones were real,” Silver said, pointing to two of the graphs on the screen behind him. “If you’re good, you should apply to CNBC instead of completing your education at Cornell.”
At other times, people who are flooded with data can miss pertinent pieces of information and reach false conclusions, Silver said.
“It’s like the apartment I just bought — I thought it was underpriced, but it turns out, it was the neighbors … for reasons I will not say,” he said.
Despite there being countless examples documenting the tendency of politicians, economists and even statisticians to be overconfident with data, “there is happiness in the end … potentially,” Silver said.
“If you’re willing to test your ideas through making actually verifiable predictions … and [to] revise your beliefs — people tend to be very stubborn — there is progress in the end,” Silver said. “We’ll always have technological developments to help us, but we also need smart people who know how to … manipulate big data and produce benefits for society, rather than just for our personal lives.”
Original Author: Akane Otani