Perhaps the hottest lingo in Silicon Valley these days is “big data”. Once upon a time, the only folks who needed to worry about massive, unstructured (or semi-structured) data sets were scientists running weather simulators or investment banks designing high-frequency trading strategies. Not anymore.
Sunday’s NY Times ran an article on the increasing ubiquity of sprawling data sets, and the growing need for analysts who know how to work with them. Yes, some of this is hype. But the underlying trend is an incredibly important one, and it’s not a fad. I’m talking about the rise of data-driven decision-making.
For millions of years, humans have gotten by on our skill at making semi-informed hunches. To be fair, as a species we’re really good at it. We don’t always understand how everything connects, but a combination of intuition, experience, and logic leads us to reasonable conclusions more often than not.
The world we live in, however, is getting more complicated all the time. Just about everything moves faster, and societies and industries are more interconnected every day. At the same time, the amount of “data noise” is growing exponentially. The 2011 Digital Universe Study reports that in 2011, humans created of 1.8 zetabytes of data (1 zetabyte = 1 trillion gigabytes). By 2020, we’ll be producing 50 times that every year. Extracting meaningful signals from that earsplitting noise, and using that information to make smart, informed decisions, is among the most critical challenges of the 21st century.
Needless to say, the arts are not at the vanguard of this trend. Art itself is among the most analysis-resistant of human activities, so perhaps it’s unsurprising that our field tends toward a fondness for intuition and a distrust of cold analysis. Alas, this is a luxury we cannot afford if we want to remain relevant in the future. To thrive in a big data world, funders, policymakers, and even individual arts administrators need more and better information about the context in which their work takes place.
Fractured Atlas has been working on this problem for a couple of years now, on a couple of different fronts. The centerpiece of our strategy is Archipelago, which is a technology platform for cultural sector data aggregation, visualization, reporting, and analysis. Archipelago cross-references data from the IRS, the US Census, the Bureau of Labor Statistics, the Cultural Data Project, mailing list co-ops, Spaces, and original research, among other sources. The result is a multi-faceted tool that can help funders, policymakers, advocates, and even individual organizations get a more comprehensive picture of the cultural assets and activity in a given geographic region. Think of it as Moneyball for the arts!
Of course, we’re not the only ones tackling big data in the arts, nor are we naive enough to think that any single tool can crack this nut. But hopefully we’re doing our part to nudge the field towards a better informed, more analytical future.