If you’ve been paying any attention at all to technology trends the past few years, you know that we live in the era of Big Data. All of those videos we upload to YouTube, hard drives we fill with government secrets (or cat photos, take your pick), and tweets we awkwardly punch out on touchscreen keyboards add up to a whole lot of gigabytes, the bulk of which are stored by someone, somewhere, indefinitely.
By some estimates, human beings generate more data every two days than we did in the entire history of civilization prior to 2003 — and that was as of three years ago! Indeed, these are exciting times for data nerds, and data nerds in the arts are no exception. Initiatives such as the Cultural Data Project, Southern Methodist University’s National Center for Arts Research, and the Americans for the Arts National Arts Index seek to collect or organize relevant indicators pertaining to everything from arts organizations’ financial health to audience reach and characteristics to long-term trends for musical instrument purchases.
Fractured Atlas is no stranger to data initiatives in the arts. Our Archipelago data visualization software is one of the largest such efforts, bringing together information on arts nonprofits, for-profits, fiscally sponsored projects, funding, audience distributions, and community context all in one place in the service of better understanding the arts ecosystem in a region. Facilitating data-driven decisions is a major long-term objective of Artful.ly, our just-launched cloud-based arts management tool, and a present-day reality for Spaces, our venue listing and booking service that can promote spaces with last-minute availability to users.
Through our research advisory services work, we’ve helped funders such as ArtsWave organize their entire grantmaking process around principles of data-driven decision-making in order to further their philanthropic objectives. Everyone benefits when funders, organizations and individuals in the arts ecosystem make thoughtful decisions about resource allocation, setting up and responding to incentives, and more. At Fractured Atlas, we believe that data can and should be a crucial input into that thoughtful decision-making process, and we’ve been increasingly vocal in evangelizing for data-driven decision making throughout the arts and cultural sector.
There’s just one problem. Up until now, Fractured Atlas has not had any formal guidelines in place to ensure that we use data in our own decision making, with the result that our internal decisions — relating to management, marketing, strategy, and the like — have been guided primarily by managerial intuition. In a “doctor, heal thyself!” moment, we’ve agreed that is time for our practices to reflect our preaching, at both the program and institutional levels. In 2013, the scope of our operations, the size of the community we serve, and the financial stakes in our work demand informed analysis at a level of rigor that we have not historically practiced. (This directive was immortalized by our fearless leader Adam Huttler in the organization’s annual Strategic Priorities Memo with the colorful title, “Eating Our Data-Driven Dog Food.”)
So between now and next summer, Fractured Atlas is embarking on a pilot initiative to explore how we can use data and evidence to improve our decision-making process at all levels. We’re calling it Fractured Atlas as a Learning Organization, and through this and future blog posts, we’re giving you the opportunity to be a fly on the wall as use this process as a way of grappling with issues of organization identity, strategy, culture, and impact. What Is a Learning Organization?
As I define it*, a learning organization is one for which information and strategy are joined at the hip. It is, quite literally, an organization that has successfully forged a culture of learning and integrated that culture into its decision-making process at all levels. Why is this integration between information and strategy important? Because every organization operates in an environment of uncertainty about what is going to result from its decisions, and every decision we make on behalf of an organization is based on a prediction, whether explicitly articulated or not, about the results of that decision.
If you can reduce the uncertainty associated with your decisions, the chances that you will make the right decision will increase. Of particular interest here are what I call decisions of consequence: dilemmas for which the consequences of making the wrong decision and uncertainty about the nature of the right decision are both high. So, how do you reduce that uncertainty? Why, through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile.
In fact, I subscribe to the notion that research is only valuable insofar as it helps to answer a question that matters. I’m not the only one who thinks so, either: Jake Porway, the founder of a nonprofit that connects data scientists with social enterprises in need, wrote this past spring that “any data scientist worth their salary will tell you that you should start [a data project] with a question, NOT the data.” In fact, all of the excitement around Big Data notwithstanding, data divorced from strategy is not likely to be very useful.
A learning organization solves this problem by forging a powerful feedback loop between information and strategy, with each feeding the other and adapting in relation to the other. The more obvious implication of this symbiosis is that organizational decisions must adapt in response to new information, as discussed above. But the less obvious implication is no less important: information-gathering must be directed by the organization’s decision-making needs. Without that intimate connection, there are no real safeguards to prevent organizations from thinking they are making data-driven decisions without really putting much thought into either the data or the decisions.
More broadly, a learning organization develops a culture of seeking out and using information thoughtfully from the highest levels to the organization’s grassroots. The most effective organizations are conscious about the impact they are trying to achieve, and willing to be open-minded regarding the paths they take to maximizing that impact. *Some readers may be familiar with the term “learning organization” as defined by MIT management scientist Peter Senge in his well-known 1990 book The Fifth Discipline. My use of the phrase is broadly in the same spirit as Senge’s, but he sets out a very specific formula for what constitutes a learning organization that I don’t make use of here. Fractured Atlas as a Learning Organization This fiscal year, which started in September and goes through next summer, we are undertaking a pilot project to put some of these principles into practice. The primary goal of the pilot is to develop a conceptual framework and a toolkit of situation-adaptable methods for reducing uncertainty about decisions of consequence. If we can reduce the uncertainty we have about those decisions through strategic measurement and information-gathering efforts, over time we’re likely to make better decisions that will in turn lead to better outcomes for Fractured Atlas and the people who benefit from our work.
As powerful as this idea is, it only works if we have a very concrete sense of what we’re trying to accomplish as an organization. While we’ve had a mission statement for some time now, the huge variety of programs and services Fractured Atlas offers is virtually impossible to fully capture in a single sentence. Accordingly, the first step in this process is to create a theory of change for every program at the organization, from which we’ll roll up an overall theory of change and logic model for the organization as a whole. This will allow us to define our overall goals as well as some key success metrics at various levels of operation, taking into account both Fractured Atlas’s mission objectives and its focus on developing programs that are sustainable with earned income.
Meanwhile, we’ve formed an internal task force to work on this project at a deeper level of engagement throughout the year. Affectionately called the Data-Driven D.O.G. Force (the “D.O.G.” stands for Data Over Gut), the group will meet every 6–8 weeks to receive calibrated probability assessment training, identify real-world decisions of consequence to use as case studies, and come up with measurement experiments to gather information relevant to those decisions. In doing so, we’ll be using a modified version of a methodology called Applied Information Economics as described in Douglas W. Hubbard’s book How to Measure Anything: Finding the Value of “Intangibles” in Business. One major advantage of AIE is that it explicitly takes into account the cost-benefit of measurement strategies by calculating something called the value of information, which we’ll be exploring further in a future post. At the end, we’ll attempt to formalize a process for identifying decisions of consequence in the future and fitting measurement strategies to the situation at hand.
We’ll also present some recommendations for building infrastructure in the form of ongoing data collection, to address those questions that are likely to be asked again and again. And through it all, I’ll be writing about it here — so that anyone who wants to can learn alongside us. Learning in Context: Why Philosophy Matters as Much as Performance Data-driven decision-making isn’t just about crunching numbers. It’s a practice that requires certain values in order to work. The hard part of being data-driven is not the “data” but the “driven” — you have to be willing to question your assumptions and actually change your behavior in response to the new information coming in.
Put another way, a learning organization is, well, open to learning new things -even things that suggest that the way that we’re currently doing things isn’t working as well as it could, or that we’re missing important opportunities to increase our impact. It’s much easier to attain that kind of open stance if we train ourselves to expect failure upfront. In general, organizations as well as people have a tendency to be far too risk-averse. Being a learning organization means embracing a culture of intentional experimentation and productive failure: we’re likely not going to hit upon the secret sauce the very first time we try something — or, sometimes, at all.
Being a learning organization similarly requires that we think about ourselves from a system perspective — how are we making a difference in light of what everyone else is doing? And how can our experiences shed light on those of others? That’s why we’re not just going down this path on our own and in private. If the specific activities of the pilot project turn out to be a big waste of time (and I can’t guarantee that they won’t), we won’t be able to hide that from you or the world. But even that would ultimately be a good thing — because, in true learning organization fashion, it would cause us to reconsider the limitations of a data-driven approach. Embracing change is hard, but one of the very best things about it is that it can allow us to extract just as much (if not more) value from failure as success. For me, personally, this project is very exciting. Of course I’m eager to find out what we’ll learn. But more than that, Fractured Atlas as a Learning Organization is an opportunity for us to exercise leadership in a way that reaffirms our highest standards for ourselves and for the field. I’m looking forward to sharing our journey with you.
.svg)