Whither the Time Machine? Considering the Counterfactual in Arts Marketing

(Cross-posted from the National Arts Marketing Project Conference Blog Salon over at ARTSBlog.) The hardest question to answer in arts research is “what would have happened if we had done things differently?” Researchers call this question the “counterfactual,” since it refers to a scenario that doesn’t actually exist. Generally speaking, it’s hard to measure things that don’t exist; hence the difficulty for arts research.

We can’t measure that scenario directly, but we can get close to it through experimental designs that include a control group. In a marketing-specific context, counterfactual scenarios come into play when considering alternative strategies aimed at driving sales or conversions. One technique that a number of organizations have used is called A/B testing, which is when two different versions of, say, a newsletter or a website get sent to random segments of your target audience. Internet technology makes A/B testing relatively painless to execute: in the case of a newsletter, for example, all it requires is a random sorting algorithm in Excel to divide the list in two before sending the slightly different newsletter versions to the lists as you normally would. You could test which design results in more clickthroughs to a specific link or which subject line results in a higher open rate. By creating an A group and a B group, you are finding a way to test the counterfactual without the use of a time machine to go back and try things a different way. Assuming the groups truly are random and the sample size isn’t tiny, it’s a really great way of getting reliable information on what you’re doing. A/B testing is not the only way of pursuing this kind of inquiry, however. Sometimes it’s not that easy to simply divide your target audience into two. What about testing the effectiveness of newspaper advertising for a musical? The ad is either in the paper or it’s not. You can’t control who sees it and who doesn’t. You could try putting an ad in for one production but not for the next and compare ticket sales between the two, of course, but that is vulnerable to all sorts of noise like differences between the two productions, the time of year, etc. You could even try putting the ad in once the show has already opened and examine the before-and-after trendline for ticket sales, but you don’t know in that instance whether tickets would have risen anyway because of good word-of-mouth and/or good reviews. It’s also impossible to distinguish in that scenario between the effect of having an ad at all and the effect of the specific content and design of the ad. As you can see, evaluating the effectiveness of a newspaper ad is a much harder problem than evaluating which photo in your newsletter is a better click-driver. One way you could go about it would be to reach out to random people — literally — and ask them whether they read the paper, and whether they’ve heard of the show. The people who don’t read the paper serve as your de facto control group for whether the advertising has any effect. Ideally you would do this twice — the day before and the day after the ad runs. Normally, you would want to test the same group before and after, but since you’re messing with the results by making survey participants aware of the show, you might have to conduct the test with two different random samples. You’ll probably want to know how often they see comparable shows as well, including from your own company, so that you can control for how likely they were to be on the lookout for your show. (That step should also help account for differences between newspaper readers and non-readers.) This option is more expensive since you’ll probably have to hire a firm to make some phone calls for you, but for a larger institution the knowledge gained will likely be well worth the expense. Plus, you could offer each participant a voucher to see the show for free or at a steep discount once they’ve completed the survey, turning this into an audience development tool at the same time. Another question you might be interested in exploring is whether the ad increases ticket sales, not just awareness. For this one you’re probably better off creating a controlled environment in which one group is shown the ad you’ve chosen to run, and the other group is shown an ad for a different theater company, or given some neutral activity. Ask them before and afterwards whether they are interested in purchasing a ticket, and you’re on your way.