At Mozilla, we are exploring ways to evaluate public relations programs to determine both short and long term efficacy. How do we know if we’re getting better if there’s no baseline indication of impact? The answer isn’t a pure science. True PR metrics are not just quantitative but qualitative as well. Quantitative analysis explores things like total number of articles, mentions in the press, coverage by country, etc. Qualitative analysis includes message penetration as well as audience and tone analysis. It turns out they are equally important in evaluating the success of Mozilla’s major PR initiatives.
PR is notoriously difficult to quantify. Part of the challenge is developing PR metrics that will be meaningful over time. If PR objectives are constantly shifting, certain programs will undoubtedly get more attention than others. Setting PR metrics is a semi-existential undertaking. Will things that matter today still matter a year from now? Two years? Ten? Fifty?
The first step in setting PR metrics is to look at our overarching PR/Communications goals. From there we can determine a solid methodology for tracking them in the short and long term. The idea is to compare apples to apples.
In the pre-Internet days, every PR professional had a ruler at his or her desk and
would simply count newspaper and magazine column inches. If the number of inches increased quarter over quarter or year over year, it was presumed that the PR team was doing a good job. Apples to apples? Yes. Meaningful? Not really. Print inches are great if every story is glowing but not if the content written in those inches says lousy things about the company/product/executive team/etc.
Quantitative Analysis
Quantitative analysis is decidedly less labor intensive than qualitative analysis and can be broken down into a number of subcategories. This is useful for media analysis in countries where we lack native speakers helping with PR. Much of the analysis can be done using structured query language and complex media databases. There are a number of these media database programs, but the people in the US tend to be most familiar with Lexis Nexis because of its prevalence on college campuses. The resulting analysis offers a breadth of insights about where, when, and on which topics we’re getting coverage.
Examples of quantitative reports we can generate based on media tracking systems in place at Mozilla:
• Number of articles*
• Coverage by topic/keyword*
• Coverage by country*
• Coverage by language*
• Coverage by media type*
• Coverage within a specific date range*
• Overlays of active user data
• Competitive coverage
• Analysis of outreach
o Number of reporters contacted v. number of reporters briefed
o Number of reporters briefed v. number who wrote articles
• Apollo tracking – Mozilla mentions compared to other major technology companies (170,000 Web and print technology articles in 13 countries)
• Firefox market share (
Xiti for Europe,
Net Applications for worldwide)
*Coverage maintained through
Meltwater News System (only online coverage)
Use case for quantitative PR analysis:
The number of articles around Firefox during June 2007 looks very different than the number of articles written during June 2008. This is because Firefox 3 launched in 2008 but there was not a comparable launch happening at the same time a year prior. To make these numbers meaningful, we compared coverage for the Firefox 2 launch to the Firefox 3 launch. This gives us a much better sense of our PR reach over time.
The analysis of Firefox 2 v Firefox 3 coverage shows steady media interest throughout the betas followed by a statistically significant spike in launch day coverage. We know the media covering betas are primarily technology press and that more media outlets write articles about Firefox once it reaches general availability (GA). The graph is in keeping with what we expected to see, but it’s very interesting to see it demonstrated visually.
Qualitative Analysis
Qualitative analysis can be very time and labor intensive, but often covers the things ordinary article counts miss. This type of analysis attempts to answer the following questions:
• Who did these articles reach and were they the target audience for our news?
• What key messages are they going to take away from the article?
• Was the tone of the article positive, negative, or neutral?
Qualitative analysis is much more subjective but covers changes in types of coverage, rather than simply volume. Qualitative offers a “depth” counterpart to the “breadth” that quantitative analysis offers. Through qualitative analysis, we learn more about which messages are resonating with specific audiences. From this, we can uncover future targets for PR outreach.
Examples of qualitative reports we can generate based on media tracking systems in place at Mozilla:
• Key message penetration
• Coverage in top publications: tech, business, consumer
• Tone analysis: positive, negative, neutral
Use case for qualitative PR analysis:
For the Firefox 3 launch, there were literally thousands of articles from around the globe. The sheer volume of coverage was impressive but we wanted to focus on getting answers to the qualitative questions above. It would have taken our entire PR team working around the clock for months on end to fully analyze every Firefox 3 article. Instead, we opted to review non-tech coverage to see which messages non-users were most likely to have encountered during the launch. The analysis is available
here in PDF format.
Upon review, we found that all of the top non-tech publications had positive to neutral articles about Firefox 3. Key messages were picked up throughout. We did not conduct this type of review for Firefox 2 so we do not have a second data point to contrast the launches.
Conclusion
The qualitative and quantitative analyses from the Firefox 3 launch were Mozilla’s first look at how our PR activities relate to press coverage. We long suspected that the structure put in place for mainstream PR outreach and the increase in spokesperson training across the organization would result in a net positive change in press coverage; only we had no way of demonstrating that numerically. Until now…
This post is meant to kick off a conversation about PR metrics. If we’re not analyzing PR activities, it’s hard to tell if we’re improving. Our goal is to post ongoing analysis of how we’re mapping to Mozilla’s PR objectives from both a quantitative and qualitative standpoint. The program is new and we’re starting with only a few data points. We hope to grow over time and generate better analysis to increase the impact of our PR campaigns.
