Advocacy evaluation – top resources
Today I spoke to the students of the Executive Certificate of Advocacy in International Affairs at the Graduate Institute of Geneva on advocacy evaluation. I promised the students to list the top resources I’d recommend on advocacy evaluation, here they are:
Practical Guide to Advocacy Evaluation from Innovation Network (pdf)>>
Guide on measuring advocacy and policy (pdf) from the Annie E. Casey Foundation
“A guide to monitoring and evaluating policy influence (pdf)” of the UK-based Overseas Development Institute describes the different approaches to evaluating policy influence.
“Advocacy Impact Evaluation” (pdf) by Michael Q. Patton – an interesting case study on influencing the US Supreme Court.
“Lessons in Evaluating Communication Campaigns: Five Case Studies” from the Harvard Family Research Project looks at evaluating advocacy campaigns ranging from gun safety to emmissions (ozone) reduction.
A pragmatic guide to monitoring and evaluating research communications using digital tools
The approach taken relates online measurement tools to four levels of assessing influence of communications on policy (an aim of research communications):
- Management, outputs, uptake, outcomes and impact.
The last level, outcomes and impact is of course the hardest to measure with digital tools. But I think if you have access to your target audiences, this can be done through in-depth interviews or more simply through email surveys to ask how they have used the research products – which can give then provide an indication of the role they have taken in influencing policy.
Can media coverage predict a company’s reputation?
The extent of influence of media coverage on public opinion is a hotly debated topic. For communication professionals, their intepretation of how influential the media is can influence how much effort they put into media relations compared to other activities. Here is an interesting case study (pdf) that shows the link between media coverage and company reputation (or more precisely people’s opinion of a company).
The case study (Winner of the 2011 Jack Felton Golden Ruler Award for excellence in research, measurement and evaluation), is on Toyota and “uses a mathematical model of the impact of persuasive information on opinion formation to show how Toyota’s corporate reputation, as measured by surveys, can be directly predicted by document sentiment (i.e. media coverage)”.
The study also makes comes to some interesting findings that have implications for wider media relations and monitoring:
- Information favorable to Toyota is about twice as persuasive as unfavorable information.
- Blogs appear to be a leading indicator of negative issues, yet have limited impact on Toyota’s corporate reputation at the national level.
- The research suggests that any representative sample of media outlets can be used to gauge opinion, and that automated sentiment scoring is sufficient.
The study concludes with some implications for communication practice in general:
“Quite simply, the model demonstrates that media relations works. Persuasive information in the media drives opinion formation. The model is truly predictive – not just a correlation – in the sense that this week’s media results are used to reliably predict next week’s corporate brand reputation”
Likert scale & surveys – best practices – 2
i’ve written previously about the Likert scale and surveys – and received literally 100s of enquiries about it. A reader has now pointed me towards this excellent article on survey questions and Likert scales that adds some interesting points to the discussion.
From my previous post, I listed the following best practices on using the Likert Scale in survey questions:
- More than seven points on a scale are too much.
- Numbered scales are difficult for people
- Labelled scales need to be as accurate as possible
And here are some further points to add drawn from this article:
- Be careful with the choice of words for labels:
“Occasionally” has been found to be very different than “seldom” but relatively close in meaning to “sometimes” (quote from article)
- Include a “don’t know” if for a point where people may simply not have an opinion:
“Providing a “don’t know” choice significantly reduced the number of meaningless responses.”
- People will respond more often to those items on the left hand side of the scale:
“There is evidence of a bias towards the left side of the scale”
On that last point, I always write my scales left to right – bad to good… This means that people may tend to select more easily the “bad” ratings. I haven’t found that to be the case (respondents often seem to be over-positive in their ratings I feel), but I stand corrected…
outcome-focused event evaluation
Here is an interesting post on conference/event evaluation from the IDS impact and learning blog.
Interesting in that they propose to go beyond evaluating just participants’ reactions and learnings from events to consider also the impact on organisers and the contributors (speakers, etc.).
Read the full post here>>
Progress on Setting Social Media Measurement Standards
There is an ongoing debate about setting standards for social media measurement. Here is an interesting contribution from the Measurement Standard that reports on a recent meeting of specialists in the US organised by PR measurement guru K.D Payne.
One conclusion which I agree with is the need for the so-called “Generally Accepted Accounting Practices (GAAP)” for social media reporting, particularly in defining the following areas:
- Content
- Reach /Engagement
- Influence/Relevancy
- Sentiment/Advocacy
- Impact and Value
Read more on the Measurement Standard >>
Conference evaluation: mapping success stories
Global standards for social media measurement?
Three leading leading communications associations have formed a new coalition to define, develop and promote industry-leading standards for social media measurement. The coalition is comprised of the International Association for Measurement and Evaluation of Communication (AMEC), Council of PR Firms and the Institute for Public Relations.
Coalition chair, Tim Marklein commented:
“Too many of the social media measurement options available today are based on proprietary tools and methods that can’t be easily replicated across brands, campaigns and organizations. As a coalition, we intend to break down those barriers and map out a path to standards that address key social media measurement challenges, including content sourcing, influence, sentiment, engagement and ROI among others.”
We couldn’t agree more. It will be interesting to see what the coalition comes up with as they move now into the consulting phase.
Measuring the impact of NGO programmes
An ongoing debate focuses on how NGOs can measure the impact of their work. The International Initiative for Impact Evaluation (3ie) and Oxfam have recently produced a very interesting paper on this subject:
Using examples from campaigns and other programmes, the paper sets out the challenges and options in evaluating impact and proposes four options for improving impact evaluation:
1) partnering with research institutions to rigorously evaluate “strategic” interventions;
2) pursuing more evidence informed programming;
3) using what evaluation resources they do have more effectively;
4) making modest investments in additional impact evaluation capacity.
View the paper (pdf)
EU Manual: Evaluating Legislation and Non-Spending Interventions in the Area of Information Society and Media
A very interesting manual published by the European Union:
Despite the wordy title, the manual is really about how to evaluate the effects of legislation and initiatives taken by governments (in this case the regional body – EU).
The toolbox at page 72 is well worth a look.
