Practical Guide to Advocacy Evaluation from Innovation Network (pdf)>>

Guide on measuring advocacy and policy (pdf) from the Annie E. Casey Foundation

“A guide to monitoring and evaluating policy influence (pdf)” of the UK-based Overseas Development Institute describes the different approaches to evaluating policy influence.

“Advocacy Impact Evaluation” (pdf) by Michael Q. Patton – an interesting case study on influencing the US Supreme Court.

“Lessons in Evaluating Communication Campaigns: Five Case Studies” from the Harvard Family Research Project looks at evaluating advocacy campaigns ranging from gun safety to emmissions (ozone) reduction.

The approach taken relates online measurement tools to four levels of assessing influence of communications on policy (an aim of research communications):

• Management, outputs, uptake, outcomes and impact.

The last level, outcomes and impact is of course the hardest to measure with digital tools. But I think if you have access to your target audiences, this can be done through in-depth interviews or more simply through email surveys to ask how they have used the research products  – which can give then provide an indication of the role they have taken in influencing policy.

View the full post here>>

The case study (Winner of the 2011 Jack Felton Golden Ruler Award for excellence in research, measurement and evaluation),  is on Toyota and “uses a mathematical model of the impact of persuasive information on opinion formation to show how Toyota’s corporate reputation, as measured by surveys, can be directly predicted by document sentiment (i.e. media coverage)”.

The study also makes comes to some interesting findings that have implications for wider media relations and monitoring:

• Information favorable to Toyota is about twice as persuasive as unfavorable information.
• Blogs appear to be a leading indicator of negative issues, yet have limited impact on Toyota’s corporate reputation at the national level.
• The research suggests that any representative sample of media outlets can be used to gauge opinion, and that automated sentiment scoring is sufficient.

The study concludes with some implications for communication practice in general:

“Quite simply, the model demonstrates that media relations works. Persuasive information in the media drives opinion formation. The model is truly predictive – not just a correlation – in the sense that this week’s media results are used to reliably predict next week’s corporate brand reputation”

Read the full case study here (pdf)>>

From my previous post, I listed the following best practices on using the Likert Scale in survey questions:

• More than seven points on a scale are too much.
• Numbered scales are difficult for people
• Labelled scales need to be as accurate as possible

And here are some further points to add drawn from this article:

• Be careful with the choice of words for labels:

“Occasionally” has been found to be very different than “seldom” but relatively close in meaning to “sometimes” (quote from article)

• Include a “don’t know” if for a point where people may simply not have an opinion:

“Providing a “don’t know” choice significantly reduced the number of meaningless responses.”

• People will respond more often to those items on the left hand side of the scale:

“There is evidence of a bias towards the left side of the scale”

On that last point, I always write my scales left to right – bad to good… This means that people may tend to select more easily the “bad” ratings. I haven’t found that to be the case (respondents often seem to be over-positive in their ratings I feel), but I stand corrected…

View the full article here>>

Interesting in that they propose to go beyond evaluating just participants’ reactions and learnings from events to consider also the impact on organisers and the contributors (speakers, etc.).

Read the full post here>>

One conclusion which I agree with is the need for the so-called  “Generally Accepted Accounting Practices (GAAP)”  for social media reporting, particularly in defining the following areas:

• Content
• Reach /Engagement
• Influence/Relevancy
• Sentiment/Advocacy
• Impact and Value

Read more on the Measurement Standard >>

Coalition chair, Tim  Marklein commented:

“Too many of the social media measurement options available today are based on proprietary tools and methods that can’t be easily replicated across brands, campaigns and organizations. As a coalition, we intend to break down those barriers and map out a path to standards that address key social media measurement challenges, including content sourcing, influence, sentiment, engagement and ROI among others.”

We couldn’t agree more.  It will be interesting to see what the coalition comes up with as they move now into the consulting phase.

Read more on the AMEC website>>

Can we obtain the required rigour without randomisation? Oxfam GB’s non-experimental Global Performance Framework (pdf)

Using examples from campaigns and other programmes, the paper sets out the challenges and options in evaluating impact and proposes four options for improving impact evaluation:

1) partnering with research institutions to rigorously evaluate “strategic” interventions;
2) pursuing more evidence informed programming;
3) using what evaluation resources they do have more effectively;
4) making modest investments in additional impact evaluation capacity.

View the paper (pdf)

“Evaluating Legislation and Non-Spending Interventions in the Area of Information Society and MediaSociety and Media (pdf)”

Despite the wordy title, the manual is really about how to evaluate the effects of legislation and initiatives taken by governments (in this case the regional body – EU).

The toolbox at page 72 is well worth a look.

View the manual here (pdf)>>