intelligent measurement

I’ve written previously about work that others and myself have done on evaluating communication campaigns, particulary concerning campaigns that aim for both changes in individual behaviour and government/private sector policies.  In this same direction, a post from the Mission Measurement blog caught my eye on evaluating advocacy campaigns.  They make the very relevant point that although evaluating the impact of advocacy campaigns is difficult - trying to isolate the precise influence on changes being observed - what certainly can be measured is the progress towards the desired change.

They go on to provide some further insight into this issue, by looking at various  measurements undertaken, such as:

• Number of contacts established
• Intermediate changes to knowledge/attitudes
• Measuring progress of change on a continuum
• Bellweather ratings

Read the full post here >>

In the same vein, what I recommend to organisations is to set clear objectives to start with in terms of what is precisely expected from advocacy/campaigning and establish relatively simple “tracking mechanisms” to follow  “progress” on an issue - on a policy level (e..g. number of governments that publicly commit to a given issue) or at an individual level (e.g  number of people who pledge to undertake a given action). Often this information is “known” within an organisation but is not centralised or analysed - making any conclusion on a campaign’s impact difficult.

Glenn

At a meeting of communicators working in the health field, I was asked about some of the methodology concerning measuring the “position” of an organisation. As organisations frequently speak of “re-positioning” themselves, it is perhaps wise to know what is your current “position” before you move anywhere…

My approach is relatively simplistic but drawn from the research of the great thinkers in this field, J. Grunig, C. Fombrun and C. van Riel.  Basically when we speak of an organisation’s “position”, for me I interpret this to mean what attributes we use to describe an organisation (e.g. modern, creative, traditional, research-focused, event-focused, etc.).  This could also be interpreted as the “identity” of the organisation.  

Thus to determine a “position” of an organisation, a good starting point would be to question management of an organisation about what attributes they believe are important for the organisation. Then, the main external target publics can be surveyed on what are the main attributes they attribute to the given organisation. Comparing the view of the management to the main target publics can be interesting as certain “gaps” will emerge between how the management view the organisation…and how the organisation is viewed externally. To go one step further, “positioning” would be determined by looking at how similiar or competing organisation are perceived on the same type of attributes.

Actually, what I describe is the basis for most “identity”, “reputation” or “positioning” studies. There is usually a measure of attributes/values/associations internally and a measure of the same externally. And often done with similar/competing organisations to provide a comparison point.

Some resources on this issue:

The book “Principles of Corporate Communications” by C. van Riel has very good descriptions of the main methodologies in identity measurement (ISDN 0131509969)

The Gauge newsletter discussing reputation measurement (pdf) >>

“Corporate reputations can be measured and managed” by C Fombrun (pdf) >>

“Methodology for assessing corporate values” by J. van Rekom, C. van riel  & B. Wierenga (pdf) >>

And finally, an interesting opinion on the “demise of positioning“.   Certainly some valid points there that I agree with - such as most “re-positioning” campaigns fail - notably because changing a logo, font or slogan doesn’t normally change the behaviour of an organisation - a main influence on “position/brand/identity”. But that’s a whole other issue I’d rather not get into…

Glenn

As I wrote in a previous post, evaluation can be quite frustrating when all your effort and work doesn’t actually lead to any significant change in future practices. Why are evaluations not used? A new report “The Utilisation of Evaluations” (pdf) from ALNAP throws some light on this subject. Although focusing on the humanitarian sector, the report has some points that apply to all types of evaluations. I found interesting the six quality factors the author identifies that contribute to the findings of an evaluation being utilised, notably:

1. Designing carefully the purpose and approach of the evaluation
2. Managing quality participation of all stakeholders throughout the evaluation
3. Allowing enough time to have all relevant staff and stakeholders involved
4. Ensuring that the evidence is credible and the report is easy to read with clear, precise recommendations with who is responsible for what and when
5. Putting in place follow-up plans at the outset
6. Ensuring that the evaluator(s) are credible, balanced and constructive - wholesale negativity is never welcomed

Going through these six factors I can see where I’ve faced obstacles in past evaluations, notably points 2 and 5. I find managing stakeholder involvement is often difficult and so is setting out follow-up plans - it often comes as an after-thought. Certainly some factor to consider for all evaluators…

Read the full report (pdf) here >> 

Glenn

The more I work in the M&E field, the more I see the importance of presenting results in a consumable way. If you are leading an evaluation project, there is nothing more frustrating than finishing your project and finding the comprehensive report you wrote gathering dust on a manager’s desk. 

But that’s what I have learnt, the comprehensive report will perhaps only be read by one or two people of the commissioning team - but the powerpoint summarising the report will be widely distributed and viewed by many.  We may think this is a “dumbing-down” of the work undertaken but it is a reality of how our work is consumed.  Here are some points on presenting results that I find useful:

• Think carefully about the data and findings you want to present. We can often be overwhelmed by data (from survey results for example).  If in doubt, put data you consider important but not essential in report annexes.
• Make the evaluation report attractive and easy to ready - facilitate this by summarising the main points and creating a brief presentation.
• Organise an event such as a staff or team meeting to discuss the results - this could have more impact than the written document.
• Through blogs and wikis, use the evaluation results to generate more discussion and interest in the given subject.  A good example is the blog created to present the results of the 2006 Euroblog survey.  

Jim Macnamara in a recent article (pdf) touches on this subject on how presenting results with a “two-tier” approach is useful - that is, presenting to top management only key data and information while fully digesting all data at the corporate communications level.

Glenn

Cartoon from toothpaste for dinner>>