intelligent measurement

In evaluation, we often make judgements based on “feelings” or “gut reaction” without any proper inquiry or comparison with other data. That is why this story about Ludwig Wittgenstein, the Austrian philosopher appealed to me. Apparently he telephoned a friend in hospital, Fania Pascal, who told the following story:

“I had my tonsils out and was in the Evelyn Nurshing Home feeling sorry for myself. Wittgenstein called. I croaked: “I feel just like a dog that has been run over.” He was disgusted: “You don’t know what a dog that has been run over feels like.”

The point being that Fania Pascal (in the hospital) is making a comparison that she cannot possibly provide any support for - how could she know what it feels like to be a dog that has been run over?

In the same way, you often hear people saying “our results are terrible” - or “we are doing too much of XY”. But my first reaction is “How do you judge that - what are you comparing it to?” - often no real inquiry or comparative data are used (which reminds me of another quote from Groucho Marx).

For those interested, the above quote comes from the book “On Bullshit” by Harry G. Frankfurt - well worth a read.

Glenn

I’ve just spent a week in Armenia and Georgia (pictured above) for an evaluation project where I interviewed people from a cross section of society. These are both fascinating countries, if you ever get the chance to visit… During my work there, I was wondering - what do people think about evaluators? For this type of in-site evaluation, we show up, ask some questions - and leave - and they may never see us again.

From this experience and others I’ve tried to interpret how people see evaluators - and I believe people see us in multiple ways including:

The auditor: you are here to check and control how things are running. Your findings will mean drastic changes for the organisation. Many people see us in this light.

The fixer: you are here to listen to the problems and come up with solutions. You will be instrumental in changing the organisation.

The messenger: you are simply channelling what you hear back to your commissioning organisation. But this is an effective way to pass a message or an opinion to the organisation via a third party.

The researcher: you are interested in knowing what works and what doesn’t. You are looking at what causes what. This is for the greater science and not for anyone in particular.

The tourist: you are simply visiting on a “meet and greet” tour. People don’t really understanding why you are visiting and talking to them.

The teacher: you are here to tell people how to do things better. You listen and then tell them how they can improve.

We may have a clear idea of what we are trying to do as evaluators (e.g. to assess results of programmes and see how they can be improved), but we also have to be aware that people will see us in many different ways and from varied perspectives - which just makes the work more interesting….

Glenn

I’ve written previously about work that others and myself have done on evaluating communication and advocacy campaigns, particulary concerning campaigns that aim for both changes in individual behaviour and government/private sector policies.

In this area, here is an interesting article from the Journal of Multidisciplinary Evaluation, “Advocacy Impact Evaluation” (pdf) by Michael Q. Patton. The article explains how an evaluation was undertaken to evaluate the impact of an advocacy campaign to influence a decision of the US Supreme Court.

What I find interesting is how the evaluation was done - what is called the “General Elimination Method”.

This is where there is an effect (the Supreme Court decision) and an intervention (the advocacy campaign) and they search for connections between the two. They tried to eliminate alternative or rival explanations until the most compelling explanation remained. They did this through interviews, analysis of news, documents and the Court’s decision. The article explains all of this and makes for interesting reading, you can read the article here (pdf).

Glenn

Evaluators often use interviews as a primary tool to collect information. Many guides and books exist on interviewing - but not so many for evaluation projects in particular. Here are some hints on interviewing based on my own experiences:

1. Be prepared: No matter how wide-ranging you would like an interview to be, you should as a minimum note down some subjects you would like to cover or particular questions to be answered. A little bit of structure will make the analysis easier.

2. Determine what is key for you to know: Before starting the interview, you might have a number of subjects to cover. It may be wise to determine what is key for you to know - what are the three to four things you would like to know from every person interviewed? Often you will get side-tracked during an interview and later on going through your notes you may discover that you forgot to ask about a key piece of information.

3. Explain the purpose: Before launching into questions, explain in broad terms the nature of the evaluation project and how the information from the discussion will be used.

4. Take notes as you discuss: Even if it is just the main points. Do not rely on your memory as after you have done several interviews you may mix up some of the responses. Once the interview has concluded try to write further on the main points raised. Of course, recording and then transcribing interviews is recommended but not always possible.

5. Take notes about other matters: It’s important also to note down not only what a person says but how they say it - you need to look out for body language, signs of frustration, enthusiasm, etc. Any points of this nature I would normally note down at the end of my interview notes. This is also important if someone else reads your notes in order for them to understand the context.

6. Don’t offer your own opinion or indicate a bias: Your main role is to gather information and you shouldn’t try to defend a project or enter into a debate with an interviewee. Remember, listening is key!

7. Have interviewees define terms: If someone says “I’n not happy with the situation”, you have understood that they are not happy but not much more. Have them define what they are not happy about. It’s the same if an interviewew says “we need more support”. Ask them to define what they mean by “support”.

8. Ask for clarification, details and examples: Such as “why is that so?”, “can you provide me with an example?”, “can you take me through the steps of that?” etc.

Hope these hints are of use..

Glenn

I’ve put together a series of fact sheets on evaluation and related subjects - mostly inspired by posts I’ve made on this blog. Plus I’ve created two “fun” sheets - on favourite quotes - and excuses for not evaluating:

Fact sheets on evaluation:
Evaluating communication campaigns (pdf)>>
Evaluating networks (pdf)>>
Ten tips for better web surveys (pdf)>>

“Fun” sheets:
Top ten excuses for not evaluating (pdf)>>
Top ten quotes on evaluation (pdf)>>

Glenn

P.S. Those with a sharp eye will notice that these fact/fun sheets are from my new company Owl RE, which offers research and evaluation services in the communications, training/events and development fields.

In evaluation projects, we often feel the strong need to talk to people - to assess a situation or judge a phenomena by surveying or interviewing people. However, this is “manufacturing” data - we are framing questions and then putting them to people - and perhaps in doing so are influencing how they respond.

Alternatively, there is a lot to say for “found” or “natural” data - information that already exists - e.g. media reports, blog posts, conference papers, etc. We often forget about this type of data in our rush to speak to people.

Take this example. I recently saw a paper presenting “current challenges in the PR/communications field”. After surveying PR/comm. professionals, a list of five current challenges were presented by the authors. This is “manufactured” data. An approach using “found” data would be to examine recent PR/comm. conference papers and see what challenges are spoken about - or study the websites of PR/comm. agencies and see what they are presenting as the main challenges.

Another example. Imagine you would like to study the experiences of US troops in Iraq. Of course you could survey and interview military personnel. However, a rich body of data certainly exists online in blog posts, videos and photos from military personnel describing their experiences.

Of course, there are limitations to using “found” data (such as it may present only the views of a select part of a population/phenomena) - but an evaluation project combining both “manufactured” and “found” would certainly make its findings more solid.

Examples of “found” data:

• blog posts
• discussion forums
• websites
• website statistics
• photo/video archives (online or offline)
• media reporting
• conference papers
• policy documents
• records (attendance, participation, complaints, sales, calls, etc.)

If you are interested to read further on this subject, this book “A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research” by David Silverman provides more examples and information on this concept.

Glenn

We often see survey questions with likert scales using frequency estimations such as “sometimes”, “often”, “always”, “never”. etc.  However, these scales often provide inaccurate responses as I’ve written about before.  Why is that so? Well, describing frequency differs enormously from person to person. This dialogue of a couple seeing separate analysts about their relationship, taken from the Woody Allen film “Annie Hall” illustrates this point:

Woody’s analyst: How often do you sleep together?
Woody Allen: Hardly ever! Maybe three times a week
Diane’s analyst: Do you have sex often?
Diane Keaton: Constantly! I’d say three times a week

So ”three times a week” for one person is “constantly” and “hardly ever” for another! This funny dialogue illustrates the point that descriptive terms (e.g. “constantly”)  are not accurate measures of frequency.

In survey questions, one way to avoid this is by not using descriptive terms and asking people directly to provide a numerical estimate, for example:

Inaccurate: How often do you watch TV?
Never, hardly ever, sometimes,  constantly

Accurate: How many hours per week do you watch TV? 
None, under 5, 6-10, 11-15, 16-20, more than 20

Of course, such numeric scales also have problems of accuracy - e.g. people can recall incorrectly how many hours they have spent watching TV.  In this regard, it is better to focus on shorter time periods (”what have you done in the past week” or “what do you do on an average day”).

Glenn   

For those working in the NGO field, measurement and evaluation implicates different issues, often in contradiction:

- Donors, that provide funding for programmes, increasingly ask NGOs to focus on evaluating the impact of their programmes - the long term results;

- At the same time, many donors require an annual feedback from NGOs on the progress of their programmes, which often focuses on outputs - how much was spent and on what;

- NGOs often desire to focus on measuring outcomes - what has been achieved as a result of programmes - as they provide more feedback on what has actually changed than outputs - but can be measured in a shorter time frame than impact (as I’ve written about before);

- NGOs, if they want to provide both a feedback on outputs, outcomes and impact means an increase in administrative overheads for programmes - something which donors are never happy about.

These issues, the potential contradictions and possible solutions are discussed further in this article “Measure what you treasure” (pdf)” from the InterAction Monday Developments journal.

Glenn

Organisations often focus on evaluating the “outputs” of their activities (what they produce) and not on “outcomes” (what their activities actually achieve), as I’ve written about before. Many international organisations and NGOs have now adopted a “results-based management” approach involving the setting of time-bound measurable objectives which aim to focus on outcomes rather than outputs - as outcomes are ultimately a better measure of whether an activity has actually changed anything or not.

Has this approach been successful? A new report from the UN (of their development agency - UNDP) indicates that the focus is still on outputs rather than outcomes as the link between the two is not clear, as they write:

“The attempt to shift monitoring focus from outputs to outcomes failed for several reasons…For projects to contribute to outcomes there needs to be a convincing chain of results or causal path. Despite familiarity with tools such as the logframe, no new methods were developed to help country staff plan and demonstrate these linkages and handle projects collectively towards a common monitorable outcome.”
(p.45)

Interestingly, they highlight the lack of clarity in linking outputs to outcome - to show a causal path between the two. For example, the difficulty in showing that something that I planned for and implemented (e.g. a staff training program - an output) led to a desirable result (e.g. better performance of an organisation - an outcome).

One conclusion we can make from this study is that we do need more tools to help us establish the link between outputs and outcomes - that would certainly be a great advance.

Read the full UN report here >>

Glenn

For many evaluation projects, an important “deliverable” is the final evaluation report, which contains the findings, conclusions and recommendations of the evaluation. Having been through many evaluations as part of a team or as an individual, I am surprised at how often this important step gets neglected or simply messed up. Following are a couple of recommendations on putting together a final evaluation report:

• Link the findings to the original evaluation questions: Not my own idea, but something I’ve seen others do well - structure the findings of the evaluation around the original questions from the brief that defined the evaluation. In this way, people reading the report can make the connection between the questions asked and what was found out.
• Summarise the key findings in one diagram or table: Aside from reading the executive summary, people often appreciate grasping the key results in one view. Without vulgarising the findings, I find it is useful to sumarise the key findings visually. You can see an example of this idea (called a “snapshot”) on page five of this evaluation report (pdf).
• Separate the recommendations from the findings: Often you see recommendations spread throughout the main body of the report. I find it confusing and believe it is easier to go through recommendations when they are found after the findings (while still making clear reference to the findings).
• Make the executive summary a summary: An executive summary should be just that - a summary. I’m surprised at how many reports actually include new information in their executive summaries that are not found elsewhere in the reports. I recommend summarising the main findings and touching on the recommendations if space allows.
• Include all the details for the really interested and pedantic: There will be a small number of your readers that will love to look further into the details - read all the 1000s of responses to the open questions, study the way the sample was selected, etc. For these readers, I recommend including these details of the evaluation as annexes. These details, such as the survey questions, interview guidelines, description of methodology, further analysis of demographics, existing research consulted, etc. will only strengthen your report and answer some questions for a select group of readers.

Related to this topic, I’ve also written previously about how to ensure that your results are used and how to present monitoring and evaluation results.

And if you want to read further, here are some very comprehensive guidelines from the World Bank on Presenting Results (pdf).

Glenn