In the work I do to evaluate conferences and events, I have put together what I believe is a “neat” way of displaying the main results of an evaluation: an event scorecard. In the evaluation of a conference that occurs every year in Geneva, Switzerland, the LIFT conference, the scorecard summarises both qualitative and quantitative results taken from the survey of attendees. Above you can see a snapshot of the scorecard.

As I have evaluated the conference for three years now, we were also able to show some comparative data as you can see here:

If you are interested, you can view the full scorecard by clicking on the thumbnail image below:

And for the really keen, you can read the full evaluation report of the LIFT08 evaluation report (pdf). 

Greetings from Tashkent, Uzbekistan from where I write this post. I’m here for an evaluation project and off to Bishkek, Kyrgyzstan now.

Glenn

In evaluation, we often make judgements based on “feelings” or “gut reaction” without any proper inquiry or comparison with other data. That is why this story about Ludwig Wittgenstein, the Austrian philosopher appealed to me. Apparently he telephoned a friend in hospital, Fania Pascal, who told the following story:

“I had my tonsils out and was in the Evelyn Nurshing Home feeling sorry for myself. Wittgenstein called. I croaked: “I feel just like a dog that has been run over.” He was disgusted: “You don’t know what a dog that has been run over feels like.”

The point being that Fania Pascal (in the hospital) is making a comparison that she cannot possibly provide any support for - how could she know what it feels like to be a dog that has been run over?

In the same way, you often hear people saying “our results are terrible” - or “we are doing too much of XY”. But my first reaction is “How do you judge that - what are you comparing it to?” - often no real inquiry or comparative data are used (which reminds me of another quote from Groucho Marx).

For those interested, the above quote comes from the book “On Bullshit” by Harry G. Frankfurt - well worth a read.

Glenn

I’ve just spent a week in Armenia and Georgia (pictured above) for an evaluation project where I interviewed people from a cross section of society. These are both fascinating countries, if you ever get the chance to visit… During my work there, I was wondering - what do people think about evaluators? For this type of in-site evaluation, we show up, ask some questions - and leave - and they may never see us again.

From this experience and others I’ve tried to interpret how people see evaluators - and I believe people see us in multiple ways including:

The auditor: you are here to check and control how things are running. Your findings will mean drastic changes for the organisation. Many people see us in this light.

The fixer: you are here to listen to the problems and come up with solutions. You will be instrumental in changing the organisation.

The messenger: you are simply channelling what you hear back to your commissioning organisation. But this is an effective way to pass a message or an opinion to the organisation via a third party.

The researcher: you are interested in knowing what works and what doesn’t. You are looking at what causes what. This is for the greater science and not for anyone in particular.

The tourist: you are simply visiting on a “meet and greet” tour. People don’t really understanding why you are visiting and talking to them.

The teacher: you are here to tell people how to do things better. You listen and then tell them how they can improve.

We may have a clear idea of what we are trying to do as evaluators (e.g. to assess results of programmes and see how they can be improved), but we also have to be aware that people will see us in many different ways and from varied perspectives - which just makes the work more interesting….

Glenn

I’ve written previously about work that others and myself have done on evaluating communication and advocacy campaigns, particulary concerning campaigns that aim for both changes in individual behaviour and government/private sector policies.

In this area, here is an interesting article from the Journal of Multidisciplinary Evaluation, “Advocacy Impact Evaluation” (pdf) by Michael Q. Patton. The article explains how an evaluation was undertaken to evaluate the impact of an advocacy campaign to influence a decision of the US Supreme Court.

What I find interesting is how the evaluation was done - what is called the “General Elimination Method”.

This is where there is an effect (the Supreme Court decision) and an intervention (the advocacy campaign) and they search for connections between the two. They tried to eliminate alternative or rival explanations until the most compelling explanation remained. They did this through interviews, analysis of news, documents and the Court’s decision. The article explains all of this and makes for interesting reading, you can read the article here (pdf).

Glenn

Evaluators often use interviews as a primary tool to collect information. Many guides and books exist on interviewing - but not so many for evaluation projects in particular. Here are some hints on interviewing based on my own experiences:

1. Be prepared: No matter how wide-ranging you would like an interview to be, you should as a minimum note down some subjects you would like to cover or particular questions to be answered. A little bit of structure will make the analysis easier.

2. Determine what is key for you to know: Before starting the interview, you might have a number of subjects to cover. It may be wise to determine what is key for you to know - what are the three to four things you would like to know from every person interviewed? Often you will get side-tracked during an interview and later on going through your notes you may discover that you forgot to ask about a key piece of information.

3. Explain the purpose: Before launching into questions, explain in broad terms the nature of the evaluation project and how the information from the discussion will be used.

4. Take notes as you discuss: Even if it is just the main points. Do not rely on your memory as after you have done several interviews you may mix up some of the responses. Once the interview has concluded try to write further on the main points raised. Of course, recording and then transcribing interviews is recommended but not always possible.

5. Take notes about other matters: It’s important also to note down not only what a person says but how they say it - you need to look out for body language, signs of frustration, enthusiasm, etc. Any points of this nature I would normally note down at the end of my interview notes. This is also important if someone else reads your notes in order for them to understand the context.

6. Don’t offer your own opinion or indicate a bias: Your main role is to gather information and you shouldn’t try to defend a project or enter into a debate with an interviewee. Remember, listening is key!

7. Have interviewees define terms: If someone says “I’n not happy with the situation”, you have understood that they are not happy but not much more. Have them define what they are not happy about. It’s the same if an interviewew says “we need more support”. Ask them to define what they mean by “support”.

8. Ask for clarification, details and examples: Such as “why is that so?”, “can you provide me with an example?”, “can you take me through the steps of that?” etc.

Hope these hints are of use..

Glenn

I’ve put together a series of fact sheets on evaluation and related subjects - mostly inspired by posts I’ve made on this blog. Plus I’ve created two “fun” sheets - on favourite quotes - and excuses for not evaluating:

Fact sheets on evaluation:
Evaluating communication campaigns (pdf)>>
Evaluating networks (pdf)>>
Ten tips for better web surveys (pdf)>>

“Fun” sheets:
Top ten excuses for not evaluating (pdf)>>
Top ten quotes on evaluation (pdf)>>

Glenn

P.S. Those with a sharp eye will notice that these fact/fun sheets are from my new company Owl RE, which offers research and evaluation services in the communications, training/events and development fields.

In evaluation projects, we often feel the strong need to talk to people - to assess a situation or judge a phenomena by surveying or interviewing people. However, this is “manufacturing” data - we are framing questions and then putting them to people - and perhaps in doing so are influencing how they respond.

Alternatively, there is a lot to say for “found” or “natural” data - information that already exists - e.g. media reports, blog posts, conference papers, etc. We often forget about this type of data in our rush to speak to people.

Take this example. I recently saw a paper presenting “current challenges in the PR/communications field”. After surveying PR/comm. professionals, a list of five current challenges were presented by the authors. This is “manufactured” data. An approach using “found” data would be to examine recent PR/comm. conference papers and see what challenges are spoken about - or study the websites of PR/comm. agencies and see what they are presenting as the main challenges.

Another example. Imagine you would like to study the experiences of US troops in Iraq. Of course you could survey and interview military personnel. However, a rich body of data certainly exists online in blog posts, videos and photos from military personnel describing their experiences.

Of course, there are limitations to using “found” data (such as it may present only the views of a select part of a population/phenomena) - but an evaluation project combining both “manufactured” and “found” would certainly make its findings more solid.

Examples of “found” data:

• blog posts
• discussion forums
• websites
• website statistics
• photo/video archives (online or offline)
• media reporting
• conference papers
• policy documents
• records (attendance, participation, complaints, sales, calls, etc.)

If you are interested to read further on this subject, this book “A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research” by David Silverman provides more examples and information on this concept.

Glenn

We often see survey questions with likert scales using frequency estimations such as “sometimes”, “often”, “always”, “never”. etc.  However, these scales often provide inaccurate responses as I’ve written about before.  Why is that so? Well, describing frequency differs enormously from person to person. This dialogue of a couple seeing separate analysts about their relationship, taken from the Woody Allen film “Annie Hall” illustrates this point:

Woody’s analyst: How often do you sleep together?
Woody Allen: Hardly ever! Maybe three times a week
Diane’s analyst: Do you have sex often?
Diane Keaton: Constantly! I’d say three times a week

So ”three times a week” for one person is “constantly” and “hardly ever” for another! This funny dialogue illustrates the point that descriptive terms (e.g. “constantly”)  are not accurate measures of frequency.

In survey questions, one way to avoid this is by not using descriptive terms and asking people directly to provide a numerical estimate, for example:

Inaccurate: How often do you watch TV?
Never, hardly ever, sometimes,  constantly

Accurate: How many hours per week do you watch TV? 
None, under 5, 6-10, 11-15, 16-20, more than 20

Of course, such numeric scales also have problems of accuracy - e.g. people can recall incorrectly how many hours they have spent watching TV.  In this regard, it is better to focus on shorter time periods (”what have you done in the past week” or “what do you do on an average day”).

Glenn   

For those working in the NGO field, measurement and evaluation implicates different issues, often in contradiction:

- Donors, that provide funding for programmes, increasingly ask NGOs to focus on evaluating the impact of their programmes - the long term results;

- At the same time, many donors require an annual feedback from NGOs on the progress of their programmes, which often focuses on outputs - how much was spent and on what;

- NGOs often desire to focus on measuring outcomes - what has been achieved as a result of programmes - as they provide more feedback on what has actually changed than outputs - but can be measured in a shorter time frame than impact (as I’ve written about before);

- NGOs, if they want to provide both a feedback on outputs, outcomes and impact means an increase in administrative overheads for programmes - something which donors are never happy about.

These issues, the potential contradictions and possible solutions are discussed further in this article “Measure what you treasure” (pdf)” from the InterAction Monday Developments journal.

Glenn

Organisations often focus on evaluating the “outputs” of their activities (what they produce) and not on “outcomes” (what their activities actually achieve), as I’ve written about before. Many international organisations and NGOs have now adopted a “results-based management” approach involving the setting of time-bound measurable objectives which aim to focus on outcomes rather than outputs - as outcomes are ultimately a better measure of whether an activity has actually changed anything or not.

Has this approach been successful? A new report from the UN (of their development agency - UNDP) indicates that the focus is still on outputs rather than outcomes as the link between the two is not clear, as they write:

“The attempt to shift monitoring focus from outputs to outcomes failed for several reasons…For projects to contribute to outcomes there needs to be a convincing chain of results or causal path. Despite familiarity with tools such as the logframe, no new methods were developed to help country staff plan and demonstrate these linkages and handle projects collectively towards a common monitorable outcome.”
(p.45)

Interestingly, they highlight the lack of clarity in linking outputs to outcome - to show a causal path between the two. For example, the difficulty in showing that something that I planned for and implemented (e.g. a staff training program - an output) led to a desirable result (e.g. better performance of an organisation - an outcome).

One conclusion we can make from this study is that we do need more tools to help us establish the link between outputs and outcomes - that would certainly be a great advance.

Read the full UN report here >>

Glenn