In evaluation, we often make judgements based on “feelings” or “gut reaction” without any proper inquiry or comparison with other data. That is why this story about Ludwig Wittgenstein, the Austrian philosopher appealed to me. Apparently he telephoned a friend in hospital, Fania Pascal, who told the following story:
“I had my tonsils out and was in the Evelyn Nurshing Home feeling sorry for myself. Wittgenstein called. I croaked: “I feel just like a dog that has been run over.” He was disgusted: “You don’t know what a dog that has been run over feels like.”
The point being that Fania Pascal (in the hospital) is making a comparison that she cannot possibly provide any support for - how could she know what it feels like to be a dog that has been run over?
In the same way, you often hear people saying “our results are terrible” - or “we are doing too much of XY”. But my first reaction is “How do you judge that - what are you comparing it to?” - often no real inquiry or comparative data are used (which reminds me of another quote from Groucho Marx).
For those interested, the above quote comes from the book “On Bullshit” by Harry G. Frankfurt - well worth a read.
Glenn
April 28, 2008
I’ve just spent a week in Armenia and Georgia (pictured above) for an evaluation project where I interviewed people from a cross section of society. These are both fascinating countries, if you ever get the chance to visit… During my work there, I was wondering - what do people think about evaluators? For this type of in-site evaluation, we show up, ask some questions - and leave - and they may never see us again.
From this experience and others I’ve tried to interpret how people see evaluators - and I believe people see us in multiple ways including:
The auditor: you are here to check and control how things are running. Your findings will mean drastic changes for the organisation. Many people see us in this light.
The fixer: you are here to listen to the problems and come up with solutions. You will be instrumental in changing the organisation.
The messenger: you are simply channelling what you hear back to your commissioning organisation. But this is an effective way to pass a message or an opinion to the organisation via a third party.
The researcher: you are interested in knowing what works and what doesn’t. You are looking at what causes what. This is for the greater science and not for anyone in particular.
The tourist: you are simply visiting on a “meet and greet” tour. People don’t really understanding why you are visiting and talking to them.
The teacher: you are here to tell people how to do things better. You listen and then tell them how they can improve.
We may have a clear idea of what we are trying to do as evaluators (e.g. to assess results of programmes and see how they can be improved), but we also have to be aware that people will see us in many different ways and from varied perspectives - which just makes the work more interesting….
Glenn
April 21, 2008
In evaluation projects, we often feel the strong need to talk to people - to assess a situation or judge a phenomena by surveying or interviewing people. However, this is “manufacturing” data - we are framing questions and then putting them to people - and perhaps in doing so are influencing how they respond.
Alternatively, there is a lot to say for “found” or “natural” data - information that already exists - e.g. media reports, blog posts, conference papers, etc. We often forget about this type of data in our rush to speak to people.
Take this example. I recently saw a paper presenting “current challenges in the PR/communications field”. After surveying PR/comm. professionals, a list of five current challenges were presented by the authors. This is “manufactured” data. An approach using “found” data would be to examine recent PR/comm. conference papers and see what challenges are spoken about - or study the websites of PR/comm. agencies and see what they are presenting as the main challenges.
Another example. Imagine you would like to study the experiences of US troops in Iraq. Of course you could survey and interview military personnel. However, a rich body of data certainly exists online in blog posts, videos and photos from military personnel describing their experiences.
Of course, there are limitations to using “found” data (such as it may present only the views of a select part of a population/phenomena) - but an evaluation project combining both “manufactured” and “found” would certainly make its findings more solid.
Examples of “found” data:
- blog posts
- discussion forums
- websites
- website statistics
- photo/video archives (online or offline)
- media reporting
- conference papers
- policy documents
- records (attendance, participation, complaints, sales, calls, etc.)
If you are interested to read further on this subject, this book “A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research” by David Silverman provides more examples and information on this concept.
Glenn
March 20, 2008
We often see survey questions with likert scales using frequency estimations such as “sometimes”, “often”, “always”, “never”. etc. However, these scales often provide inaccurate responses as I’ve written about before. Why is that so? Well, describing frequency differs enormously from person to person. This dialogue of a couple seeing separate analysts about their relationship, taken from the Woody Allen film “Annie Hall” illustrates this point:
Woody’s analyst: How often do you sleep together?
Woody Allen: Hardly ever! Maybe three times a week
Diane’s analyst: Do you have sex often?
Diane Keaton: Constantly! I’d say three times a week
So ”three times a week” for one person is “constantly” and “hardly ever” for another! This funny dialogue illustrates the point that descriptive terms (e.g. “constantly”) are not accurate measures of frequency.
In survey questions, one way to avoid this is by not using descriptive terms and asking people directly to provide a numerical estimate, for example:
Inaccurate: How often do you watch TV?
Never, hardly ever, sometimes, constantly
Accurate: How many hours per week do you watch TV?
None, under 5, 6-10, 11-15, 16-20, more than 20
Of course, such numeric scales also have problems of accuracy - e.g. people can recall incorrectly how many hours they have spent watching TV. In this regard, it is better to focus on shorter time periods (”what have you done in the past week” or “what do you do on an average day”).
Glenn
March 11, 2008
For those working in the NGO field, measurement and evaluation implicates different issues, often in contradiction:
- Donors, that provide funding for programmes, increasingly ask NGOs to focus on evaluating the impact of their programmes - the long term results;
- At the same time, many donors require an annual feedback from NGOs on the progress of their programmes, which often focuses on outputs - how much was spent and on what;
- NGOs often desire to focus on measuring outcomes - what has been achieved as a result of programmes - as they provide more feedback on what has actually changed than outputs - but can be measured in a shorter time frame than impact (as I’ve written about before);
- NGOs, if they want to provide both a feedback on outputs, outcomes and impact means an increase in administrative overheads for programmes - something which donors are never happy about.
These issues, the potential contradictions and possible solutions are discussed further in this article “Measure what you treasure” (pdf)” from the InterAction Monday Developments journal.
Glenn
March 3, 2008
Often we attend conferences where one of the stated objectives is “increase/build/create networking” and I always found it odd that there is never any attempt to measure if networking really took place.
A possible solution is to map networks created by participants at conferences - and compare these networks to those that existed before the conferences.
This is exactly what I have done recently in a network mapping study that you can view here (pdf - 1 MB) and the above image is from. From the LIFT conference of 2007, we mapped the networks of 28 participants (out of 450 total participants) before and after the conferences. We found some quite surprising results:
-
These 28 participants had considerable networks prior to the conference - reaching some 30% of all participants.
-
These networks increased after the conference -the 28 people were then connected to some 50% of all participants.
-
Based on the sample of 28 participants, most participants doubled their networks at LIFT07 - e.g. if you went to the conference knowing five people, you would likely meet another five people at the conference - thus doubling your network to ten.
Although this is only a mapping of 28 participants, it provides some insight into conferences and how networks develop - it’s also quite interesting that 28 people can reach 50% (225 people) of the total conference participants in this case.
View the full report here (pdf - 1 MB).
If you are after further information on network mapping, I recommend Rick Davies’ webpage on network mapping. Although it focuses on development projects it contains a lot of useful information on network mapping in general.
Glenn
January 14, 2008
Well I do, actually - granularity is more important than we think for many fields…no, it’s not some sort of breakfast cereal- it’s the size or scale that characterizes an object or activity. And often we see errors made in placing activities at the same level that are not actually at the correct level … An example - I recently noticed a survey that featured the following question:
In which region are you working?
- Sub-Saharan Africa
- North Africa and Middle East
- Europe
- South Asia
- East Asia
- Russia and North Asia
- China
- North America
- Central and South America
- Australasia and Japan
- UK
- Other region
The problem is that the countries and regions mentioned are at different levels - and this is a problem of granularity. “UK” and “China” are not at the same level as “Central and South America” and “South Asia”. This creates problems for people completing the survey - If I live in the UK what do I select? UK or Europe? Both are correct.
In this example, there would be three possible solutions; 1) list all countries of the world using an ISO standard list, 2) list countries applicable to the project (using “other country” for those exceptions that will certainly arise) or 3) use broad, widely accepted regions, such as Europe, Asia, Africa, etc. These solutions resolve the issue of granularity by placing the countries/regions at the same level.
It may seem banal but if these issues are not resolved before the questions are asked, the analysis will prove difficult. This is just one example - granularity is important for many fields such as information management (libraries), website design, software and retail (you never see supermarket aisles marked “vegetables, cereals, bananas” do you?)
Glenn
December 31, 2007
To add to my previous favourite quotes on evaluation and measurement, I have collected the following quotes - enjoy!:
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”
Albert Einstein
“The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question”
Peter Drucker
“One of the great mistakes is to judge policies and programs by their intentions rather than their results”
Milton Friedman
“The pure and simple truth is rarely pure and never simple”
Oscar Wilde
“First get your facts; then you can distort them at your leisure”
Mark Twain
I know that half of my advertising dollars are wasted … I just don’t know which half”
John Wanamaker
Glenn
December 23, 2007
The new Terminal Five at Heathrow Aiport, London - which had quite a controversial birth - is going to undertake an unusual experience in “testing” there facilities before public launch. As I’ve written about before, the aspect of evaluation prior to the launch of a project or activity is often overlooked - and this is an extreme example of this principle in action.
Terminal Five is seeking 15,000 volunteers to act as test users of their new facilities. They will ask volunteers to act as “real” passengers and go through the steps of checking in just to stepping on to the aircraft. I find it fascinating that they will “test” their facilities in such a large-scale manner. Always recommended is “field testing” of new products and services - but this is going quite far. Of course, the question begs, what happens if the user testing brings up major issues with the terminal? Well, Terminal Five lists the following as their aim:
“What are we trying to achieve?: Proof that Terminal 5 is safe, secure and works like clockwork. We’ll also ensure the team who will be running the terminal get the chance to test and develop their service. We also need to identify anything we need to fix prior to opening”
So let’s see what such a trial will bring - how big will the “fixes” be - or will all work like clockwork? Regardless, I am sure they will receive interesting feedback and discover, as I have in usability testing, that people view and use products or facilities in ways totally unanticipated. Do you want to volunteer? Read more here>>
Glenn
December 4, 2007
