intelligent measurement

In the work I do to evaluate conferences and events, I have put together what I believe is a “neat” way of displaying the main results of an evaluation: an event scorecard. In the evaluation of a conference that occurs every year in Geneva, Switzerland, the LIFT conference, the scorecard summarises both qualitative and quantitative results taken from the survey of attendees. Above you can see a snapshot of the scorecard.

As I have evaluated the conference for three years now, we were also able to show some comparative data as you can see here:

If you are interested, you can view the full scorecard by clicking on the thumbnail image below:

And for the really keen, you can read the full evaluation report of the LIFT08 evaluation report (pdf). 

Greetings from Tashkent, Uzbekistan from where I write this post. I’m here for an evaluation project and off to Bishkek, Kyrgyzstan now.

Glenn

Evaluators often use interviews as a primary tool to collect information. Many guides and books exist on interviewing - but not so many for evaluation projects in particular. Here are some hints on interviewing based on my own experiences:

1. Be prepared: No matter how wide-ranging you would like an interview to be, you should as a minimum note down some subjects you would like to cover or particular questions to be answered. A little bit of structure will make the analysis easier.

2. Determine what is key for you to know: Before starting the interview, you might have a number of subjects to cover. It may be wise to determine what is key for you to know - what are the three to four things you would like to know from every person interviewed? Often you will get side-tracked during an interview and later on going through your notes you may discover that you forgot to ask about a key piece of information.

3. Explain the purpose: Before launching into questions, explain in broad terms the nature of the evaluation project and how the information from the discussion will be used.

4. Take notes as you discuss: Even if it is just the main points. Do not rely on your memory as after you have done several interviews you may mix up some of the responses. Once the interview has concluded try to write further on the main points raised. Of course, recording and then transcribing interviews is recommended but not always possible.

5. Take notes about other matters: It’s important also to note down not only what a person says but how they say it - you need to look out for body language, signs of frustration, enthusiasm, etc. Any points of this nature I would normally note down at the end of my interview notes. This is also important if someone else reads your notes in order for them to understand the context.

6. Don’t offer your own opinion or indicate a bias: Your main role is to gather information and you shouldn’t try to defend a project or enter into a debate with an interviewee. Remember, listening is key!

7. Have interviewees define terms: If someone says “I’n not happy with the situation”, you have understood that they are not happy but not much more. Have them define what they are not happy about. It’s the same if an interviewew says “we need more support”. Ask them to define what they mean by “support”.

8. Ask for clarification, details and examples: Such as “why is that so?”, “can you provide me with an example?”, “can you take me through the steps of that?” etc.

Hope these hints are of use..

Glenn

For many evaluation projects, an important “deliverable” is the final evaluation report, which contains the findings, conclusions and recommendations of the evaluation. Having been through many evaluations as part of a team or as an individual, I am surprised at how often this important step gets neglected or simply messed up. Following are a couple of recommendations on putting together a final evaluation report:

• Link the findings to the original evaluation questions: Not my own idea, but something I’ve seen others do well - structure the findings of the evaluation around the original questions from the brief that defined the evaluation. In this way, people reading the report can make the connection between the questions asked and what was found out.
• Summarise the key findings in one diagram or table: Aside from reading the executive summary, people often appreciate grasping the key results in one view. Without vulgarising the findings, I find it is useful to sumarise the key findings visually. You can see an example of this idea (called a “snapshot”) on page five of this evaluation report (pdf).
• Separate the recommendations from the findings: Often you see recommendations spread throughout the main body of the report. I find it confusing and believe it is easier to go through recommendations when they are found after the findings (while still making clear reference to the findings).
• Make the executive summary a summary: An executive summary should be just that - a summary. I’m surprised at how many reports actually include new information in their executive summaries that are not found elsewhere in the reports. I recommend summarising the main findings and touching on the recommendations if space allows.
• Include all the details for the really interested and pedantic: There will be a small number of your readers that will love to look further into the details - read all the 1000s of responses to the open questions, study the way the sample was selected, etc. For these readers, I recommend including these details of the evaluation as annexes. These details, such as the survey questions, interview guidelines, description of methodology, further analysis of demographics, existing research consulted, etc. will only strengthen your report and answer some questions for a select group of readers.

Related to this topic, I’ve also written previously about how to ensure that your results are used and how to present monitoring and evaluation results.

And if you want to read further, here are some very comprehensive guidelines from the World Bank on Presenting Results (pdf).

Glenn

Further to my earlier post on ten tips for better web surveys, the email that people receive inviting them to complete an online survey is an important factor in persuading people to complete the survey - or not. Following are some recommended practices and a model email to help you with this task:

1. Explain briefly why you want an input: it’s important that people know why you are asking their opinion or feedback on a given subject. State this clearly at the beginning of you email, e.g. “As a client of XYZ, we would appreciate your feedback on products that you have purchased from us”.  

2. Tell people who you are: it’s important that people know who you are (so they can assess whether they want to contribute or not). Even if you are a marketing firm conducting the research on behalf of a client, this can be stated in the email as a boiler plate message (see example below). In addition, the name and contact details of a “real” person signing off on the email will help.

3. Tell people how long it will take: quite simply, “this survey will take you some 10 minutes to complete”. But don’t underestimate - people do get upset if you tell them it will take 10 minutes and 30 minutes later they are still going through your survey…

4. Make sure your survey link is clickable: often survey softwares generate very long links for individual surveys.  You can often get around this by masking the link, like this “click to go to survey >>“. However, some email systems do not read correctly masked links so you may be better to copy the full link into the email as in the example below. In addition, also send your email invitation to yourself as a test - so you can click on your survey link just to make sure it works…

5. Reassure people about their privacy and confidentiality: people have to be reassured that their personal data and opinions will not be misused. A sentence covering these points should be found in the email text and repeated on the first page of the web survey (also check local legal requirements on this issue).

6. Take care with the ”From”, “To” and “Subject”: If possible, the email address featured in the ”From” field should be a real person. The problem will be if your survey comes from info@wizzbangsurveys.net it may end up in many people’s spam folders.  For the “To”, it should contain an individual email only - we still receive email invitations where we can see 100s of email addresses in the “To” field - it doesn’t really instill confidence as to how your personal data will be used. The “Subject” is important also - you need something short and straight to the point (see example below). Avoid using spam-catching terms such as “win” or “prize”.

7. Keep it short: You often can fall into the trap of over explaining your survey and hiding the link somewhere in the email text or right at the bottom. Try and keep your text brief - most people will decide in seconds if they want to participate or not - and they need to be able to understand why they should, for whom, how long it will take and how (”Where is the survey link?!).

Model email invitation:   

From: j.jones@xyzcompany.net 
To: glenn.oneil@gmail.com
Subject: XYZ Summit 2008 - Seeking your feedback

Dear participant,

On behalf of XYZ, we thank you for your participation in the XYZ Summit.

We would very much appreciate your feedback on the Summit by completing a brief online survey. This survey will take some 10 minutes to complete. All replies are anonymous and will be treated confidentially.

To complete the survey, please click here >>

If this link does not work, please copy and paste the following link into your internet window:
http://optima.benchpoint.com/optima/SurveyPop.aspx?query=view&SurveyID=75&SS=0ZJk1RORb

Thank you in advance; your feedback is very valuable to us.

Kind regards,
J. Jones
Corporate Communications
XYZ Company
email: j.jones@xyzcompany.net 
tel: ++ 1 123 456 789

****
Benchpoint has been commissioned by XYZ to undertak this survey. Please contact Glenn O’Neil of Benchpoint Ltd. if you have any questions: oneil@benchpoint.com

The following article from Quirks Marketing Research Review also contains some good tips on writing email invitations.

Glenn

I recently had a discussion with a colleague about how we should represent the findings of an evaluation study. I am a big fan of using graphs and diagrams to explain the findings - as they say a “picture tells a thousand words”.

But we often see many misuses of graphs and diagrams that can provide an incorrect idea for the reader. I came across an example from a report recently which I have reproduced here:

This is an interesting example of a pictogram or scatter chart to represent two variables: 1) level of opportunity/risk (vertical scale) and 2) size of share value (size of bubbles).

But examing this chart, it made me wonder - what does the horizontal scale represent? In other words, on what basis are the bubbles placed left to right? I cannot see any logical basis in the chart for the horizontal location of the bubbles. I think that’s unfortunate as such a chart could use the horizontal scale to reinforce the share value variable or distribute the bubbles on another basis (e.g. sector of interest).

For those interested in graphic presentation of information, some key texts to read are found on the website of Edward Tufte, a leading specialist in this area.

Glenn

Further to what I wrote last week  about checklists and their use in evaluation, I have found an excellent article on the logic and methodology of checklists.

Dr Michael Scriven of the Evaluation Centre of Western Michigan University describes the different types of  checklists and how good checklists are put together. In particular, I like his list of the seven values of checklists, of which I summarise as follows: 

1. Reduces the chance of forgetting to check something important
2. Are easier for the lay stakeholder to understand and evaluate
3. Reduces the “halo effect”- it forces an evaluator to look at all criteria and not be overwhelmed by one highly valued feature
4. Reduces the influence of the “rorschach effect” - that is the tendancy to see what one wants to see in a mass of data - evaluators have to look at all dimensions
5. Avoids criteria being counted twice or given too much importance
6. Summarises a huge amount of professional knowledge and experience
7. Assists in evaluating what we cannot explain

As Dr Scriven points out, checklists are very useful tools in getting us to think through the “performance criteria” of all kinds of processes, projects or occurences, e.g. what are the key criteria that make a good trainer - and what criteria are more important than other?

Read the full article here >>

Glenn

Often in evaluation, we are asked to evaluate projects and programmes from several different perspectives: the end user, the implementer or that of an external specialist or “expert”.  I always favour the idea that evaluation is representing the *target audiences* point of view - as is often the case in evaluating training or communications programmes - we are trying to explain the effects of a given programme or project on target audiences.  However, often a complementary point of view from an “expert” can be useful.  A simple example - imagine if you making an assessment of a company website - a useful comparison would be comparing the feedback from site visitors with that of an  “expert” who examines the the website and gives his/her opinion.

However, often opinions of “experts” are mixed in with feedback from audiences and comes across as unstructured opinions and impressions. A way of avoiding this is for “experts” to use checklists -  a structured way to assess the overall merit, worth or importance of something.  

Now many would consider checklists as being a simple tool not worthy of discussion. But actually a checklist is often a representation of a huge body of knowledge or experience: e.g. how do you determine and  describe the key criteria for a successful website?

Most checklists used in evaluation are criteria of merit checklists - where a series of criteria are established and given a standard scale (e.g. very poor to excellent) and are weighed equally or not (e.g. one criteria is equal or more crucial than the next one).  Here are several examples where checklists could be useful in evaluation:

• Evaluating an event: you determine “success criteria” for the event and have several experts use a checklist and then compare results.
• Project implementation: a team of evaluators are interviewing staff/partners on how a project is being implemented. The evaluators use a checklist to assess the progress themselves.
• Evaluating services/products: commonly used, where a checklist is used by a selection panel to determine the most appropriate product/services for their needs.

This post by Rick Davies actually got me thinking about this subject and discusses the use of checklists in assessing the functioning of health centres.

Glenn  

When speaking of achieving objectives for a project, I’ve heard a lot of people speak of the “intended impact” and I’ve read quite some “impact reports”. I know it’s a question of language, but often people use the word “impact” when in fact they should use the word “results”. Impact in the evaluation field has a specific application to long term effects of a project. The DAC Glossary of Key Terms in Evaluation and Results Based Management (pdf) produced by the OECD contains the most widely accepted definitions in this field. Impact is defined as:

“Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.

And “results” is defined as

“The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention”.

Consequently  I believe that when we produce a report that shows media visibility generated by a project, this is a short term output and should be called “results” rather than “impact” which applies to more long-term effects.

Glenn

I came across the EvaluationWiki recently which provides interesting background information and resources in all fields of, what else, evaluation. I particularly enjoyed reading the Origins of Evaluation page.

And while speaking of wikis, we shouldn’t forget the newPRwiki’s page on PR measurement for those interested in communcations and evaluation.

Glenn

I read of interest on this post about a network mapping service that has created relational maps about investors, companies and people in the Silicon Valley.

For example, they made a map of the capital links between the three big social networking sites  (Facebook, Friendster & LinkedIn).  You can view more examples here.

What’s interesting is that this is a paying service - to fully access and use the maps - which points out the value that network mapping can have - for analysing complex situations.

Thanks to Kushtrim Xhakli for bringing this to my attention.

Glenn