intelligent measurement

For many evaluation projects, an important “deliverable” is the final evaluation report, which contains the findings, conclusions and recommendations of the evaluation. Having been through many evaluations as part of a team or as an individual, I am surprised at how often this important step gets neglected or simply messed up. Following are a couple of recommendations on putting together a final evaluation report:

• Link the findings to the original evaluation questions: Not my own idea, but something I’ve seen others do well - structure the findings of the evaluation around the original questions from the brief that defined the evaluation. In this way, people reading the report can make the connection between the questions asked and what was found out.
• Summarise the key findings in one diagram or table: Aside from reading the executive summary, people often appreciate grasping the key results in one view. Without vulgarising the findings, I find it is useful to sumarise the key findings visually. You can see an example of this idea (called a “snapshot”) on page five of this evaluation report (pdf).
• Separate the recommendations from the findings: Often you see recommendations spread throughout the main body of the report. I find it confusing and believe it is easier to go through recommendations when they are found after the findings (while still making clear reference to the findings).
• Make the executive summary a summary: An executive summary should be just that - a summary. I’m surprised at how many reports actually include new information in their executive summaries that are not found elsewhere in the reports. I recommend summarising the main findings and touching on the recommendations if space allows.
• Include all the details for the really interested and pedantic: There will be a small number of your readers that will love to look further into the details - read all the 1000s of responses to the open questions, study the way the sample was selected, etc. For these readers, I recommend including these details of the evaluation as annexes. These details, such as the survey questions, interview guidelines, description of methodology, further analysis of demographics, existing research consulted, etc. will only strengthen your report and answer some questions for a select group of readers.

Related to this topic, I’ve also written previously about how to ensure that your results are used and how to present monitoring and evaluation results.

And if you want to read further, here are some very comprehensive guidelines from the World Bank on Presenting Results (pdf).

Glenn

Further to my earlier post on ten tips for better web surveys, the email that people receive inviting them to complete an online survey is an important factor in persuading people to complete the survey - or not. Following are some recommended practices and a model email to help you with this task:

1. Explain briefly why you want an input: it’s important that people know why you are asking their opinion or feedback on a given subject. State this clearly at the beginning of you email, e.g. “As a client of XYZ, we would appreciate your feedback on products that you have purchased from us”.  

2. Tell people who you are: it’s important that people know who you are (so they can assess whether they want to contribute or not). Even if you are a marketing firm conducting the research on behalf of a client, this can be stated in the email as a boiler plate message (see example below). In addition, the name and contact details of a “real” person signing off on the email will help.

3. Tell people how long it will take: quite simply, “this survey will take you some 10 minutes to complete”. But don’t underestimate - people do get upset if you tell them it will take 10 minutes and 30 minutes later they are still going through your survey…

4. Make sure your survey link is clickable: often survey softwares generate very long links for individual surveys.  You can often get around this by masking the link, like this “click to go to survey >>“. However, some email systems do not read correctly masked links so you may be better to copy the full link into the email as in the example below. In addition, also send your email invitation to yourself as a test - so you can click on your survey link just to make sure it works…

5. Reassure people about their privacy and confidentiality: people have to be reassured that their personal data and opinions will not be misused. A sentence covering these points should be found in the email text and repeated on the first page of the web survey (also check local legal requirements on this issue).

6. Take care with the ”From”, “To” and “Subject”: If possible, the email address featured in the ”From” field should be a real person. The problem will be if your survey comes from info@wizzbangsurveys.net it may end up in many people’s spam folders.  For the “To”, it should contain an individual email only - we still receive email invitations where we can see 100s of email addresses in the “To” field - it doesn’t really instill confidence as to how your personal data will be used. The “Subject” is important also - you need something short and straight to the point (see example below). Avoid using spam-catching terms such as “win” or “prize”.

7. Keep it short: You often can fall into the trap of over explaining your survey and hiding the link somewhere in the email text or right at the bottom. Try and keep your text brief - most people will decide in seconds if they want to participate or not - and they need to be able to understand why they should, for whom, how long it will take and how (”Where is the survey link?!).

Model email invitation:   

From: j.jones@xyzcompany.net 
To: glenn.oneil@gmail.com
Subject: XYZ Summit 2008 - Seeking your feedback

Dear participant,

On behalf of XYZ, we thank you for your participation in the XYZ Summit.

We would very much appreciate your feedback on the Summit by completing a brief online survey. This survey will take some 10 minutes to complete. All replies are anonymous and will be treated confidentially.

To complete the survey, please click here >>

If this link does not work, please copy and paste the following link into your internet window:
http://optima.benchpoint.com/optima/SurveyPop.aspx?query=view&SurveyID=75&SS=0ZJk1RORb

Thank you in advance; your feedback is very valuable to us.

Kind regards,
J. Jones
Corporate Communications
XYZ Company
email: j.jones@xyzcompany.net 
tel: ++ 1 123 456 789

****
Benchpoint has been commissioned by XYZ to undertak this survey. Please contact Glenn O’Neil of Benchpoint Ltd. if you have any questions: oneil@benchpoint.com

The following article from Quirks Marketing Research Review also contains some good tips on writing email invitations.

Glenn

An advantage of the Internet age is that it is much easier to undertake surveys by using online web services that are relatively inexpensive. A disadvantage is that the quality of many web surveys are questionable. To help you write better web surveys, following are ten tips for better web surveys drawn from years of experience:

1. Explain why: When receiving an invitation (often by email) to complete a web survey, the respondents must understand why they are being asked to fill out the survey. You need to clearly state the purpose of the survey and how the results will be used. This can usually be stated in one to two sentences, e.g. “This survey is to collect your thoughts on the seminar you attended yesterday. Your feedback will help us to improve future events”.

2. Promise confidentiality: Most people will respond to your survey if they know their personal details and opinions will not be shared with the whole worldwide web. If needed, you can ask demographic questions such as age, education, country of residence and income but these need to be phrased sensitively (e.g. for age, ask for the year of birth and for salary offer a range e.g. “do you earn between $40,000 - 50,000, 51,000 - 60,000, etc.”). On your email invitation and first page of your survey you need to reassure respondents of confidentiality and anonymity. A simple sentence will do, such as “All feedback provided is anonymous and will be treated confidentially”. 

3. Tell people how long it will take: Often people get frustrated completing surveys as they don’t know how long it will take. It is better to state up front in the email invitation how long the survey will take, such as “This survey will take some 10 minutes to complete”. In addition, within the survey, you should activate the progress meter feature (which most online survey systems have), that shows respondents how much of the survey they have completed often with a small graph, e.g.: “30% of the survey completed”.

4. Keep it short: People often abandon surveys because they are too long. A good rule of thumb is that if you go over 25 questions you are asking quite a lot of respondents. Of course it depends on your subject and the potential respondents: if the subject is important to people they will spend more time responding to the survey.

5. Vary the type of questions: Many web surveys often ask use the same type of question repeatedly, such as using a scale “poor to excellent” with a long list of subjects to check off. This can induce survey fatigue where the respondents simply click down the columns vertically (e.g. they choose “good” for every subject) just to complete the survey. If possible, question types should be varied in order to avoid such a problem.

6. Always include at least one open question: These are questions where people can type in their own responses. Often web surveys only have closed questions where respondents check off the answers. Open questions, although requiring more time for analysis, often provide much more in-depth feedback and some insight into the “how” and “why”. If you are not sure how to place an open question in a survey, add one at the end of the survey requesting comments, such as “This survey has been about XYZ. Do you have anything else you would like to add?”

7. Place demographic questions last: To be able to make some useful analysis of the data you collect, you will need to collect some demographic data - in most cases this would be the country/state of residence and type of work as a minimum. These questions should be placed at the end of the survey by which time respondents will feel more comfortable answering such questions. This is even more important for questions on more sensitive demographic information such as age, income and ethnic background. 

8. “Other” may be your most useful response: When providing respondents with a pre-defined list of responses (e.g. what type of work do you do? Legal, finance, IT, PR, etc.), include an “Other, please specify_____” option. This helps to clarify if your pre-selected responses covered all possible answers and you may well be surprised by new groups of responses placed in “Other” that emerge.

9. Always give people a way out if they can’t answer: Sometimes in a survey you will arrive at a question with pre-defined responses and you will think “well, none of these apply to me”. Respondents are then forced to select a false response. Always read through your questions and imagine the range of responses possible. If in doubt, place a “none of the above” or “Not applicable” as a possible response for questions with pre-defined responses.

10. Always email invitations on a Tuesday - and send a reminder: Studies show that email invitations that are sent on a Tuesday will more likely be opened than on other days. It is also important to send a reminder - say 10 days after the initial invitation - if you can filter out those respondents that have already responded all the better. Further, our experience shows that you can double your response rate with an email reminder.

And here are 20 more tips on writing better web surveys from Userfocus>> 

Glenn

I recently had a discussion with a colleague about how we should represent the findings of an evaluation study. I am a big fan of using graphs and diagrams to explain the findings - as they say a “picture tells a thousand words”.

But we often see many misuses of graphs and diagrams that can provide an incorrect idea for the reader. I came across an example from a report recently which I have reproduced here:

This is an interesting example of a pictogram or scatter chart to represent two variables: 1) level of opportunity/risk (vertical scale) and 2) size of share value (size of bubbles).

But examing this chart, it made me wonder - what does the horizontal scale represent? In other words, on what basis are the bubbles placed left to right? I cannot see any logical basis in the chart for the horizontal location of the bubbles. I think that’s unfortunate as such a chart could use the horizontal scale to reinforce the share value variable or distribute the bubbles on another basis (e.g. sector of interest).

For those interested in graphic presentation of information, some key texts to read are found on the website of Edward Tufte, a leading specialist in this area.

Glenn

Often we attend conferences where one of the stated objectives is “increase/build/create networking” and I always found it odd that there is never any attempt to measure if networking really took place.

A possible solution is to map networks created by participants at conferences - and compare these networks to those that existed before the conferences.

This is exactly what I have done recently in a network mapping study that you can view here (pdf -  1 MB) and the above image is from. From the LIFT conference of 2007, we mapped the networks of 28 participants (out of 450 total participants) before and after the conferences. We found some quite surprising results:

• These 28 participants had considerable networks prior to the conference - reaching some 30% of all participants.
• These networks increased after the conference -the 28 people were then connected to some 50% of all participants.
• Based on the sample of  28 participants, most participants doubled their networks at LIFT07 - e.g. if you went to the conference knowing five people, you would likely meet another five people at the conference - thus doubling your network to ten.

Although this is only a mapping of 28 participants, it provides some insight into conferences and how networks develop - it’s also quite interesting that 28 people can reach 50% (225 people) of the total conference participants in this case.

View the full report here (pdf - 1 MB).

If you are after further information on network mapping, I recommend Rick Davies’ webpage on network mapping. Although it focuses on development projects it contains a lot of useful information on network mapping in general.  

Glenn  

An interesting post from the Buzz Bin which provides a good summary of current thoughts on how to measure social media such as blogs, social networks and podcasts. Well worth a read…

 Glenn

Well I do, actually - granularity is more important than we think for many fields…no, it’s not some sort of breakfast cereal- it’s the size or scale that characterizes an object or activity. And often we see errors made in placing activities at the same level that are not actually at the correct level … An example - I recently noticed a survey that featured the following question:

In which region are you working?
- Sub-Saharan Africa
- North Africa and Middle East
- Europe
- South Asia
- East Asia
- Russia and North Asia
- China
- North America
- Central and South America
- Australasia and Japan
- UK
- Other region

The problem is that the countries and regions mentioned are at different levels - and this is a problem of granularity. “UK” and “China” are not at the same level as “Central and South America” and “South Asia”. This creates problems for people completing the survey - If I live in the UK what do I select? UK or Europe? Both are correct.

In this example, there would be three possible solutions; 1) list all countries of the world using an ISO standard list, 2) list countries applicable to the project (using “other country” for those exceptions that will certainly arise) or 3) use broad, widely accepted regions, such as Europe, Asia, Africa, etc.  These solutions resolve the issue of granularity by placing the countries/regions at the same level.  

It may seem banal but if these issues are not resolved before the questions are asked, the analysis will prove difficult. This is just one example - granularity is important for many fields such as information management (libraries), website design, software and retail (you never see supermarket aisles marked “vegetables, cereals, bananas” do you?)

Glenn

To add to my previous favourite quotes on evaluation and measurement, I have collected the following quotes - enjoy!:

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”
Albert Einstein

“The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question”
Peter Drucker

“One of the great mistakes is to judge policies and programs by their intentions rather than their results”
Milton Friedman

“The pure and simple truth is rarely pure and never simple”
Oscar Wilde

“First get your facts; then you can distort them at your leisure”
Mark Twain

I know that half of my advertising dollars are wasted … I just don’t know which half”
John Wanamaker

Glenn

In an earlier post, I wrote about the advantages of using open-ended questions in surveys. The challenge is once you have 100s (or 1000s) of responses from your target audience - how do you analyze the answers to open-ended questions?

Basically, we draw on techniques developed for analyzing qualitative data - we are looking for patterns and trends in the responses so we can reach some conclusions as to what we are seeing. I summarise the main steps that I would usually undertake:

1. Read through the responses.Yes, as laborious as it may seem, you must read through each response to get a feeling for the data. As you read through the responses, you will probably see some common themes emerging.

2. Create response categories. The second step is to develop categories for the different themes you are seeing. For example, with a question asking for people’s feedback on a website, you will probably be able to group comments into categories such as “content”, “design”, “features”, “service”, etc.

3. Label each comment with one or several categories. As you read through the comments, assign at least one category to each response. This is what is called “coding” and best done in an excel sheet with responses in one column and your category (s) in the next column.

4. Look at what you have. In the example about feedback for a website, you might label half of your responses as “content”.  You can then divide the responses on “content” into smaller categories, e.g. “corporate content”, “product content”, etc. By doing this you will start to see what are the trends in the data and the main  issues raised by your respondents.

5. Think what are the responses about? Once you have categorised and coded data, it doesn’t do you much credit just to say “some half of people spoke about  content; most of these people spoke about the corporate pages on the website”. You must be able to explain what is being said about the subject or theme. For example in the case of “content” - what were people saying about content? Imagine if a respondent said:

“I consult regularly the corporate pages. This information is well-presented but not up-to-date. I never seem to be able to find information on latest priorities and management profiles”

This example contains different comments on aspects related to design, site updating, navigation and missing content. Notice that the comment on navigation is actually not a “content” issue - but would be considered as a “design” issue and needs to be coded accordingly.

6. Identifying the patterns and trends: once the data has been studied and categories determined, the next step is to see what categories are related and where can trends and patterns be identified: are there common themes emerging? Or are there a series of unrelated points being mentioned?  

7. Writing up the analysis: Once you have analyzed the data and identified the major patterns and trends your next step is to write a summary of what you have found. This would normally be a descriptive text incorporating comments directly from the respondents. For example:

“In providing feedback on the website, some half of the respondents spoke about content. The main issues raised included the inability to find content and the lack of up-to-date content on management themes. To a lesser extent, the high quality of the product information and the desire for more information on the management team were mentioned. The following comment from a respondent illustrates these points:

“I find the quality of the product information very good. However, the information is often difficult to find and is hidden on the website”.

As you see, I use terms such as “some half”,  ”main issues” and “to a lesser extent” to illustrate the magnitude of the trends identified. Some prefer to transfer such an analysis into quantifiable terms - such as “some 50%” or “under 30%”. I prefer not to -  but if you are dealing with very few responses, it’s better to mention the precise numbers such as “5 out of 20 responses preferred…”.

Good luck with your analysis!

Glenn        

Having previously written about best practices for using likert scale questions in surveys, I’d like to say something in favour of using open-ended questions. An open-ended question allows respondents to answer a question in their own words. In web surveys, this involves having a text field/box where respondents can write in their answer to a question posed. 

Open-ended questions have the advantage over close-ended questions (that use pre-defined answers, such as “good”, “excellent”, etc.) in that they provide an insight into the “how” and “why” aspects of an issue. Close-ended questions typically answer the “how much/many” and “when” aspects.

In my opinion, a survey should contain at least one open-ended question. Imagine if you are asking people about a product and they have to rate it on a satisfaction scale. It would be very interesting to go behind the numbers and ask them ”describe for me the two major advantages of using this product”. Matched to your satisfaction scale (take particular note of what the very satisfied and very unsatisfied customers are answering), this information is highly valuable.

I also advocate finishng a survey with an open-ended question, such as “This survey has been about your experience with XYZ product. Do you have anything else you would like to say?”.

You would be surprised at the number of people that do have something to say! I am always told that people don’t like to give feedback; they are fed-up with answering surveys. But my experience has shown that if you really are interested in an issue / product / service / company, you will give a feedback - open-ended questions are perfect for that.

Of course, the downside is how do you analyse the answers you get? How can you draw useful actionable points from the answers? That’s another story that I’ll cover in my next post…

In the meantime, here is a good summary of best practices for open-ended questions>>

Glenn