intelligent measurement

Having previously written about best practices for using likert scale questions in surveys, I’d like to say something in favour of using open-ended questions. An open-ended question allows respondents to answer a question in their own words. In web surveys, this involves having a text field/box where respondents can write in their answer to a question posed. 

Open-ended questions have the advantage over close-ended questions (that use pre-defined answers, such as “good”, “excellent”, etc.) in that they provide an insight into the “how” and “why” aspects of an issue. Close-ended questions typically answer the “how much/many” and “when” aspects.

In my opinion, a survey should contain at least one open-ended question. Imagine if you are asking people about a product and they have to rate it on a satisfaction scale. It would be very interesting to go behind the numbers and ask them ”describe for me the two major advantages of using this product”. Matched to your satisfaction scale (take particular note of what the very satisfied and very unsatisfied customers are answering), this information is highly valuable.

I also advocate finishng a survey with an open-ended question, such as “This survey has been about your experience with XYZ product. Do you have anything else you would like to say?”.

You would be surprised at the number of people that do have something to say! I am always told that people don’t like to give feedback; they are fed-up with answering surveys. But my experience has shown that if you really are interested in an issue / product / service / company, you will give a feedback - open-ended questions are perfect for that.

Of course, the downside is how do you analyse the answers you get? How can you draw useful actionable points from the answers? That’s another story that I’ll cover in my next post…

In the meantime, here is a good summary of best practices for open-ended questions>>

Glenn
 

In an earlier post on best practices for likert scale questions, I made reference to an article “The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information”.  A reader pointed out to me that this is one the most famous social psychology articles (by George A. Miller, 1956).  

A main conclusion of the article is that most people can only deal with seven ideas at a time. Very clever people can manage up to nine and less skilled people may only be able to concentrate on five ideas at once.

Putting this theory into practice, I can see plenty of examples which call out for the “seven point limit”, such as:

• Number of chapters in a report 
• Number of recommendations in reports (if there are more than seven then they should be broken down under sub-headings)
• Number of points on a likert scale question (see my previous post on this)
• Lists of variables/aspects that must be ranked/rated on surveys (e.g. ”please rate the following aspects of the website..”)
• Items on a checklist to be assessed (see my previous post on checklists)

And that’s only five ideas so us less skilled people can cope…

Glenn

I’ve been looking into the best practices for using the Likert scale type of question,  probably the most widely used response scale featured in surveys - often used to measure attitudes and other factors (e.g. ”Excellent” to “Poor”). Created by Rensis Likert (pictured above) in the 1930s, his original scale featured five points. Over time, there has been many discussions and disagreements focused on one central question:  What works best with the Likert scale to give you the most accurate responses?  

I have read a number of studies on this question (sorry, I don’t link to them as they are all books or academic journals (that require a fee) but if you are interested write to me and I’ll give you the references) and the following are the points that most (but not all) scholars agree on:

More than seven points on a scale are too much. Studies show that people are not able to place their point of view on a scale greater than seven. So go for seven or less. What is the perfect number? Studies are not conclusive on this, most commonly mentioned are five, four or three point scales.

Numbered scales are difficult for people.  For example, scales that are marked “1 to 5, with 5 being the highest” result in less accurate results than scales with labels such as “good” or “poor”.  If numbered scales are used, signposts are recommended (e.g. put “poor” above 1, “satisfactory” above 3 and “excellent” above 5).

Labelled scales need to be as accurate as possible. Commonly uses labels such as “often” or “sometimes” often result in inaccurate responses. As these terms mean different notions of engagement from person to person, culture to culture (not to add the complexity of translating these terms).  Scholars recommend using time-bound labels for frequency measures such as “once a week” (although problems of correct recall are also an issue). In addition, studies show that people find it difficult to differentiate between “very good” and “good” - better to use “good” and “excellent”.

And that’s it! Basically, there are inconclusive results on the use of a middle or neutral point (e.g. four point vs. a five point scale). Some scholars advocate a five point scale where respondents can have a “neutral” middle point whereas others prefer to “force” people to select a negative or positive position with a four point scale. In addition, the use of a “don’t know” option is inconclusive. I personally believe that a “don’t know” option is essential on some scales where people may simply not have an opinion. However, studies are inconclusive on if a “don’t know” option increases accuracy of responses.

Further information on the Likert Scale:

Examples of commonly-used Likert Scales >>

More examples of commonly-used Likert Scales >>

“Designing a useful Likert Scale” (pdf)>>

“The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information”>>

Glenn

Further to what I wrote last week  about checklists and their use in evaluation, I have found an excellent article on the logic and methodology of checklists.

Dr Michael Scriven of the Evaluation Centre of Western Michigan University describes the different types of  checklists and how good checklists are put together. In particular, I like his list of the seven values of checklists, of which I summarise as follows: 

1. Reduces the chance of forgetting to check something important
2. Are easier for the lay stakeholder to understand and evaluate
3. Reduces the “halo effect”- it forces an evaluator to look at all criteria and not be overwhelmed by one highly valued feature
4. Reduces the influence of the “rorschach effect” - that is the tendancy to see what one wants to see in a mass of data - evaluators have to look at all dimensions
5. Avoids criteria being counted twice or given too much importance
6. Summarises a huge amount of professional knowledge and experience
7. Assists in evaluating what we cannot explain

As Dr Scriven points out, checklists are very useful tools in getting us to think through the “performance criteria” of all kinds of processes, projects or occurences, e.g. what are the key criteria that make a good trainer - and what criteria are more important than other?

Read the full article here >>

Glenn

Often in evaluation, we are asked to evaluate projects and programmes from several different perspectives: the end user, the implementer or that of an external specialist or “expert”.  I always favour the idea that evaluation is representing the *target audiences* point of view - as is often the case in evaluating training or communications programmes - we are trying to explain the effects of a given programme or project on target audiences.  However, often a complementary point of view from an “expert” can be useful.  A simple example - imagine if you making an assessment of a company website - a useful comparison would be comparing the feedback from site visitors with that of an  “expert” who examines the the website and gives his/her opinion.

However, often opinions of “experts” are mixed in with feedback from audiences and comes across as unstructured opinions and impressions. A way of avoiding this is for “experts” to use checklists -  a structured way to assess the overall merit, worth or importance of something.  

Now many would consider checklists as being a simple tool not worthy of discussion. But actually a checklist is often a representation of a huge body of knowledge or experience: e.g. how do you determine and  describe the key criteria for a successful website?

Most checklists used in evaluation are criteria of merit checklists - where a series of criteria are established and given a standard scale (e.g. very poor to excellent) and are weighed equally or not (e.g. one criteria is equal or more crucial than the next one).  Here are several examples where checklists could be useful in evaluation:

• Evaluating an event: you determine “success criteria” for the event and have several experts use a checklist and then compare results.
• Project implementation: a team of evaluators are interviewing staff/partners on how a project is being implemented. The evaluators use a checklist to assess the progress themselves.
• Evaluating services/products: commonly used, where a checklist is used by a selection panel to determine the most appropriate product/services for their needs.

This post by Rick Davies actually got me thinking about this subject and discusses the use of checklists in assessing the functioning of health centres.

Glenn  

It is often difficult to get organisations away from simply measuring “outputs” - what is produced - to measuring “outcomes” - what are the effects of outputs.

Funny enough, many organisations want to go from the very superficial measuring of output (e.g. how many news articles did we generate) to the very in-depth measuring of impact (e.g. the long term effect of our media visibility on audiences). Impact is feasible but difficult to measure, as I’ve written about before. However, instead of focusing on the two ends of the measurement scale, organisations would perhaps be wise to focus on “outcome” measurement.

I think this quote from a UN Development Programme Evaluation Manual (pdf) sums up why outcome is an appropriate level to measure for most organisations:

“Today, the focus of UNDP evaluations is on outcomes, because this level of results reveals more about how effective UNDP’s actions are in achieving real development changes. A focus on outcomes also promises a shorter timeframe and more credible linkages between UNDP action and an eventual effect than does a focus on the level of overall improvement in people’s lives, which represent much longer-term and diffuse impacts .”

The notion of the shorter timeframe and more credible linkages is certainly appealing for many organisations considering their focus of evaluation.

Glenn

As I mentioned in an earlier post, people often confuse “impact” with “results”. Is it possible to measure “long term impact” of projects? It is, however for most projects it is unrealistic to do so for two reasons: time and cost.

To evaluate impact, you would usually need to wait some 12 months after the major elements of a project have been implemented. Many organisations cannot simply wait that long. In term of costs, an impact study requires a triangulation methodology that uses various quantitative and qualitative research methods which could be costly. However, if time and cost are not issues, an impact evaluation is possible, keeping in mind the following points:

Was the impact desired defined at the beginning of the project?

 For example, greater organisation efficiency; change in the way a target audience and/or an organisation behaves; or improvements in how services for a given audience are managed?

What have been the other elements influencing the impact you want to measure?

Your project cannot be viewed in isolation; there must have been other factors influencing the changes being observed. Identifying these factors will help you to assess the level of influence of your project compared to other factors.

Do you have a mandate to measure impact?

When assessing impact, you will be looking at long term effects that probably go outside of your own responsibilities and into the realms of other projects and units - you are looking at an area of the wider effects of your organisation’s activities and this needs to be taken into consideration. For example, if you are looking at the longer term effects of a training program, you would want to look at how individuals and the organisation as a whole are more efficent as a result of the training. Do you have the political mandate to do so? - As you may discover effects that go way beyond your own responsibilities.

Evaluating impact is a daunting but not impossible task. For most projects, it would be more realistic to focus on measuring outputs and preferably outcomes - and think of short term outcomes as I have written about previously.

Glenn

When speaking of achieving objectives for a project, I’ve heard a lot of people speak of the “intended impact” and I’ve read quite some “impact reports”. I know it’s a question of language, but often people use the word “impact” when in fact they should use the word “results”. Impact in the evaluation field has a specific application to long term effects of a project. The DAC Glossary of Key Terms in Evaluation and Results Based Management (pdf) produced by the OECD contains the most widely accepted definitions in this field. Impact is defined as:

“Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.

And “results” is defined as

“The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention”.

Consequently  I believe that when we produce a report that shows media visibility generated by a project, this is a short term output and should be called “results” rather than “impact” which applies to more long-term effects.

Glenn

Adding to what I wrote about last week concerning measuring behaviour changes that result from communication campaigns - and why I recommend to consider looking at immediate responses (or “outtakes”) as an alternative to long-term changes - I can see parallels in areas other than in campaigns.

As you may know, a favourite of mine is measuring the impact of conferences and meetings. Industry conferences are traditionally sold as being great places to learn something and network, network -  and network. But I’m always surprised when attending such conferences at how organisers, if they measure something, focus on measuring the reactions to the conferences, usually in terms of satisfaction. No attempt is made to measure immediate changes to behaviour (such as extending a network) or longer term behaviour or impact in general.

But it is certainly possible, this diagram (pdf) illustrates what I did to measure immediate and mid-term changes to behaviour following a conference (LIFT). Despite the limitations of the research as I explain here, I was able to track some responses following the conference that could be largely contributed to participating in the conference - such as meeting new people or using new social media in their work. One year after the conference, participants also provided us with types of actions that they believed were influenced largely by their participation. Actions included:
- launching a new project
- launching a new service/product
- establishing a new partnership
- Initiating a career change
- Invitations for speaking engagements

Some of these actions were anticipated by the conference organisers - but many were not. It shows that it can be done and is certainly worth thinking about in conference evaluation.

Glenn

In an interesting post by Tom Watson on the Dummyspit blog,  he writes about the difficulty of changing behaviour through PR campaigns - in this case to encourage better eating habits in UK schools (staying away from hamburgers for example). He also notes the rather depressing statistic that PR campaigns normally have only a 0.04% success rate of changing behaviour! In setting campaign objectives, I encourage organisations to be realistic about the targets they set - for example, a modest 2 - 10%  behaviour change for public health campaigns is cited as being a realistic target (Rice & Paisley 1982).

But before you abandon your PR activities in despair, we have to be clear about what we define as “behaviour change”. Changing eating habits is a long-term behaviour change. PR campaigns often focus on trying to change long term behaviour and don’t consider setting goals in terms of immediate responses -  what can people be expected to do when reached by the campaign? In PR terminology this is called “outtakes” (different from long-term behaviour which would be “outcomes”).

Following are some practical examples of “outtakes” or immediate responses (which are short term behaviours) that can be measured. I believe that the percentage of change desired amongst a target audience can can be set at higher than 10% for many of these:

- % of people that sign a petition
- % of people that refer  a web page to a friend
- % people that go online to participate in an online discussion
- % of organisations/individuals that publicly support a campaign
- % of people who report undertaking a new initiative as a result of a campaign

Glenn