The magical number seven, plus or minus two

In an earlier post on best practices for likert scale questions, I made reference to an article “The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information”. A reader pointed out to me that this is one the most famous social psychology articles (by George A. Miller, 1956).

A main conclusion of the article is that most people can only deal with seven ideas at a time. Very clever people can manage up to nine and less skilled people may only be able to concentrate on five ideas at once.

Putting this theory into practice, I can see plenty of examples which call out for the “seven point limit”, such as:

• Number of chapters in a report
• Number of recommendations in reports (if there are more than seven then they should be broken down under sub-headings)
• Number of points on a likert scale question (see my previous post on this)
• Lists of variables/aspects that must be ranked/rated on surveys (e.g. “please rate the following aspects of the website..”)
• Items on a checklist to be assessed (see my previous post on checklists)

And that’s only five ideas so us less skilled people can cope…

Glenn

Likert scale & surveys – best practices

I’ve been looking into the best practices for using the Likert scale type of question, probably the most widely used response scale featured in surveys – often used to measure attitudes and other factors (e.g. “Excellent” to “Poor”). Created by Rensis Likert (pictured above) in the 1930s, his original scale featured five points. Over time, there has been many discussions and disagreements focused on one central question: What works best with the Likert scale to give you the most accurate responses?

I have read a number of studies on this question (sorry, I don’t link to them as they are all books or academic journals (that require a fee) but if you are interested write to me and I’ll give you the references) and the following are the points that most (but not all) scholars agree on:

More than seven points on a scale are too much. Studies show that people are not able to place their point of view on a scale greater than seven. So go for seven or less. What is the perfect number? Studies are not conclusive on this, most commonly mentioned are five, four or three point scales.

Numbered scales are difficult for people. For example, scales that are marked “1 to 5, with 5 being the highest” result in less accurate results than scales with labels such as “good” or “poor”. If numbered scales are used, signposts are recommended (e.g. put “poor” above 1, “satisfactory” above 3 and “excellent” above 5).

Labelled scales need to be as accurate as possible. Commonly uses labels such as “often” or “sometimes” often result in inaccurate responses. As these terms mean different notions of engagement from person to person, culture to culture (not to add the complexity of translating these terms). Scholars recommend using time-bound labels for frequency measures such as “once a week” (although problems of correct recall are also an issue). In addition, studies show that people find it difficult to differentiate between “very good” and “good” – better to use “good” and “excellent”.

And that’s it! Basically, there are inconclusive results on the use of a middle or neutral point (e.g. four point vs. a five point scale). Some scholars advocate a five point scale where respondents can have a “neutral” middle point whereas others prefer to “force” people to select a negative or positive position with a four point scale. In addition, the use of a “don’t know” option is inconclusive. I personally believe that a “don’t know” option is essential on some scales where people may simply not have an opinion. However, studies are inconclusive on if a “don’t know” option increases accuracy of responses.

Further information on the Likert Scale:

Examples of commonly-used Likert Scales >>

More examples of commonly-used Likert Scales >>

“Designing a useful Likert Scale” (pdf)>>

“The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information”>>

Glenn

The value of checklists and evaluation: 7 reasons

Further to what I wrote last week  about checklists and their use in evaluation, I have found an excellent article on the logic and methodology of checklists.

Dr Michael Scriven of the Evaluation Centre of Western Michigan University describes the different types of  checklists and how good checklists are put together. In particular, I like his list of the seven values of checklists, of which I summarise as follows: 

1. Reduces the chance of forgetting to check something important
2. Are easier for the lay stakeholder to understand and evaluate
3. Reduces the “halo effect”- it forces an evaluator to look at all criteria and not be overwhelmed by one highly valued feature
4. Reduces the influence of the “rorschach effect” – that is the tendancy to see what one wants to see in a mass of data – evaluators have to look at all dimensions
5. Avoids criteria being counted twice or given too much importance
6. Summarises a huge amount of professional knowledge and experience
7. Assists in evaluating what we cannot explain

As Dr Scriven points out, checklists are very useful tools in getting us to think through the “performance criteria” of all kinds of processes, projects or occurences, e.g. what are the key criteria that make a good trainer – and what criteria are more important than other?

Read the full article here >>

Glenn

Checklists and evaluation

Often in evaluation, we are asked to evaluate projects and programmes from several different perspectives: the end user, the implementer or that of an external specialist or “expert”. I always favour the idea that evaluation is representing the *target audiences* point of view – as is often the case in evaluating training or communications programmes – we are trying to explain the effects of a given programme or project on target audiences. However, often a complementary point of view from an “expert” can be useful. A simple example – imagine if you making an assessment of a company website – a useful comparison would be comparing the feedback from site visitors with that of an “expert” who examines the the website and gives his/her opinion.

However, often opinions of “experts” are mixed in with feedback from audiences and comes across as unstructured opinions and impressions. A way of avoiding this is for “experts” to use checklists – a structured way to assess the overall merit, worth or importance of something.

Now many would consider checklists as being a simple tool not worthy of discussion. But actually a checklist is often a representation of a huge body of knowledge or experience: e.g. how do you determine and describe the key criteria for a successful website?

Most checklists used in evaluation are criteria of merit checklists – where a series of criteria are established and given a standard scale (e.g. very poor to excellent) and are weighed equally or not (e.g. one criteria is equal or more crucial than the next one). Here are several examples where checklists could be useful in evaluation:

• Evaluating an event: you determine “success criteria” for the event and have several experts use a checklist and then compare results.
• Project implementation: a team of evaluators are interviewing staff/partners on how a project is being implemented. The evaluators use a checklist to assess the progress themselves.
• Evaluating services/products: commonly used, where a checklist is used by a selection panel to determine the most appropriate product/services for their needs.

This post by Rick Davies actually got me thinking about this subject and discusses the use of checklists in assessing the functioning of health centres.

Glenn