intelligent measurement

Well I do, actually - granularity is more important than we think for many fields…no, it’s not some sort of breakfast cereal- it’s the size or scale that characterizes an object or activity. And often we see errors made in placing activities at the same level that are not actually at the correct level … An example - I recently noticed a survey that featured the following question:

In which region are you working?
- Sub-Saharan Africa
- North Africa and Middle East
- Europe
- South Asia
- East Asia
- Russia and North Asia
- China
- North America
- Central and South America
- Australasia and Japan
- UK
- Other region

The problem is that the countries and regions mentioned are at different levels - and this is a problem of granularity. “UK” and “China” are not at the same level as “Central and South America” and “South Asia”. This creates problems for people completing the survey - If I live in the UK what do I select? UK or Europe? Both are correct.

In this example, there would be three possible solutions; 1) list all countries of the world using an ISO standard list, 2) list countries applicable to the project (using “other country” for those exceptions that will certainly arise) or 3) use broad, widely accepted regions, such as Europe, Asia, Africa, etc.  These solutions resolve the issue of granularity by placing the countries/regions at the same level.  

It may seem banal but if these issues are not resolved before the questions are asked, the analysis will prove difficult. This is just one example - granularity is important for many fields such as information management (libraries), website design, software and retail (you never see supermarket aisles marked “vegetables, cereals, bananas” do you?)

Glenn

To add to my previous favourite quotes on evaluation and measurement, I have collected the following quotes - enjoy!:

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”
Albert Einstein

“The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong question”
Peter Drucker

“One of the great mistakes is to judge policies and programs by their intentions rather than their results”
Milton Friedman

“The pure and simple truth is rarely pure and never simple”
Oscar Wilde

“First get your facts; then you can distort them at your leisure”
Mark Twain

I know that half of my advertising dollars are wasted … I just don’t know which half”
John Wanamaker

Glenn

In an earlier post, I wrote about the advantages of using open-ended questions in surveys. The challenge is once you have 100s (or 1000s) of responses from your target audience - how do you analyze the answers to open-ended questions?

Basically, we draw on techniques developed for analyzing qualitative data - we are looking for patterns and trends in the responses so we can reach some conclusions as to what we are seeing. I summarise the main steps that I would usually undertake:

1. Read through the responses.Yes, as laborious as it may seem, you must read through each response to get a feeling for the data. As you read through the responses, you will probably see some common themes emerging.

2. Create response categories. The second step is to develop categories for the different themes you are seeing. For example, with a question asking for people’s feedback on a website, you will probably be able to group comments into categories such as “content”, “design”, “features”, “service”, etc.

3. Label each comment with one or several categories. As you read through the comments, assign at least one category to each response. This is what is called “coding” and best done in an excel sheet with responses in one column and your category (s) in the next column.

4. Look at what you have. In the example about feedback for a website, you might label half of your responses as “content”.  You can then divide the responses on “content” into smaller categories, e.g. “corporate content”, “product content”, etc. By doing this you will start to see what are the trends in the data and the main  issues raised by your respondents.

5. Think what are the responses about? Once you have categorised and coded data, it doesn’t do you much credit just to say “some half of people spoke about  content; most of these people spoke about the corporate pages on the website”. You must be able to explain what is being said about the subject or theme. For example in the case of “content” - what were people saying about content? Imagine if a respondent said:

“I consult regularly the corporate pages. This information is well-presented but not up-to-date. I never seem to be able to find information on latest priorities and management profiles”

This example contains different comments on aspects related to design, site updating, navigation and missing content. Notice that the comment on navigation is actually not a “content” issue - but would be considered as a “design” issue and needs to be coded accordingly.

6. Identifying the patterns and trends: once the data has been studied and categories determined, the next step is to see what categories are related and where can trends and patterns be identified: are there common themes emerging? Or are there a series of unrelated points being mentioned?  

7. Writing up the analysis: Once you have analyzed the data and identified the major patterns and trends your next step is to write a summary of what you have found. This would normally be a descriptive text incorporating comments directly from the respondents. For example:

“In providing feedback on the website, some half of the respondents spoke about content. The main issues raised included the inability to find content and the lack of up-to-date content on management themes. To a lesser extent, the high quality of the product information and the desire for more information on the management team were mentioned. The following comment from a respondent illustrates these points:

“I find the quality of the product information very good. However, the information is often difficult to find and is hidden on the website”.

As you see, I use terms such as “some half”,  ”main issues” and “to a lesser extent” to illustrate the magnitude of the trends identified. Some prefer to transfer such an analysis into quantifiable terms - such as “some 50%” or “under 30%”. I prefer not to -  but if you are dealing with very few responses, it’s better to mention the precise numbers such as “5 out of 20 responses preferred…”.

Good luck with your analysis!

Glenn        

Having previously written about best practices for using likert scale questions in surveys, I’d like to say something in favour of using open-ended questions. An open-ended question allows respondents to answer a question in their own words. In web surveys, this involves having a text field/box where respondents can write in their answer to a question posed. 

Open-ended questions have the advantage over close-ended questions (that use pre-defined answers, such as “good”, “excellent”, etc.) in that they provide an insight into the “how” and “why” aspects of an issue. Close-ended questions typically answer the “how much/many” and “when” aspects.

In my opinion, a survey should contain at least one open-ended question. Imagine if you are asking people about a product and they have to rate it on a satisfaction scale. It would be very interesting to go behind the numbers and ask them ”describe for me the two major advantages of using this product”. Matched to your satisfaction scale (take particular note of what the very satisfied and very unsatisfied customers are answering), this information is highly valuable.

I also advocate finishng a survey with an open-ended question, such as “This survey has been about your experience with XYZ product. Do you have anything else you would like to say?”.

You would be surprised at the number of people that do have something to say! I am always told that people don’t like to give feedback; they are fed-up with answering surveys. But my experience has shown that if you really are interested in an issue / product / service / company, you will give a feedback - open-ended questions are perfect for that.

Of course, the downside is how do you analyse the answers you get? How can you draw useful actionable points from the answers? That’s another story that I’ll cover in my next post…

In the meantime, here is a good summary of best practices for open-ended questions>>

Glenn
 

The new Terminal Five at Heathrow Aiport, London - which had quite a controversial birth - is going to undertake an unusual experience in “testing” there facilities before public launch. As I’ve written about before, the aspect of evaluation prior to the launch of a project or activity is often overlooked - and this is an extreme example of this principle in action.

Terminal Five is seeking 15,000 volunteers to act as test users of their new facilities. They will ask volunteers to act as “real” passengers and go through the steps of checking in just to stepping on to the aircraft. I find it fascinating that they will “test” their facilities in such a large-scale manner. Always recommended is “field testing” of new products and services - but this is going quite far. Of course, the question begs, what happens if the user testing brings up major issues with the terminal?  Well, Terminal Five lists the following as their aim:

“What are we trying to achieve?: Proof that Terminal 5 is safe, secure and works like clockwork. We’ll also ensure the team who will be running the terminal get the chance to test and develop their service. We also need to identify anything we need to fix prior to opening”

So let’s see what such a trial will bring - how big will the “fixes” be - or will all work like clockwork? Regardless, I am sure they will receive interesting feedback and discover, as I have in usability testing,  that people view and use products or facilities in ways totally unanticipated.  Do you want to volunteer? Read more here>>

Glenn