intelligent measurement

We often see survey questions with likert scales using frequency estimations such as “sometimes”, “often”, “always”, “never”. etc.  However, these scales often provide inaccurate responses as I’ve written about before.  Why is that so? Well, describing frequency differs enormously from person to person. This dialogue of a couple seeing separate analysts about their relationship, taken from the Woody Allen film “Annie Hall” illustrates this point:

Woody’s analyst: How often do you sleep together?
Woody Allen: Hardly ever! Maybe three times a week
Diane’s analyst: Do you have sex often?
Diane Keaton: Constantly! I’d say three times a week

So ”three times a week” for one person is “constantly” and “hardly ever” for another! This funny dialogue illustrates the point that descriptive terms (e.g. “constantly”)  are not accurate measures of frequency.

In survey questions, one way to avoid this is by not using descriptive terms and asking people directly to provide a numerical estimate, for example:

Inaccurate: How often do you watch TV?
Never, hardly ever, sometimes,  constantly

Accurate: How many hours per week do you watch TV? 
None, under 5, 6-10, 11-15, 16-20, more than 20

Of course, such numeric scales also have problems of accuracy - e.g. people can recall incorrectly how many hours they have spent watching TV.  In this regard, it is better to focus on shorter time periods (”what have you done in the past week” or “what do you do on an average day”).

Glenn