intelligent measurement

As I wrote in a previous post, research results are sometimes misused (that’s nothing new…) and we are often given scant details on how the results were gathered and analysed.

I came across a study undertaken by a bank in Geneva, Switzerland (where I am living) that makes a series of claims about e-banking, web surfing habits and computer use in general. I was surprised to learn that these claims were based on a sample of 300 residents. Now Geneva has some 440,000 residents and I seem to recall from Statistics 101 that 300 people doesn’t really make a representative sample of 440,000 (it would be closer to 600 people depending upon the confidence level and intervals you are aiming at).

I’m not such a stickler on samples given that often the audiences we are looking at can be broken down into sub-populations that are often relatively small in number (so we look for highest participation as possible) - but if you do have a uniform finite population, try using this online sample size calculator to estimate the sample needed, it’s quite useful.

Glenn

I’m always surprised at the number of published research results that fail to explain how they came to gather/analyse the results they are promoting. Related to this, you have the issue of results being misused, embellished or taken out of context. Constantin Basturea tells the interesting tale of how results from a quick poll of 50 website visitors became a poll of “300 business communicators” in a later publication. He only found this out after being curious about the poll results and requesting details of the methodology.

Personally, I think it’s always wise to publish information about your methodology for evaluation projects, particularly if the results are published and freely available. That’s what we did for the evaluation of the LIFT06 conference, publishing the results and the methodology. Then hopefully your results are not taken out of context and the methodology is available for review and criticism.

Glenn  

I see the that program of the 4th annual measurement summit (to be held in NH, US) has been announced by the Institute for Public Relations. This is an important conference for people interested in communications and evaluation. There are certainly some interesting topics being addressed: Shell Corporation on reputation tracking,  Procter and Gamble on ROI and PR and a panel discussion featuring three of my favourite “scholars” in the field of evaluation: Larissa Grunig, James Grunig and Jim MacNamara. The full program is available online (pdf).

For those of us who will not be able to make it, let’s hope they publish the main presentations and conclusions (they usually are kind enough to do so so).

Glenn

Can we make the logical step to “output” with media monitoring – measuring changes to knowledge, behaviour or attitudes? With traditional media monitoring we cannot. And that’s the missing link of most media monitoring – how can we tell if the media exposure led to a change with a given audience?  Polling of audiences and making an informed assumption linking their media use with changes observed is possible – but cost and complexity are the main deterrents for many organisations.

But with the online environment, there are some interesting developments in the ability to link media exposure with an actual behaviour change of an audience. Take this example: people who read an article online and then link to it in their blog have made a behaviour change - a simple example. If we could show the path from media exposure to the triggering of thoughts, comments, actions and ideas we are heading in the “outcome” direction. David Phillips of Leaverwealth blog is working in this area and is developing software to summarise content of RSS feeds under subject headings and show the path to the original stories and posts. This uses a statistical/mathematical technique, Latent Semantic Analysis which extracts and represents the similarity of meaning of words and passages. Now, that’s much more valuable than clip counting.

Glenn

Some further thoughts on media monitoring:

Michael Blower at the Media evaluation blog writes about audience reaction to the media as an “outtake” indicator with the new “BBC most popular stories” tool:

“For too long media content analysis has been driven by media output. This new tool makes it possible to do something which, up to now has been an expensive luxury – see an exact measure of media out-take.”

I find this a refreshing point of view from a media evaluator, moving the focus from “output” to “outtake”. If we consider “outtake” to mean understanding, reaction and favourability to a message, such website tools can provide a feedback – albeit not complete – on this level of measurement. I guess there must be a web-based tool that can collate and prioritise the popularity of news stories (based on visitor traffic) from the main news sites. Googletrends does this with search results and links it to news stories as I wrote about previously, a step in this direction.  Of course we have to factor in the limitations of web metrics including pass-word protected content of news sites.

Glenn  

Prompted by a client asking me to look at how they evalute their media relations, I’ve been examining closer at the whole media monitoring subject. I’ve followed the debate on the new media monitoring system proposed in Canada (called “PR Measurement“ ”MRP” - it’s actually PRess monitoring and not Public Relations measurement) and I see that K D Paine has some wise comments in this area. What do I think? - the amount of time, budget and resources spent on media monitoring is completely out of proportion. For me, media monitoring is an indication of efficiency (number of messages placed, supported and received) and not an indication of effectiveness (did our message reach our given audiences and change knowledge, attitudes or behaviour).

Instead of spending time to manually analyse clips or budgets to pay for monitoring software, I would prefer to see media relations professionals tackling the harder questions - what was the effect of a given media campaign on their target audiences? Media monitoring conducted jointly with surveying target audiences is an interesting solution.

This can work particularly well with targeted campaigns; my colleagues at Benchpoint in the UK recently did a joint media analysis (with Mantra International) on the retail sector linked with surveys of the target audience (customers). The problem of isolating the effect of a given media campaign/event/activity is not simple as I have written about previously but at least - at a minimum - you will have indications as to the influence of the different media and be able to make reasonable assumptions - with supporting evidence - about your media work and its impact.

There are interesting developments in this area which I will write further about in the coming days.

Glenn