intelligent measurement

Here is something from the mainstream media, the International Herald Tribune has published an interesting article about the legitimacy or not of online polling - basically the use of online survey tools to conduct polling with broad audiences.

The article pits two main players in the field -  Yougov against TNS.

The representative from TNS says about internet polling:

“Internet polling is like the Far West, with no rules, no sheriff and no reference points.”

 Of course, he has a point although the counter argument is that *offline* polling  - usually done by calling people on their fixed phone line - is fastly becoming obsolete as countries create “no call” lists and people use increasingly their mobile phones.

From my perspective, I would see the debate from a slightly different angle:

• We shouldn’t forget the approach of triangulation, that is the combination of different research methods to bring us closer to the *truth*.  Internet polling, if combined with other research methods (interviews, focus groups, observations) can become more useful.
• The whole article focuses only on surveying *unknown* audiences - that is members of the public as a whole. However, most evaluation that I see undertaken is done with *known” audiences, e.g. staff, partners, customers, members, etc. For *known* audiences, the use of internet polling is efficient and reliable, assuming of course that your audience does have access to the Internet. Of course, if you are trying to gauge the opinions of audiences that are obviously not using the Internet, then another approach would be appropriate.

Glenn  

I’ve written previously about Return on Investment (ROI) and particularly its application to blogging (which I consider as flawed).

At a broader level, there has been discussion for some time on ROI for public relations/communications programmes. Tom Watson of the dummyspit blog has written about this issue and the difficulty of applying what is essentially a financial concept to a non-financial activity, as is the case for marketing and PR.
As he states.

“For marketers, the application of ROI limits their role to sales support and ignores the brand and reputational issues. In PR, I’ve long argued that the use of business language is a fundamental sign of insecurity and a lack of confidence. It seems that marketing has the same affliction.”

Read the full post here >>

Glenn

I was recently at a presentation where the speaker commented on the SMART approach to writing objectives and said that that “A” stood for “Ambitious”. The person next to me nudged me and whispered “I think he means “Achievable”". And that got me thinking that the A should stand for ”Ambitious but Achievable”.

For it to be possible to evaluate a programme or project, it is necessary to set SMART objectives that allow the indentification of indicators that facilitate evaluation. What do SMART objectives look like? I’ve created several examples (pdf) if you are interested at my interpretation of SMART.

But there is a potential conflict in the concept of ”Ambitious but Achievable”: we need to set realistic objectives that can be evaluated - but at the same time we want to make our objectives ambitious enough in order to contribute significantly to the overall aims of a project or programme.

Glenn

In the past months I’ve had many discussions with project managers about evaluation where they clearly feel uncomfortable linking their objectives to outcomes - and now after reading a recent article in the Evaluation journal, the penny has dropped (i.e. I am able to link up the dots and understand a little more…).

In an article by John Mayne, “Challenges and Lessons in Implementing Results-Based Management”, he discusses the issue of accountability and outcomes:

“People are generally comfortable with being accountable for things they can control. Thus, managers can see themselves as being accountable for the outputs produced by the activities they control. When the focus turns to outcomes, they are considerably less comfortable, since the outcomes to be achieved are affected by many factors not under the control of the manager.”

And that’s it - a communications manager prefers to be accountable for the number of press releases s/he publishes and not the changes to an audiences knowledge or attitude, a training manager prefers to be accountable for the number of courses s/he organises and not the impact on an organisation’s efficency, etc.

So is there a solution? John Mayne speaks of a more sophisticated approach to accountability, notably to look at the extent to which a programme has influenced and contributed to the outcomes observed. And that leads us to the next question - how much influence is good enough?

Food for thought…you can read an earlier version of the article here (pdf).

Glenn