intelligent measurement

I’ve just finished a very interesting project - the evaluation of the impact of the LIFT06 conference that took place in Geneva in February 2006. In a true open source spirit, the evaluation report is available for everyone to consult. With this evaluation, we tried to go beyond the standard assessment of reactions to a conference. We looked at changes to knowledge, attitudes and behaviours. Using a triangulation approach combining quantitative and qualitative research methods, I believe we could identify the influence of LIFT06 on these above variables. We were aware of the limitations to the evaluation given that it was a punctual evaluation and based largely on self-assessment of attitudinal and behavioural changes, which I explain here in the report.

What sort of changes could we identify?

Changes to awareness and attitudes: Through and online survey, the majority of attendees (82%) agreed that LIFT06 provided them with interesting information on the usage of emerging technologies and 70% agreed that LIFT06 influenced what they thought about the subject. This quote taken from an attendee’s blog illustrates this point: 

“And just think; if I had never gone to Lift06 I would not be feeling anything like this strongly about the issue”

 Changes to Behavior: Evaluations of conferences are rarely able to show a direct relation between the event and changes in behaviour of attendees. With LIFT06, some attendees indicated a change in behaviour, such as starting a blog or getting a new partnership.  another key objective of LIFT06 was to “connect” people - 94% of attendees reported that they met new people at LIFT06.

 As I'm setting up and running online surveys using the Benchpoint system (commercial plug), I am always interested to see examples of best practices in online surveying.  Sometimes we come across examples of worst practice also. Below I've copied in a screenshot of a survey I was asked to complete. It breaks a simple rule of surveying that dates back to phone and street surveying: personal / demographic information should normally be requested at the end of a survey. As it has been found that people are more comfortable answering such questions once they are familiar with the survey theme and if done in person, with the interviewer. I found the amount of personal information this survey asks even before you "start survey" is excessive and was the reason why I didn't complete the survey. Hopefully general standards on online surveying are emerging to avoid this type of issue.

 Glenn

Chris Mykrantz of Watson Wyatt writes about his company’s latest communication ROI study on the Simply-Communicate website.

For years internal communications people have been arguing for a seat at the top table where decisions are taken, and several have made it. But, warns Chris,

“Be careful what you wish for. Senior leadership is now asking what value they’re contributing by sitting there”.

He mentions large companies who are actively looking for a financial–based ROI measurement of communication, a statement which may have a few communicators who measure by instinct quaking in their shoes!

One of his key points, with which we heartily agree, is :

“Start at the end and think hard cash - the first step in developing a communication strategy that includes measurement should be to envision the successful outcome. What would be different in your organization if the communication was on target? How would it change the business? How would it change employees? Don't settle for awareness or satisfaction. Find a desired state that you can define in dollar terms and design your strategy around that.”

Read the full article here.

Richard

This post at the IABC measurement blog caught my attention, as its author asks the question:

"So why don’t we measure more? Is it budget, competencies, time or the risk of accountability?"

People usually don't evaluate for various reasons, but the most common excuses I've heard are the following:

1. "It's too expensive".
   With the amount of free advice, excellent guidelines and cheap research solutions available, this doesn't pass anymore.
2. "I don't know how to".
   Fair enough, but you can learn a lot yourself without having to engage expensive consultants. 
3. "I'm too busy "doing" to be bothered with measuring".
   Frightening. People love doing things, it's natural. But sometimes you have to stop and take a step back to see what you have achieved. 
4. "What I'm doing couldn't possibly be measured".
   Often heard from the Creative Type. People who create their own fonts, too clever campaigns and beautiful artwork that impresses other Creative Types. But my question is - what did you change?
5. "I don't see the value of it".
   How else can you judge the value of your work if you don't attempt to analyse and assess it?
6. "I'm scared what of what I will find out".
   But I think it will be scarier for you if you don't evaluate and someone else does.
7. "People are fed up with giving their opinion".
   I don't think people are - as I've written about before.
8. "My gut feeling tells me I'm doing a good job".
   There is a certain vogue that says out intuition is often our best call. But research often brings out issues that were not even on your radar.
9. "All my work is vetoed by the CEO, if s/he's happy so am I".
   The CEO sees the organisation through the same rose-colored glasses as you do. In PR, it's your public's perception of your communication that counts.
10. "You can't prove anything anyway". 
    You can rarely obtain 100% proof that your programme caused the change seen. But what you do is collect evidence that indicates the role your activity played, as I further explained in this post.

Glenn