Workshop participation & short term impact

An interest of mine is looking at the short & long term impact of conferences and workshops. A lot of work has been done on evaluating the impact of training that I have written about before. Basically, we can look at four levels of impact: 1. Reaction, 2. Learning, 3. Behavior & 4. Results. A lot of conference/workshop evaluation focus on the “reaction” aspect – what did participants like/prefer about an event.

But more interesting is to look at the learning, behavior and- if possible – results aspect. This usually takes time – however, if we are clear about what a workshop/conference is trying to achieve, we can often identify changes in learning/behavior in the short term.

A practical example. When I ran the “Do-It-Yourself Monitoring and Evaluation” workshop (pictured above that’s – David Washburn and myself at the workshop) at the LIFT07 conference, my main objective was to get people thinking about how they could integrate monitoring and evaluation into their own projects. Using a basic evaluation framework (pdf) groups worked to break down projects into the main steps needed for evaluation.

So was the “learning” aspect succesful? – I’d like to think so. Quite a few people commented to me how it got them thinking about monitoring/evaluation and what they could do with their own projects. Also, the following participants blogged about the workshop, an indication of what they took away from the workshop – and also crossing into the “behaviour” area: they processed some thoughts and took the action (behaviour) of writing about it:

• Internet Next Generation blog (David Washburn)>>
• The world we live in (Victoria Visser) >>
• Profession web.ch >>
• Chiperoni.org >>

Even more so, one participant told me about how he used the information from the workshop the same week, which supports my idea about the possiblity to identify short term impact, even in terms of behaviour:

“When we got back from the workshop, I took out the evaluation framework and sat down with my colleagues and planned out what we were going to monitor and evaluate in our major projects, setting down objectives and evaluation indicators. So we can use the framework as a guide in the coming six months.”

Glenn

Beware: dodgy Blog ROI in circulation

Forrester Research have published a new report on the “ROI of blogging” (at USD $ 379 a pop). And I’ve seen that many bloggers have jumped on this with utmost enthusiasm.

Well hold on….

Although Charlene Li of Forrester explains well the ROI model there are some fundamental flaws of the ROI calculation that KD Paine and David Phillips explain further. As KD Paine put it:

“The false assumptions and inaccuracies in this report are scary”

What is the main flaw? Well, the whole blog ROI calculation falls down as it is based on comparing purchased advertising to editorial content, which is a highly discredited way of measuring PR value (read more about comparing advertising to editorial content in this report (pdf) by some leading scholars).

The report does have some interesting points in that it attempts to pull out some of benefits of blogging (such as customer insights) and comparing this to the cost of market research). Certainly the idea of showing how visibility grows from a blog post (through generating comments, thoughts and referrals) to changes in attitudes and behaviors is heading in the right direction, as I’ve written about before.

And as for blogging ROI, I would look more at the cost of working hours in blogging and comparing it to working hours needed to mount a traditional campaign – and comparing the changes to behaviour and attitude using both methods (admittedly easier said than done). That would be more a measure of “efficiency” than anything else.

Glenn

Using employee opinion surveys to guide HR policies

In a previous post Glenn wrote that evaluation is frustrating when no changes result.

Quite so. I have seen many an employee survey gather dust or been quietly forgotten.
Why should employees bother? Often the survey will avoid touchy issues- “The elephant in the room”.  There is often a huge delay in making the survey results public. And nothing changes anyway.

Well, here’s a little case history to cast a ray of hope.

A Benchpoint client in the City of London’s financial sector (often characterised by macho management cultures) wanted to run an anonymous poll of employee attitudes on just about everything relating to the job and the relationship with the employer.

The company had very enlightened HR policies by traditional City standards. A “listening culture”, personal development goals, work/life balance, continuous personal feedback etc. But were the policies working? What needed to change?

More than “just a survey”, The Benchpoint poll enabled important decisions to be taken quickly with the benefit of employee input. The company announced operational changes just 5 days after the survey closed, and immediately after a presentation of the results to all employees.

Employee response and feedback was positive; just one employee was unable to take part. For the first time they had an opportunity to voice their true opinions anonymously and confidentially. And they were impressed by the speed of their management’s reaction.

For management, this was a relatively low-cost exercise, and was particularly economic of senior people’s time. It was also an excellent way of “walking the talk” and demonstrating that HR policies regarding inclusivity, development and teamwork were real.

The data collected provided an insight into the behaviour drivers of groups of employees, and a dynamic tool for future goal setting and benchmarking.

Conclusions

• Don’t do a survey unless you are prepared to communicate the results and make changes.
• Don’t make organisation or operational changes unless you have surveyed and understood the real issues.
• Do it quickly, share the results quickly and make your announcements quickly.
• Don’t assume that everything is OK in your organisation. Measure intelligently to find out what the real issues are.

Richard

D.I.Y. Workshop on Monitoring and Evaluation in Geneva

As part of the LIFT07 conference in Geneva, I ran a workshop on “Do-It-Yourself Monitoring and Evaluation” today. Some 40 people attended and the participants did some interesting work in designing evaluation frameworks for their projects. My presentation can be found here:

D.I.Y. Monitoring and Evaluation (pdf – 690 kb)

There was also some interest in the following areas:

• Network mapping; Rick Davies has an excellent web page on this methodology.
• Relationship measurement; a good start are the IPR guidelines.
• Most Significant Change Method and Outcome Mapping are described on this web page quite well (scroll down and you can download the pfd with more details).
• And there was a lot of interest in Blog Return-on-investment. You can read about the work of Forrester Research in this area, but you’d be wise to read the counterpoints on the blogs of K D Paine and LeverWealth.

Glenn