I’ve just finished an interesting evaluation study on LIFT07, an international conference on emerging technology and communications that was held in Geneva during February 2007 - you can view the complete evaluation report here (pdf -339 kb). Our main evaluation tool was a survey of the conference attendees (48% of participants completed the survey).
Apart from providing useful feedback for the conference organisers that will assist them in improving future conferences, the study set out to find out what was the longer term impact of the first LIFT conference (held in February 2006). By surveying attendees that participated in both the 2006 and 2007 conferences we were able to “track” some key points of changes in attitudes and behaviours and to what extent they could be attributed to the LIFT conference. My findings are summarised in this diagram (pdf).
What I found very interesting is that one year after the conference, 28% of attendees (of a 50 person sample) said they started new activities partly due to LIFT06, such as forming a partnership, creating a blog or launching a new partnership. Further, 90% of attendees said that the conference influenced them in finding and exchanging information.
Of course, we have to recognise the limitations of the study, notably that it is self reported (and not backed-up by independent confirmation) and it is a relatively small sample (i.e. 17.5% - 50 people out of 285 participants). Nevertheless, we can make certain conclusions that the conference did have longer term impact in quite precise areas with some participants; establishing new contacts, inspiring new ideas and ways to find and exchange information.
Glenn
June 19, 2007
An interest of mine is looking at the short & long term impact of conferences and workshops. A lot of work has been done on evaluating the impact of training that I have written about before. Basically, we can look at four levels of impact: 1. Reaction, 2. Learning, 3. Behavior & 4. Results. A lot of conference/workshop evaluation focus on the “reaction” aspect - what did participants like/prefer about an event.
But more interesting is to look at the learning, behavior and- if possible - results aspect. This usually takes time - however, if we are clear about what a workshop/conference is trying to achieve, we can often identify changes in learning/behavior in the short term.
A practical example. When I ran the “Do-It-Yourself Monitoring and Evaluation” workshop (pictured above that’s - David Washburn and myself at the workshop) at the LIFT07 conference, my main objective was to get people thinking about how they could integrate monitoring and evaluation into their own projects. Using a basic evaluation framework (pdf) groups worked to break down projects into the main steps needed for evaluation.
So was the “learning” aspect succesful? - I’d like to think so. Quite a few people commented to me how it got them thinking about monitoring/evaluation and what they could do with their own projects. Also, the following participants blogged about the workshop, an indication of what they took away from the workshop - and also crossing into the “behaviour” area: they processed some thoughts and took the action (behaviour) of writing about it:
Even more so, one participant told me about how he used the information from the workshop the same week, which supports my idea about the possiblity to identify short term impact, even in terms of behaviour:
“When we got back from the workshop, I took out the evaluation framework and sat down with my colleagues and planned out what we were going to monitor and evaluate in our major projects, setting down objectives and evaluation indicators. So we can use the framework as a guide in the coming six months.”
Glenn
February 23, 2007
As part of the LIFT07 conference in Geneva, I ran a workshop on “Do-It-Yourself Monitoring and Evaluation” today. Some 40 people attended and the participants did some interesting work in designing evaluation frameworks for their projects. My presentation can be found here:
D.I.Y. Monitoring and Evaluation (pdf - 690 kb)
There was also some interest in the following areas:
- Network mapping; Rick Davies has an excellent web page on this methodology.
- Relationship measurement; a good start are the IPR guidelines.
- Most Significant Change Method and Outcome Mapping are described on this web page quite well (scroll down and you can download the pfd with more details).
- And there was a lot of interest in Blog Return-on-investment. You can read about the work of Forrester Research in this area, but you’d be wise to read the counterpoints on the blogs of K D Paine and LeverWealth.
Glenn
February 7, 2007
Last week I gave a presentation on evaluation at the First ISO and IEC Marketing and Communication Forum which took place in Geneva. The forum gathered communications and marketing professionals from all over the world working in the field of standards development.
My presentation focused on some of my favourite topics of evaluation, notably:
- Why don’t marketing/communication professionals evaluate
- The need for clarity in setting marketing/communication objectives
- How low cost evaluation can be undertaken
- The risks of being over creative in communications
On the last point, I used the example of the Got Milk campaign which has been lauded as one of the most visible and creative ad campaigns of all time (notably by the advertising industry). However, did the highly creative ads actually help achieve campaign objectives? That is, to get people to drink more milk? Well, milk consumption continues to decline and the ads have been criticised for not addressing a key concern for teenagers - that they consider milk to be fattening.
And that’s the point I tried to make, that creativity all is well and good - but it has to help communicators achieve their campaign goals - and be measurable.
My full presentation can be downloaded here:
Presentation: Effective Marketing & Communications through Evaluation (pdf - 1 MB)
Glenn
Acknowledgement: the example of the Got Milk campaign comes from the book “The Fall of Advertising and the Rise of PR”, A & L Ries.
December 11, 2006
The LIFT blog beat me to it in announcing that I will be involved in working with the LIFT team in evaluating the 2007 event that will take place in February 2007. LIFT is an international conference that takes place in Geneva and looks at the relationship between technology and society.
My experience with LIFT 06 in evaluating the reactions and initial impact to the event is written up in this journal paper (pdf) or directly on the LIFT website.
In 2007, I hope to go further by exploring the impact of social media on the event setting and looking at networks that develop. It should be fun!
Glenn
November 6, 2006
For those of you in the Geneva (Switzerland) here is an event I’m helping organise that you may be interested to attend:
As part of the Future Series, the International University in Geneva and Geneva Women in International Trade present a roundtable:
“The Future of Corporate Communications”
Tuesday 14 November 2006, 6 p.m.
Movenpick Hotel, ICC, Rte de Bois 20, 1215 Geneva
Panel members:
- Cary Adams, Managing Director, Lloyds TSB International Private Banking
- Charlotte Lindsey, Deputy Director of Communications, International Committee of the Red Cross
- Peter Warne, Senior External Communications Manager, Nestle
“Through blogs, wikis and podcasts, publics are conversing amongst themselves and building influential sources of information as alternatives to traditional media. Are organisations ready and willing to converse with publics? Can corporate communications cope with competing sources of influence and use social media to build relationships with their key audiences?”
Entry fee: 25 CHF (15 CHF for GWIT members and IUN Alumni)
Agenda:
6.00 p.m. Welcome
6.30 p.m. Presentations followed by discussions
7.30 p.m. Cocktail and networking
Register online for the event.
Glenn
October 30, 2006
At the international conference of the European Evaluation Society, I attended an interesting workshop on Assumptions Based Comprehensive Development Evaluation Framework (ABCDEF) presented by Professor Osvaldo Feinstein, evaluation consultant. The main thrust of his workshop was to challenge us to consider fully the assumptions that are made in development projects - and consequently the impact on evaluation. He has created a guide to what we should consider when exploring assumptions, namely: Incentives, Capacities, Adoption, Risk, Uncertainty and Sustainability. Cleverly, it makes the acronym Icarus, whom we all know flew too close to the sun which melted the wax holding together his wings. And that is the underlying theory of Professor Feinstein, as he put it:
“An unexamined assumption can be very dangerous!”
More information on ABCDEF can be found in the Sage Handbook of Evaluation.
Glenn
October 9, 2006
The International Conference of the European Evaluation Society is underway in London and I just participated in several of the pre-conference workshops.
In one workshop, Elliot Stern, a senior Evaluation Consultant made a very poignant point - is the aim of evaluation “to prove” or “improve”?
A simple but interesting distinction - traditionally he pointed out most evaluation aimed to “prove” if an intervention changed anything - and that’s it (a gasp when through the room…)
It’s only a rather recent development that evaluation has been asked to focus also on “improve” - how can an intervention be more effective.
Most likely you are thinking that it can’t be so - how can an evaluation not make the step from ”prove” to “improve” ? But often the terms of reference for an evaluator is only to “prove” - in other words, evaluate but please no recommendations.
Glenn
October 4, 2006
For communication professionals in the Geneva region, Switzerland, the following event may be of interest to you:
“The Executive MScom Program (University of Lugano) in collaboration with the Société Romande de Relations Publiques (SRRP) and the International Public Relations Association (IPRA) is launching the MScom Geneva Communications Forum. The conference series, to take place twice yearly, will provide expert perspectives on the fast-changing field of communication management. The theme of the 1st Geneva communications forum will be:
“The changing communications landscape in the age of global business”
Keynote speakers:
Harold Burson, Founder and Chairman of Burson-Marsteller
Keith Rockwell, Spokesperson of the World Trade Organization
Date and location:
Wednesday, October 11, 2006, 18:00 hrs
Swissôtel Métropole Geneva
Cost:
CHF 75 - members of the public
CHF 50 - members of SRRP, IPRA & MScom Alumni Association”
More information can be found in the invitation (pdf) and online registration is possible on the mscom website.
Disclaimer: I am a graduate of the Executive MScom Program.
Hope to see some of you there.
Glenn
September 12, 2006
I see the that program of the 4th annual measurement summit (to be held in NH, US) has been announced by the Institute for Public Relations. This is an important conference for people interested in communications and evaluation. There are certainly some interesting topics being addressed: Shell Corporation on reputation tracking, Procter and Gamble on ROI and PR and a panel discussion featuring three of my favourite “scholars” in the field of evaluation: Larissa Grunig, James Grunig and Jim MacNamara. The full program is available online (pdf).
For those of us who will not be able to make it, let’s hope they publish the main presentations and conclusions (they usually are kind enough to do so so).
Glenn
July 19, 2006
