Thoughts from the Berlin Measurement Summit

The 1st European Measurement Summit was a great success.  Delegates are busy doing a survey (organised by Benchpoint), and the feedback is looking very positive.
My Highlights:
Neil Martinson, head of press and PR in the UK Government’s Central Office of Information (COI), spends £25million (€29.3 million) on PR every year, so is fairly interested in knowing which half is wasted. He asked five  media measurement and evaluation specialists to do some test measurements on a recent campaign. The result? Five very different measurements, and no agreement on criteria or methodology.
David Rockland’s sprited defence of AVE’s (Advertising Value Equivalents). Actually, it’s quite a good measure of penetration, reach and performance. The only trouble is the V word. People manipulate the figures to give the impression that editorial is worth more than advertising, which is hardly objective or honest. And no two people seem to do it the same way (see above) But there has to be a way of integrating this figure with other measures to give a true index of success. By the way David is MD of Ketchum’s global research network, and knows a thing or two. Methinks the detractors are a little over the top on this one.
Social Media. Half the conference were struggling to understand what Social Media is and how to use it, while the other half were on line to each other commenting on what each speaker was saying, without the bother of joining in the discussion with other delegates. I was chuffed when the delegate in front of me started reading this blog  during one of the presentations. Should I join Twitter? Or is it just people shouting, and no-one else listening?

I shall be returning to more serious content in future posts. But meanwhile, please comment or contribute to the ongoing debate.

Richard

Summer School in Italy on Monitoring and Evaluation

What could be better than spending some summer weeks in Bologna, Italy (pictured above) and learning about monitoring and evaluation?

Well, the University of Bologna in Italy is organising a summer school on monitoring and evaluation of international programmes and public policies from June 8 – 13  2009.

Further information on the programme:

“The 2009 Summer School On Monitoring And Evaluation Of International Programmes And Public Policies is part of the University of Bologna International Summer School Programme. It is organised by the Centre for International Development (www.cid-bo.org) in cooperation with the Faculty of Political Sciences, the Department of Economics, and the Department of Political Science of the University of Bologna and with the Office for International Programmes of the State University of New York at Albany.

Enrolment fee is € 1,500, and scholarships are available for participants from low-income countries.”

More information and registration>>

Evaluation Professional Development Course on the “Concept Mapping” Method – 18 & 19 June 2009

For those in Switzerland, there is an interesting course (in English) coming up on “Concept mapping” organised by the Swiss Evaluation Society, details as follows:

The concept Systems method guides action plan development, strategic planning, implementation, evaluation and other activities in which group input is valued. Concept mapping takes the ideas of individuals and combines them to form unique visual representations called Concept Maps.

Dates: 18 & 19 June 2009

Location: University of Fribourg, Switzerland

Facilitators: Prof. Bill Trochim, Cornell University and Mary Kane, President of Concept Systems USA

Cost: CHF 700 (reduction of CHF 50 for Swiss Evaluation Society members

View the flyer (pdf)

More information and registration>>

Online courses in evaluation

Claremont Graduate University (USA) is offering three online courses in evaluation in the next months at very reasonable prices:

* March 28-29: Hard-Core Qualitative Research Methods, Part A (Michael Scriven)
* April 4-5: Hard-Core Qualitative Research Methods, Part B (Michael Scriven)
* May 19-20: Introduction to Practical Program Evaluation: A Theory-Driven Approach (Stewart I. Donaldson)

Each course includes two 3-hour training sessions in an online virtual classroom  environment, available to anyone with aninternet connection anywhere in the world.

These courses are offered at USD $90 each.

Full descriptions are available on their website>>

(the authors of this blog have no commercial relationship with the courses – they just seem interesting and at a good price!)

First European Summit on Communication Measurement 10 – 12 June 2009

The first European Summit on Communication Measurement is planned for 10 – 12 June 2009 in Berlin, Germany and the program looks very interesting so far. The summit starts with a full day of workshops and the speakers over the next two days include:

* Bjorn Edlund, VP Communications, Shell International
* Pablo Fernández Calvo, Director of Communication, BBVA
* David Michaelson, President USA, Echo Research
* Neil Martinson, Director of News & PR, UK’s Central Office of Information
* Frank Oviatt, President & CEO of the Institute for Public Relations
* Katie Paine,CEO, KD Paine & Associates
* David Rockland,Partner,Managing Director,Ketchum
* Dr Christopher Storck, Principal, HeringSchuppener
* Dr Tom Watson, Bournemouth University
* Dr Donald K Wright, Boston University
* Professor Dr Ansgar Zerfass, EUPRERA/University of Leipzig

Some of the topics to be covered include:

* Social Media – The Big New Opportunity for Communications
* Traditional Media is Adapting with Confidence to Embrace the Challenge of Social Media
* How Clients see Media Change and Experiments with Communications Models
* International Measurement – an Insight into the Latin American Regional Analysis for BBVA
* In Defence of AVE’s
* What Media Evaluation will look like in the future: the next generation
* Public Relations in Europe – Status Quo and Future Challenges
* The Comparative Communications Effectiveness of Advertising and PR
* Social Media Measurement Case Studies
* The Role of Research in Communications
* Predicting the Future for Communications-What International Research Tells Us
The conference is organised by the International Association for Measurement and Evaluation of Communications , the Institute for Public Relations (US) and the German Public Relations Public Relations Association (DPRG).

More information and registration>>

Conference – building for the future: evaluation in governance, development & progress

The European Evaluation Society has it’s biennial conference (full title above) scheduled for 1-3 October 2008 in Lisbon, Portugal. The conference kicks off with a series of hands-on workshops, that I attended at the last conference and found very useful. Read more on the conference website.

I’ll be presenting a paper at the conference on “Evaluating Public Information and Advocacy Campaigns”, so if you are going, please drop me a line…

Glenn

Workshop participation & short term impact

An interest of mine is looking at the short & long term impact of conferences and workshops. A lot of work has been done on evaluating the impact of training that I have written about before. Basically, we can look at four levels of impact: 1. Reaction, 2. Learning, 3. Behavior & 4. Results. A lot of conference/workshop evaluation focus on the “reaction” aspect – what did participants like/prefer about an event.

But more interesting is to look at the learning, behavior and- if possible – results aspect. This usually takes time – however, if we are clear about what a workshop/conference is trying to achieve, we can often identify changes in learning/behavior in the short term.

A practical example. When I ran the “Do-It-Yourself Monitoring and Evaluation” workshop (pictured above that’s – David Washburn and myself at the workshop) at the LIFT07 conference, my main objective was to get people thinking about how they could integrate monitoring and evaluation into their own projects. Using a basic evaluation framework (pdf) groups worked to break down projects into the main steps needed for evaluation.

So was the “learning” aspect succesful? – I’d like to think so. Quite a few people commented to me how it got them thinking about monitoring/evaluation and what they could do with their own projects. Also, the following participants blogged about the workshop, an indication of what they took away from the workshop – and also crossing into the “behaviour” area: they processed some thoughts and took the action (behaviour) of writing about it:

• Internet Next Generation blog (David Washburn)>>
• The world we live in (Victoria Visser) >>
• Profession web.ch >>
• Chiperoni.org >>

Even more so, one participant told me about how he used the information from the workshop the same week, which supports my idea about the possiblity to identify short term impact, even in terms of behaviour:

“When we got back from the workshop, I took out the evaluation framework and sat down with my colleagues and planned out what we were going to monitor and evaluate in our major projects, setting down objectives and evaluation indicators. So we can use the framework as a guide in the coming six months.”

Glenn

D.I.Y. Workshop on Monitoring and Evaluation in Geneva

As part of the LIFT07 conference in Geneva, I ran a workshop on “Do-It-Yourself Monitoring and Evaluation” today. Some 40 people attended and the participants did some interesting work in designing evaluation frameworks for their projects. My presentation can be found here:

D.I.Y. Monitoring and Evaluation (pdf – 690 kb)

There was also some interest in the following areas:

• Network mapping; Rick Davies has an excellent web page on this methodology.
• Relationship measurement; a good start are the IPR guidelines.
• Most Significant Change Method and Outcome Mapping are described on this web page quite well (scroll down and you can download the pfd with more details).
• And there was a lot of interest in Blog Return-on-investment. You can read about the work of Forrester Research in this area, but you’d be wise to read the counterpoints on the blogs of K D Paine and LeverWealth.

Glenn

Does Creativity Equal Results?

Last week I gave a presentation on evaluation at the First ISO and IEC Marketing and Communication Forum which took place in Geneva. The forum gathered communications and marketing professionals from all over the world working in the field of standards development.

My presentation focused on some of my favourite topics of evaluation, notably:

• Why don’t marketing/communication professionals evaluate
• The need for clarity in setting marketing/communication objectives
• How low cost evaluation can be undertaken
• The risks of being over creative in communications

On the last point, I used the example of the Got Milk campaign which has been lauded as one of the most visible and creative ad campaigns of all time (notably by the advertising industry). However, did the highly creative ads actually help achieve campaign objectives? That is, to get people to drink more milk? Well, milk consumption continues to decline and the ads have been criticised for not addressing a key concern for teenagers – that they consider milk to be fattening.

And that’s the point I tried to make, that creativity all is well and good – but it has to help communicators achieve their campaign goals – and be measurable.

My full presentation can be downloaded here:
Presentation: Effective Marketing & Communications through Evaluation (pdf – 1 MB)

Glenn

Acknowledgement: the example of the Got Milk campaign comes from the book “The Fall of Advertising and the Rise of PR”, A & L Ries.

LIFT 07 – evaluation, networks & social media

The LIFT blog beat me to it in announcing that I will be involved in working with the LIFT team in evaluating the 2007 event that will take place in February 2007. LIFT is an international conference that takes place in Geneva and looks at the relationship between technology and society.

My experience with LIFT 06 in evaluating the reactions and initial impact to the event is written up in this journal paper (pdf) or directly on the LIFT website.

In 2007, I hope to go further by exploring the impact of social media on the event setting and looking at networks that develop. It should be fun!

Glenn