I recently conducted a one day training workshop for the staff of Gellis Communications on communications evaluation. We looked at several aspects including:

• How to evaluate communication  programmes,  products and campaigns;
• How to use the “theory of change” concept;
• Methods specific to communication evaluation including expert reviews, network mapping and tracking mechanisms;
• Options for reporting evaluation findings;
• Case studies and examples on all of the above.

Gellis Communications and myself are happy to share the presentation slides used during the workshop – just see  below (these were combined with practical exercises – write to me if you would like copies)

Online tools, such as corporate websites, members’ directories or portals increasingly play an important role in communications’ strategies.  And of course,  they are increasingly important to evaluate.

I just concluded an evaluation of an online tool, created to facilitate the exchange of information amongst a specific community. The tool in question, the Central Register of Disaster Management Capacities is managed by the United Nations Office for the Coordination of Humanitarian Affairs.

The evaluation methodology that I used for evaluating this online tool is interesting as it combines:

• Content analysis
• Network mapping
• Online survey
• Interviews
• Expert review
• Web metrics

And for once, you can dig into the methodology and findings as the evaluation report is available publicly: View the full report here (pdf) >>

I’m currently in Brussels for some evaluation training with Gellis Communications and in our discussions the use of Likert Scale in surveys. As I’ve written about before, the Likert scale (named after its creator pictured above) is widely used response scale in surveys. My earlier post spoke about the importance of labelling points on the scales and not to use too many points (most people can’t place their opinion on a scale of more than seven). Here are several other issues that have come up recently:

To use an even or odd scale: there is an ongoing debate on the Likert scale as to whether you should use an odd (five point for example) or even (four point for example).  Some advocate and odd scale where respondents can have a “neutral” middle point whereas others prefer to “force” people to select a negative or positive position with an even scale (e.g four points). In addition, the use of a “don’t know” option is inconclusive.  I personally believe that a “don’t know” option is essential on some scales where people may simply not have an opinion.  However, studies are inconclusive if such an option increases accuracy of responses.

Left to right or right to left: I always advocate displaying scales from the negative to the positive, left to right. It seems more logic to me and some automated survey software mark your answers and calculate the responses for graphs on this basis, e.g. that the first point is the lowest. But I’ve had heard others argue that it should be the opposite way around – put positive to negative, left to right – as people will click on the first point by default in online surveys – which I personally don’t believe.  I’ve not yet found any academic reference supporting either way but looking at all examples in academic articles, 95% are written as negative to positive, left to right – some evidence in itself!

Mr Likert you have a lot to answer for!

An interesting online course on monitoring and evaluation:

Title: Results Based Monitoring and Evaluation

(6 two hour sessions over 6 weeks from 04/11/09 to 16/12/09)

This course will help participants understand the key components for effective M&E in order to improve management performance and evidenced based decision making. By the end of the course participants will have gained the skills and experience to design and implement a Results Based Monitoring and Evaluation system which they can use in their work. To view course content click on the link below.

http://www.uk.locallivelihoods.com/Moduls/WebSite/Page/Default.aspx?Pag_Id=132

Note: this blog has no commercial association with the course organisers – it just seems an interesting course!

As I’ve written about before, the way in which we present evaluation findings – usually in a long undigestable report – certainly has its limitations.  It’s been sometime I’ve been thinking that with the developments in multimedia there must be better ways than the written document to communicate evalution findings – and here it is! We’ve just completed a multimedia video report on the evaluation of the LIFT France conference:

This is certainly the way forward. Thanks to Patricia (concept & inteviews) , Thierry (filming & production), Benchpoint (survey) and Yona (graphics).

A colleague is looking at how people are using social media and what do they value in it – you can help by filling in this quick survey (it takes about one minute).

We will report on the results here in the coming weeks – stay tuned!

I just read some interesting survey results that found that 84% of professionals don’t measure the Return-On-Investment for the social media programs that they run.

It’s not a surprising result, but is it the right question? I would have asked how many professionals are measuring the success of their social media programs and what and how they are measuring. That would be a good starting point before an eventual ROI measurement. Follow further the debate on Mashable.

This introductory course to monitoring and evaluation could be of interest:

Venue: Online at http://pcmitraining.com/course/category.php?id=2

Date: 1st October and 1st of the month thereafter

Introduction to Monitoring and Evaluation

Introduction to Monitoring and Evaluation is a four week interactive e-learning workshop for individuals who are new to the field of monitoring and evaluation or those who wish to formalise their existing understanding which has been developed through work based experience. The course considers important concepts and activities from inception through to outcome evaluation and additionally includes valuable project management and leadership techniques. The course is accredited and requires approximately 7.5 hours participant input per week, including assignments, and is priced at £750. The course runs once a month, starting on the 1st of each month. 

Further information: http://pcmitraining.com/course/category.php?id=2

  An interesting new publication is available from the UK National Council for Voluntary Organisations (NCVO)  on campaign evaluation. “Is your campaign making a difference?”  (cover pictured to the left) focuses on explaining how measurement can be done simply and effectively for campaigns.

The publication costs 25 GDP but you can download the first section for free (pdf)>>