World Metrology Day and Evaluation

As a blog dedicated to the art of intelligent measurement, we cannot let World Metrology Day go by without a fanfare.

According to the day’s organisers Metrologists are scientists who specialise in measurement techniques, so the team at “Intelligent Measurement” and Benchpoint can justifiably add a new descriptor to their various activities.

I can demonstrate how far intelligent measurement has come by describing my day’s activities. I am finalising charts and data for a big survey we are running for the Berlin Measurement Summit on how PR people and communicators measure their success. (There is still time to take part if you are in the business – www.benchpoint.com/measure.html).

I am also working on data to measure the effectiveness of the European Broadcasting Union’s recent conference in Lucerne, and putting together the finishing touches to a study measuring the effectiveness and reach of a UK professional association’s communications activities. For another client, a large international construction group, we are halfway through a measurement of its organisational effectiveness; for another we are putting together a programme of regular monthly surveys to take the management “pulse” in a fast changing environment.

These are all measurements which will put a substantive value to things which were only recently regarded as intangible, and immeasurable.  Many organisational functions were judged by subjective discussion, and levered by political manoeuvring. Through intelligent measurement, it is now possible to allocate budgets and resources to projects and techniques which are the most likely to bear fruit in achieving organisational change or excellence.

Happy World Metrology Day!

Richard

US summer program on evaluation

 If you are not able to go to Turin and if you are based in the US or close by, you might want to consider the summer evaluation courses taking place in Atlanta, Georgia, USA (pictured above) from 14-17 June 2009.

Organised by the American Evaluation Association and the Centers for Disease Control and Prevention,  there are more than 40 workshops on offer over four days, in addition to keynote speakers. Here are some workshops that I would love to attend:

• What Counts as Credible Evidence in Contemporary Evaluation
• How Policy Is Made and How Evaluators Can Affect It
• Making Your Final Report Useful and Used
• Using Social Network Analysis in Program Evaluation
• Taking it Global: Tips for International Evaluation
• Every Picture Tells a Story: Flow Charts, LogFrames 
• Evaluation 2.0: Measuring Connected Communications  

More information and registration>>

11 hints for a successful evaluation

What makes a successful evaluation? I don’t mean a *success* in terms of producing *pleasing* results  – but more so an evaluation that is successfully managed, respects all stakeholders and is of practical use to end users…I’ve been giving it some thought and I’ve come up with 11 hints for a successful evaluation, summarised in below and available with explanatory texts as a fact sheet (pdf):

1. Make clear the unclear from the start
2. Learn about what you are evaluating from the start
3. Invest in planning
4. Verify verify verify
5. Communicate communicate communicate
6. Collect only information you need
7. Expect the unexpected
8. Report your findings in a brief illustrative way
9. Don’t hide the limits of your evaluation
10. Don’t be disappointed if nothing happens immediately with your findings
11. Give something back for all that you are taking

View the fact sheet with explanatory texts (pdf)>>

Do you have any other hints to add?

Eleven evaluation blogs

The American Evaluation Association has published a list of 11 blogs that focus on evaluation – we are happy to see that Intelligent Measurement blog is amongst them – along with many other interesting blogs. Certainly more evaluators could/should be blogging?

View the full list here >>

95 theses on evaluation

Disturbed by the state of affairs in evaluation, Professor Cronbach and colleagues wrote a 95 theses on reform in evaluation (inspired by Martin Luther’s 95 theses in 1517). They speak of the need for:

“A thoroughgoing transformation. Its priests and patrons have sought from from evaluation what it cannot, probably should not, give.”

Although written 28 years ago, the 95 theses (pdf) makes may pertinent points still valid today.

Here are several favourites that have stood the test of time (no. 75 is my favourite):

9. Commissioners of evaluations complain that the messages from evaluations are not useful, while evaluators complain that the messages are not used.

35. “Evaluate this program” is often a vague charge because a program or a system frequently has no clear boundaries.

49. Communication overload is a common fault; many an evaluation is reported with self-defeating thoroughness.

75. Though the information from an evaluation is typically not used at a foreseeable moment to make a foreseen choice, in many evaluations a deadline set at the start of the study dominates the effort.

95. Scientific quality is not the principle standard; an evaluation should aim to be comprehensible, correct and complete, and credible to partisans on all sides.

Read the full 95 theses (pdf) – despite this poor copy it’s well worth a read. The 95 theses originally appeared in the book “Towards reform of program evaluation“.

Glenn

From broad goals to specific indicators

No doubt you have heard of the Millenium Development Goals (MDGs), eight broad goals on poverty, ill-health, etc, agreed upon by all countries to try and reach by 2015.

From a monitoring and evaluation point-of-view, what is interesting is that these goals are broad sweeping statements, such as:

Goal 1: Eradicate Extreme Hunger and Poverty

Goal 3: Promote Gender Equality and Empower Women

One could ask – how can these broad goals be possibly monitored and evaluated?

As detailed on this MDGs monitoring website, what has been done is to set specific indicators for each goal, for example:

Goal 3: Promote Gender Equality and Empower Women

Description: Eliminate gender disparity in primary and secondary education, preferably by 2005, and in all levels of education no later than 2015

Indicators:
3.1 Ratios of girls to boys in primary, secondary and tertiary education
3.2 Share of women in wage employment in the non-agricultural sector
3.3 Proportion of seats held by women in national parliament

So from broad goals, the MDGs focus on two to seven specific indicators per goal that they are monitoring. That’s an interesting approach, as often we see broad goals set by organisations and then no attempt made to actually detail any indicators.

the MDGs monitoring website plays an active role in monitoring these indicators combining quantitative data (statistics) and qualitative data (case studies) – also an interesting approach to show how such indicators can be tracked.

Glenn

Social network analysis and evaluation

Measuring networks can have many applications: how influence works, how change happens within a community, how people meet, etc. I’m interested in measuring networks as indicator of how contacts are established amongst people, particularly in events and conferences, as I’ve written about previously.

In this area, there is a new resource page available on social network analysis and evaluation from M&E news. The page contains many useful resources and examples of network analysis and evaluation for non-profit organisations, education, events and research and development – including one from myself.

(Above image is from a network analysis of a conference, further information is available here>> )

Glenn

Cultural issues in evaluation

Having spent the last week in the Congo – mostly in Kisangani (pictured above) for an evaluation project, I’ve been thinking about cultural issues and evaluation – in particular how evaluators are perceived in different societies as I’ve written about before.

Interestingly, when I was recently in Central Asia, it was explained to me that evaluation in the Soviet tradition was traditionally seen as an inspection-like function which would search for small mistakes for which people could then be punished for (demotion or worse..).

In Africa, the perception is quite different. People see you as coming to listen, investigate and relay what you have found. Those working with NGOs are now familiar with evaluation.

Of course, cultural issues and how you are percieved can affect your evaluation. I don’t believe there are any quick learning points except to understand as much as you can about the cultural you are working in – and to test your evaluation methodology and questions by discussing with local people prior to any gathering of data.

This article (pdf) has some interesting points on evaluating across cultures, for example, explaining local relevance and usefulness of the evaluation and to be careful in the use of questionnaire types (such as the Likert scale) which may be misunderstood in some contexts.

Glenn

network mapping tool

As regular readers will now, I am interested in network mapping and have undertaken some projects where I have used network mapping to assess networks that have emerged as a result of conferences.

Here is quite an interesting tool, Net-Map, an interview-based mapping tool. The creators of this tool state that it is a “tool that helps people understand, visualize, discuss, and improve situations in which many different actors influence outcomes”.

Read further about the tool and view many of the illustrative images here>>

Glenn

Tonsils, run over dogs and comparisons

In evaluation, we often make judgements based on “feelings” or “gut reaction” without any proper inquiry or comparison with other data. That is why this story about Ludwig Wittgenstein, the Austrian philosopher appealed to me. Apparently he telephoned a friend in hospital, Fania Pascal, who told the following story:

“I had my tonsils out and was in the Evelyn Nurshing Home feeling sorry for myself. Wittgenstein called. I croaked: “I feel just like a dog that has been run over.” He was disgusted: “You don’t know what a dog that has been run over feels like.”

The point being that Fania Pascal (in the hospital) is making a comparison that she cannot possibly provide any support for – how could she know what it feels like to be a dog that has been run over?

In the same way, you often hear people saying “our results are terrible” – or “we are doing too much of XY”. But my first reaction is “How do you judge that – what are you comparing it to?” – often no real inquiry or comparative data are used (which reminds me of another quote from Groucho Marx).

For those interested, the above quote comes from the book “On Bullshit” by Harry G. Frankfurt – well worth a read.

Glenn