intelligent measurement

I've written previously about measuring networks and how it can be very useful for an organisation to assess the different links between their key stakeholders. I came across another approach to mapping stakeholders which has some very good elements: the clarity concept identifies stakeholders and maps the relationships based on attitude, importance and influence of publics and the strength of the relationships (see the example above). You can read further about this concept on the stakeholder relationship management blog.

Although this is essentially a proprietary solution (offered by Clarity CS), what I appreciate is that the people behind the solution offer all their thoughts, theories and ideas on the subject so you can learn a lot from their approach - at no cost.

Disclaimer: there are no links between the authors of this blog and Clarity CS. Although one of the founders of this solution, Jon White, was a lecturer at my masters programme).

Glenn  

Looking further into outcome evaluation in different fields, I came across a very good resource with a rather quaint title: The Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources.  Interesting to note the similiarities between my own top ten excuses for not evaluating and the common myths about evaluation that the author lists; as follows:

Myth: Evaluation is a complex science. I don’t have time to learn it! No! It’s a practical activity. If you can run an organization, you can surely implement an evaluation process!

Myth: It’s an event to get over with and then move on! No! Outcomes evaluation is an ongoing process. It takes months to develop, test and polish — however, many of the activities required to carry out outcomes evaluation are activities that you're either already doing or you should be doing. Read on …

Myth: Evaluation is a whole new set of activities – we don’t have the resources No! Most of these activities in the outcomes evaluation process are normal management activities that need to be carried out anyway in order to evolve your organization to the next level.

Myth: There’s a "right" way to do outcomes evaluation. What if I don’t get it right? No! Each outcomes evaluation process is somewhat different, depending on the needs and nature of the nonprofit organization and its programs. Consequently, each nonprofit is the "expert" at their outcomes plan. Therefore, start simple, but start and learn as you go along in your outcomes planning and implementation.  

Myth: Funders will accept or reject my outcomes plan No! Enlightened funders will (at least, should?) work with you, for example, to polish your outcomes, indicators and outcomes targets. Especially if your's is a new nonprofit and/or new program, then you very likely will need some help — and time — to develop and polish your outcomes plan.

Myth: I always know what my clients need – I don't need outcomes evaluation to tell me if I'm really meeting the needs of my clients or not You don’t always know what you don’t know about the needs of your clients – outcomes evaluation helps ensure that you always know the needs of your clients. Outcomes evaluation sets up structures in your organization so that you and your organization are very likely always focused on the current needs of your clients. Also, you won’t always be around – outcomes help ensure that your organization is always focused on the most appropriate, current needs of clients even after you've left your organization.

Glenn

The new "Google Trends" product is a nice complementary tool for monitoring the "noise" out there on an issue. It may prove useful for organisations that want to track general interest on a trend, or on related issues to see if there is any correlation.

What is interesting is that we can get an idea of the impact of events and policy announcements on issues - and how it spikes interest in an issue - and thus increased search results and news stories.

In the graph below, I have chartered WWF (blue line) and climate change (red line). The letters (A, B, C etc.) indicate major news stories on events and announcements. What is interesting is the peak around certain events (like D, a climate change conference) and both searches on WWF and climate change rise slightly. Also there are unexplained peaks in the search and news volume.

What can we conclude from this? Firstly, the search and news volumes on WWF and climate change do mirror each other often in terms of peaks and troughs which could be reassuring to the organisation as climate change is one of its key campaigns. Secondly, it supports the notion that events and policy announcements influence a public's awareness and interest in an issue and may help track and explain these spikes. I wrote about this before in what I termed the "Kylie effect". Thirdly, it shows that there are still spikes in public interest that are not traceable to news/policy announcements (look at the spike in early 2004 for both WWF and climate change). Could it be partly explained because Google Trends does not include blogs in its analysis? They could be a possible source of some peaks (i.e. a blogger writes about an issue, links to a story or WWF site inciting interest in the issue).  Steve Rubel points out this weakness in the analytical power of the new tool. KD Paine has also written about the tool and PR measurement.

Glenn

I’ve written in previous posts about my work in evaluating the impact of events.  A very interesting paper on this subject  “A Guide to Measuring Event Sponsorship” has been published by US-based Institute for Public Relations. The title is misleading as the paper focuses on how to measure the effectiveness of an event and not on sponsorship evaluation (a separate subject, don’t get me started on it…).

The guide states:

“There are four central questions to keep in mind concerning
event evaluation:
1. How effective was the event? To what extent did the event impact the target public in the desired manner?
2. Did the event change the targeted public in unexpected ways,
whether desirable or undesirable?
3. How cost effective was the event?
4. What was learned that will help improve future events? ”

The Guide goes further than I have done in event evaluation by looking at calculating ROI and at the impact on sales (applicable for a commercially focused event). It also confirms my general opinion on event evaluation - we have to go further than simply counting attendees, general reactions and press coverage - we have to look at the impact on attendees’ knowledge, attitudes, behaviours and anticipated behaviour (e.g. intention to purchase a product).

Glenn  

If you are based in Europe and interested in learning more about evaluation, the European Evaluation Society is organising their biennial conference in London from 4-6 October 2006. Even more interesting is the series of Professional Development Training Workshops they are organising around the event. One workshop looks particularly interesting - "innovative evaluation approaches" covering the following subjects:

• Theory based approaches
• Participative evaluation and ‘social inclusion’
• Integrating qualitative and quantitative methods
• Monitoring and Evaluation in international development settings
• Synthesis reviews and meta-evaluations
• Experimental methods and RCTs
• Modelling and risk analysis
• Self evaluations
• Cost benefit analysis  

Glenn