intelligent measurement

Professor James Grunig, a leading thinker in the field of communications / PR evaluation was in Switzerland last week. While here, he gave an interview where he spoke about the notion of PR being "research intensive" amongst a wide range of subjects in the PR field.

Glenn

People (read: potential clients) often tell me that their publics are fed up filling out online survey, being interview or queried about their opinions. Rubbish I say. People are willing to spend time giving their opinion about a website, service or issues that is important to them.

In a recent online survey of external audiences for an international organisation, 25% of the potential audience responded to the survey. And of those who responded, 44% requested a copy of the results. In an evaluation of an international event, 55% of the participants responded to survey. And of those who responded, 60% requested a copy of the results. 

In the survey for the international organisation, 80% of respondents said that the organisation's website was very important or important for their work. Wouldn't you like to give an opinion on something that important to you? I would.

My experience is that people are willing to participate in research if it is something that is important to them and if they believe something will be done with the results.

In this article on the simply-communicate.com site, the author points out that: 

“In a recent survey of reasons for non-response to employee questionnaires, the biggest driver of non-participation was found to be 'Nothing would happen as a result'".

So people do want to see that their opinion is valued. The fact that such a large percentage of people want to receive the evaluation results indicate that they are interested in the subject in question. In addition, I imagine that people are interested to see if what they think corresponds to the norm - and to judge what the organisation will do as a consequence of the evaluation.

Glenn

If your focus is on evaluating PR, training or another business competency, it is sometimes helpful to learn more about evaluation by looking further than your particular focus. Look at international aid. I just read a review of a new book The White Man’s Burden - Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good By William Easterly. Based on the book review, he raises two interesting points about evaluation:

Planners vs. Searchers: he says most aid projects have either one of these approaches. He writes “A Planner thinks he already knows the answer. A Searcher admits he doesn’t know the answers in advance”. He explains that Searchers treat problem-solving as an incremental discovery process, relying on competition and feedback to figure out what works.

Measuring Success: Easterly argues that aid projects rarely get enough feedback whether from competition or complaint. Instead of introducing outcome evaluation where results are relatively easy to measure (e.g. public health, school attendance, etc), advocates of aid measure success by looking at how much money rich countries spend. He says this is like reviewing movies based on their budgets.

You can read the full review of Easterly’s book on the IHT website. 

These are some excellent points that we can apply to evaluation across the board. Evaluators can probably classify many projects they have evaluated as either of a Planner or Searcher nature. The Searcher approach integrates feedback and evaluation throughout the whole process - look at PR activities, we may not know what communication works best with our target audience to begin with, but by integrating feedback and evaluation we can soon find out.

His analogy about reviewing movies also strikes a chord. How many projects are evaluated on expenditure alone? 

 Aside from the evaluation aspects, his thoughts on international aid are interesting. I spent my formative years as an aid worker in Africa, Eastern Europe and Asia and a lot of what he says confirms my own experiences and conclusions.

Glenn  

In presentations on evaluation and measurement, I find the use of quotes from experts or well-known personalities assists in clarifying the importance of the field for the listeners. Here are some of my favourite quotes:

“True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information” 
Winston Churchill

“Fear cannot be banished, but it can be calm and without panic; it can be mitigated by reason and evaluation”
Vannevar Bush

“The only man who behaves sensibly is my tailor; he takes my measurements anew every time he sees me, while all the rest go on with their old measurements and expect me to fit them”  
George Bernard Shaw 

And a favorite from Monitoring and Evaluation News:

Friend to Groucho Marx: “Life is difficult!” 

Marx to Friend: “Compared to what?”

In evaluation, we often choose between using qualitative (e.g. focus group) and quantitative (e.g. survey) methods. In fact, we should always try and use both approaches. This is what is referred to as triangulation: the combination of several research methods in the study of the same phenomenon. My experience has been that a combination of research methods helps provide more data to work with and ultimately a more accurate evaluation. In a recent project, I was able to use interviews combined with surveys to assess participant reaction to training. I found that the information we could draw from the interviews was complimentary - and of added value - to what we discovered through the surveys.

Even if you are only conducting online surveys, the inclusion of open questions (where respondents put in comments in a free text field) is not quite triangulation but will provide you with insight into the phenomenon being evaluated. In a recent online survey project, we were able to clarify important issues by sorting and classifying the comments made in open questions.  This proved invaluable information and gave the evaluation heightened status within the organisation.

Glenn  

The USA’s  Institute for Public Relations, which offers all of its research and publications free on the web, has published a new edition of the Dictionary of Public Relations Measurement and Research (pdf - 198kb), edited by Dr. Don W. Stacks of the University of Miami.

We all measure different things and we measure them all differently (and not always intelligently). Not a good idea if you want to benchmark or compare data.

Don Stacks and his colleagues have done a magnificent job in assembling, codifying and defining most of the common terms we come across when talking about measurement, from “Algorithm” to “Z-score”.

It’s a giant step forward in defining some standard methodologies which everyone can use.

The Institute’s website, http://www.instituteforpr.org also contains a large number of downloadable papers on every aspect of communications measurement and evaluation. Well worth a visit.

Richard

 One company that is a good example of “intelligent measurement” is Innovest Strategic Value Advisors, Inc.  an investment research and advisory firm (disclosure: there are no links between the authors of this blog and Innovest). This company specializes in analyzing “non traditional” drivers of risk and shareholder value, including companies’ performance on environmental, social and strategic governance issues.

Innovest’s research is focused on those factors which contribute most heavily to financial performance. Environmental and social performance measures are used as leading indicators for management quality and long-term financial performance, not as commentaries on the intrinsic ethical worth of the companies. At the heart of Innovest’s analytical model is the attempt to balance the level of environmentally and socially driven investment risk with the companies’ managerial and financial capacity to manage that risk successfully and profitably into the future.

Environmental assessment criteria:
In total, the Innovest EcoValue‘21™ model synthesizes over 60 data points and performance metrics, grouped together under six key value drivers – Historical contingent liabilities, Financial Risk assessment, Operating risk exposure, Sustainability risk, Strategic management capability and Sustainable profit opportunities.

Social assessment criteria:
Over 50 individual performance indicators are addressed in Innovest’s IVA™ rating model. The principal value drivers are Sustainable Governance, Stakeholder management, Human Capital management, Products and Services, and behaviour in relation to oppressive regimes or exploitative labour markets in emerging markets.

The measurement process is complex, intensive and time consuming. Once the interview/data gathering process is completed, each company is rated relative to its industry competitors. Companies are rated against the Innovest performance criteria, and given a weighted score, as well as a letter grade (AAA, BB etc.). Each of the factors has an industry-specific weighting, based in part on a regression-based factor attribution analysis examining recent (5 year) stock market performance.

This approach is intelligent measurement at its finest.  The methodology is rigorous, and the outputs have enormous strategic value to investors, and management of any capital intensive company trying to understand the correlation between investment in traditionally regarded “soft” management issues and shareholder value.

Which why Innovest commands a high price for its reports and is generally regarded as the leader in its field.

Richard

The International Association of Business Communicators (IABC) has launched a new blog with three orientations: branding, employee and measurement. Interesting to note that PR measurement has been given a high profile by the IABC - an indication of its importance to communicators today. 

The measurement blog has already some interesting posts and comments on measuring employee communications and context and evaluation, something I’ve written about before.

Glenn