Qualitative monitoring

Why do it?

  1. One lesson that has been learned from the exercises completed to date is ‘It is not advisable to leave the task of content analysis until after all iterations have been completed“. The reasoning behind this argument is that if you have ten iterations with ten participants you could have 10 x 10 x 150 words = 15,000 words of text to analyse at the end of such an exercise. The alternative is to start coding the contents of contributions as soon as they are received (150 words at a time).
  2. Also, if the content of contributions are analysed as soon as they are received there is much greater potential for identifying emerging trends in the content that might be of concern/interest, and which could be quickly fed back to participants via the Guidance text at the top of the exercise, which is usually updated with each new iteration.

How to do it?

By monitoring I mean the periodic identification of what attributes of interest/concern are present or absent in each contribution in a given iteration. This can be done using a simple Excel sheet to construct an affiliation matrix, where:

  • Column headings list the numbered contributions, in numerical order, color coded by iteration number
  • Row headings down the left list an accumulating list of attributes of interest seen in the contributions.
  • Cell values indicate whether the row attribute was present in the column contribution, or not (using 0/1)
  • Row totals on the far right column indicate the total number of times an attribute has been noted as present
  • Column totals at the bottom row indicate “content richness” the extent to which individual contributions exhibited few versus many of the coded attributes

The attributes described in the row headings do not need to be defined in advance, they can be progressively added to (and revised) as each new contributions becomes available with each new iteration. I will include an example of such a matrix here shortly.

Types of attributes to code as present/absent

So far, these are some of the types we have used:

  • Types of actors mentioned in the contribution
  • Types of actions undertaken by actors within the contribution
  • Types of wider influences, e.g. COVID
  • Types of outcomes
  • Types of problems in the text, requiring email contact with the participants

Tags

A subset of the listed attributes be used to describe tags that the facilitator or participants have applied to individual contributions

Where is the qualitative bit?

The chosen description of each attribute, and its designated membership in larger categorisations = the qualitative content