Ten stages of a ParEvo exercise

Update 2020 03 17: There is now a 12 minute YouTube video explaining how ParEvo works here

In summary

1. Clarifying the aim of a ParEvo exercise

2. Identifying who will be involved

3. Describing the starting point of the process

4. Defining the endpoint

5. The facilitator provides guidance to participants

6. Participants make their contributions

7. Developing storylines are shared

8. Re-iteration of 5, 6, 7

9. Evaluation

10. Using the products of the ParEvo process

In detail

Planning a ParEvo exercise is important. At almost every stage described below, there are design choices which can make a significant difference to how the exercise proceeds, and the value it provides to the participants.

1. Clarifying the aim of a ParEvo exercise

These are of two kinds: (a) within-exercise aims, and (b) post-exercise aims

1.1 Within-exercise aims

Two types of objectives can be pursued, in parallel:

  • Content objectives: These are about the nature of the contents of the storylines that are to be developed
  • Process objectives: These are about the ways in which the participants might be expected to interact, and the effects of those interactions.

Re content objectives, ParEVo can be used to develop alternative views of:

  • What might happen in the future, or
  • What has happened in the past

Alternative futures can be of two types:

  • Forecasting, where there is no prior view of what the desired end state is, for any particular time in the future.
  • Backcasting, where there is an agreed end state, which any scenarios being developed should lead to.

Here is a Wikipedia explanation of the difference between forecasting and backcasting.

Process objectives can be about how people participate. For example:

  • Identifying future scenarios which have maximum ownership by all participants
  • Identifying which participants are particularly good at making contributions valued by others, and vice versa
  • Identifying which participants are most similar and most different in their perspectives on a given issue
  • Doing research on what forms of participation are associated with the development of scenarios that are positively evaluated on criteria like probability and desirability, or the opposite

Meta-cognition: This is the third type of objective that may be relevant, one that bridges the content and process distinction. Meta-cognition is thinking about how we think. In this instance (and quite importantly)  – about how we think about the future (described by some as “futures literacy“)  There are two ParEvo facilities that may help with this objective:

  • The Comment facility, which can be used throughout a ParEvo exercise
  • The Evaluation function, which would typically be used at the end of a ParEvo exercise

 

1.2 Post-exercise aims

These are all about how you want to make use of the exercise once it has been completed. For more on these expectations, skip to section 10. Using the results of the ParEvo process

2. Identifying who will be involved

Three types of people will normally be involved:

  • Participants, who generate the contents of scenarios within a ParEvo exercise, and who evaluate the overall results.
  • A Facilitator(s) who invites participants, sets up the framework within which they can participate, and provides continuing guidance throughout a ParEvo exercise.
  • The ParEvo Administrator who approves requests from people to act as a Facilitator, and provides them with the parameters they can control. This is a background role thereafter. Technical support and advice can also be provided to facilitators on how to make the best use of ParEvo.

Postscript: Facilitators can now allow a forth group, known as Observers, to view the contents generated by a ParEvo process, in real-time and after completion. But not to add content in any way. This is done by sharing an exercise-specific hypertext link.

Participants can participate as individuals, representing their own views. Or they can take on roles, representing different stakeholders. Or, in each iteration, they can voice the views/behaviour of different actors who could be involved in the unfolding events. Or as individuals, participants can each represent a whole team or unit with a particular interest or perspective. Approaches which maximise the diversity of views are likely to be helpful.  But all within the constraint that participants should be expected to have a shared interest in the scenarios being developed.

The minimum number seems likely to be four or more. Larger numbers will generate more diversity of views. For more on this question see How many is too many”   Diversity is what drives the ParEvo process.  With really large numbers it may be best for these to be broken into a number of small teams, each acting as a quasi-individual. There is some evidence that diverse teams (each with more homogenous members) may be the best way way to solve complex problems (Pescetelli, et al, 2020)

As mentioned above, all participants must initially be invited by a ParEvo exercise Facilitator. They then register as participants on the ParEvo website, to obtain a password.  This then enables them to log onto the ParEvo website thereafter and gain access to any of the exercises they are involved in.

2020 03 26: There needs to be some degree of fit between the characteristics of the group of participants and the purpose of a ParEvo exercise. The purpose has to be motivational in one respect or another. Asking people to speculate on alternate futures that they either know little about or whose contents will have little consequences for them, may not be very productive.

3. Describing the starting point of the process

This is a seed paragraph of text, providing a common starting point for all subsequent scenarios. This can be a real event or an imagined event. Think of it as the opening paragraph in the first chapter of a novel.

4. Defining the endpoint

At the planning stage, this is an option, not a requirement. The endpoint can be defined as (a) a point in time and/or (b) a specific number of iterations of the ParEvo process (see below). As above, it may or may not include a description of what is expected to happen at that endpoint (backcasting) or not (forecasting)

The optimal number of iterations will depend partly on the number of participants. If the number of iterations equals the number of participants +1 then this means that each participant will have had the opportunity to build on the contributions of each of the other participants, at on at least one occasion.  This might represent a minimal ideal level of exploration of the diversity of ideas presented by the diversity of participants. In practice, in the exercises completed so far, only one of the exercises has extended this far.

5. The facilitator provides guidance to participants

In each iteration, from the beginning onwards, the facilitator needs to provide participants with some guidance. It can include the following:

  • Minimal requirements for participants’  contributions:
    • maximum length,
    • plausibility/probability and consistency requirements,
    • deadlines for contributions
    • guidance on civility, etc
  • Context setting information.:
    • Reminding participants of the overall purpose of the exercise
    • (Optionally) providing information on “surrounding developments” that the emerging storylines might need to take into account.
  • Privacy policy:

6. Participants make their contributions 

In this first iteration, all participants:

  • Receive and read the guidance from the facilitator and then
  • Contribute an additional section of text describing what they think might happen next
    • Note: There will be length limits on contributions, defined by the facilitator when they set up the exercise

7. Developing storylines are displayed and shared

When participants log onto the ParEvo webpage and then access the particular exercise they are involved in they will see a view like the one below. This provides a tree structure, on the left side, to enable people to navigate along and between different storylines, while seeing how they connect to each other. This is supplemented by a scrollable column of text in the center of the page, representing the currently selected storyline of interest. On the right side, there is space for comments on these contributions, this is a more optional feature of the overall process.

rick-test-exercise-comments-open
Figure 1: The ParEvo user interface

The identity of the contributor of each paragraph to each storyline is not made visible. The intention is that the participants’ focus should be on the content of the contributions, uninfluenced by knowledge of who the contributors are.  Their identity will be known to the facilitator (see more below).

Participants do not get to see each other’s contributions until all have contributed to a given iteration. At that point, these are all displayed at once, as above.

8. Re-iteration of 5,6,7

A new iteration only begins when all participants have contributed to the previous iteration, and they have been displayed.

Facilitator updates guidance to participants

This may or may not be needed, depending on participants previous behavior and the need to introduce any new information about the imagined “surrounding context”

Participants add new contributions

All participants are asked to look at each of the developing storylines and choose one which they would like to extend with a second contribution of their own. Each participant can only make one contribution, to one existing storyline, per iteration. But with each new iteration, they can change their mind about which storyline they now want to contribute to. As above, their text contribution could be a good, bad or neutral development, as seen from any stakeholder’s perspective. Participants could choose to add to any previous contribution, made by others or by themselves. As before, these contributions are anonymous.

Previously I had thought that fast iterations would be a good thing, making fewer time demands on participants and delivering results sooner than later. But a paper by Bernstein et al ( 2018) titled “How Intermittent Breaks in Interaction Improve Collective Intelligence.” suggests otherwise, that delays between iterations might be beneficial.

Display and sharing of contributions

After all participants have made their next contribution the display is updated to show the extended contents of each of the storylines. If more than one participant chooses to add their next contribution to the same existing storyline then that storyline now becomes two (or more) storylines. On the other hand, if some existing storylines did not receive any new contributions they remain as viewable storylines but are treated as “extinct”. These storylines can no longer be added to in subsequent iterations of participant contributions. The total number of storylines in an iteration will always equal the number of participants.

9. Evaluation

Evaluation of content

The progress and achievements of a ParEvo exercise can be assessed in two ways::

  1. Participants can be enabled to post Comments the contributions that have been made. The use of the Comment facility is optional. The Facilitator decides if and when to allow Comments and individual participants can choose if and when to make comments during any given iteration. They can make a maximum of one comment per storyline in a given iteration. This comment facility can be used in two ways:
    1. In parallel: As the second part of each iteration of the ParEvo process. But only after all new contributions are received and displayed in a given iteration. These contributions are anonymised and all displayed at once, prior to the commencement of the next iteration.
    2. In sequence: After the completion of all iterations, the facilitator re-sets the exercise to Iteration1 and enables comments on contributions made in iteration 1. They then proceed to do the same with iteration 2 and so on to the last iteration. It is probably advisable to do this before triggering the Evaluation stage (see below) .
  2. At the end of the ParEvo process, the facilitator triggers an evaluation stage, where participants are asked to rate the surviving storylines, in terms of (a) their probability of happening in real life, (b) and their desirability of happening, or any other relevant criteria. See Figure 2 below.
evaluation response form
Figure 2: Form that participants use to express evaluation judgements.

A new version of ParEvo now allows the Facilitator to edit and change the above default evaluation criteria. Other criteria, such as novelty, may be more useful in some circumstances. (See Pugh, 2016)

The aggregated responses of all participants to the built-in evaluation questions are shared with all participants, through a display as seen in Figure 3 below

evaluation feedback
Figure 3: Display of aggregated evaluation responses. The display changes according to the selected storyline

This screen can now also be shared with non-participants through an optionally created “observers” hypertext link.

The participants responses can also be downloaded and analysed as a dataset, as is explained on the Evaluation page of this website.

Survey Monkey (or similar) can be used to ask participants additional and more open-ended evaluation questions. See the example use of online pile sorting on the Evaluation page of this website for one possible approach.

Evaluation of process

During a ParEvo exercise, two types of anonymised data are automatically collected about how people participated, during each iteration:

  • Which participant contributes to which other participants’ most recent contribution.
  • Which participant contributes to which developing storyline

The facilitator can download this data in a matrix format. It can then analysed to generate measures that describe how participants have helped construct the different storylines. These are described in detail on the Analysis of Participation page.

10. Using the products of the ParEvo process

The contents generated by a ParEvo exercise can be used at three stages of a development project or intervention of some kind:

1. Planning

Programme design

Carried out at an early stage, a ParEvo exercise can inform the design of how a programme is expected to work. Storylines, especially those evaluated as more desirable and more likely to occur, can help articulate in considerable detail the sequence of events, and actors involved, which connect the implementation of activities, and subsequent short and longer-term effects, outcome and impacts.  A simple walk-through a storyline which is probable and desirable could provide numerous prompts for the necessary contents of an evaluable Theory of Change. Branching points along that storyline might be indicative of particular risks and opportunities that a Theory of Change might need to incorporate.

More systematic approaches may also be useful. The narrative content of a storyline could be systematically “translated” into a diagrammatic representation of the kind typically used to describe a Theory of Change (Davies, 2018). There seem to be at least two ways of doing this:

  • By analysing the text of a single storyline, pulling out key actors and activities that link them. These can be captured and represented as network structures. Firstly as a “temporal network” (showing actors interactions x time intervals), then as a consolidated static network – as shown in Figures 1 and 2  here.
  • By doing a content analysis (i.e. coding of key themes in the text) of all the storylines. A matrix showing the co-occurrence of particular of themes in individual participants contributions that accumulate during an exercise can be used to generate a network diagram showing their inter-relationships. See Figures 5 and 6 in this discussion of an example content analysis This is more akin to a systems map.

Risk management

A more traditional use of scenario planning is to identify alternative futures that may need to be forfended. That is, prepared for well in advance. For example, by developing strategies for responding to scenarios, should they actually happen. Two different types of scenarios can be considered:

  1. Scenarios where some probabilities can be assigned
  2. Scenarios where it is not possible to do so

This corresponds to a distinction often made between risk and uncertainty 

Note that with the existing evaluation feature built into ParEvo it is possible to distinguish between storylines with:(a)  agreed probabilities, (b) disagreed probabilities, (c) less extreme but unknown probabilities.  The difference between (a), and (b) & (c) approximates the risk and uncertainty distinction. For more on this, see the section on using predefined evaluation criteria

Identifiable risks

Where these risks are identifiable, a specific strategy may be needed to respond to each different scenario of this kind. These different kinds of scenarios may vary not only in terms of their estimated likelihood but possibly also on other criteria that may have consequences that can be forfended e.g undesirability.

Experience with ParEvo exercises so far suggests that only a minority of storylines will have identified and uncontested probabilities. The same is the case for other attributes such as undesirability. So other complementary strategies will also be needed.

Uncertainties

For the second type of scenario, characterised by uncertainty.  what is needed are more generalised and robust strategies.  Here is a useful idea from Sandra Mitchell’s book Unsimple Truths: Science Complexity and Policy (2009).

Rather than maximise expected utility, Popper, Lempert and Bankes (2002:423) recommend identifying and adopting what they called the most robust strategies. These strategies might not have been the best possible option available as any one outcome but their satisfactory outcomes occur in the largest range of future contingencies. Robustness analysis requires one to consider models take into account what we do know without pretending that we have precise probability assignments for what we don’t know. Rather it analyses a range of diverse but possible scenarios and the ways in which a policy decision today would play out in each of them. As they put it, a key insight from scenario-based planning is that multiple highly differentiated views of the future can capture the information we have about the future better than any single best estimate” p93 [underlining added]

How could such robust strategies be developed? There are two options. One would be to treat all the remaining storylines as one group. Then to find a robust way of responding, regardless of which individual storyline actually occurred. The larger the number of storylines in this set, the more difficult this is likely to be.

Where this is too big a challenge the storylines could be split into two groups, where the members of each group would share some common characteristics, and which made them different from the other group. A common and feasible robust response might then be identifiable for each group.  One way of doing this is by using participatory pile sorting exercises, where participants could be asked to “sort these storylines into two different different groups, according to what you see as the”most significant difference between them – which could make a difference to how we should respond“.  One version of pile sorting, known as hierarchical card sorting, identifies 2 groups, then 4 sub-groups, then 8 sub-sub-groups, etc, to be identified i.e. a nested hierarchy.

Regarding the above options for identifiable risks and for uncertainties, there may be some science that is relevant here, to do with “bet-hedging” strategies  – a particular evolutionary strategy where there is a high level of environmental unpredictability.

The difference between adaptive plasticity and bet-hedging is that plastic norms of reaction result in the expression of an optimal phenotype over a range of environments, whereas bet-hedging expresses a single phenotype (that may be a fixed level of diversification) that is neither optimal nor a failure across all environments“(Simons, 2011)

For more, see this Wikipedia article on bet-hedging in biology 

2. Monitoring and adaptation

Monitoring

Once an intervention has begun, progress with implementation normally needs to be monitored. Ideally,  along with the identification of events in the surrounding environment that may be necessary, supportive or obstructive.

Traditional monitoring and evaluation tools, notably the Logical Framework, have severe limitations. While they can be economic in terms of the number of events that need to be observed and measured this economy comes at the price of tunnel vision.

In contrast, the branching structure of ParEvo storylines can provide a much wider view of possible developments that could have consequences for the success of the intervention. The ongoing scanning for signs of any of these developments being realised may not necessarily be that costly.  Events simply have to be observed as happening or not, not repeatedly measured with some care.

Some of the events in ParEvo storylines might be of sufficient concern and probability that they warrant consistent attention i.e. being continually looked out for. These could be, in effect, built into an organisation’s M&A system.

Other events might be less predictable, and less clearly identifiable, but still of concern. In these circumstances, it could make sense to make use of the Most Significant Change technique as a horizon scanning tool. The MSC question might ask “What was the most significant change that took place in the last x period that could have consequences for how our organisation/intervention works in the next y period?” The answer would be, as with any MSC application, a description of the event and an explanation of its significance. The organisation would then need to plan a management response, i.e. ideas/proposals for how the organisation should respond in anticipation of those consequences.

With some uses of MSC, one or more domains of change are identified in advance. These domains are added onto the end of the MSC question, as in ‘most significant change in… (domain description)….”. They give the search for significant changes a degree of focus without being too prescriptive. In the application proposed above domains could be particular themes/kinds of events seen in the ParEvo storylines. These could be identified by third-party content analysts or by the participants themselves e.g. using pile sorting methods.

Adaptation

When a ParEvo exercise has finished there will be a set of storylines describing a number of alternative futures. In those storylines there may be references to how the participants own organisation has responded. If so, questions then need to be asked about those responses:

  • Was that a realistic response, i.e. would the organisation actually have the capacity to do that?
  • Was it an appropriate response, i.e. would there have been a more relevant response?

There will also be other events in the storylines where there is no mention of a response by the participant’s own organisation. The events which might be particular interest may be those at the branching points of the storylines, where the subsequent storylines go in different directions, with quite different consequences. These events could be examined, one at a time, and questioned in a similar manner as above:

  • How could the participants own organisation have responded in this situation, in a way that might have influenced which way the storyline subsequently went?
  • What existing capacity was present, or additional capacity would be needed, for the organisation to be able to respond in that manner?

3. Evaluation

Reconstructed histories

All interventions have a history, and any evaluation of those interventions will need to pay attention to that history. But where multiple different stakeholders have been involved then their views on the key events in that history, and the consequences of those events, may vary. The planning of an evaluation should be informed by knowledge of those differences. That means paying attention to all branches of the storylines generated by a ParEvo exercise.

As an evaluation progresses, attention may justifiably focus on a particular storyline, addressing what happened to a particular intervention. In those circumstances, the contents of the storyline might provide useful initial raw material for a more in-depth analysis of causal mechanisms using process-tracing .