Ten stages of a ParEvo exercise

This is a re-edited version, building on the experience of exercises that have been completed so far. It is likely to go through further revisions.

In summary

1. Clarifying the aim of a ParEvo exercise

2. Identifying who will be involved

3. Describing the starting point of the process

4. Defining the endpoint

5. The facilitator provides guidance to participants

6. Participants make their contributions

7. Developing storylines are shared

8. Re-iteration of 5, 6, 7

9. Evaluation

12. Using the results of the ParEvo process

In detail

1. Clarifying the aim of a ParEvo exercise

Two types of objectives can be pursued, in parallel:

  • Content objectives: These are about the nature of the contents of the storylines that are to be developed
  • Process objectives: These are about the ways in which the participants might be expected to interact, and the effects of those interactions.

Re content objectives, ParEVo can be used to develop alternative views of:

  • What might happen in the future, or
  • What has happened in the past

Alternative futures can be of two types:

  • Forecasting, where there is no prior view of what the desired end state is, for any particular time in the future.
  • Backcasting, where there is an agreed end state, which any scenarios being developed should lead to.

Here is a Wikipedia explanation of the difference between forecasting and backcasting.

Process objectives can be about how people participate. For example:

  • Identifying future scenarios which have maximum ownership by all participants
  • Identifying which participants are particularly good at making contributions valued by others, and vice versa
  • Identifying which participants are most similar and most different in their perspectives on a given issue
  • Doing research on what forms of participation are associated with the development of scenarios that are positively evaluated on criteria like probability and desirability, or the opposite

2. Identifying who will be involved

Three types of people will be involved:

  • Participants, who generate the contents of scenarios within a ParEvo exercise, and who evaluate the overall results.
  • A Facilitator(s) who invites participants, sets up the framework within which they can participate, and provides continuing guidance throughout a ParEvo exercise.
  • The ParEvo Administrator who approves requests from people to act as a Facilitator, and provides them with the parameters they can control. This is a background role thereafter. Technical support and advice can also be provided to facilitators on how to make the best use of ParEvo.

Participants can participate as individuals, representing their own views. Or they can take on roles, representing different stakeholders. In either case, they should be expected to have a shared interest in the scenarios being developed.

The minimum number seems likely to be four or more. Larger numbers will generate more diversity of views. For more on this question see How many is too many”  

Diversity is what drives the ParEvo process. Participants can be individuals or small teams.  With really large numbers it may be best for these to be broken into a number of small teams, each acting as a quasi-individual. There is some evidence that diverse teams (each with more homogenous members) may be the best way way to solve complex problems (Pescetelli, et al, 2020)

As mentioned above, all participants must be invited by a ParEvo exercise Facilitator. They then register as participants on the ParEvo website, to obtain a password.  This then enables them to log onto the ParEvo website thereafter and gain access to any of the exercises they are involved in.

3. Describing the starting point of the process

This is a seed paragraph of text, providing a common starting point for all subsequent scenarios. This can be a real event or an imagined event.

4. Defining the endpoint

This is an option, not a requirement. The endpoint can be defined as: (a) a point in time and/or (b) a specific number of iterations of the ParEvo process (see below). As above, it may or may not include a description of what is expected to happen at that endpoint (backcasting) or not (forecasting)

5. The facilitator provides guidance to participants

In each iteration, from the beginning onwards, the facilitator needs to provide participants with some guidance. It can include the following:

  • Minimal requirements for participants’  contributions:
    • maximum length,
    • plausibility and consistency requirements,
    • deadlines for contributions
    • guidance on civility, etc
  • Context setting information.:
    • Reminding participants of the overall purpose of the exercise
    • (Optionally) providing information on “surrounding developments” that the emerging storylines might need to take into account.
  • Privacy policy:

6. Participants make their contributions 

In this first iteration, all participants:

  • Receive and read the guidance from the facilitator and then
  • Contribute an additional section of text describing what they think might happen next.

7. Developing storylines are displayed and shared

When participants log onto the ParEvo webpage and then access the particular exercise they are involved in they will see a view like the one below. This provides a tree structure, on the left side, to enable people to navigate along and between different storylines, while seeing how they connect to each other. This is supplemented by a scrollable column of text in the center of the page, representing the currently selected storyline of interest. On the right side, there is space for comments on these contributions, this is a more optional feature of the overall process.

rick-test-exercise-comments-open

The identity of the contributor of each paragraph to each storyline is not made visible. The intention is that the participants’ focus should be on the content of the contributions, uninfluenced by knowledge of who the contributors are.  Their identity will be known to the facilitator (see more below)

Participants do not get to see each other’s contributions until all have contributed to a given iteration and these are then displayed, as above.

8. Re-iteration of 5,6,7

A new iteration only begins when all participants have contributed to the previous iteration, and they have been displayed.

Facilitator updates guidance to participants

This may or may not be needed, depending on participants previous behavior and the need to introduce any new information about the imagined “surrounding context”

Participants add new contributions

All participants are asked to look at each of the developing storylines and choose one which they would like to extend with a second contribution of their own. Each participant can only make one contribution, to one existing storyline, per iteration. But with each new iteration, they can change their mind about which storyline they now want to contribute to. As above, their text contribution could be a good, bad or neutral development, as seen from any stakeholder’s perspective. Participants could choose to add to any previous contribution, made by others or by themselves. As before, these contributions are anonymous.

Previously I had thought that fast iterations would be a good thing, making fewer time demands on participants and delivering results sooner than later. But a paper by Bernstein et al ( 2018) titled “How Intermittent Breaks in Interaction Improve Collective Intelligence.” suggests otherwise, that delays between iterations might be beneficial.

Display and sharing of contributions

After all participants have made their next contribution the display is updated to show the extended contents of each of the storylines. If more than one participant choses to add their next contribution to the same existing storyline then that storyline now becomes two (or more) storylines. On the other hand, if some existing storylines did not receive any new contributions they remain as viewable storylines but are treated as “extinct”. These storylines can no longer be added to in subsequent iterations of participant contributions. The total number of storylines in an iteration will always equal the number of participants.

9. Evaluation

Evaluation of content

The progress and achievements of a ParEvo exercise can be assessed in two ways::

  1. Participants can be enabled to post Comments on each contribution during the current iteration of the ParEvo process, but only after all new contributions are received and displayed in a given iteration. These contributions are anonymised and all displayed at once, prior to the commencement of the next iteration.
    1. The use of the Comment facility is optional. The Facilitator decides if and when to allow Comments and individual participants can choose if and when to make comments during any given iteration. They can make a maximum of one comment per storyline in a given iteration.
  2. At the end of the ParEvo process, the facilitator triggers an evaluation stage, where participants are asked to rate the surviving storylines, in terms of (a) their probability of happening in real life, (b) and their desirability of happening, or any other relevant criteria.  This survey data can be then downloaded and analysed, as is explained on the Evaluation page of this website.
    1. Future versions of ParEvo will allow Facilitator to choose their own preferred evaluation criteria
    2. Survey Monkey (or similar) can be used to ask participants additional and more open-ended evaluation questions. See the example use of online pile sorting on the Evaluation page of this website for one possible approach.

The facilitator will then shares the results of the evaluation with all participants along with a link to the completed exercise with all storylines visible. This should remain available to participants after completion of the exercise i..e from this point.

Evaluation of process

During a ParEvo exercise, two types of anonymised data are automatically collected about how people participated, during each iteration:

  • Which participant contributes to which other participants’ most recent contribution.
  • Which participant contributes to which developing storyline

The facilitator can download this data in a matrix format. It can then analysed to generate measures that describe how participants have helped construct the different storylines. These are described in detail on the Analysis of Participation page.

10. Using the results of the ParEvo process

The content and process data generated by a ParEvo exercise can be used at three stages of a development project or intervention of some kind:

  1. Planning

Carried out at an early stage, a ParEvo exercise can inform the design or modification of a Theory of Change of how a programme is expected o work. Storylines, especially those evaluated as more likely to occur, can help articulate in considerable detail the sequence of events connecting the implementation of activities, and subsequent short and longer-term effects. The more such detail is provided the more evaluable the intervention will be.

ParEvo storylines can provide a manageable perspective in between an overly simplistic single linear chain model and more dauntingly complex network models.

4. Monitoring

Once an intervention has begun, progress with implementation needs to be monitored, along with events in the surrounding environment that may be necessary, supportive or obstructive. ParEvo storylines can provide a wider view of the possible developments at any given time that may need attention.   Most agencies have M&E plan based on one particular scenario of how events will take place. But the branching perspective generated by a ParEvo exercise suggests that at different points in time there might be bifurcations, where real events follow a different route than expected. This possibility strongly suggests that M&E systems need to be looking to the right and left and not just straight ahead, so to speak. Alternative scenarios developed during a ParEvo exercise could suggest what else to be looking out for, as events unfold.

3. Evaluation

All interventions have a history, and any evaluation of those interventions need to pay attention to that history. But where multiple different stakeholders have been involved their views on the key events in that history, and their consequences may vary. The use of a ParEvo exercise to collectively reconstructive the history of an intervention can provide an evaluation team with alternative views of historical events aka causal process that led to particular outcomes of interest.