Ten stages of a ParEvo exercise

Update 2020 03 17: There is now an 12 minute YouTube video explaining how ParEvo works HERE

In summary

1. Clarifying the aim of a ParEvo exercise

2. Identifying who will be involved

3. Describing the starting point of the process

4. Defining the endpoint

5. The facilitator provides guidance to participants

6. Participants make their contributions

7. Developing storylines are shared

8. Re-iteration of 5, 6, 7

9. Evaluation

10. Using the results of the ParEvo process

In detail

1. Clarifying the aim of a ParEvo exercise

Two types of objectives can be pursued, in parallel:

  • Content objectives: These are about the nature of the contents of the storylines that are to be developed
  • Process objectives: These are about the ways in which the participants might be expected to interact, and the effects of those interactions.

Re content objectives, ParEVo can be used to develop alternative views of:

  • What might happen in the future, or
  • What has happened in the past

Alternative futures can be of two types:

  • Forecasting, where there is no prior view of what the desired end state is, for any particular time in the future.
  • Backcasting, where there is an agreed end state, which any scenarios being developed should lead to.

Here is a Wikipedia explanation of the difference between forecasting and backcasting.

Process objectives can be about how people participate. For example:

  • Identifying future scenarios which have maximum ownership by all participants
  • Identifying which participants are particularly good at making contributions valued by others, and vice versa
  • Identifying which participants are most similar and most different in their perspectives on a given issue
  • Doing research on what forms of participation are associated with the development of scenarios that are positively evaluated on criteria like probability and desirability, or the opposite

Meta-cognition: This is the third type of objective that may be relevant, one that bridges the content and process distinction. Meta-cognition is thinking about how we think. In this instance (and quite importantly)  about how we think about the future. There are two ParEvo facilities that may help with this objective:

  • The Comment facility, which can be used throughout a ParEvo exercise
  • The Evaluation function, which would typically be used at the end of a ParEvo exercise

2. Identifying who will be involved

Three types of people will normally be involved:

  • Participants, who generate the contents of scenarios within a ParEvo exercise, and who evaluate the overall results.
  • A Facilitator(s) who invites participants, sets up the framework within which they can participate, and provides continuing guidance throughout a ParEvo exercise.
  • The ParEvo Administrator who approves requests from people to act as a Facilitator, and provides them with the parameters they can control. This is a background role thereafter. Technical support and advice can also be provided to facilitators on how to make the best use of ParEvo.

In addition, a forthcoming feature will allow Observers to view the contents generated by a ParEvo process, in real-time and after completion. But not to intervene in any way.

Participants can participate as individuals, representing their own views. Or they can take on roles, representing different stakeholders. Or, in each iteration, they can voice the views/behaviour of different actors who could be involved in the unfolding events. Or as individuals, participants can each represent a whole team or unit with a particular interest or perspective. Approaches which maximise the diversity of views are likely to be helpful.  But all within the constraint that participants should be expected to have a shared interest in the scenarios being developed.

The minimum number seems likely to be four or more. Larger numbers will generate more diversity of views. For more on this question see How many is too many”   Diversity is what drives the ParEvo process.  With really large numbers it may be best for these to be broken into a number of small teams, each acting as a quasi-individual. There is some evidence that diverse teams (each with more homogenous members) may be the best way way to solve complex problems (Pescetelli, et al, 2020)

As mentioned above, all participants must initially be invited by a ParEvo exercise Facilitator. They then register as participants on the ParEvo website, to obtain a password.  This then enables them to log onto the ParEvo website thereafter and gain access to any of the exercises they are involved in.

2020 03 26: There needs to be some degree of fit between the characteristics of the group of participants and the purpose of a ParEvo exercise. The purpose has to be motivational in one respect or another. Asking people to speculate on alternate futures that they either know little about or whose contents will have little consequences for them, may not be very productive.

3. Describing the starting point of the process

This is a seed paragraph of text, providing a common starting point for all subsequent scenarios. This can be a real event or an imagined event. Think of it as the opening paragraph in the first chapter of a novel.

4. Defining the endpoint

This is an option, not a requirement. The endpoint can be defined as (a) a point in time and/or (b) a specific number of iterations of the ParEvo process (see below). As above, it may or may not include a description of what is expected to happen at that endpoint (backcasting) or not (forecasting)

5. The facilitator provides guidance to participants

In each iteration, from the beginning onwards, the facilitator needs to provide participants with some guidance. It can include the following:

  • Minimal requirements for participants’  contributions:
    • maximum length,
    • plausibility/probability and consistency requirements,
    • deadlines for contributions
    • guidance on civility, etc
  • Context setting information.:
    • Reminding participants of the overall purpose of the exercise
    • (Optionally) providing information on “surrounding developments” that the emerging storylines might need to take into account.
  • Privacy policy:

6. Participants make their contributions 

In this first iteration, all participants:

  • Receive and read the guidance from the facilitator and then
  • Contribute an additional section of text describing what they think might happen next
    • Note: There will be length limits on contributions, defined by the facilitator when they set up the exercise

7. Developing storylines are displayed and shared

When participants log onto the ParEvo webpage and then access the particular exercise they are involved in they will see a view like the one below. This provides a tree structure, on the left side, to enable people to navigate along and between different storylines, while seeing how they connect to each other. This is supplemented by a scrollable column of text in the center of the page, representing the currently selected storyline of interest. On the right side, there is space for comments on these contributions, this is a more optional feature of the overall process.

rick-test-exercise-comments-open

The identity of the contributor of each paragraph to each storyline is not made visible. The intention is that the participants’ focus should be on the content of the contributions, uninfluenced by knowledge of who the contributors are.  Their identity will be known to the facilitator (see more below).

Participants do not get to see each other’s contributions until all have contributed to a given iteration. At that point, these are all displayed at once, as above.

8. Re-iteration of 5,6,7

A new iteration only begins when all participants have contributed to the previous iteration, and they have been displayed.

Facilitator updates guidance to participants

This may or may not be needed, depending on participants previous behavior and the need to introduce any new information about the imagined “surrounding context”

Participants add new contributions

All participants are asked to look at each of the developing storylines and choose one which they would like to extend with a second contribution of their own. Each participant can only make one contribution, to one existing storyline, per iteration. But with each new iteration, they can change their mind about which storyline they now want to contribute to. As above, their text contribution could be a good, bad or neutral development, as seen from any stakeholder’s perspective. Participants could choose to add to any previous contribution, made by others or by themselves. As before, these contributions are anonymous.

Previously I had thought that fast iterations would be a good thing, making fewer time demands on participants and delivering results sooner than later. But a paper by Bernstein et al ( 2018) titled “How Intermittent Breaks in Interaction Improve Collective Intelligence.” suggests otherwise, that delays between iterations might be beneficial.

Display and sharing of contributions

After all participants have made their next contribution the display is updated to show the extended contents of each of the storylines. If more than one participant chooses to add their next contribution to the same existing storyline then that storyline now becomes two (or more) storylines. On the other hand, if some existing storylines did not receive any new contributions they remain as viewable storylines but are treated as “extinct”. These storylines can no longer be added to in subsequent iterations of participant contributions. The total number of storylines in an iteration will always equal the number of participants.

9. Evaluation

Evaluation of content

The progress and achievements of a ParEvo exercise can be assessed in two ways::

  1. Participants can be enabled to post Comments the contributions that have been made. The use of the Comment facility is optional. The Facilitator decides if and when to allow Comments and individual participants can choose if and when to make comments during any given iteration. They can make a maximum of one comment per storyline in a given iteration. This comment facility can be used in two ways:
    1. In parallel: As the second part of each iteration of the ParEvo process. But only after all new contributions are received and displayed in a given iteration. These contributions are anonymised and all displayed at once, prior to the commencement of the next iteration.
    2. In sequence: After the completion of all iterations, the facilitator re-sets the exercise to Iteration1 and enables comments on contributions made in iteration 1. They then proceed to do the same with iteration 2 and so on to the last iteration. It is probably advisable to do this before triggering the Evaluation stage (see below) .
  2. At the end of the ParEvo process, the facilitator triggers an evaluation stage, where participants are asked to rate the surviving storylines, in terms of (a) their probability of happening in real life, (b) and their desirability of happening, or any other relevant criteria.  This survey data can be then downloaded and analysed, as is explained on the Evaluation page of this website.
    1. Future versions of ParEvo will allow Facilitator to choose their own preferred evaluation criteria
    2. Survey Monkey (or similar) can be used to ask participants additional and more open-ended evaluation questions. See the example use of online pile sorting on the Evaluation page of this website for one possible approach.

The facilitator will then shares the results of the evaluation with all participants along with a link to the completed exercise with all storylines visible. This should remain available to participants after completion of the exercise i..e from this point.

Evaluation of process

During a ParEvo exercise, two types of anonymised data are automatically collected about how people participated, during each iteration:

  • Which participant contributes to which other participants’ most recent contribution.
  • Which participant contributes to which developing storyline

The facilitator can download this data in a matrix format. It can then analysed to generate measures that describe how participants have helped construct the different storylines. These are described in detail on the Analysis of Participation page.

10. Using the results of the ParEvo process

The content and process data generated by a ParEvo exercise can be used at three stages of a development project or intervention of some kind:

Planning

Carried out at an early stage, a ParEvo exercise can inform the design, or the modification of the design of how a programme is expected to work. Storylines, especially those evaluated as more likely to occur, can help articulate in considerable detail the sequence of events, and actors involved, which connect the implementation of activities, and subsequent short and longer-term effects, outcome and impacts. The more such detail is provided the more evaluable the intervention will be.

PS: On a separate website, I have explained how a narrative could be converted into a diagrammatic representation, showing how people and events are connected over time.

Monitoring and adaptation

Once an intervention has begun, progress with implementation normally needs to be monitored. Ideally,  along with the identification of events in the surrounding environment that may be necessary, supportive or obstructive.

Because they are plural, ParEvo storylines can provide a wider view of the possible developments at any given time that may need attention.   Most agencies have M&E plan based on one particular scenario of how events will take place. But the branching perspective generated by a ParEvo exercise suggests that at different points in time there might be various bifurcations, where real events follow a different route than expected. This possibility strongly suggests that M&E systems need to be looking to the right and left and not just straight ahead, so to speak. Alternative scenarios developed during a ParEvo exercise could suggest what else to be looking out for, as events unfold.

In practical terms, I would suggest carrying out two types of examinations of storylines generated by a  completed ParEvo exercise. Both involve systematically “walking” up each branch and sub-branch of the tree, from the seed, and ask two types of questions at each branching point:

  • Knowledge requirements:
    • How soon would we be able to tell if and when each of these alternatives is likely to happen?
    • What can we do to improve our capacity to know what is likely to happen
  • Action implications:
    • What will be the consequences of each alternative?
    • How can we prepare for each alternative?

Evaluation

Reconstructed histories

All interventions have a history, and any evaluation of those interventions need to pay attention to that history. But where multiple different stakeholders have been involved then their views on the key events in that history, and their consequences may vary. The use of a ParEvo exercise to collectively reconstructive the history of an intervention can provide an evaluation team with alternative views of historical events aka causal process that led to particular outcomes of interest. When a ParEvo exercise is focused on the consequences of a particular intervention the storylines that are generated might provide useful raw material for a process-tracing type analysis of causal mechanisms involved.

Evaluation of anticipated futures

At some stage in the future, it may be useful to look back at the contents of a ParEvo exercise and ask which elements of which storylines actually happened in reality. Then, if a real event occurs in more than one storyline, which description of that event has the best fit with the real event. Such a review should pay attention to extinct as well as surviving storylines. The distribution of predicted-events-that-happened could then be compared to the evaluations judgements that were previously made about each storylines, especially their likelihood of actually happening.