Ten stages of a ParEvo exercise

In summary

1. Clarifying the aim of a ParEvo exercise

2. Identifying who will be involved

3. Describing the starting point of the process

4. Defining the endpoint

5. The facilitator provides guidance to participants

6. Participants make their contributions

7. Developing storylines are shared

8. Re-iteration of 5, 6, 7

9. Evaluation

10. Using the products of the ParEvo process

In detail

Planning a ParEvo exercise is important. At almost every stage described below, there are design choices which can make a significant difference to how the exercise proceeds, and the value it provides to the participants.

1. Clarifying the aim of a ParEvo exercise

These are of two kinds: (a) within-exercise aims, and (b) post-exercise aims

1.1 Within-exercise aims

Two types of objectives can be pursued, in parallel:

  • Content objectives: These are about the nature of the contents of the storylines that are to be developed
  • Process objectives: These are about the ways in which the participants might be expected to interact, and the effects of those interactions.

Content objectives, ParEvo can be used to develop alternative views of:

  • What might happen in the future, or
  • What has happened in the past

Alternative futures can be of two types:

  • Forecasting, where there is no prior view of what the desired end state is, for any particular time in the future.
  • Backcasting, where there is an agreed end state, which any scenarios being developed should lead to.

Here is a Wikipedia explanation of the difference between forecasting and backcasting.

Process objectives can be about how people participate. For example:

  • Identifying future scenarios which have maximum ownership by all participants
  • Identifying which participants are particularly good at making contributions valued by others, and vice versa
  • Identifying which participants are most similar and most different in their perspectives on a given issue
  • Doing research on what forms of participation are associated with the development of scenarios that are positively evaluated on criteria like probability and desirability, or the opposite

Meta-cognition: This is the third type of objective that may be relevant, one that bridges the content and process distinction. Meta-cognition is thinking about how we think. In this instance (and quite importantly)  – about how we think about the future (described by some as “futures literacy)  There are two ParEvo facilities that may help with this objective:

  • The Comment facility, which can be used throughout a ParEvo exercise
  • The Evaluation function, which would typically be used at the end of a ParEvo exercise

1.2 Post-exercise aims

These are all about how you want to make use of the exercise once it has been completed. For more on these expectations, skip to section 10. Using the results of the ParEvo process

2. Identifying who will be involved

People can be involved in 6 different roles.

  • The ParEvo Administrator who approves requests from people to act as a Facilitator, and provides them with the parameters they can control. This is a background role thereafter. Technical support and advice can also be provided to Facilitators on how to make the best use of ParEvo.
  • A Facilitator(s) who invites participants, sets up the framework within which they can participate, and provides continuing guidance throughout a ParEvo exercise.
  • Participants (aka Contributors), who generate the contents of scenarios within a ParEvo exercise
  • Commentators, who comment on the contributions made by participants. Commentators can be Contributors or other parties.
  • Evaluators, who evaluate the storylines at the end of a ParEvo exercise. They can be Contributors and/or Commentators
  • Observers, who can view the contents generated by a ParEvo process, in real-time and after completion. But not to add content in any way. This is done by sharing an exercise-specific hypertext link.

Two roles are essential: A Facilitators and Participants, all others are optional

Individuals can take on more than one role. Typically participants will also be evaluators. Commentators may also be allowed this role

Participants can participate as individuals, representing their own views. Or they can represent different stakeholders. Or, in each iteration, they can voice the views/behaviour of different actors who could be involved in the unfolding events. Or as individuals, participants can each represent a whole team or unit , within an organisation, voicing a particular interest or perspective. Approaches which maximise the diversity of views are likely to be helpful.  But all within the constraint that participants should be expected to have a shared interest in the scenarios being developed.

The minimum number seems likely to be four or more. Larger numbers will generate more diversity of views. For more on this question see How many is too many”   Diversity is what drives the ParEvo process.  With really large numbers it may be best for these to be broken into a number of small teams, each acting as a quasi-individual. There is some evidence that diverse teams (each with more homogenous members) may be the best way way to solve complex problems (Pescetelli, et al, 2020)

As mentioned above, all participants must initially be invited by a ParEvo exercise Facilitator. They then register as participants on the ParEvo website, to obtain a password.  This then enables them to log onto the ParEvo website thereafter and gain access to any of the exercises they are involved in.

2020 03 26: There needs to be some degree of fit between the characteristics of the group of participants and the purpose of a ParEvo exercise. The purpose has to be motivational in one respect or another. Asking people to speculate on alternate futures that they either know little about or whose contents will have little consequences for them, may not be very productive.

3. Describing the starting point of the process

This is a seed paragraph of text, providing a common starting point for all subsequent contributions. This can be a real event or an imagined event. Think of it as the opening paragraph in the first chapter of a novel. The event will typically happen at a particular point in time, that is made quite clear. The seed paragraph will normally be drafted by the Facilitator.

4. Defining the endpoint

At the planning stage, this is an option, not a requirement. The endpoint can be defined as (a) a point in time and/or (b) a specific number of iterations of the ParEvo process (see below). As above, it may or may not include a description of what is expected to happen at that endpoint (backcasting) or not (forecasting)

The optimal number of iterations will depend partly on the number of participants. If the number of iterations equals the number of participants +1 then this means that each participant will have had the opportunity to build on the contributions of each of the other participants, at on at least one occasion.  This might represent a minimal ideal level of opportunity to explore and exploit the diversity of ideas presented by the diversity of participants. In practice, in the exercises completed so far, only one of the exercises has extended this far.

A note on time: Each iteration can be introduced as a consecutive period of calendar time, a temporal framework within which all participants have to work within. Or, no time definition can be given for each iteration. The only requirement being that the events in the current iteration have to be events that took place after the previous iteration. Within this option each participant will be describing events in their own “subjective” time, which may move at different speeds within one iteration and the next, and it may move a a different pace to that of other participants. The main requirement here is that participants do continue to signal, one way of another, when the events they described, did happen.

5. The Facilitator provides guidance to participants

In each iteration, from the beginning onwards, the facilitator needs to provide participants with some guidance. This will be found at the top of the ParEvo user interface, as seen in this example. It can include the following:

  • Minimal requirements for participants’  contributions:
    • maximum length,
    • plausibility/probability and consistency requirements,
    • deadlines for contributions
    • guidance on civility, etc
  • Context setting information.:
    • Reminding participants of the overall purpose of the exercise
    • (Optionally) providing information on “surrounding developments” that the emerging storylines might need to take into account.
  • Privacy policy:

6. Participants make their contributions 

In this first iteration, all participants:

  • Receive and read the guidance from the facilitator and then
  • Contribute an additional section of text describing what they think might happen next. This action develops the beginning of N storylines, where N = the number of participants. It contributes “variation”, one of the three essential parts of the evolutionary algorithm

7. Developing storylines are displayed and shared

When participants log onto the ParEvo webpage and then access the particular exercise they are involved in they will see a view like the one below, with five different parts:

  1. The Facilitators guidance in the centre top area, with the exercise title above it
  2. A graphic representing the exercise theme, on the top left
  3. The seed text, underneath the Facilitators guidance.
  4. A  tree structure, on the left side, enabling participants to navigate along and between different storylines, while seeing how they connect to each other. This is supplemented by a scrollable column of text in the center of the page, representing the currently selected storyline of interest.
  5. Though not shown here, there is also a space for comments on contributions.
parevo first view

The identity of the contributor of each paragraph to each storyline is not made visible. The intention is that the participants’ focus should be on the content of the contributions, uninfluenced by knowledge of who the contributors are.  Their identity will be known to the Facilitator (see more below).

Participants do not get to see each other’s contributions until all have contributed to a given iteration. At that point, these are all displayed at once, as above.

8. Re-iteration of 5,6,7

A new iteration only begins when all participants have contributed to the previous iteration, and they have been displayed.

Facilitator updates guidance to participants

This may or may not be needed at the beginning of each new iteration, depending on participants previous behavior and the need to introduce any new information about the imagined “surrounding context”

Participants add new contributions

At the beginning of each new iteration all participants are asked to look at each of the developing storylines and choose one which they would like to extend with a new contribution of their own. Each participant can only make one contribution, to one existing storyline, per iteration. But with each new iteration, they can change their mind about which storyline they now want to contribute to. As above, their text contribution could be a good, bad or neutral development, as seen from any stakeholder’s perspective. Participants can choose to add to any previous contribution, made by others or by themselves. As before, these contributions are anonymous.

Previously I had thought that fast iterations would be a good thing, making fewer time demands on participants and delivering results sooner than later. But a paper by Bernstein et al ( 2018) titled “How Intermittent Breaks in Interaction Improve Collective Intelligence.” suggests otherwise, that delays between iterations might be beneficial.

Display and sharing of contributions

After all participants have made their next contribution the display is updated to show the extended contents of each of the storylines. If more than one participant chooses to add their next contribution to the same existing storyline then that storyline now branches and becomes two (or more) storylines. On the other hand, if some existing storylines did not receive any new contributions they remain as viewable storylines but are now treated as “extinct”. These storylines can no longer be added to in subsequent iterations of participant contributions. The total number of storylines in an iteration will always equal the number of participants.

9. Evaluation

Evaluation of content

The progress and achievements of a ParEvo exercise can be assessed in three ways:

1. Participants Comments

Participants can be enabled to post Comments the contributions that have been made. The use of the Comment facility is optional. The Facilitator decides if and when to allow Comments and individual participants can choose if and when to make comments during any given iteration. They can make a maximum of one comment per storyline in a given iteration. This comment facility can be used as the second part of each iteration of the ParEvo process. But only after all new contributions are received and displayed in a given iteration. These contributions are anonymised and all displayed at once, prior to the commencement of the next iteration.

2. Participants evaluations – within ParEvo

At the end of the ParEvo process, the facilitator triggers an evaluation stage, where participants are asked to rate the surviving storylines, on two default criteria: (a) their probability of happening in real life, (b) and their desirability of happening, or any other relevant criteria. See Figure 2 below.

evaluation response form
Figure 2: Form that participants use to express evaluation judgements.

The Facilitator can edit and change the default evaluation criteria. Other criteria, such as novelty, or observability, may be more useful in some circumstances. (See Pugh, 2016)

After all responses have been received the aggregated responses of all participants to the built-in evaluation questions are shared with all participants, through a display as seen in Figure 3 below. The ratings of each storyline can be viewed by clicking on the right and arrows above the evaluation panel.

evaluation feedback
Figure 3: Display of aggregated evaluation responses. The display changes according to the selected storyline
3. Participants evaluations – via external survey

Survey Monkey (or similar) can be used to ask participants additional and more open-ended evaluation questions. See the design and results of such a survey associated Alternate futures for the USA 2020+ exercise. The following types of questions can be asked:

  • Questions about specific storylines
    • Closed ended questions, like those above about desirability and probability
    • Open ended questions, such as the “most significant difference between the storylines”
  • Questions abut the whole set of storylines
    • What was most surprising about the content, or what was absent from the content
    • How much the events were likely to affect the participant, and how much the participant could affect the events
  • Questions about participant’s judgements versus that of others
    • How optimistic their own contributions were, versus those of their own

Postscript: 2021 07 28: There is a now a dedicated web page here on this site with a growing menu of different types of evaluation questions.

You can also see an example of an external survey used in association with the Alternate futures for the USA 2020+ exercise

Downloadable data

At the end of a ParEvo exercise, the following kinds of data can be downloaded in an Excel file format, and subject to further analysis:

  • Participant x Participant matrix – showing who built on whose previous contributions
  • Participant x Storyline matrix – showing who built on which storylines
  • Text of each storyline – organised by storyline
  • Evaluation matrix – showing which storylines were rates as most/ least desirable/probable
  • Participants evaluation judgements – showing which participant made which  judgements
  • Participant list – of all those involved
  • Export search history – showing which key words were searched for, their aggregate incidence and were found in which specific contributions

10. Using the products of the ParEvo process


  • This part of the ParEvo workflow is not expected to happen online, within the ParEvo app. It could take place in face to face meetings or via video conference calls
  • The list of steps below is provisional and is very likely to be revised in the light of experience gained from new ParEvo exercises.

1. Identifying major disagreements about the storylines

When participants are asked to identify the most likely and least likely storylines, it is not unusual that there will be some disagreement in these judgements. For example five people might rate a particular storyline as most likely, and one other might rate it as least likely.  Where outright contradictions of this kind occur these need to be resolved before there can be any coherent thinking about an appropriate response to a particular storyline. This is most likely to involve discussion between participants, and perhaps other parties.

There are two possible outcomes of these discussions. Firstly, agreement will be reached about the likelihood of the storyline happening, in which case discussion can proceed about how to best respond to the possibility ( see more on this below). The second possibility is that no agreement can be reached about likelihood, so storylines in this category can be characterised as having what is called Knightian uncertainty

For scenarios characterised by uncertainty responses need to be generalised and robust. That is, they should be widely applicable across different circumstances, even though this may be at the cost of being the optimal response in a particular situation.   For example, when a company is concerned about the uncertainty of its future it might decide to increase its capital reserves. Even though this will be at the cost of efficiency in the use of capital i.e it will not be an optimal response.  In the social and biological world these strategies are sometimes called “bet hedging” strategies. For more, see this Wikipedia article on bet-hedging in biology.

Randomisation of choices can also be an option, where a menu of available choices can be identified. For more on this (seriously proposed) option see On the usefulness of deliberate (but bounded) randomness in decision making”

2. Identifying the kinds of storylines that are missing

When a set of surviving storylines have been located in a scatterplot, with axes such as likelihood and desirability, some attention should be given to the nature of that distribution. Are there some conspicuously empty spaces in the scatterplot, where there is an absence of storylines describing what might be happening there? If so, this may have implications for the design of any subsequent ParEvo type exercises. The assumption here is that a diversity of storylines i.e. occupying a whole range of possible types, is desirable, because it will mean the people or organisations involved in an exercise will have to think more widely and flexibly about their possible responses.

Caveat: As the number of iterations increased the proportion of storylines that have not survived will increase, and soon exceed the number of surviving storylines. These need attention, because they represent gaps, i.e. potential themes and directions that have deliberately ignored. Key word searches can be used to test ideas about what is being neglected i.e. more common in extinct storylines. For example :

3. Prioritising analysis of specific kinds of storylines

When a set of surviving storylines has been plotted on a likelihood x desirability scatterplot there will be four broad categories of storylines that can be prioritised for more detailed discussion. This is my suggested priority order, which would probably bets be checked with the participants in any ParEvo exercise. :

  1. Least desirable but more likely storylines
  2. More desirable but less likely storylines
  3. More desirable and more likely storylines
  4. Less desirable and less likely storylines

4. Responding to specific storylines

Within each of these categories different types of responses would seem to be appropriate:

  1. With storylines which are least desirable but more likely the concern here will be with risk,
    • and how it could proactively be inhibited or retroactively be mitigated.
  2. With storylines which are more desirable but less likely the concern here will be with opportunity,
    • and how it could proactively be enabled or retroactively capitalised
  3. With storylines which are more desirable and more likely, perhaps the focus will be on checking against, and updating, any prior strategies about where the organisation wanted to go.
  4. With storylines which are least desirable and least likely, the most appropriate response may be to periodically revisit, and if necessary, update this assessments about likelihood.

5. Identify responses that will work across multiple storylines

A suggestion borrowed from Scoblic (2020): For example isolation and tracing are two strategies that are appropriate across many different kinds of epidemics and pandemics. These events (as described by different storylines) may be undesirable but vary in probability, or be desirable and vary in probability..