Questions and Answers

Questions asked below:

  • When will the app be publicly available?
  • Where does expert knowledge come into the picture?
  • What about what we don’t know?
  • How many participants are too many?
  • How many participants can ParEvo work with?
  • What sort of participant feedback would be most useful?
  • Could participation in a ParEvo exercise be open-ended?

When will the app be publicly available?

The app is already available, at  We are now looking for earlier adopters / beta-testers, i.e. people willing to try out the version that is now available. If you are interested then contact All early adopters will get as much skype video-based tech support and advice as they need. We want to learn from your experience, as well as our own. A Facilitators Protocol is also available. This spells out all the steps needed for a facilitator to conduct a ParEvo exercise.

Where does expert knowledge come into the picture?

In many traditional approaches to scenario development and planning, expert knowledge is a key part of the scenario development process. In more participatory approaches there may need to be tradeoffs. The Oteros-Rozas (2015) paper on participatory scenario planning examines a set of 23 case studies  “Nine cases cited the lack of quantitative information, statistical and data-based testing, or modelling to support trends analysis as weaknesses. Five cases reported as a relevant weakness the unavoidable trade-off between the accuracy requested by the science base, which includes high complexity of scientific information, versus the social relevance of the process

In ParEvo there are different opportunities for expert knowledge, however, defined:

  1. Participants can be given a pre-exercise briefing by relevant experts, and/or…
  2. Experts can be represented as one of the groups of stakeholders participating in an exercise, and/or…
  3. Experts can provide additional context-setting information, shared with all participants, by the facilitator, at the start of each new iteration, and/or…
  4. Experts can use the Comment facility after all contributions have been received in a given iteration, and/or…
  5. Experts can be involved in the evaluation of surviving storylines, especially in relation to plausibility and probability of the described sequence of events.
  6. Experts can be involved in subsequent content analysis of all the storylines, surviving and extinct

What about what we don’t know?

What if you are not impressed by the knowledge displayed by participants in a ParEvo exercise?

People have their limitations. Maybe often we don’t know very much about a subject of concern. Perhaps we should not necessarily expect the ParEvo process to always deliver impressive results, in the form of creative and collectively constructed scenarios.

Perhaps we should also treat a ParEVo exercise as a means of explicating the limits of our collective knowledge. If so, this suggests that almost as a default procedure, we should always get a third party to examine what has been produced, to identify what is missing.

How many are too many?

For some time I have been cautious about having too many participants, thinking that once the number gets large no one will be able to read all the current storylines and thus make an informed choice. And there was also another argument, listening to Scott Page on “collective intelligence”, it seems that there may be diminishing returns as the number of participants increases. 

But I now think this was a mistaken approach. As the number of participants, and thus storylines, grow, participants will have to resort to sampling storylines (if they have not done so already). In the absence of cues about authorship, this may well be a quasi-random process. If so, this may not be a bad thing.  “Lorenz et al. (2011) proved that the diversity of views of the group would decline when the group was fully exchanging information,” says Yu, et al (2018) in their literature review on “collective intelligence”. So taking multiple samples of available information may avoid this trap.

There seems to be a parallel here with the use of “bagging” in ensemble methods of prediction modelling, where multiple different random samples of attributes in a data set are used to generate an aggregate prediction that is better than any model based on a single sample.

2019 04 10: For more on the effects of different sized groups on collective performance, see Vercammen, Ans, Yan Ji, and Mark Burgman. 2019. “The Collective Intelligence of Random Small Crowds: A Partial Replication of Kosinski et Al. (2012). Judgment and Decision Making 14 (1): 91–98.

How many participants can ParEvo work with, in practice?

Up to now, the maximum number I have worked with is 12. The structure of the ParEvo app layout may impose some upper limits. I have yet to test this out, but guess that it might be around 15. I will explore this possible constraint soon.

Assuming there is a practical constraint on numbers, how could this be managed as in worked around? One approach, which I am keen to explore, is to treat those participants (e.g. number 16 or more) as members of a queue. If a storyline becomes extinct (i.e. is not added to in an existing iteration) then the last contributor to that storyline drops out of the list of active participants and joins the bottom of the queue. In the next iteration, his/her place in the list of active participants would be taken by a person on the top of the queue. This approach could have the effect of increasing the diversity of contributions available to the participants to build on.

This approach has a connection to the claim that “science progresses one funeral at a time“. In other words, it is not only the content of a scientific idea but also who carries it, which makes a difference. See the recent supporting research by Azoulay et al (2019) 

What sort of participant feedback would be the most useful?

The queue model proposed above provides the deselected participant with quite clear negative feedback. There are other less radical forms of feedback available.

Two are easily calculated, using data generated during a ParEvo exercise:

  1. The number of other contributions that were added on to one’s own contributions.  An egotistical perspective, perhaps.
  2. The number of contributions one added on to others’ contributions. An altruistic perspective, perhaps.

This information is already visible to individual participants. What is not automatically visible to each participant is the same information about the performance of the other participants.

What would happen if this information was collectively available? Making it so could be seen as a type of “gamification“. That is, it could affect the nature of the incentives affecting how people participate. Would this be a net positive or a net negative in its effects? 

Could participation in a ParEvo exercise be open-ended?

In its current form, any ParEvo exercise involves a defined number of participants. A more open version of selective participation would not require the number of participants to be defined either at the beginning or thereafter. On the surface, this would be problematic because the limitation on the number of contributions during any given iteration is a necessary part of the implementation of the evolutionary algorithm – i.e. selection. If numbers of participants varied from one iteration to the next, so to would the selection pressure

However, a hybrid form may be feasible. That is, the number of people contributing during any iteration could be subject to a standard limit. But in each iteration, a different group of people, taken from the top of a queue of all intended participants, could be the active participants. Previous active participants could be recycled through the bottom of the queue.


Leave a comment or ask a question

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.