This was the 10th ParEvo exercise
Purpose
The general purpose of this study was:
- To identify and understand more about the many different ways that people think about the future
- To identify how different forms of cooperation between participants contribute to the development of different kinds of alternate futures
- To refine the design of the ParEvo process, which is intended to facilitate the participatory exploration of alternate futures
More specifically:
- To identify what types of storylines and participant behavior could occur if participants were crowdsourced. And as such, had no prior knowledge of each others’ interests and views
Participants
The exercise participants were crowdsourced through Prolific.co (an online participant recruitment for surveys and market research). Criteria used to find participants via that platform were:
- Age: Greater than 20
- Nationality: United States
- Education: Undergraduate degree or higher
Once the participants were recruited Prolific.co then provided this additional information on the self-selected participants :
- Gender: 6 women, 5 men
- Age: median 31
- Employment: 8 full-time, 1 part-time, 1 unemployed-seeking work
- Country of birth: 5 USA, 5 other
- The approval rate for submissions in previous studies: 9 of 10 had a 100% approval rate
Process
The exercise involved eight iterations plus a final evaluation stage. Participants were introduced to the exercise, and to each iteration within the exercise, through the contents of a dedicated Prolific.co web page (as is required). They were then directed to a Survey Monkey survey where they made their contributions to each iteration (again as required by Prolific). On completion, they were returned to another Prolific web page, to have their work contributions reviewed and payments triggered . The participants contributions were then manually entered by the Facilitator into a ParEvo.org exercise and the ParEvo exercise web page was then shared with all participants at the start of the next iteration. The final evaluation involved the same sequence but used a different Survey Monkey survey page.
All participants were asked to give their consent at the start of each iteration (See the survey instrument link below). All contribution were anonymous, in the eyes of other participants. All participants were paid at a rate above the UK minimum wage rate (per hour), with each iteration being paid for a 30 minute span of time. In practice, the median time spent per participant per iteration was 15 minutes, thus in effect doubling the hourly rate. Aggregated results of the evaluation stage were shared with all participants via this web page, and all participants retained access to the completed exercise on https://www.parevo.org/
Instructions to participants
Pdf copies are available for the following:
- Instructions for the first iteration
- Instructions for the following iterations
- Instructions for the evaluation stage
- Survey Monkey survey instrument
Exercise results
Contents
The storylines generated by this exercise can be seen here: https://parevo.org/exercise/public/c016a2b8-bdfc-4c54-9590-1111333253fe
Key word searches
Anyone can do their own very simple content analysis of ParEvo storylines using the keyword search facility built into the ParEvo website. Results are shown in the following format:
- 8 of 8 iterations contain ‘Biden’
- 53 of 80 contributions contain ‘Biden’
- 9 of 10 live storylines contain ‘Biden’
- 17 of 19 extinct storylines contain ‘Biden’
Ad hoc searches showed for example…
- The number of references to China and Russia were surprisingly few (3 and 2 respectively out of 80)
- References to Biden and Trump were understandably much more common (53 and 28 out of 80)
- References to a Covid vaccine were very common in early iterations, then disappeared, then reappeared in the last two iterations
Relationships between keyword search results.
The results of multiple key word searches can also be downloaded from ParEvo.org in the form of an “adjacency matrix”, in an Excel file, where key words are listed by rows and contributions are listed by columns, and cell values indicate if the row key word was found in the column contribution. Using network analysis software such as Ucinet/Netdraw that data can then be visualised as a network structure. Two useful types of network diagrams can be generated:
- Two mode or affiliation network, showing which key words were found in which contributions (Figure 1)
- One mode network, showing which key words were co-occurring in the same contributions (Figure 2)
Filtered views: Network contents can also be filtered, using specific node or link relationships of interest. Figure 3 has filtered Figure 1 to show only those contributions that mention Trump, Biden or both (in the center). The contribution numbers indicate in which iteration they were made e.g. 11,12,14 were in the second iteration, 65,68 were in the seventh iteration.
Text extraction
Text extraction uses machine learning to automatically scan text and extract relevant or core words and phrases from unstructured data like news articles, or contributions by ParEvo participants. These lists are a useful means of generating a menu of search terms, which might be wider in ambit than my initial top-of-mind list.
Entity extraction. With text2data.com it is possible to use a pre-trained general purpose predictive model to find and categorise entities mentioned within ParEvo contributions – as persons, organisations and locations. Using text2Data.com my original top-of-mind list of 6 countries was expanded to 12, and a list of two people was expanded to 14. The same extended list could have been developed by manual search, but this would have taken much more time.
Theme extraction: The same software program can also extracts “themes”, which may be relevant to the analysis of ParEvo storylines. In the first ten contributions the following themes were selected as being of interest from among the 35 that were extracted):Artificial intelligence, budget deficit, multiple lawsuits, promote unions, restricted by Twitter, corporate tax increases, gained seats and control, Senate run-offs.
Word frequencies: Online word count services, as offered by Rapid Table and others, are less nuanced and more literal. Frequencies are generated for all the words in a text, plus pairs and triplets of words. Most may not be search worthy, but those that are can be prioritized by their frequency of occurrence.
Participants’ evaluations
A Survey Monkey survey was used to obtain assessments from all 10 participants. A pdf copy of the survey instrument is available here.
1. Probability and desirability of the storylines
- Forty percent of likelihood judgements and 20% of desirability judgements included some contradictory judgements. This was higher than was found in recent exercises involving staff within one organisation (Exercise 9), where more consistency of views might be expected.
- Storyline 77 stands out as the storytelling with the most pronounced features, being most likely and least desirable. It was built by three participants, with one accounting for 75% of the contents. This storyline has this final paragraph for late 2022:
- In 2022 when the year was rounding up, the country faced a complicated political uprising. Republicans held the edge in the senate but lost some good numbers to the Democrats. Biden faced challenges on his proposed corporate tax increase. Although Democrats managed to gain control in a 50-50 Senate, he still face heavy objections from conservative Democrats. The legislative arm started a war to truncate and undermine his administration. With federal budget deficit of $5.8 trillion in the year ending, he will have a tough time rejuvenating the economy come 2023.
- Rick Davies: How different is this from the likely situation in early 2021?
- In 2022 when the year was rounding up, the country faced a complicated political uprising. Republicans held the edge in the senate but lost some good numbers to the Democrats. Biden faced challenges on his proposed corporate tax increase. Although Democrats managed to gain control in a 50-50 Senate, he still face heavy objections from conservative Democrats. The legislative arm started a war to truncate and undermine his administration. With federal budget deficit of $5.8 trillion in the year ending, he will have a tough time rejuvenating the economy come 2023.
- Storyline 80 was the most conspicuous storyline that was both unlikely and undesirable. It was built by three participants, with one accounting for 75% of the contents. This storyline has this final paragraph for late 2022:
- The US military became extremely powerful, rated one of the most powerful military in the world. With the help classified intelligence, Joe Biden got infiltrated North Korea successfully and compelled them to back out completely from nuclear weapon exploration. A big win for the Democrats. Joe Biden however maintained it was a big win for the great American people.
2. Optimism – Pessimism
Participants were asked “Looking at the 10 surviving storylines, do you think that overall they were too optimistic or too pessimistic” and to answer use a 0-100 scale where 0 was “Too pessimistic” and 100 was “Too optimistic”
- The median rating was 53, the lowest rating was 6 and the highest rating was 88
- Rick Davies: My reaction was that participants were way too optimistic, underestimating the difficulty the Biden administration would have with an obstructive Senate
- Readers’ reactions?: Use the Comment facility below
3. Confidence in own contributed events happening
Participants were asked “How confident are you that most of the events that you described in your contributions happening will actually happen?” and to answer using a 0 to 100 scale where 0 was “Very doubtful” and 100 was “Very confident”
- The median rating was 66, the lowest rating was 40 and the highest rating was 100
There was a weak “positive” correlation between how optimistic participants felt most storylines were, and how confident they were that the events they added to the storylines would actually happen. That is, people who were felt most storylines were too pessimistic tended to be more confident about their own contributed events actually happening. Optimism was a characteristic people shared regarding their own and others judgements
Worth exploring? Those who felt others were too optimistic while they were very confident.
On reflection, the two questions about optimism and confidence could have been better phrased so they both referred to the same sentiment i.e. confident or optimism, but not both.
4. Significant differences between storylines
Participants were asked to “sort ALL the 10 surviving storylines into 2 groups, of any size, according to what you think is an important difference in the storylines. Then use the Comment box below, to explain how you think the first group of storylines is significantly different from the second“
Differences that were identified
- Stories of the nation continuing on more or less as we have been. Though there is strife and political conflict in these stories, Americans still have the rights and freedoms they’re accustomed to. The overall tone is one of progress or normalcy. Versus
- Pile 2 contains dystopian stories. There might be an emphasis on military dominance. Or the entire populace has delightedly taken the least tested vaccine in the history of western medicine, manufactured for profit by companies that have immunity from liability. The tone of these stories reminds me a little of the novel 1984.
- The difference between these two groups is overall tone of negativity or positivity. I found that those in pile 1 tended to be a bit more negative, pessimistic, or realistic about what would happen in the next few years, while those in pile 2 tended to be positive, optimistic, and patriotic. Pile 1 showed our political system as partisan and not united, with power going back and forth to the parties, while Pile 2 tended to favor a strong national identity and unity while being an exemplary world leader.
- Pile 1 is more positive and realistic. They represent a more idealized future with positive consequences. I think pile 2 is more out there and more dramatic representing a less hopeful future.
- Outcome of the upcoming 2022 election (who controls the House) and extent to which Republicans fight against implementation of the Biden administration’s policy priorities
- pile 1 seemed to have a more optimistic and positive world view of what was happening with the presidency and had joe biden doing a good job with the USA, pile 2 seemed to be more pessimistic, and had more struggle fighting with the senate, and Joe Biden seemed to be more struggling in his presidency
- Pile 1: I think is not that real or convincing
- I think the pile 1 storylines were smooth and realistic
- Pile 1 had concrete storylines that appeared real. I also feel that writers put a good amount of effort into it. Pile 2 didn’t exactly impress me but they were well written
- The contents of Pile 2 are exclusively the most Republican sided storylines. After seeing how weak the Republican party is as they bended to the will of a tyrant rather than stand up to him, I believe any pile 2 scenario would be disastrous to the country. Pile 1 may not lead to a world of normalcy but, at least the cowards wont have control of any branches.
- Pile 1 contains humorous but relatable storylines while pile 2 contains somewhat concrete plots
5. Surprising contents within the storylines
Participants were ask “Which of any of the 10 surviving storylines (or part therein) did you find most surprising?”and then “Why did you think that storyline was the most surprising?”
- This story gives tremendous credit to the Covid vaccines for slowing a pandemic that lasts into 2022. I think this is unlikely. The current vaccines do nothing to prevent transmission. Dr. Fauci said recently that social distancing and masks would still need to be used by the vaccinated population.
- The terrorism threat and war was surprising.
- I was surprised that someone went in the direction of the vaccine being unsafe, and took such a negative focus on the future.
- There were several major foreign policy breakthroughs that I had not even considered in likely outcomes for the next two years
- Mainly because it went into detail about the vaccine and the effect that it had on the people and our society. Priority was placed on everybody and to me seemed the most realistic.
- It spoke a lot about the government.
- The storyline has a great theme with an almost-perfect grammar. By “it had a great theme” I meant the central idea behind the story was surprising to me. It talked about how Biden took stern actions against Russia and how flu was fought
- The write-up appeared like Biden performed a magic where he suddenly caused covid-19 to disappear just like that
- The first time I read the Republicans taking the house, I almost fell out laughing. Honestly, their party is so fractured and the damage is so strong, I don’t see how anyone could expect them to gain the upper hand in any department.
- The storyline seemed authentic
6. Surprising omissions from the storylines
Participants were asked “What kinds of events were you surprised to not see in any of the storylines, but which you think might be expected in the 2021-2022 future?”
- All of us seemed to completely ignore immigration. I just noticed this reading over the stories a final time. I’m sure immigration policies will continue to be areas of contention.
- The future of physical money and cryptocurrency
- I was surprised not to see more about climate destruction, lack of resources, and terrorist attacks (domestic and foreign). I was really surprised not to see anything about mass shootings which have become a very unfortunate common occurrence in the United States.
- Efforts to pass criminal justice and policing reforms, which previously had bipartisan support
- If I remember correctly I do not remember seeing any riots or mass chaos, I expect to an extent some people will riot especially because Joe Biden won and Donald Trump lost
- None none
- I was surprised not to see love and sex scandal stories
- I surprised that I don’t see fairytales
- Biden’s health declining from the stress of the presidency. I know we all don’t wish for it, but, for a man of his age and the stressors of the job, it is a possibility that hasn’t been explored.
- Fiction
7. Likelihood to be affected by the events
Participants were asked “To what extent do you think any of the events described in any of the storylines are likely to affect your life in the next two years?” and to answer using a 0 to 100 scale where 0 was “Not at all” and 100 was “A lot”. The mid-point was labelled as “I don’t know”
- The median rating was 87, the lowest rating was 19, the highest rating was 100
8. Ability to affect the events
Participants were asked “To what extent do you think you will be able to have an effect on any of the events described in the storylines? ” and to answer using a 0 to 100 scale where 0 was “None at all” and 100 was “A lot”. The mid-point was labelled as “I don’t know”
- The median rating was 59, just above the mid-point of “I don’t know”, the lowest rating was 1. and the highest rating was 99
Net Agency
Overall, participants reported being more likely to be influenced by the storyline events, compared to having an influence over those events. There was one conspicuous outlier, the participant who expected the storyline events to have little effect on him/her, but who expected to have a lot of effect on some of those events. This outlier should have been investigated through direct contact with the participant, to rule out the possibility that they simply misunderstood the question they were responding to.
9. Participants views of the exercise
Participants were asked: “Looking back at the contributions you were asked to write over the 8 iterations, did you think your tasks were too easy or too difficult?” and to answer using a 0 to 100 scale where 0 was “Too easy” and 100 was “Too difficult”. The mid-point was labelled as “I don’t know”
- The median value was 57, the highest value was 97 and the lowest value was 31
In future exercises this question will be replaced by one which asks participants how important it was to them that other participants subsequently added a contribution to their contribution. I have often assumed that this is the case for most participants, but this could be quite wrong and having said of participants with ranging views about this may be important.
Relationships between participants
1. Cooperation structure
In this exercise a relatively large proportion of participants were “isolating” in the sense that they often built on their own previous contributions (bottom left quadrant). However there was a significant proportion were “bridging” i.e building on the contributions of others, who also built on their contributions (top right quadrant) (#2,#3#,#4). There were very few “following” i.e. only building on the contributions of others, who did not build on their contributions. There was one participant who was conspicuously “leading” i.e whose contributions were being built on by others, but who did not build on the contributions of others (#1)
In Figure 8 the bridging role of participants 2,3 and 4 can be clearly seen
Participant 1 is “leading” in Figure 1 above, and in this Figure 9 network diagram is also the most connected to all the surviving storylines
2. Diversity of interactions between participants
Given 10 participants and 8 iterations this mean there were 70 (i.e. 80-10) different possible types of interactions between them.The actual number of types of interactions that took place was 29 (41% of 70), which is lower than previous ParEvo exercises. This is a simple measure of network density, which is being used here as a proxy for “variety diversity”.
3. Relationships between participants’ evaluation judgements
When Figures 8 and 10 are compared there is no simple correspondence between how participants collaborate in the construction of storylines and how they agree on the subsequent evaluation of the surviving storylines. The process of participatory storyline development seems to influence their evaluation judgements.
Network structure of storylines
1. Exploration – Exploitation (after James March)
Exploratory storylines = extinct storylines = 19 (66% of 29 actual) The maximum possible = 79% = (63/80) , minimum possible = 0%
Exploitation storylines = surviving storylines = 10 (34% of 29) The maximum possible = 100%, the minimum possible = 16%
2. Diversity (after Andy Stirling)
Disparity diversity among surviving storylines = distance between surviving storylines, as measured by number of links connecting surviving storylines = 45 = 56% of the maximum possible (=80) . Higher percentage = more disparity
This measure seems higher than in other exercises, largely because many of the participants were building on their own storylines (49% of all contributions were on to participants own previous contributions)
Analysis
1. Storyline contents
Key words as predictors of surviving storylines
The “keywords x contributions” affiliation matrix data mentioned above can be analysed to identify if there are any key word which are good predictors of participants contributions being part of extinct versus surviving storylines. EvalC3, an Excel base predictive modeling app, was used to generate the following set of predictive models.
Figure 11 show seven predictive models, each one being a separate branch of the tree (on that is lying sideways!). Only one third of the contributions belonging to surviving storylines could be predicted with confidence i.e. there were no false positives (green branches). Very few (3/44) of the contributions belonging to extinct storylines could be predicted with confidence (red branches). The best predictor of membership of surviving storylines was the presence of the keyword “health” but the absence of the word “global” and “Biden”. This model finds four True Positives and no False Positives (4:0), these key words are sufficient but not necessary.
In summary, the most that can be said about this type of analysis is that it is possible, but that the characteristics of most contributions contributing to surviving storylines are difficult to identify (N=36). Even more so with predictive models for membership of extinct storylines (N=44).
Participant IDs as predictors of surviving storylines
Another data set generated by a ParEvo exercise is an affiliation matrix showing the relationships between participants IDs and the storylines their contributions built.
- 2 storylines were developed by a single participant
- 3 storylines were developed by 2 participants
- 4 storylines were developed by 3 participants
- 1 storyline was developed by 5 participants
- 0 storylines were developed by 8 different participants (the maximum possible)
Some participants were more strongly associated with a storyline becoming extinct i.e. by being the last contributor
- Participant 9: 26% of extinct storylines
- Participant 2: 21%
- Participants 6: 16%
- Participant 4: 10%
- Participant 7: 10%
- Participant 3: 5%
- Participant 8: 5%
- Participant 5: 0%
- Participant 1: 0%
2. Participant behaviors
Relationship between participants’ contributions and aggregated evaluation judgements
There was a low and insignificant correlation between the number of participants who contributed to a storyline and aggregate judgements of how likely that storyline was to happen (r=0.172).
There was no correlation at all (r=0.0) between the number of participants who contributed to a storyline and aggregate judgements about storyline desirability.
Both findings suggests that participants evaluation judgements were independent of their contribution behavior.
Some observations…
- Participants understandings of the exercise requirements relating to contributions were good. There were no major misunderstandings
- Participants understanding of the evaluation questions was more variable. Text comments ranged from the detailed and meaningful to very short and uninformative (and requiring one to one follow up questions)
- Many of the descriptive measures in the results section would be more meaningful if they were compared to the same measures from other exercises. Comparisons could be useful with exercises involving three different types of participants:
- Self-selected crowdsourced participants
- Self-selected Community of Interest participants.
- Self-selected staff members from within a single organisation
- [To be continued]
Your comments…
Please use the Comment facility below to make any comments about the process and products (storylines and evaluations and analyses)
ParEvo 