Evaluation Questions

What you will find here

An accumulating list of questions that can be asked of ParEvo exercise participants, when an exercise reaches the evaluation stage. This should be seen as a menu of questions, not a set of necessary questions. It is not exhaustive, some facilitators have designed and run their own surveys on other platforms.

How to ask

There are four options

Use the widget built into the ParEvo app. Widely used in the past.
Use a third party online survey service. More often used recently
Group discussions, online or face to face. Less common, but should be used more often
One to one interviews. Infrequent, but still useful

Types of questions

1. ParEvo widget

Facilitators can set up one or more binary questions that ask the participants to select a storyline that is, their view:
- most likely, and least likely
- most desirable, and least desirable
- most equitable, and least equitable
- most sustainable, and least sustainable
- etc…
Now available: Participants are able to annotate their choices with text explanations of their reasons

2. Online surveys

Essential questions
- A question asking for the unique ID e.g. email, of the respondent
- A question asking for the respondents consent to the use of their survey responses, under conditions as described in a following paragraph
Optional questions
- Open ended questions
  - What was the most surprising event seen within the storylines that you read?
    - Answer = description of surprise and IDs of the storylines
  - What was the most surprising omission from all the storylines that you read?
    - Answer = description of surprise
  - What was the most significant difference between the ends of all the storylines that you read?
    - Answer = description of difference and IDs of two groups of storylines
- Rating scale questions
  - Contents of storylines
    - How optimistic do you think your/other participants contributions were?
    - How plausible do you think your/other participants contributions were?
    - To what extent do you think you might be able to influence these events?
    - To what extent do you think these events might influence your decisions?
    - How often did you see inconsistencies between two connected contributions?
    - When you saw inconsistencies to what extent did they influence the plausibility of the storyline?
    - How often did you borrow ideas from previous contributions? (not counting the contribution you were directly adding to)
    - Do you think the contributions were too imaginative / insufficiently imaginative?
  - Consequences
    - How likely are you to change any of your future plans as a result of this exercise?
    - To what extent has this exercise changed what you think about the exercise topic?
    - To what extent has this exercise changed what you think about how other people think about this topic?
  - Process questions
    - How important was it to you that other people built on your contributions?
    - How important was it to you that you built on other participants contributions?
    - On average, how many minutes did you spend during a single iteration reading, then adding your own new contribution?
    - How much did you enjoy reading the storylines and then adding your contributions to the storylines?
    - How useful were the Comments under the different contributions?
      - Which Comment did you find the most useful, and why so?

3. Group discussions

Content questions
- What explanation is there for the fact that some participants disagreed in their evaluations of storyline x, in relation to its desirability/likelihood/etc?
- What responses might be feasible with the different individual or types of storylines e.g
  - Preventive or enabling responses to events and in what way?
  - Mitigation or exploitation of events, and in what way?
Process questions
- Was the facilitation too ..
  - Directive or Non-directive
- Timing
  - Duration of individual iterations
  - Total number of iterations
- Guidance text on web page and emails
  - Clarity
  - Length

4. One to one interviews

Content questions
- What actions have you taken / are you likely to take in the near future, that have been prompted by the exercise?
Process questions
- How did you decide to choose which storyline to continue?
- Are you likely to make use of a similar process in the future?
- Or encourage others to do so?