Evaluation Questions

What you will find here

An accumulating list of questions that can be asked of ParEvo exercise participants, when an exercise reaches the evaluation stage. This should be seen as a menu of questions, not a set of necessary questions.

How to ask

There are four options

  1. Use the widget built into the ParEvo app. Most widely used.
  2. Use a third party online survey service. Commonly used, but not always
  3. Group discussions, online or face to face. Less common
  4. One to one interviews. Infrequent

Types of questions

1. ParEvo widget

  1. Facilitators can set up one or more binary questions that ask the participants to select a storyline that is, their view:
    • most likely, and least likely
    • most desirable, and least desirable
    • most equitable, and least equitable
    • most sustainable, and least sustainable
    • etc…

2. Online surveys

  • Essential questions
    • A question asking for the unique ID e.g. email, of the respondent
    • A question asking for the respondents consent to the use of their survey responses, under conditions as described in a following paragraph
  • Optional questions
    • Open ended questions
      • What was the most surprising event seen within the storylines that you read?
        • Answer = description of surprise and IDs of the storylines
      • What was the most surprising omission from all the storylines that you read?
        • Answer = description of surprise
      • What was the most significant difference between the ends of all the storylines that you read?
        • Answer = description of difference and IDs of two groups of storylines
    • Rating scale questions
      • Contents of storylines
        • How optimistic do you think your/other participants contributions were?
        • How realistic do you think your/other participants contributions were?
        • To what extent do you think you might be able to influence these events?
        • To what extent do you think these events might influence your decisions?
        • How often did you see inconsistencies within storylines?
        • How often did you see inconsistencies between two connected contributions?
        • How often did you borrow ideas from previous contributions? (not counting the contribution you were directly adding to)
      • Consequences
        • How likely are you to change any of your future plans as a result of this exercise?
        • To what extent has this exercise changed what you think about the exercise topic?
        • To what extent has this exercise changed what you think about how other people think about this topic?
      • Process questions
        • How important was it to you that other people built on your contributions?
        • How important was it to you that you built on other participants contributions?
        • On average, how many minutes did you spend during a single iteration reading, then adding your own new contribution?
        • How much did you enjoy reading the storylines and then adding your contributions to the storylines?

3. Group discussions

  • Content questions
    • What explanation is there for the fact that some participants disagreed in their evaluations of storyline x, in relation to its desirability/likelihood/etc?
    • What responses might be feasible with the different individual or types of storylines e.g
      • Preventive or enabling responses to events and in what way?
      • Mitigation or exploitation of events, and in what way?
  • Process questions
    • Was the facilitation too ..
      • Directive or Non-directive
    • Timing
      • Duration of individual iterations
      • Total number of iterations
    • Guidance text on web page and emails
      • Clarity
      • Length

4. One to one interviews

  • Content questions
    • What actions have you taken / are you likely to take in the near future, that have been prompted by the exercise?
  • Process questions
    • How did you decide to choose which storyline to continue?
    • Are you likely to make use of a similar process in the future?
    • Or encourage others to do so?