How Eco Heroes impact studies work
Schools can measure what Eco Heroes changes for their pupils. This page explains, step by step, how an impact study is run, what is asked, how the results are calculated and how pupils' privacy is protected.
The study in one minute
Before and after
Pupils answer the same short questionnaire before the programme, straight after it and about ten weeks later.
A fair comparison
Their change is compared with similar classes that start Eco Heroes one term later. Every class gets the programme.
No names
Answers are linked over time by an anonymous code from the pupil's seat in the class, never by a name or e-mail address.
Results from 10 pupils
A teacher sees the class result only when at least 10 pupils answered before and after, and never one pupil's answers.
The questions the study answers
Main question: do pupils in Eco Heroes classes gain more knowledge of the 17 Sustainable Development Goals than pupils in comparison classes?
- Do their attitudes to nature and their belief that they can make a difference change more?
- Do they report more real-world sustainable actions, and do these last ten weeks later?
- Do teachers feel more confident teaching the goals, and how easy do they find the platform?
- Does more use (goals completed, missions done) go with larger gains?
Design: a waitlist comparison
Participating classes are split into two groups. The first starts Eco Heroes now; the second starts one term later and is the comparison until then. Where a school or network agrees, classes are allocated by lot, the strongest design; otherwise comparison classes are matched on age band, school type and country.
Select a round to see what happens.
T0 · Before. Round T0, before the first lesson. Both groups answer. This is the starting point for every measure.
T1 · After. Round T1, at the end of the ten-week programme. Both groups answer; the difference between the groups is the main result.
T2 · 10 weeks later. Round T2, about ten weeks later. Shows whether the effect lasts. For the comparison classes it is also their own 'before', so the study gets a second before-and-after.
Two versions of the knowledge test
The knowledge test has two versions of equal difficulty, A and B. Half of each class answers A first and B later, the other half the reverse, so a higher score cannot come from remembering the questions.
What pupils and teachers answer
About 10 to 12 minutes in class for pupils and about 10 minutes for teachers. Where a published, tested scale exists, it is used as published, in the pupil's language.
Pupils 9–12
- Knowledge of the 17 goals: 15 multiple-choice questions (version A or B).
- Attitudes to nature (2-MEV): 16 statements on caring for and using nature. Published scale, Manoli and others, 2019.
- Belief that you can make a difference: 4 statements with a picture scale.
- Actions in the last 7 days: 8 questions, such as switching off lights or talking about a goal at home.
- Enjoyment and engagement: 5 statements, after the programme.
- Three open questions, after the programme: what you learned, what you changed, what you would improve.
Pupils 13–17
- Knowledge of the 17 goals: 15 multiple-choice questions (version A or B).
- Action competence (SPACS-Q): 12 statements on knowing what to do, believing it matters and being willing to act. Published scale, Olsson and others, 2020.
- Actions in the last 7 days: 8 questions, such as switching off lights or talking about a goal at home.
- Enjoyment and engagement: 5 statements, after the programme.
- Three open questions, after the programme: what you learned, what you changed, what you would improve.
Pupils 6–8
- Younger pupils do not fill in a questionnaire. Their teacher completes a short observation sheet for the whole class (6 statements: knowledge shown, taking part, actions).
Teachers
- Confidence teaching the 17 goals: 8 statements, before and after.
- Sustainability teaching time and practice this term: 4 questions, before and after.
- What was delivered (lessons, goals, missions, minutes): 4 questions, after.
- How easy the platform is to use: the 10-item System Usability Scale (Brooke, 1986), after.
- Three open questions: what worked, what did not, how pupils reacted.
Platform data add an objective layer: goals completed, quiz scores and missions logged, linked to the same anonymous code.
How the results are calculated
Main result: the knowledge score after the programme is compared between the two groups, taking each pupil's starting score and age band into account and allowing for pupils being grouped in classes. It is reported as an effect size (Hedges' g) with its 95% confidence interval.
- The same method is used for attitudes, action competence, actions and teacher confidence. Because several secondary results are tested, p-values are adjusted (Holm method) so that chance findings are not reported as effects.
- Each scale's reliability is checked (Cronbach's alpha, or KR-20 for the knowledge test) with 0.70 as the minimum.
- Open answers are coded into themes by two people independently on part of the answers; agreement must reach Cohen's kappa 0.70 before one person codes the rest.
- Answers given unusually fast, a failed attention question or identical answers to a whole scale are flagged and checked, not deleted automatically.
How many classes are needed
Move the slider to choose the smallest effect worth detecting. The numbers assume 22 pupils per class, 10% of the variation between classes, a starting test that explains half of the final result, 15% of pupils lost, 80% power and a 5% significance level.
| Smallest effect worth detecting | Classes per group | Pupils per group |
|---|---|---|
| 0.40 (moderate to large) | 9 | 198 |
| 0.30 (moderate) | 15 | 330 |
| 0.25 (small to moderate) | 21 | 462 |
Privacy and safeguarding
The study is part of the school's own evaluation of its teaching. The school is in charge of the data; Eco Heroes processes it for the school under the data processing agreement.
Pupil
Answers on a class device after entering the class code. Can skip any question or stop. There are no marks.
Teacher
Sees how many pupils have answered, and the class result (average before and after, share of pupils who improved, the goals the class knows least) once 10 or more pupils answered both times. Never sees one pupil's answers.
Parent
Is told by the school before the first round and can opt the child out; where the school's law or policy requires it, written consent is asked instead.
Eco Heroes
Sees pooled, anonymised results across classes for research and reports. Free-text answers are filtered automatically when submitted: e-mail addresses and phone numbers are removed, and answers matching a short list of words that may show a child at risk are taken out of the study data and passed only to the school's designated safeguarding contact. Names are not removed automatically; the researchers who read answers flag possible names and never report them. A possible disclosure is passed to the school's designated contact within one school day. If the school has not named anyone, these answers go to the school's administrators; for a class not linked to a school (for example one used only through a learning platform), to Eco Heroes' safeguarding lead.
No names, e-mail addresses or dates of birth are collected.
E-mail addresses and phone numbers typed into an answer are removed.
No result is shown for fewer than 10 pupils.
Answers are deleted 24 months after the study's last round.
Targets set in advance
These targets are fixed in the public study plan before the first round, so they cannot be moved after the results are in. Results below a target are published too.
| Measure | Target |
|---|---|
| Knowledge, Eco Heroes classes compared with comparison classes after the programme | Effect size of 0.30 or more |
| Knowledge gain within Eco Heroes classes | At least 20 percentage points; 70% of pupils improve |
| Knowledge kept ten weeks later | At least 80% of the gain |
| Attitudes and action competence | Effect size of 0.15 or more |
| Pupils who complete at least one real-world mission | 60% or more |
| Teacher confidence teaching the goals | +0.5 on a 5-point scale |
| Ease of use (System Usability Scale) | 75 or more (average across studies: 68) |
| Response rate | 85% before; 75% of pupils matched before and after |
Questions
Does a class that waits miss out?
No. Every class gets Eco Heroes; the comparison classes start one term later. That is why this design is fair to pupils.
Can a pupil refuse?
Yes. Any question can be skipped, a pupil can stop at any time, and parents can opt their child out in advance.
Will the results be published?
Yes, as pooled results that cannot identify a class or a pupil, including results below target. The study plan is registered publicly before the first round.
Why ten pupils?
Averages from fewer pupils could reveal individual answers and are too uncertain to mean anything.
Who designed the questionnaires?
The knowledge test and the action questions were written by Eco Heroes from the lesson objectives, and a pilot with pupils checks every question before a study starts. The attitude and action-competence scales are published scales, used under their open licences.
Words used on this page
- Effect size
- How big a difference is, in standard deviations. 0.2 is small, 0.5 medium, 0.8 large.
- Confidence interval
- The range in which the true value lies with 95% confidence.
- Reliability
- How consistently a scale measures. 0.70 or more is the accepted minimum.
- Waitlist comparison
- A design where comparison classes receive the programme later instead of not at all.