After implementing your cigarette butt littering reduction project and collecting feedback and results, it's time to evaluate: did it reach its goal, what have we learned and how could we do it better next time?
Project effectiveness
The effectiveness of your outcomes in meeting the project goal is indicated by the results of measuring butt littering behaviour, butt litter on the ground, and reported behaviour change (Measurement) in identified hotspots (Assessment).
Simple analysis of Measurement data can provide the following examples of results:
- A __% reduction in butt littering behaviour was measured in identified hotspots.
(Source: Behavioural observation data) - A __% reduction in butt litter was measured in identified hotspots.
(Source: Butt litter audit- accumulation) - __% of people surveyed saw project advertising or were approached by educators, heard it on radio (__%) or both (__%).
(Source: Recall survey) - __% of these people retained at least one message from the project.
(Source: Recall survey) - __% of these people were influenced to change their littering behaviour.
(Source: Recall survey)
Be careful not to outright state that the project caused these results. It is very difficult for any project, or indeed science, to show a causal link between an action and a result. A conservative yet valid statement is: 'these changes in butt littering, butt litter and reported behavioural change, coincided directly with the project period'.
We can report with confidence that these results are robust as they are a 'triangulation' of methods to measure, in our best efforts, a single audience.
Project Impact – Long-term
The ButtFREE approach has factored within its design, tools shown in behavioural studies to foster long-term behavioural change including instalment of infrastructure, Commitment and face-to-face education. Other factors, not explicit in the design, have also been factored in such as social diffusion of messages and expectations on litterers encouraging a social norm of correct disposal of cigarette butts.
The Measure Section of the ButtFREE Toolkit produces results that describe changes in littering behaviour and litter in hotspots, it also includes, through the questions within the survey, indication of long term impact of butt littering behaviour.
Obtaining long-term impact data is challenging and fraught with assumptions in any community based social marketing exercise and is an approach that Butt Free Australia has deemed outside the scope of the project design offered in this Toolkit.
Please contact us if further information is required.
Project efficiency
It's as important to measure the project process – development, design, management, implementation and/or delivery – as it is the project outcomes.
This includes measurement of the efficient application of the ButtFREE Solutions in the project plan:
- Was the Context and audience identification accurate as indicated in the Measurement methods?
- Was the education effective, was litter reduced in areas where infrastructure was installed?
- Did partners feel adequately included in the planning process?
- Did partners feel well represented?
Suggestions of how these might be measured was supplied in the Measure Section
What have we learned?
Evaluating what has or hasn't worked and why, is relevant both for the project effectiveness and efficiency. Seeking feedback from the project team and partners is a useful exercise in gaining such insights which can then be compiled with outcomes to produce an evaluation summary.
Recommendations
Any suggestions for what can be done better next time from your 'What have we learned’ exercise will serve as recommendations that can be of great value to future projects.
If you were to publish your report, this information can be extremely valuable to others planning a similar project.
Evaluating for an audience – internal or external
In preparing an evaluation summary, remember the audience it's intended for. Will you provide evaluation internally to assess whether or not the project was a good investment, or report to a wider community on its success? Or perhaps both?
Effectiveness and efficiency is of interest both internally and externally. External audiences would particularly benefit from the 'learnings' or recommendations with an emphasis on effectiveness. Internal audiences may be particularly interested in cost efficiency against impact.
The Evaluation summary
Brevity and clarity are key to a good evaluation summary. Beware of analysis paralysis: only note areas of particular value or concern, those that can be built upon and those to improve for future projects.
A Draft evaluation TEMPLATE to gain team/partner feedback is provided in the Tools list below.
An Evaluation summary EXAMPLE is provided in the Tools list below
Action - Evaluate
- Collate the results of your measurement to provide statements of results (project effectiveness)
- Circulate the Draft evaluation TEMPLATE to team members and partners to gain feedback on project efficiency
- Prepare an Evaluation summary using project effectiveness and efficiency results and feedback, including any recommendations
Downloads
Section
Evaluate Section (576 KB)
Tools
Evaluation summary EXAMPLE.doc
ButtFREE Toolkit: Overview | << Previous Section: Implement | Next Section: Report >>
