Evaluating public engagement
This toolkit is for University staff and students who are undertaking public engagement and want to evaluate their activities.
This page covers:
- Why bother with evaluation?
- Thinking about your evaluation
- Planning your evaluation
- Data collection and reporting
Why bother with evaluation?
Most research councils require public engagement to be included in funding applications and some evidence to be provided at the end of the funding to demonstrate its impact. Funding for public engagement activities and the evaluation of those activities should be included in the application and fully costed in URMS (University Research Management System). Evaluation can help you demonstrate the impact of your project and provide evidence to the research council to support your activity.
- Provide assistance with planning future events
- Provide evidence of the audience’s experience of the event
- Gather views from others involved in your event, such as partners or stakeholders
- Demonstrate value for money
- Help with reporting to funders
- Generate evidence for the Research Excellence Framework (REF)
- Help to inform learning that can be shared with others and inform future events, research or the development of new research questions
- Help you decide whether your event has achieved what it set out to do
Evaluation is about critical reflection and it can be: formative (to support your activity, e.g. identifying gaps or barriers, gathering information to identify the need for your activity, testing ideas and concepts), summative (i.e. a summary – final impact of your activity) or about a process.
The evaluation tree
Fran Marshall, Research and Evaluation Manager in the Public Engagement and Impact Team recently presented a poster - 'Public engagement evaluation: can you see the wood for the trees?' - at Engage 2015, the national public engagement conference organised by the National Coordinating Centre for Public Engagement (NCCPE). The evaluation tree poster, shown in thumbnail on the right, is available to download in PDF format by clicking on the image or from the 'Downloads' box on the right of the page.
Before embarking on evaluation, you will need to consider various factors to ensure your evaluation is robust. It may be helpful to think about evaluation as a process illustrated by this evaluation tree metaphor.
Thinking about your evaluation
Start at the roots of the tree and ask yourself what kind of evaluation is it (e.g. a one off activity or something more long term), what do you want to know (e.g. you might want to know about learning or what the audience may do as a result of coming to your event) and what are the aims and objectives of your event (e.g. raise awareness of your subject). Just like a living tree, these questions anchor your evaluation activity making it strong and stable.
You will probably have some objectives or purpose of undertaking the event and this should be a focus for your evaluation. For example, if your event was held to promote learning of research in a particular field, your evaluation might what to know ‘how much did you learn about the subject’ or ‘did you know that the University does this kind of work?’ If you think about evaluation as a means to seeing if you have achieved your objectives, then it can be a useful way to see if your objectives are realistic and measurable.
Once you have clear answers to these questions, you can begin to plan your evaluation.
Planning your evaluation
The evaluation plan is represented by the trunk of the tree and should be robust and strong to support your evaluation activities. Just like a living tree, the trunk connects and supports the roots (the purpose, aims and what you want to know) to the branches (data collection methods).
The main questions are:
- Who do you want to evaluate? Usually the audience. If your event involves other stakeholders such as external partners, other staff or volunteers it may also be useful to ask them about their experience so that future activities can be improved.
- Who are your public? Consider how many people will attend, whether they are from particular groups and if they will pre-book.
- What resources are available? It is important to have someone to undertake the evaluation on the day and analyse the results afterwards.
- What type of event is it? Different data collection methods are more appropriate for different events. For example a pre-booked audience for a lecture is different to a drop in audience at an exhibition.
- What questions should you ask? In general, keep it simple and ask people about the event. Nothing is perfect so chose what fits best.
To summarise, focus on one or two things that really want to know, remember the unexpected and be prepared for some unanticipated findings.
Data collection and reporting
As a minimum you should:
- Count your audience
- Record your event using photographs
- Use at least one technique to provide evidence
Methods of data collection
The evaluation tree shows some of the data collection methods that you could consider when evaluating your event. Many can be used for a low cost and you may want to mix some methods to provide a rich diversity to your data. Whichever method you choose, it should answer your questions and fit practically with your event.
Here are some examples of evaluation methods:
- Voting - audience feedback can be obtained using 'Responseware' with clickers or mobile devices and provide a good interactive method or asking questions which can yield a good response rate
- Written comments either in a comments book, or on a poster or postcard
- Analysis of social media discussion and media coverage - e.g. with Storify, Twitonomy, Google analytics
- Focus groups and interviews can be good for more in-depth research or provide sound bites
- Films and videos
- Creative techniques, such as sticker exercises, graffiti walls or something built into the activity
- Questionnaires or surveys - online questionnaires are useful because most software will do some of the analysis for you and these can be easily sent out when attendees have pre-booked and given their email address. Tablet devices can also be used to collect feedback and can be less onerous that asking attendees to provide written comments. Paper surveys can also be useful but you will need someone to be responsible for managing them at the event and the input and analysis of data can be time consuming
- Observation and photography - you can observe the audience during your event and see how they react. You should always count your visitor numbers and take photographs if possible. Whilst these will not necessarily provide evaluation data observation and photography will help you to show an overall picture of the event and can be used in post-event promotion and news stories
When organising your evaluation, ensure that you have enough staff or volunteers that are confident in engaging with the audience to collect the data. Most evaluation can be undertaken free of charge or at a low cost. You may incur costs of printing, hiring equipment such as poster boards, pens or recording or filming charges. The cost of undertaking evaluation should be included in your funding application.
The evaluation report
After your event and when your data collection is complete, you should compile your evaluation data and analysis into a report. This part of the process is represented by the basket.
In addition to your evaluation findings and analysis, you may want to include the following information in your report: details about the event and any highlights, finance, media coverage, marketing activities, visitor numbers, visitor feedback and staff feedback.
Your report should also reference the aims and objectives of your event.
An example report is available to download from the 'Downloads' box on the right of the page.
Guidance for evaluation elements in grant application forms
Aims and objectives of proposed activity
Please ensure that aims and objectives are SMART (specific, measureable, achievable, realistic and time bound) and can be evaluated as part of the planned activity. E.g., ‘to inspire children to be future scientists’ or ‘to encourage attendees to study science’ are virtually impossible to measure at an event unless long term follow up is planned. Smarter objectives would be ‘to increase knowledge of x’ or ‘to make attendees more aware of x’.
Evaluation sections of application forms tend to be brief and when this is the case, be driven by the aims and objectives rather than the methodology. E.g., ‘the evaluation will collect data from attendees about how much they learnt’, rather than ‘we will collect qualitative data by questionnaire on learning’.
If there is scope within the word limit, you might want to identify the main groups you need feedback from, e.g. audience, partners or academics followed by the objectives and key questions for each group.
|Audience||To provide an enjoyable experience||How much did you enjoy the event?|
|Academics||To provide an opportunity for academics to share their research with the public||To what extent did the event help you engage the public with your research?|
Where possible, include evaluation throughout the project not just as the end or as part of the event.
Key question guidance which you might want use as a starting point is available to download in the box on the right.
Don’t forget to cost in evaluation materials and any staff time where possible.
Please discuss the evaluation part of the application with the Public Engagement Team at an early stage. If the award is successful, please discuss the development of a more detailed evaluation plan and reporting requirements with the Public Engagement Team (firstname.lastname@example.org).
- Start to think about evaluation when you are applying for grant funding and during the event planning process rather than at the end - if public engagement is part of a grant application it should be fully costed at the application stage
- There is no single answer or way to undertake evaluation, try out some different techniques beforehand and try to put yourself in the position of your audience to develop a fun and interactive way to gain feedback
- Use the information you gather to learn how to improve your event for next time.