Impact evaluation
Introduction
The information here provides an introduction to an impact evaluation approach for the evaluation of learning and teaching (L&T) projects, primarily those which are allocated resources through the Learning and Teaching Development Grant (LTDG) Scheme. However, the approach is also being used in other contexts (e.g. CILASS, SeeChange).
For those individuals or teams who have been allocated resources through the LTDG scheme, resource will have been included for the LeTS evaluation team to work with you in undertaking the evaluation. This guidance has been written with the twofold purpose of providing background information about the approach, and to help those undertaking their own impact evaluation. The references and recommended reading provides more detailed information for those who wish to find out more.
Background
The need for an appropriate impact evaluation approach for L&T change initiatives within the University was identified in 2005 when LDMU (now part of LeTS) conducted a meta-evaluation of projects allocated resources through the LTDG scheme. This found that although formative evaluation was helping project leaders to improve L&T within their local context, it was not helping to lead to broader understanding of transferable good practice. Summative evaluations, which evaluated against the stated aims and objectives of a project, were not providing information about the longer term and wider impact of projects on L&T within the institution, and were not responsive to emergent outcomes. They were often considered as if they were ‘audit’ activities rather than inquiry that would help to contribute to organisational learning about good practice in learning and teaching.
The functions of the formative and summative evaluations are still considered as important and relevant in the context of curriculum development, but there is an identified need to provide good quality data to help with decision making at the strategic level. This has prompted development of an evaluation framework with the emphasis on impact and which was endorsed by the Learning and Teaching Committee (via the Learning and Teaching Development Group) in Spring 2006.
Aims and objectives
The questions that impact evaluation seeks to answer for the University at the strategic level are:
How do the change strategies employed by staff members who are undertaking curriculum development projects affect teaching practice, and what effect do such changes on practice have on the student learning experience?
Do the activities of curriculum development at project and programme (e.g. LTDG scheme and CILASS) levels contribute towards transformation and change at the institutional level?
The key objectives are therefore:-
- to lead to understanding about the impact curriculum development activities have on the student experience, learning, teaching practice, departmental L&T cultures and the institution;
- to identify successful and transferable curriculum interventions with a view to facilitating the sharing and adoption of good practice;
- to enable comparisons across a diversity of curriculum development projects in order to establish the relative merits of alternative approaches;
- to inform University strategy and decision making about where to focus activity and resources for enhancing L&T quality in the future;
- to lead to impact evaluation being an integral and natural process for those engaging in curriculum development activity.
The impact evaluation that has been adopted is based on the ‘Theory of Change’ approach (Connell and Kubisch, 1998). Further information about this approach and how this has informed impact evaluation in the context of curriculum development at the University of Sheffield is given in the ‘Theoretical background’ section. However, the overarching philosophy is that the evaluation should be embedded within a participative, holistic, reflective and reflexive approach to curriculum development.