Gathering Evidence of Impact
Why evaluating impact matters
- It helps identify effective methods and activities for engaging people with your research, maximising impact generation
- Evaluation enables anticipation of challenges and improvement of future endeavours by learning from both successes and failures
- It allows you to demonstrate the value/return of investment of your research to funders, media and other stakeholders
- It helps meet obligations and informs decision-making processes
- For universities, it contributes to national and international rankings and provides evidence for future funding applications
- It improves the quality and effectiveness of research impact strategies
- It provides researchers with a sense of accomplishment and can boost job satisfaction by showing how their work makes a difference
What to evaluate?
What do you need in order to determine the impact of your research, and be able to tell a convincing impact story?
Research impact evaluation is defined as
... the process of assessing the significance and reach ... of both positive and negative effects of research
(Reed et al., 2021, p3)
Significance: The degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or wellbeing of the beneficiaries
Reach: The extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact.
You will also need to consider the starting point to measure from. What is the baseline? What is the situation now/after your research, compared to the situation before?
Causality: To complete the story, you need to be able to demonstrate that there is a causal relationship between your research and the changes you've identified. Research is often only one factor among many influencing outcomes and leading to impact.
- The causal relationship may be one of attribution
if you can show that the research was entirely sufficient and directly responsible for causing the impact. - Alternatively, the relationship may be one of contribution
where you can show that the research was a necessary factor (among others) that contributed to causing the impact.
Embedding evaluation across the project lifecycle
Integrate evaluation into your project by:
- Using the impact pathway model to map out evaluation points across the project timeline.
- Evaluating the design of your impact pathway to identify potential flaws early.
- Implementing ongoing monitoring and data collection processes.
- Monitoring the delivery of activities and their immediate outcomes to make timeline adjustments.
- Assessing the broader and longer-term impacts to ensure the project's success beyond initial activities.
- Being alert to unexpected outcomes or impacts
- Conducting regular check-ins with stakeholders and partners.
- Building processes to assess what works and what doesn't.
- Using evaluation insights to inform project decisions and be prepared to adjust strategies based on evaluation findings.
There are two types of evaluation
Summative
It's tempting to think that all that is required to identify the impact of a project or programme of research is a summative evaluation at the end
Formative
But it's also useful to establish an ongoing formative evaluation throughout the project or programme
How to build an effective evaluation plan
- Align evaluation with project design: Start planning the evaluation from the beginning of your project. Align it with the project's impact pathway to ensure consistency between planned activities, outputs, and desired outcomes.
- Define objectives and desired impacts: Clearly outline the goals of your project and the impacts you aim to achieve. These should guide all evaluation activities
- Formulate evaluation questions: Develop specific evaluation questions that will help determine whether and how your project is achieving its intended results.
- Identify indicators and sources of data: Select key indicators that reflect your goals. Determine what data you will need and where it can be sourced to support your impact claims.
- Choose an appropriate evaluation design: Select a methodology or methodologies appropriate to your project context and that can effectively assess both the significance and reach of impacts.
- Plan data collection and timelines: Outline the methods for collecting both quantitative and qualitative data. Set clear timelines for each data collection activity.
- Assign roles and responsibilities: Clarify who is responsible for each evaluation activity to ensure accountability and efficiency.
- Plan for short-term and long-term impact assessment: Include strategies to capture both immediate outcomes and longer-term impacts over time.
- Capture and document evidence throughout the project: Establish methods to gather and store relevant data and documentation as the project progresses.
- Analyse and interpret results: Plan how you will analyse the data collected to draw meaningful conclusions about your project's impact.
- Communicate findings to stakeholders: Develop a communication strategy to share evaluation findings with stakeholders in an accessible and actionable format.
Evaluation methods and approaches
Quantitative
Experimental and statistical methods
Qualitative
Textual, oral and arts-based methods
Systems analysis
Combining qualitative and quantitative methods
Indicator-based approaches
Identifying variables that indicate the achievement of impacts.
Evidence synthesis approaches
Systematic aggregation across multiple evaluations in different contexts
Data and evidence
Evidencing impact involves systematically collecting and analysing data to show how research has influenced or benefitted the various stakeholders including communities, policymakers, and other researchers.
Use robust impact evidence to substantiate your impact claims, demonstrate significance, and illustrate the reach of your research outcomes and impacts. Possible sources include:
- Surveys
- Interviews or focus groups
- Tracking attendance at events
- Gathering testimonials and statements from research users and beneficiaries
- Tracking citations of research in policy documents, clinical guidelines, other research reports
- Using social media metrics and Altmetrics to show online engagement and reach
- Capturing stories and case studies to illustrate specific impact examples
- Using photographs or video to showcase impacts visually
Impact Pathway Model - Indicators and evidence
Download a blank template to plan your own indicators, data collection and capture evidence across the impact pathway model, including a worked example.
References
A list of references and further reading