Impact Pipeline: Investigating downstream leaks and preventing upstream leaks

A MEL and impact measurement system is like plumbing: there are many steps throughout a project lifecycle where potential impact can “leak” (go unachieved, unmeasured, or unreported).   

The Challenge 

CARE tracks progress towards its Vision 2030 targets in  impact – the number of people experiencing lasting change due to an intervention. Although CARE has been able to demonstrate impact on millions of lives during Vision 2030, we were also aware of many potential leakages in the impact pipeline. 

With this in mind, we investigated our leaks downstream, by reviewing evaluations and reports for projects with suspected unreported/underreported impact.  

What we did 

Reviews of project evaluations from mid-2022 to January 2024 focused on the last part of the impact stream, “Reporting the evidence”. The focus was on four types of projects, with the amount of impact recovered varying and decreasing with each review: 

• Covid-19 specific projects: A mix of direct implementation and systems level advocacy for vaccine delivery (evidence of 21.2M additional people impacted) 
• Humanitarian: Large task force focused on humanitarian projects/mostly direct implementation (evidence of 8.8M additional people impacted) 
• Systems level impact: Targeted outreach to projects that did systems level impact (evidence of 6.4M additional people impacted) 
• Missing Impact: Recently closed, mostly direct implementation projects that reported high reach numbers, but no impact (evidence of 1.5M additional people impacted) 

What we learned 

Direct Programming

From reviews of evaluations for direct programming projects (Humanitarian, Missing Impact, and some Covid-19 projects), we learned to focus efforts upstream: 

Downstream, there is only so much existing evidence available to “recover” impact. We saw diminishing amounts of impact recovered, with a large number from the Humanitarian project review and less from the Missing Impact review later on. Eventually, there was a limit to how much evidence was collected and left unreported by projects.  

When impact was not evidenced, it was largely due to upstream leaks: 

• a lack of funding to complete impact evaluations,  
• recording measures for donor indicators that did not match CARE’s impact indicators,  
• recording mismatched baseline and endline measures 

We can prevent these upstream leaks by incorporating appropriate impact evaluation into proposals, even when donors don’t require it. This not only requires processes (such as budgets with MEL funding, measurements that align with both donor and organizational indicators), but also buy-in from project teams and MEL staff to support robust impact measurement.  

Systems Level Work

From the reviews of projects doing systems level work (Systems Level and some Covid-19 projects), we learned to focus efforts downstream: 

Much of the impact recovered was recorded from projects after they ended, when the results of systems level work (i.e. advocacy and social norms change) could be observed. An example of a past successful recovery of systems level work that did not initially report impact is shown in this 2020 outcome harvesting systems-level evaluation in Ethiopia. We need to continue to invest in targeted outreach and technical advisors who know the portfolios of projects, to record systems level impact when it can be observed after a project closes. 

Scale Up

There are resources to facilitate efforts in different parts of the impact stream. For example, AIIR tools are specifically designed to help record impact for systems level work, and existing data recording tools like KoboToolbox can be used to proactively capture impact measurements. As CARE and other INGOs continue to work towards targets like Vision 2030, the focus should shift from only reviewing projects with missing impact to scaling up existing resources where appropriate.