Reflections on a USAID Development Journey #4: The long game - tracking a Theory Of Change and performance beyond our funding

Introduction to these informal notes

On Halloween 2022, I retired from working with USAID after almost 40 years, as a technical officer, a Project Design Officer and supporting the Agency in what was the former Bureau of Policy and Program Coordination (PPC) and then the Bureau for Policy, Planning, and Learning (PPL).  I also had a lot of fun working on renewable energy programs as well as Natural Resource Management (NRM) policy programs throughout Africa, as well as part of the reengineering team developing the Agency’s programming guidance and co-designed the Results Framework.  I also had a  lot  of work on training, knowledge management and a wide range of other things.  

One of my most fun jobs over the last several years has been working with a number of the PPL Communities of Practice (COPs), including the Collaborating, Learning, and Adapting COP and the Program Cycle Implementation COP, which are internal knowledge sharing spaces for USAID staff.  I decided to leave by summarizing some of my thoughts over the…decades, through a series of posts. Here are the posts, with just some irrelevant timing logistics deleted and grammar and typos corrected, as well as reducing the level of USAID-centric situations and acronyms. 

Please remember that they are informal posts, and represent my personal perspective, and do not represent USAID policy.

Reflections on a USAID Development Journey #4: The long game - tracking a Theory Of Change and performance beyond our funding 

Here goes a longish email. I've raised this a couple of times in the past but I thought I'd mention it again, to see what conversation it generates.  

The timeline and life cycle of a development problem/challenge: In many situations, a development problem (or opportunity) takes more than five years to be resolved.  Most capital projects (bridges, power stations, for instance) usually take from 12 to 20 years before the switch can be thrown. In terms of agricultural research, starting a new program can take decades to see impact, especially if the country needs to develop a cadre of researchers, etc. And as for protecting biodiversity…?  This isn't to say that there aren't problems that can be addressed and "solved" in five years, but there are many with a life cycle far longer than five years.  Or in other words, while USAID often programs and finances in five-year chunks, the world isn't necessarily so accommodating.

Add to that, we often are focused on OUR investment and OUR role. We often do not fully grasp that in many situations, we are part of the solution (and sometimes only a bit player) and the need to track the impact of our money tends to lead to understating the role of others. Most importantly, we tend to define the problem and its resolution as neatly fitting the five-year constraint of our planning, budgeting and procurement processes.  Put another way, we often let our business process "tail" wag the development "dog". We tend to define our interventions to fit the limitations of how we plan, fund and procure, sometimes forcing the problem to fit us, rather than the other way around.

Impact on M, E and L.  OK, so what does this mean for M, E and L?  Over the years, I've noticed a conflation between monitoring results and monitoring-funded interventions. And a reluctance to keep monitoring after our money is gone.  Often, in fact, missions will stop funding M&E once our investments stop - it's simply hard to justify. But sometimes, by defining results around the timeline of our funding leads us to measuring the performance of our interventions, as opposed to the overall results we should actually be caring about.  

And this isn't just an issue for USAID. I remember a terrific forester with FAO who led planning and analytic support for a massive, long-term program supporting agroforestry in 7 countries in the 1970s and 1980s; the overall program was well-thought through and included pretty massive investments. What was remarkable was his funding of a review fully 15 years AFTER the FAO money stopped. Why? Because in that subject, all the FAO really had been supporting with their money were key enabling conditions and initiating foundational work; developmentally what was important was what happened well after the FAO stopped. For example, the FAO was often funding nurseries and tree seedling plantings, but not the final harvesting a decade or so later. 

What did he find? A wide range of results. In some instances, the results pretty much met the Theory Of Change (TOC) built into the FAO's initial program. In a couple of cases though, the overall results in terms of trees harvested far exceeded their expected targets, but the beneficiary was not as intended (for example, the program was wildly embraced by the middle class in India, not the targeted poor small farmer), and in another country, the program was so successful that it led to new problems that no one could have anticipated.

How then do we manage, monitor and evaluate the long game?  This is something we used to fund to some extent via the long-departed CDIE team in PPC (PPL's predecessor) and certain sectors have done this from time to time across several programs, but I don't know of missions doing this on their own.  Not that they don't care, but it's so hard to justify the expenditure after our money has been spent. I do remember, as I outlined in an earlier thread, that USAID/Madagascar was part of a 20-year, multi-donor effort supporting biodiversity which was extremely successful and that did indeed look at the long game, but ironically, the USAID staff coming and going in much shorter increments (every 4 years at best) had a hard time remembering that any five-year funding and design increment was part of a much longer plan. (Sort of like the Easter Island heads; I know these are here for a reason but it was before my time...). More on this topic later.

To be fair, our "shareholders" (be it Congress or the American public) aren't looking for long-term systemic change; usually, they want to see that their investment pays off, now. I don't blame them. But given that, is there a way a mission can look far enough back, and forward, to capture the long game?