Supporting Local Partners with MEL: Measure What Matters Most
As new Local Works partners launch their activities focusing on strengthening community participation in newly consolidated communities in Armenia, they connect with community members, strategically plan their work, and decide how they will meaningfully measure their progress through their Monitoring, Evaluation, and Learning (MEL) planning. Taking the time to plan upfront for meaningful MEL can set both partners and USAID up for successful learning throughout their activities. However, USAID-specific MEL terms and processes often pose challenges for many of USAID’s partners around the world.
This was the case in Armenia, where three out of the four implementing Local Works program partners had no previous experience working with USAID, nor its MEL processes. According to Anahit Gevorgyan from the Women’s Support Council, “[Initially] the MEL concept first looked alien and quite complicated.”
The Mission, driven to right-size MEL requirements for new partners, adopted a participatory approach to support partners in developing simplified MEL plans, identifying and designing custom indicators, and strengthening partners’ capacity to measure what matters most to them. The Mission’s MEL Specialist played an active role in facilitating by:
- Sharing a folder of MEL resources with partners, including details on the Missions’ Strategy; templates for a Performance Indicator Summary Table and an Activity MEL Plan Template; performance indicator reference sheets for key indicators; and helpful links to Learning Lab resources like the How-To Note for Activity MEL Plans.
- Walking through resources and requirements with each partner on a virtual call, where they spent significant time on the MEL Plan Template and its components.
- Encouraging partners to share draft versions of their MEL Plans prior to the official submissions and ask questions as they arose.
Gevorgyan noted, “when the details [of the MEL plan] were presented, we [better understood] our potential to work on it.”
The Challenge
As the four partners began sharing their early draft MEL Plans with the Mission, several common challenges emerged. To begin, they proposed a lot of indicators–more than were required in the award, and many did not align well with the activities’ key objectives. Additionally, indicators focused more on the outputs of the activities, rather than outcomes. While a few output indicators may be useful for program management, these did not meaningfully reflect the community-level priorities and change that activities seek to achieve. The partners also struggled identifying and capturing data that would reflect outcome-level change.
Naira Arakelyan, of the Armavir Development Center, reflected, “The [drafting] process was very interesting, requiring in-depth knowledge of program management and MEL competencies to make the right choice of meaningful indicators and further develop them. It was a learning process. USAID’s support and provided resources were particularly instrumental for developing the MEL plans.”
Collaborating and Refining: Virtual Accompaniment
According to USAID/Armenia’s MEL Specialist, Arpine Porsughyan, “We realized in the process that we need a gradual approach and multi-stage support. Only one training session on the MEL Plan template was not sufficient, though it [was] a good start. Being adaptable was key as we continued to identify the challenges our partners were facing in the process of MEL Plan development.”
Through discussions with partners, the Mission identified several priority areas to right-size MEL plans, and to strengthen partner MEL capacity moving forward. The Mission MEL Specialist and Agreement Officer’s Representatives (AORs) worked with local partners virtually to refine the Activity MEL plans. This included:
1. Shifting away from a heavy focus on output-level measures and towards tracking outcome-level change
While output level indicators are helpful to track what is immediately produced because of the activity (e.g., participants trained, or number of community forums held), the Mission encouraged partners to focus more on relevant outcomes, which are typically a result of more than one intervention or output, and usually require a longer time period to be observed (e.g., improved capacity of local governments to engage citizens). They worked with partners to reduce output indicators to the “need-to-knows,” which helped limit the data collection burden, and facilitated discussions around meaningful outcome measures that might be included over the life of the activity.
2. Prioritizing custom indicators that matter most to local communities
With Local Works focusing on measuring what matters to local constituents, not only what matters to USAID, the Mission worked with each partner to co-create new indicators and explore existing measures of success that are directly relevant and useful to local constituents. USAID/Armenia also shared the custom indicators of each project with all partners for cross learning and consistency. For example, one custom indicator measures the number of participatory mechanisms for community decision making–such as public hearings, town halls, and collecting citizen feedback– introduced in the community as a result of the activity.
3. Identifying meaningful data collection methods that engage local communities in the learning process
USAID/Armenia knew it was important that the data collected for indicators help local partners track their progress towards their own purposes and objectives, rather than serving only as a reporting exercise for USAID. They collaborated virtually with partners to determine the types of data collection efforts that would align with their planned activities. The goal was to ensure that data collection efforts were feasible, high quality, and fit-for-purpose. As a result, partners are planning to use various approaches for local communities to share how useful they are finding activities and recommendations for improvement, ranging from suggestion boxes, surveys, phone calls and text messages, focus group discussions, one-on-one interviews, community meetings, and notice boards.
Diving Into the Weeds: Workshopping Performance Indicator Reference Sheets (PIRS)
The Mission recognized that USAID’s specific requirements for a Performance Indicator Reference Sheet (“PIRS”) – are very new to local partners, and frequently confusing. “PIRSs [were] the biggest pain-points for most of the partners,” Porsughyan recounted. The Mission determined that a more collaborative approach would help. To ensure indicators were rigorous and meaningful to local communities, USAID/Armenia convened a workshop with all partners, AORs, and the Mission MEL Specialist. The workshop focused on understanding data quality standards for performance indicators and the role of PIRSs in ensuring rigor of the data. First, they worked together on practical exercises to help partners improve their understanding of data collection and reporting for select standard indicators. Once the partners were more comfortable with the concept of PIRSs, the bulk of the workshop agenda created space for partners to develop PIRS for their own custom indicators through small breakout groups, during which partners with overlapping approaches were encouraged to collaborate.
The workshop was a learning opportunity: while all four Local Works activities are guided by the overall goal of citizen engagement in local planning and decision-making processes, each partner will take different approaches to achieve it based on their own experience and deep understanding of the local context. The Mission is excited to see how partners’ approaches to monitoring their efforts generates learning about what works in locally led development and why.
Hasmik Azibekyan of the Community Center for Development noted that the workshop “was very useful in terms of getting to know USAID staff and get their insights, as well as discuss possible improvements in outreach and communication, programmatic reporting and creating opportunities for the Local Works partners to network and find new opportunities for cooperation and learning.”
As the set of virtual consultations and the workshop drew to a close, the new partners finalized their Activity MEL Plans, indicators, and data collection plans. Susanna Shahnazaryan of the Goris Press Club reflected, “The MEL process was very useful, as we jointly came up with a set of reasonable and applicable indicators, which we’re now in the process of refining.” The learning will not stop here: each partner received a budget line for assessments and learning, which they will collaboratively plan during their first year of implementation. The Mission and partners will integrate these details into Activity MEL plans during year one of implementation. Looking ahead, USAID/Armenia hopes that through ongoing partnership, each of the four organizations will strengthen their capacities further, from monitoring, to learning – and beyond!
Planning to co-create MEL Plans and indicators with local partners? Here are some resources to help you dive into the process:
- Activity MEL Plan Template for Local Partners
- Example agenda for a collaborative PIRS workshop and Example custom PIRS