Measuring the Impact of Learning Events and Publications

Describe The Approaches Utilized To Measure / Assess This KM Initiative: 

Making Cents uses various approaches to assess the outcomes and impacts the conferences and publications have on those working to increase and improve economic opportunities for young people. We start with a global consultation that asks members of the global community working on topics the conference covers about the priorities that they would like the conference to address. We also conduct targeted surveys by phone and email to increase the response rate and obtain more detailed information. While Making Cents is designing and conducting the consultation, we invite leaders in the fields the conference addresses to join the conference’s Global Advisory Committee (GAC). GAC members contribute directly to the development of the conference’s learning agenda, and therefore the content of the resulting publication. GAC membership grows after the consultation, which determines the conference’s themes, to ensure the GAC represents the range of stakeholders involved in the fields and contains expertise related to the conference themes. At the conclusion of the conference, Making Cents collects an on-site all-conference evaluation and session evaluations. The all-conference evaluation asks questions on participants’ overall conference experience. The session evaluations are on individual sessions at the conference. Making Cents also conducts a four-month post-conference evaluation to ask those who participated in the conference whether the information, resources, or connections they obtained at the learning event have directly resulted in new partnerships or programmatic changes. In regards to the “State of the Field” publications, Making Cents requires those interested in downloading the publications to share their contact information through a dialogue box. This information helps Making Cents understand the publications’ reach, and enables us to conduct evaluations of the readership. These evaluations inform Making Cents on the conference’s and publication’s outcomes and impacts, as well as the design of the conferences and publications that follow. They show that as a direct result of participating in the conference and/or reading the publication, individuals have: 1) gained new technical capacity; 2) established new partnerships; 3) disseminated or received invitations to submit funding proposals; and/or 4) revamped programs and policies as a result of the new information they obtained.

What Do You Think Are The Main Unanswered Questions Or Challenges Related To This Field Of Work?: 

It would be helpful to have a central location to find other effective models for evaluating learning events and publications, and to learn from others’ experiences in this area. When we conducted initial research on this, I did not find any models that are considered “best practice”. We therefore look forward to learning from others on what has worked well for them, and how others effectively “make the case” as to the importance of investing in knowledge exchange.

What Was The Purpose Or Motivation For Assessing This KM Initiative?: 

Making Cents wants to know whether the information that is shared via the conferences and publications are practical and useful to those who design, implement, monitor, evaluate, and/or fund programs and policies. This is important for understanding whether these learning initiatives are helping to build and strengthen the fields.

What Were The Most Important Lessons Learned About The Assessment Process?: 

Making Cents has learned how important it is to create mechanisms to engage stakeholders in this knowledge exchange initiative right from the beginning and throughout the process in order to build their ownership in it and to ensure we all have a shared vision for it. Since this initiative is truly “by the field and for the field”, Making Cents goes to significant lengths to involve those working on topics the conference and publication cover in the design, implementation, monitoring, and evaluation of it. We have also learned that it is not enough to simply disseminate an electronic survey to evaluate these initiatives, even if your target audience has regular access to the Internet. Competing priorities often prevent a sizable and meaningful online survey response. Instead, we use a multifaceted approach to hear directly from stakeholders, as we utilize online surveys, the phone, email, and increasingly, social networking platforms. Additionally, we have learned the importance of clearly articulating what participants will get out of participating in these assessment processes. While contributing to the “greater good” of building and strengthening the field is generally seen as valuable, participants understand how contributing to the conference’s design enables them to have a say in the type of information they will obtain at the conference and through the publication and the potential partners they will meet. Finally, we recognize the importance of showing stakeholders how we use their input and feedback in the design and implementation of these learning initiatives. We want to ensure our stakeholders see how they are directly involved in the process that aims to contribute to programs and policies achieving greater impact in a sustainable way at scale.

What Would You Do Differently Next Time?: 

If we could start our monitoring and evaluation process again, I would consult with a professionally trained survey designer and M&E expert at the beginning. Asking the right question in the right way will provide you with the answers that can really help inform your programming. We learned a lot after engaging a professional a few years ago and were subsequently able to better demonstrate the “return on investment” conference and publication sponsors received. It also helped us know the real impact these knowledge exchange initiatives were having at the organizational level and at the field level.

What Advice Would You Give To Others Based On Your Experience?: 

I suggest approaching knowledge management initiatives as one would approach other kinds of projects: generate trust and ownership among stakeholders through a demand-driven and participatory design, implementation, monitoring, and evaluation process; integrate M&E into the design of the initiative; create a shared vision and shared expectations; demonstrate the benefits of engaging to all partners; focus on and show results regularly; and aim to catalyze transformational change. Aim to be able to share positive feedback from participants on their experience, which you capture through evaluations, as their words will powerfully demonstrate the value of your knowledge exchange initiative.

Describe The KM Initiative: 

When Making Cents International created the first annual Global Youth Enterprise & Livelihoods Development Conference and post-conference “State of the Field” publication in 2007, we intentionally built a monitoring and evaluation system into the design and implementation process of the learning event and product in order to ensure they were demand-driven, participatory, and impactful. The conference and resulting publication aim to build and strengthen the fields of youth enterprise, employment and livelihoods development; workforce development; and youth-inclusive financial services while supporting programs and policies achieving greater impact within the fields. At a time when these fields are expanding and evolving, these learning events and products provide important platforms for knowledge exchange and partnership building. In just four years, this conference has become the “go to” place for practitioners, funders, policymakers, researchers, and others who design, implement, monitor, evaluate, and fund programs in these fields. The 2010 Conference -- which took place September 14-16 at the Inter-American Development Bank in Washington, DC -- convened more than 400 participants from 63 countries. They shared lessons learned, promising practices and innovative ideas related to the following conference themes:

• Workforce Development
• Youth-Inclusive Financial Services & Financial Capabilities
• Monitoring, Evaluation & Impact Assessment
• Youth Enterprise Development
• Adolescent Girls & Young Women

After each conference, Making Cents synthesizes the guidance and programmatic examples participants shared at the event to develop and disseminate a practical “State of the Field” publication. To date, these publications have been downloaded more than 2,500 times in 146 countries.