Measuring the Hard-to-measure: Embracing complexity and empowering communities through data

When, as research, monitoring, and evaluation professionals, we are tasked with measuring programs with multiple, non-linear pathways within complex environments, our first instinct may be to run for the hills. This is true especially if we have been trained in and closely adhere to traditional principles of epidemiology and frequentist statistics. In reality, though, most development programs, components, and contexts are complex. Whether we work in fragile settings experiencing rapid, unanticipated consequences of climate change or the vestiges of war, or in stable settings where a new health or social innovation is being tested and rolled out, the complexity of program and setting often obscures the causal pathways between intervention inputs and outcomes. The challenge of measuring what is hard to measure creates the risk that we may miss critical pieces of information that could improve program implementation, effectiveness, and reach. The people meant to be served by health and social programs may be excluded from the larger data narrative when we miss these important pieces of information. Employing only traditional methods that do not holistically capture their reality is also a disservice to the global development community that is striving to achieve the 2030 Sustainable Development Goals.

What can we do?

Researchers and monitoring and evaluation (M&E) practitioners in global health are looking beyond the traditional methods that focus on pre-conceived, static causal pathways. Complexity aware monitoring (CAM) approaches to M&E are designed to navigate the tricky, ever-changing waters of settings in which causal pathways are unclear. A variety of approaches fall under CAM: some use mapping techniques, others are narrative-based, and still others are indicator-based. Many are participatory in nature, relying on program participants and service providers to share their stories and perspectives. This often promotes adaptive learning and management, which are catalysts of program course-correction and strengthening. Gaining the perspective of those directly served and affected illuminates causal pathways and identifies outcomes that may not be easily recognizable to the linear-minded evaluator or program designer.

CAM approaches can be combined with traditional M&E approaches to gain insight into some of the unknowns. Additionally, CAM approaches are adaptable and have been used in a variety of sectors including global health, agriculture, wildlife and fisheries conservation and management, and democracy and governance. 

How can it help?

USAID’s Measure Evaluation project conducted ripple effects mapping (REM) with orphans and other vulnerable youth in Botswana in a multi-pronged HIV, health, education, and economic intervention. The REM groups elicited the experiences and “ripples” that emerged from the youth’s involvement in the program. For example, the REM found that youth who participated in teen clubs (support groups for youth with HIV) felt heard and supported by their peers during club meetings, which helped them accept their HIV status and reduced self-stigma and shame. This, along with knowledge they gained from the teen club facilitators about the benefits of antiretroviral (ARV) medication, led youth who had previously taken the medication inconsistently to start doing so on a daily basis. Some said that taking ARV consistently helped them focus in school. REM helped to uncover the sequence of outcomes resulting from the program that may not have emerged using traditional methods exclusively. 

JSI has also used techniques like pause and reflect in a variety of settings to elicit stakeholder perspective on both the intended and unintended outcomes of projects and initiatives, as well as the pathways for change as part of the Innovations for Maternal, Newborn and Child Health Initiative. These pause and reflect meetings were held as regular check-ins between the monitoring and evaluation team, supported by JSI, and the program team, supported by Concern Worldwide, implementing a social and behavior change program focused on improving maternal and child health outcomes in rural Upper West Ghana. The program, Community Benefits Health, focused on using non-monetary incentives, in addition to social and behavior change communications to foster community support for maternal and child health. The pause and reflect meetings were an opportunity to discuss progress, barriers, and data about satisfaction with the program gleaned from interviews with community members. The pause and reflect meetings also allowed the program implementers to adjust programming to reflect the needs of the community, for example by modifying targets to achieve the incentive based on contextual factors like migration, and modifying the incentive itself (selecting emergency transport instead of a community borehole). 

In June and July 2021, JSI organized a five-session CAM workshop webinar series through the USAID-funded MOMENTUM Knowledge Accelerator. The workshops strengthened the capacity of MOMENTUM field and headquarter staff, USAID staff, and other participants in the use of CAM approaches. 

CAM approaches help us understand the complex nature of the settings where we work. Engaging vulnerable populations in M&E helps ensure that findings accurately reflect what happens in fluid programming environments. CAM approaches also add value to program M&E by bringing the data to those who need it most—stakeholders and beneficiaries. With the data directly in their hands, they are better represented and able to make decisions about programs and policies that affect their lives. 

Written by Mahua Mandal and Emily Stammer