“Multi-level Mixed Evaluation Method” as a case study of innovative evaluation approaches

In one of its most important discussion papers on innovation, UNDP presents innovation in M&E as the utilization of a new idea, method, approach or instrument / device, that meets at least two of the following criteria: a) it brings significant impact to the evaluation practice, b) it constitutes a catalytic factor of change meaning that it facilitates paradigmatic rethinking of the evaluation process or c) it is concrete as it is defined by a new idea, method, approach or instrument that has been already used / tested and has proved it’s added value (Discussion Paper: Innovation in Monitoring and Evaluation, UNDP, 2013). In this context, the aim of the event is to explore and bring to the attention of researchers, public administrators/evaluations commissions and practitioners one of the innovative approaches used in recent years in evaluation practice, namely Multi-level Mixed Evaluation Method, but for which the specialized literature and thus, the theoretical conceptualization lacks of a common, unified perspective (Multilevel Mixed Methods Research Designs: Advancing a Refined Definition, Headley and Plano Clark, 2020). While mixed methods design of evaluations of public interventions is a general practice nowadays in the field of evaluation, its utilization on a multi-level evaluation design is less common, due to its complexity. The method, that brings together the two complementary approaches, often used separately – mixed methods design and multi-level design, is a complex, very technical and innovative approach that brings added value to the outcomes of an evaluation by aiming to capture the complexity of socio-economic systems and their dynamic. According to the UNDP, Multi-level Mixed Evaluation Method is an approach that includes the deliberate, massive and creative use of mixed (quantitative and qualitative) methods on multiple levels for complex evaluations, particularly for service delivery systems (UNDP, 2013).

The purpose of the event is to explore the methodology, utilization and limitations of this innovative method based on the case study of the Multi-Country Programme Evaluation ECA Region – Republic of Moldova, Bulgaria, Ukraine and Kyrgyzstan, commissioned by UNICEF ECARO and conducted by QURES Quality Research and Support, which took place in the period January 2021 – January 2022. The evaluation employed various quantitative and qualitative data analysis methods, which were applied on a multi-layer evaluation design. The multi-level design of the evaluation was based on the following two aspects:
a) the diverse UNICEF change strategies used in each addressed Country Programme, from using advocacy to win support for the cause of children from decision-makers to fostering innovation in programming by modeling interventions aiming to improve service delivery and engage communities, citizens and civil society organizations in public decision-making;
b) the two-folded purpose of the evaluation, namely the aim of assessing the relevance, coherence and impact achieved throughout the implementation of the 4 addressed country programmes (CPs) and the aim of identifying common lessons learned, transferable findings and conclusions that can support the programming process conducted by UNICEF at regional level.

The study methodology included among others the following two innovative methods: Process Tracing (PT) and Social Network Analysis (SNA), which will be presented within the event. In this regard, the discussions will be focused on the specificities of these methods, the difficulties encountered when applying them and the extent to which these methods brought added value to the evaluation. Thus, it is important to emphasize that the general approach of the event will not be a theoretical one, but a more practical one, being based on examples and aiming at bringing into discussion a comparative view of innovative methods used, the added value brought for the purpose of the evaluation and their limitations. The discussions will also highlight difficulties encountered when applying together these methods within a multi-level evaluation design and most important, the lessons learned in the context of the aforementioned multi-country evaluation.

Moreover, the session will bring together the supply and demand side of evaluation and will facilitate exchange of first hand experiences among the participants. In the first part, the discussion will be organized by different type of innovative evaluation methods used within the multi-level mixed evaluation approach. In the second part, the discussion will be focused on the needs of the potential users of the evaluation / demand side and their perspective regarding the extent to which these methods could facilitate the utilization of the evaluation results, due to their capacity to better capture the complexity of socio-economic interventions, that take place in a multi-layer system and a dynamic context. In the end, one of the overarching aims of such reflections, as intended to be facilitated through this event, should be to identify how the evaluation process can better meet the needs and expectations of its main intended users.