Project

Reflexive Monitoring in Action

Reflexive Monitoring in Action (RMA) is an integrated methodology to encourage learning within multi-actor groups or networks as well as institutional change in order to deal with complex problems. Key is to trigger recurrent collective reflection on the results of actions undertaken in the light of systemic barriers as well as opportunities. These system change initiatives will thus be able to change their practices, relations and rules and contribute to the long term ambition of sustainability.

Over the past decade, the methodology has been applied in many initiatives for system change and programmes in a diversity of domains, such as sustainable agriculture, health, international development, the knowledge infrastructure, and learning for sustainability. It has developed into a coherent body of basic starting points, principles and intervention strategies. On this site, you find the basic guide, additional tools, practical results and the main scientific articles about RMA.

Guide and tools

The guide on Reflexive Monitoring in Action offers principles, practical guidelines as well as theory and tools. Additional tools, developed more recently, are provided separately.

RMA Guide
English
Nederlands
Portugués
Deutsch (adapted)

Learning Mirror New tool RMA, supplement to the RMA guide.

Examples

Primary Innovation
New Zealand 2012-2017 by Agresearch, New Zealand. Website with links to guide, toolbox and general information.
Connecting Nature
Urban by Nature, EU Horizon 2020 Innovation Action Programme. Guide for Reflexive Monitoring. Framework overall project. Introduction RMA on YouTube (ignore the subtitling!)
Nordwin and Wellant schools
Output RMA (in Dutch).
Education for Sustainable Development 2030, Germany.
Website on monitoring.
Public Innovation Lab of the Uruguayan government
Article about the evaluation tool.
Leadership in healthcare institutes
Report of the research based on RMA (in Dutch).
Natuurpact (2014-2027)
Evaluation for BPL Netherlands Environmental Assessment Agency by Athena Institute, VU, Amsterdam. Output of RMA in Dutch; Review of the RMA evaluation in English; and its summary in Dutch.
Natal care. Elaboration of Dynamic Learning Agenda (Dutch)
Energy Cooperative Lochem
Report learning history smart grid project
'Voedselfamilies' (Foodfamilies), Provincie Zuid-Holland
Website with links to the monitoring report and more (in Dutch)

Explanation

Short introduction to the masterclass Reflexive Monitoring, by PJ Beers, in Dutch

RMA as part of a co-innovation approach by Denise Bewsell. 2020
The value of RMA, short interview Barbara van Mierlo, 2019. In Dutch
Interview Barbara van Mierlo at CDI Conference 'M&E for Responsible Innovation', 2015

Shared presentation slides on RMA by Marlen Arkesteijn for HumidTropics. 2014

Reflexive monitoring in action from ILRI

Scientific articles

Key articles

Van Mierlo, B., Beers, P. J., & Hoes, A. C. (2020). Inclusion in responsible innovation: revisiting the desirability of opening up. Journal of Responsible Innovation, 7(3), 361-383.

We investigate opening up, a crucial aim of responsible innovation, in the situation of companies initiating sector-wide change in order to take societal responsibility. Two case studies in agriculture were conducted, using a framing perspective that enlightens how issues are (re-)defined and acquire meaning in conversations. For both industry-led innovation initiatives, this showed when and how the initiatives’ issue frames opened up and closed down. The results suggest that the inclusion of actors is not the panacea for opening up the innovation processes, given that responsiveness seemed more relevant. Furthermore, we confirm that closing down may occur simultaneously with opening up and as such is an inherent part of responsible innovation processes. Hence, in addition to the advocated opening up, the question of how to balance it with closing down in order to arrive at collaborative action deserves full attention.

Klaassen, P., Verwoerd, L., Kupper, F., & Regeer, B. (2020). Chapter 9 Reflexive monitoring in action as a methodology for learning and enacting Responsible Research and Innovation. Assessment of Responsible Innovation: Methods and Practices, 222 - 243.

In parallel to the conceptual work undertaken on and in RRI, increasing attempts were also made to practise RRI. Much of this work builds on older traditions of thought and action, which can be found in disciplines or fields such as (constructive) technology assessment, (applied) ethics, science communication or gender studies, and the conceptual work RRI borrows from, adds to or relates to this. Indeed, in our view one of the merits of RRI is precisely located in the fact that it brings together valuable conceptual and methodological resources that previously were dispersed across various communities. With this merit, however, there is also an associated risk – namely, that with the ensuing complexity of RRI it becomes difficult to monitor or assess how people, institutions, companies, fields or the R&I system as a whole are influenced by RRI. In this chapter, we argue that reflexive monitoring in action (RMA), an interactive and action-oriented methodology for monitoring complex projects that aim to contribute to sustainable system innovation in the context of ‘wicked problems’, could well be used to monitor and assess RRI. To make our case, we build both on our experiences with formulating and using criteria for assessing RRI gained in the RRI Tools project, funded by the EC, as well as on our experiences with practising RMA during a research project in the context of a Dutch nature policy programme, Natuurpact.

Verwoerd, L., Klaassen, P., & Regeer, B. J. (2020). How to normalize reflexive evaluation? Navigating between legitimacy and integrity. Evaluation, 1356389020969721.

While hybrid evaluation practices are increasingly common, many Western countries continue to favor modernist evaluation logics focused on performance management—hampering the normalization of reflexive logics revolving around system change. We use Normalization Process Theory to analyze the work evaluators from a policy assessment agency undertook to accomplish the alignment between the prevailing and proposed logics guiding evaluation practice, while implementing a reflexive evaluation approach. Ad hoc alignment strategies and insufficient investment in mutual sense-making regarding reflexive evaluation hindered normalization. We conclude that alignment requires developing reflexive evaluation legitimacy in the context of application and guarding reflexive evaluation integrity, while contextual structures and cultures and reflexive evaluation components are being negotiated. Elasticity (of contextual structures and cultures) and plasticity (of reflexive evaluation components) are introduced as helpful concepts to further understand how reflexive evaluation practices can become normalized. We reflect on the use of Normalization Process Theory for studying the normalization of reflexive evaluation.

Beers, P. J., Turner, J. A., Rijswijk, K., Williams, T., Barnard, T., & Beechener, S. (2019). Learning or evaluating? Towards a negotiation-of-meaning approach to learning in transition go vernance. Technological Forecasting and Social Change, 145, 229-239.

Welfare state transitions require not only experimentation and social learning to explore the potential of innovations for sustainable development but also monitoring to draw vital lessons from experiments for application now and in the future. However, current approaches to learning are mostly concerned with what happens within the boundaries of innovation experiments, whereas transitions in the making feature important boundary-crossing processes between initiatives and their environment. In this paper, we explore the learning approach of negotiation of meaning as a basis for transition monitoring in a New Zealand programme for agricultural innovation called ‘Primary Innovation’. We used a mix of specifically crafted value add documents and workshops to foster learning processes between innovation projects and incumbents. Results suggest that a negotiation-of-meaning perspective offers an important complement to our understanding of learning processes in transition contexts and how to facilitate them.

van Mierlo, B. & P.J. Beers. (2018). Understanding and governing learning in sustainability transitions: A review. Environmental innovation and societal transitions. In press. https://doi.org/10.1016/j.eist.2018.08.002

Many transitions scholars underscore the importance of learning in sustainability transitions, but the associated learning processes have hardly been conceptualised. The diverse, well-established research fields related to learning are broadly ignored or loosely applied. In this paper, we systematically explore four interesting learning traditions in terms of their value for gaining an in-depth understanding of learning in sustainability transitions and their relevance for fostering learning, by connecting them to key features of transitions. The selected learning traditions from different disciplinary backgrounds provide valuable insights. None of them sufficiently addresses the complexity of transitions. They include, however, a diversity of relevant learning contexts. We conclude that they have value for investigating new areas such as learning in socio-technological regimes and in later phases of a transition, while enlightening forms of learning that have not yet been fully recognised in transition studies, such as superficial learning, unlearning, and learning to resist change.

Beers, P. J., & van Mierlo, B. (2017). Reflexivity and Learning in System Innovation Processes. Sociologia Ruralis, 57(3), 415-436.

In‐depth insight into the relationship between reflexivity, learning and reflection is needed to enrich governance approaches for persistent sustainability issues. However, current conceptualisations of reflexivity seem limited for the understanding of system innovation processes – reflexivity remains highly abstract and often gets conflated with reflection, and, by extension, learning. In this article, we critique the concepts reflexivity and learning and then present an exploratory case‐study of greenhouse vegetable production in the Netherlands to examine their relations. Results suggest a shifting compatibility between initiative and context – reflexivity alignment: the extent to which an innovation initiative shares an orientation towards structural change with its institutional setting. Furthermore, learning sometimes appeared to increase reflexivity, as often assumed, but we found evidence of the opposite as well – reflexivity changes preceding learning. Synthesising the results, we posit three archetypal modes that describe the relation between learning in and reflexivity of a system innovation initiative.

Beers, P. J., Mierlo, B., & Hoes, A. C. (2016). Toward an integrative perspective on social learning in system innovation initiatives. Ecology and Society, 21(1): 33.

Sustainability transitions go hand in hand with learning. Theories in the realm of sustainability sciences mostly concentrate on diversity and learning outcomes, whereas theories from the educational sciences mostly focus on learning as an interactive process. In this contribution, we aim to benefit from an integration of these perspectives in order to better understand how different interaction patterns contribute to learning. We studied STAP, an innovation initiative of Dutch greenhouse growers. The Dutch greenhouse sector is predominantly focused on production and efficiency, which causes problems for its future viability. STAP aimed to make the sector more market-oriented while at the same time increasing its societal acceptability (societally responsible innovation). To that end, STAP focused on the development of integrated value chains (primary production, sales, trade) that can contribute to a transition towards a societally sensitive greenhouse sector. As action researchers, we collected extensive transcripts of meetings, interviews, and various other documents. We used an open coding strategy to identify different patterns of interaction and the learning outcomes produced by the initiative. We then linked the interaction patterns to the outcomes. Analysis suggests that seemingly negative attack-and-defend patterns of interaction certainly can result in substantial learning results, while seemingly positive synthetic interaction patterns, where participants strive to build on each other, can result in rather bland interaction without substantial outcomes. The results offer an empirical basis to our approach of linking learning interactions to learning outcomes, and it suggests that learning for sustainability can be enhanced by focusing on interaction patterns.

Regeer, B. J., de Wildt-Liesveld, R., van Mierlo, B., & Bunders, J. F. (2016). Exploring ways to reconcile accountability and learning in the evaluation of niche experiments. Evaluation, 22(1), 6-28.

While evaluation is seen as a mechanism for both accountability and learning, it is not self-evident that the evaluation of niche experiments focuses on both accountability and learning at the same time. Tensions exist between the accountability-oriented needs of funders and the learning needs of managers of niche experiments. This article explores the differences in needs and expectations of funders and managers in terms of upwards, downwards and internal accountability. The article shows that as the multi-stakeholder contexts in which niche experiments take place give rise to various requirements, tensions in evaluation are essentially a specific manifestation of tensions between niche experiments and their multiple contexts. Based on our findings, an adjusted accountability framework is proposed, including several strategies that can reconcile a learning approach with accountability needs in niche experiments aiming to change current practices in a more sustainable direction.

Arkesteijn, M., B. van Mierlo and C. Leeuwis (2015). The need for reflexive evaluation approaches in development cooperation. Evaluation 21(1): 99-115.

Within development cooperation, development issues are increasingly recognized as complex problems requiring new paths towards solving them. In addition to the commonly used two dimensions of complex problems (uncertainty and disagreement), we introduce a third dimension: systemic stability; that is, stability provided by rules, relations and complementary technology. This article reflects on how development evaluation methodologies and especially those introducing a complexity perspective address these three dimensions. Inferring that this third dimension deserves more attention, we explore the characteristics of reflexive evaluation approaches that challenge systemic stability and support processes of learning and institutional change. We conclude that reflexive evaluation approaches may well complement current system approaches in development evaluation practice.

Veen, S. C. V., R. D. Wildt–Liesveld, J. F. Bunders and B. J. Regeer (2014). Supporting reflective practices in social change processes with the dynamic learning agenda: an example of learning about the process towards disability inclusive development. International Journal of Learning and Change, 7(3-4): 211-233.

Change processes are increasingly seen as the solution to entrenched (social) problems. However, change is difficult to realise while dealing with multiple actors, values, and approaches. (Inter)organisational learning is seen as a way to facilitate reflective practices in social change that support emergent changes, vicarious learning, and dialogues in a participatory manner. The inclusion of persons with disability in development cooperation requires a social change process to transform current attitudes, practices, and organisational cultures. This paper aims to show how the dynamic learning agenda (DLA) integrates learning with social change by analysing and reflecting on the application of the DLA in the case of facilitating learning on the process towards disability inclusive development. In this paper, we will show the theoretical foundations, purpose, practical implementation and added value of the DLA. We will reflect on the application of the DLA and we will discuss how the DLA supports the integration of learning in social change.

van Mierlo, B., M. Arkesteijn and C. Leeuwis (2010). Enhancing the reflexivity of system innovation projects with system analyses. American Journal of Evaluation 31(2): 143-161.

Networks aiming for fundamental changes bring together a variety of actors who are part and parcel of a problematic context. These system innovation projects need to be accompanied by a monitoring and evaluation approach that supports and maintains reflexivity to be able to deal with uncertainties and conflicts while challenging current practices and related institutions. This article reports on experiences with reflexive process monitoring (RPM)—an approach that has been applied in several networks in the Dutch agricultural sector, which strive for sustainable development. Particular attention is paid to conducting system analyses—a core element of the methodology. The first results show that system analyses indeed have the potential to enhance reflexivity if carried out collectively. However, regular patterns of thinking and acting within projects interfere in subtle ways with the new knowledge generated and limit the transformation of the reflexive feedback and insights into action.

Network activities

Future and past network activities dedicated to share experiences and insights with people interested, to foster competence building among reflexive monitors and to develop RMA further.

Future network activities
In May 2024 we will organise a meeting for the Community of Practice of Reflexive Monitors. It is on invitation.

Past network activities
September 1, we had a lively dialogue with practitioners of Reflexive Monitoring (researchers as well as others) and people interested at the International Sustainability Transitions Conference.

It was the first activity of the recently established CoP RMA. We discussed what it meant to operate at the boundaries of a system change initiative and its institutional setting and how practitioners navigate both in- and external resistance. We learned about the need to involve middle management, how to start subtly if conditions for making room for change are not yet in place, how to relate to commissioners and much more. Thanks to my co-chair TJ Schuitmaker-Warnaar, the audience and the panel members Corine Quarles van Ufford, Carolien de Vries, Eva Kunseler, PJ Beers, Barbara Regeer, Anne Loeber, Kris Kok and Lieselot Vandenbussche.

Seminar Reflexive monitoring for transformative change, 9 April 2014.
Theatre debate Sustainable Agriculture Innovation, 17 April 2015

Trainings

Future and past trainings for competence building. We also provide tailored in-company training and coaching.

Future trainings
The Masterclass Reflexive Monitoring of 2023 (in Dutch) is fully booked. A new edition is expected in 2024. If you are interested, register for News on RMA (2-3 times a year).

Past trainings
Module in UvA Academy Masterclass Policy Research - Dutch, 2021 - 2023

In-company training Municipality Heerhugowaard (2020), Freie Universität Berlin (2020)

Masterclass Advanced Course Transitiemonitoring, 14 januari en 4 februari 2016, for the VNG (Association of Dutch municipalities), Utrecht

Contact person dr. BC (Barbara) van Mierlo

Contact

Project information