Let's talk about measuring impact (even if it’s hard)

By Laura Vidal

Research & Project Specialist,
Latin America

IRIS

If measuring impact and evaluating success is already complex in any project, in the world of narrative change, there’s an extra layer. The idea of measuring what changes because of the work of a specific organization or group is as fascinating as it is difficult.

That's why we gathered last July: to talk impact and MEL (monitoring, evaluation and learning) with our friends from La Sobremesa Anca Matioc, Laura Lehman e Tania Altamirano.

If measuring impact and evaluating success is already complex in any project, in the world of narrative change, there’s an extra layer.

The idea of measuring what changes because of the work of a specific organization or group is as fascinating as it is difficult. That's why we gathered last July: to talk impact and MEL—monitoring, evaluation and learning. 

Can the impact of a change in narrative be measured in numbers? What do clicks, views, retweets, comments tell us? In this world where we seek to change representations, sow ideas about our communities or make excluded people more visible, concrete changes often happen slowly, over time, only visible through subjective, personal, and collective experiences. If we throw ourselves into bean counting, dance to the rhythm of corporate tech algorithms, target virality or nothing: aren’t the numbers just the tree hiding the forest?

To reflect together on this, we called on our friends from La Sobremesa—Anca, Laura, and Tania—to kick us off. Our goal is for this conversation about impact—and how to evaluate it—to be the first of many. Through them, we’ll learn how to keep evaluation strategies central from the inception of the project.

Growing inward

La Sobremesa invited us to think not only about the impact of our work on the communities we work with but also to think about the impact that work has on us and the ways it can strengthen our own ecosystem. 

La Sobremesa highlighted that an important element in the process is aiming to understand the problem we are aiming to tackle; and that’s something that can be done best when working together with—and not only for—those who are facing those challenges every day. In all of this, the keyword is exchange. 

It's about creating a project design that invites us to ask ourselves questions about those involved; and also, paying attention to who is not present or represented in this particular context and ask ourselves why.

Fewer answers and more questions

It would seem that even our own ideas around “impact” are in need of their own narrative change. We were excited – though not necessarily surprised – at the range of ideas that came out of this first conversation. The experiences of those who participated in the gathering gave us an important clue to understanding the key elements in the design and development of narrative change projects. We could say these are projects that aim to change perceptions and even imaginings: measuring such changes is quite the complex task. 

Nearly every project present in this Tertulia benefited from a special kind of intuition, something that proved essential in the development of their methodology...but this intuition would have been impossible without an extensive and well-connected experience with the communities themselves.

We were left in the end with a question, one that often swings on a pendulum between the quantitative and the qualitative: how do wepresent impact that isn’t primarily numeric. The importance of learning to understand that in social change, numbers do not always answer the questions we have, or at least not necessarily. Indicators, by their nature, are imperfect and uncomfortable, and we have to learn to integrate them into more flexible, more fluid observations.

It's about learning where to shine the light and what to do with the sources we have. To make visible the data we can learn from and determine which elements truly give us a full portrait, with all of its light and shadow.

So far, we have no fixed schemes or ready-made answers. Which means that it is possible we may have to create them by ourselves. It is our hope that these conversations become a first step towards that goal, and that this new learning cycle will show us the way.