Designing our Evaluation Process: Meeting #1 SUMMARY

Creating the Future is stepping into a year-long process to document what we have learned from the demonstration projects at the heart of our mission.

The first step in that effort is to determine what the whole process will entail. That design phase will answer questions like…

  • Who are we evaluating for?
  • What do those individuals and groups need our evaluation effort to make possible for them?
  • What projects shall we evaluate? All 30 of them? If not, which ones?
  • What questions will our evaluation answer?
  • What will be our approach to answering those questions?

Several key features have been at the heart of all our work at Creating the Future, including our demonstration projects. Those features will be at the heart of the evaluation process as well.

1) Our mission is to demonstrate the power of the question-based Catalytic Thinking framework, and to make that framework a go-to for social justice advocates and changemakers around the world.

2) Our timeframe: Our mission has a finite timeframe. Rather than a program that will live on forever, we see our mission as a project to be accomplished within 10 years. That timeframe is coming to an end in the next few years. To complete our work, we are documenting what we’ve learned, using that information to complete a library of both theory and case studies. At that time, we will connect with folks who are already teaching in universities, nonprofit resource centers, and other centers of learning, for them to carry forward the work of sharing the powerful questions of Catalytic Thinking.

3) Our approach: All our demonstration projects, are rooted in the questions of Catalytic Thinking. This evaluation process will be rooted in those questions as well.

4) Open and participatory: All our strategy conversations at Creating the Future, including all board meetings and fund development strategy meetings, are open for anyone in our community to participate. This has been our modus operandi since 2013, and will apply to the evaluation effort as well.

The meeting schedule for the design phase of this project can be found at this link.

Meeting #1: Summary
You can watch or listen to the recording of our first meeting at this link.

During our Evaluation Process Design Team’s first meeting, we took some time to get to know each other. So many projects begin by bringing a new team together, and, supposedly in the interest of “valuing people’s time,” they dive right into the work.

Unfortunately, without taking time to get to know who is in the room and what has led them to be part of the work, it is difficult to build the kind of trust and camaraderie that helps make the work go smoothly. That is actually the opposite of valuing people’s time! As you watch the recording, you will therefore see that we took the first 30 minutes to connect as humans.

From there, we dove in to Question #1 of the Catalytic Thinking framework. The following are the answers the group discussed.

Who will be affected by the evaluation process we design? Whose lives will be touched by our evaluation effort?

As we answered this question, it became clear that there are two processes going on simultaneously and overlapping.

1) Evaluating Creating the Future’s demonstration projects.

2) Demonstrating Catalytic Thinking as an approach to evaluation, including doing so as a participatory process.

That led to the following categories of people who will be affected by the process we design, and by the end result of the evaluation – case studies in a resource library.

People who are interested in evaluation / who perform evaluation in their work

Teachers. Teachers are asked to evaluate in the craziest forms and fashions continually. We’re not numbers. We’re human beings and there have to be new ways to evaluate human beings who all learn differently, all show up differently, are on that continuum in different places all the time.

Employee evaluations

Nonprofits as intermediaries for the people in communities who receive the direct benefit of services.

Students and teachers.

University students who work for nonprofits. Graduate and undergraduate.

Workplaces for students who graduate and/or study Catalytic Thinking in their classes.

People practicing in the evaluation space as at least some part of their day to day work, who are wondering about how to do more participatory and human centered forms of evaluation

“Accidental evaluators.” Someone who does not see themselves as a traditional evaluator and maybe somebody who has the attitude of “I don’t touch evaluation. I don’t want to be any part of it.” Maybe they’ll read the case studies and say, “I do evaluate. This is what I do.” And they can then see themselves as part of this work, as opposed to some of the more restrictive or limited ways that I think evaluation often has been framed historically, or in more mainstream contexts.

 Volunteer board members. Having board members understand and appreciate the evaluation process.

Advisors to boards

Communities of practice. Informal, grassroots networks of people working together on projects.

Small orgs influencing larger orgs: The adage from the 60s, that if the people will lead, the leaders will follow. The same thing happens with small organizations. If enough small organizations are doing something differently, then larger organizations are eventually influenced. Bottom-up systems transformation.

People who are interested in participatory processes (not just evaluation)

People watching these recordings in the future, to see the process we used to design our evaluation process.

Researchers could see the recordings as a pretty rich data source.

University people working in offices of community partnerships or engagement.

Government officials who are required to have a “public participation” process.

Governments seeing what is possible, for example participatory budgeting

Community groups who are trying to engage with people. Churches.

Mutual aid groups and collective giving / giving circles. How are we all making decisions together?

Sponsors of events / activities: Partnerships where you promise some sort of engagement in return. How are you evaluating those numbers? Oftentimes it’s based on exposure to these many new clients or you get to meet people at a gala and things like that. How can that be more participatory?

Small organizations influencing larger organizations.

People who will use the case studies documenting our learnings (the end result of our evaluation efforts)

Changemakers who are facing resistance to taking new approaches, who need resources and people that can inspire them and help them.

Philanthropy / funders. “In the work in our organization, we’ve had to retrain philanthropists in how they look at us.”

People who teach in graduate programs, where the students are already working in the nonprofit sector and are looking to advance their credentials. They could use case as part of their teaching.

The students in those programs, to feed a desire of students who are looking for different ways to approach evaluation and learning and understanding.

Government funders. “So that they stop demanding a ten-page report to know how we’re spending the five dollars they gave us.”

Volunteer board members. Having board members understand and appreciate the evaluation process.

Advisors to boards

Culture: Company culture, educational culture, nonprofit culture, or the culture within a given nonprofit. What’s condoned, what’s promoted, what’s not. To impact that would be hugely beneficial.

Demonstration project participants themselves

Participants will understand the evaluation process differently. Not to just justify the work they’ve done or was it worth the money, but to think about future opportunities to work together, to document value in a way that is more authentic to what they were seeking. To align with their expectations or help serve their needs further.

Evaluation that goes beyond measuring what’s easily measurable.

Remove the power disparity and systemic power issues. “So often we chase the ripples, but we forget about the stone. And I think we have to go back to the origination of things. Where is it coming from? Too often we go into communities and say, ‘What do you want to do?’ And they say, ‘Well, what do you want us to do?’ We’re so used to being told what to do instead of actually taking the time to think about ‘What do we actually want to accomplish ourselves?’ and not what is some outside entity telling us.”

Alignment, psychological safety, coherence, empowerment. Joy. Excitement.

Preventing re-traumatization. What participants would not experience is any level of re-traumatization, whether the initial trauma was caused by the evaluation of the program or something else.

Comments that relate to Conditions for Success

As we continue this process, we will consider conditions for success. These thoughts arose during this initial conversation. We’re saving them here, to be sure to add them back into the conversation when we get to that point.

What will it take for people to feel safe to give feedback?

Am I safe? Is it safe to give feedback? Is it safe to give feedback that is actually representative of my experience? Even when the answer is yes, it is safe, people don’t necessarily feel that it’s safe.

Or we continuously give feedback and nobody listens and so it gets exhausting. so I think, yeah, those are just thoughts coming up for me.

Participatory:

Making decisions FOR people vs. WITH people. People get to make their own rules.

Years ago, we taught an online university class, and there was an issue that came up that we were supposed to decide the outcome. And we instead turned it around and asked the students. It makes such a difference when people make their own rules. We talk about it as participation and community engagement, but how much of it is people making their own rules? This comes back to FEELING empowered.

REFLECTIONS

What is standing out from our conversation today?

  • How many different people and groups are affected by evaluation. I’ve had a little bit of a narrow view, but hearing these perspectives is making me think more broadly.
  • I’m reflecting on how many different perspectives and backgrounds and are in the room here. Folks were bringing up things that I hadn’t thought of and it was pretty regular. And so it was it was nice to just hear from everybody.
  • How many types of people and groups may be impacted by this work, and also the importance to narrow it down as well. So thinking big AND thinking local, at the same time.
  • A phrase that has stayed with me today is “invitation to play.” Not only in our conversation today, but also coming up in the process and also for the future people who are going to be exposed to all of this.
  • I feel like I have a lot of unanswered questions. And so I’m sitting in that discomfort of not having clarity about what we’re doing and why and what exists already. But then I’m reminded that that’s part of the process, and embracing that.
  • The quote that’s going to stick with me is “too often we focus on the ripples and forget about the stone.”
  • The richness of the conversation itself and the cadence. It feels slower than many conversations of which I am a part these days. And I appreciate that. Some of that has to do with just letting the question be and coming back to the question and not just peeling off the layers, but exploring the fabric, both deep and wide.
  • One thing sticking with me was “participatory democracy.” Using those words in our country right now, it would be really something, but in this small group, we just had that. We just demonstrated that. And that’s why this is so much fun. We all have a voice at whatever level we’re coming into this. And everyone is holding space for each other.
  • I’m thinking what impact this project will have on everyone here and then how that might alter folks’ impacts more broadly. I’m seeing a network of ripples.
  • I’m thinking about the relationship between intention and impact. I don’t have any particular conclusions, I just think it is a part of this work, especially when it comes to evaluation and thinking about outcomes. Sometimes we downplay the importance of intent and being intentional. And sometimes we overplay the importance of intent. We often see a gap between intention and impact, but the intention is important. That’s just what’s on my mind right now.
  • I just feel like I’m in the right place at the right time. The word that came to my mind was bridging, that often we spend a lot of time on islands and in silos, and that this changes that dynamic in a real important way.

Next Steps

During Meeting #2, we will consider the next questions in the Catalytic Thinking framework:

  • Question #2: How are all the people we described currently experiencing evaluation? What is important to them about evaluation? What do they feel about evaluation in general? What have they experienced with their own evaluations?
  • Question #3: For each group we’ve defined, what would they want our evaluation process – and the case studies that will be the end result – to make possible for them?

If you’d like to be part of that conversation, you can see more at this link. We hope to see you there!

Leave a Comment