By
Jay Rothman

October 2003

Overview

Action Evaluation (AE), as we use the term, is an innovative method that uses social and computer technology to define, promote, and assess success in complex social interventions. [1] AE grew out of my work as a theorist and intervener in identity-based conflicts.[2] It was a direct response to recurrent questions that I and other conflict-resolution practitioners, participants, and funders were asking about the efficacy of conflict-resolution interventions, particularly in deep, "intractable" conflicts. We asked with increasing urgency, "Does conflict resolution really work? How can we know? What does 'work' mean, who defines it, and how?" And most important, "How can our search for answers about success increase our chances of achieving it?"

Traditional forms of evaluation stand apart from the projects that they evaluate, and illuminate gaps between initial program goals and actual outcomes. Action Evaluation joins a project by helping participants define and then formatively redefine success, to forge effective action and make success a self-fulfilling prophecy. Too often, the criteria of success are imposed upon a conflict-resolution initiative from the outside, without seeking meaningful and sustained input of the various groups involved in the conflict or intervention. Action Evaluation gathers and organizes input and ownership by those involved, by assisting them to create their own criteria for success. Thus, by defining and seeking success in a continuous, integrative way, AE is both an evaluation and an intervention tool.

Since the early 1990s, Action Evaluation has assisted thousands of participants, funders, and facilitators in nearly 100 conflict-resolution and community-reconciliation initiatives. These projects have become more successful by reaching consensus about what they seek to accomplish, why, and how.

Action Evaluation (AE) fosters passionate participation and ownership. It gathers and organizes essential input to a project, by asking questions about:

• people's goals - What?
• their values and beliefs - Why? -- and
• suggested action strategies - How?

in the context of their identity groups for the current or future initiative. For example:

• In Cincinnati, Ohio, we organized a year-long initiative to resolve conflicts between police officers and other community members. The project included 3,500 participants in eight identity groups, including youth, police, business leaders, and African-American citizens.[3]
• At the Dayton Mediation Center, people shared their ideals for the future of the center in groups of volunteers, staff, and board.[4]
• In York, Pennsylvania, following an incident that reignited a shameful history of racism in the city, civic leaders joined together and formed "YorkCounts" to improve race relations and general services in areas of education, economics, development, and community welfare.
• In these and other efforts, after each group reaches its own consensus about goals, representatives of the various groups gather to define overarching project goals including general principles for practice, core values, and proposed strategies. Next, action teams constituted of participants across all groups are formed to invent strategies and launch initiatives for action.

By asking those most directly involved in a conflict-resolution initiative to collaboratively define their goals, articulate core values, and brainstorm action strategies, the group becomes more coherent and focused. Participants who might otherwise be at odds with one another about the purposes of their joint effort, can effectively walk in step with one another and reflect together on their practice as they engage in it. As such, action evaluation is a form of collaborative social intervention. AE is also a form of conflict intervention, in that it brings to the surface any conflicts between participants' goals and strategies, and helps participants become more engaged and active. In summary, by asking stakeholders to collaboratively define their goals and objectives, Action Evaluation supports a central principle of conflict resolution: any process designed to effectively address a conflict must engender ownership and participation by those most directly affected by the conflict.

Stages in Action Evaluation

As outlined in Box 1, the three stages in Action Evaluation are: establishing baseline, formative monitoring, and summative evaluation. The first stage begins with stakeholders articulating their definitions of success, prior to the launch of the project or intervention. Individual definitions are then woven together into a single platform through a collaborative negotiation process, first within groups and then across them.

Stage One: Establishing the Baseline

Through the use of an online questionnaire, individual participants define what they perceive to be the project's major goals. They articulate what these goals are, why they care about these goals (or why they are important generally), and how they could best be accomplished.

Once the individual goals have been defined, the Action Evaluator determines whether each respondent's goal is unique, shared (between two or more respondents), or contrasting with those of other group members.

After determining the shared definitions within each identity group (in the Dayton Mediation Center, for example, the board, staff, and volunteer mediators) we repeat the same process across all stakeholder groups. Face-to-face feedback sessions assist in this process.

Stage Two: Formative Monitoring

In the baseline stage, stakeholders articulate their goals and motivations in order to arrive at clear and consensual definitions of success. During the formative stage, participants refine their goals and develop strategies for overcoming obstacles. The baseline stage ends with an action plan specifying what needs to be done (as well as by whom, and when) in order to achieve these goals.

One of the underlying assumptions of action evaluation is that goal setting is a process that continues throughout the life of a project. No matter how well project participants articulate and agree upon their goals at the baseline, they may discover new goals and opportunities as they go along. In addition, participants frequently need to reconsider goals as they encounter resistance or other obstacles to implementation. Finally, project participants may discover that there is a gap between their espoused goals (what they said they wanted) and the goals implicit in what they are actually doing in their practice.

The formative stage actually overlaps with the baseline stage because the action plan, which is the output of the baseline stage, becomes important data in the formative stage. It provides project participants with an explicit basis for comparing intentions with what is actually happening in the project. The formative stage, however, is not simply a control mechanism for keeping the project on track. Rather, it uses the awareness of discoveries, gaps, and contradictions as opportunities for reshaping and fine-tuning a project design. Project stakeholders are asked to function as "reflective practitioners" by standing outside the situation, becoming more aware of their actual goals and strategies for action, and experimenting with new ones.

Stage Three: Summative Evaluation

Finally, there is a summative evaluation stage or more traditional "evaluation-as-judgment," in which defined criteria of success are used to see how well an intervention has met its own internally derived and consciously evolved goals. As a project reaches an intermediate point, or its conclusion, participants use their evolved goals to establish criteria for retrospective assessment. Stakeholders will, for example, examine whether they have reached specified goals, and ask themselves, "why?" or "why not?" They will ask themselves how and what they could have done differently or better.

The Power of Why

Another very powerful aspect of AE is its elicitation of and focus upon individual and collective core values. We refer to this process as "The Power of Why." In the Cincinnati Collaborative, an 18-month intervention designed to address the deteriorating state of police community relations, the design enabled citizens across eight different stakeholder groups to participate (African-American citizens, white citizens, business/education/foundation leaders, youth, police officers and their families, religious and social service leaders, and other minority groups).[5] Community members were invited to answer three questions through interviews and questionnaires:

• What are your goals for future police-community relations in Cincinnati?
• Why are those goals important to you? (What experiences, values, beliefs, and feelings influence your goals?), and
• How do you think your goals can best be achieved?

Citizens were then invited to participate in feedback sessions with other members of their stakeholder groups, to listen to others' responses and to be heard. Participants selected representatives to craft a platform of goals for improving police-community relations, and this platform guided negotiators as they worked toward a settlement. The consensus goals that ultimately emerged, which are the cornerstone of the historic Collaborative Settlement Agreement, are shown in Figure 1.

In addition to crafting a platform of goals, participants also had the opportunity within the feedback sessions to engage in dialogue with others and to share their "whys" in a small group setting. Participants' whys ranged from concerns about fairness and respecting differences, to concerns about safety, and concerns about the police officers being able to perform their job duties effectively.

The powerful "why" discussions enabled the citizens of Cincinnati to experience resonance with one another, and to find commonalities between their own and others' fears, hurts, hopes, and dreams. This outlet for being heard was critical to so many participants; until that point, many had felt unheard and ignored.

As stated by a young African-American woman, "When we felt pain, no one from the city came to listen to us. We needed someone to comfort and listen to us." Healing began to happen as leaders in the city were finally listening to the people and hearing their ideas, and this element was critical to healing in Cincinnati. It will continue to be essential, as the citizens of Cincinnati begin to implement the changes that are outlined in the settlement agreement. The inclusive and participatory process, rooted in participants' core and articulated values -- or "whys" - has helped citizens to feel a sense of ownership of the agreement, and it has helped them to make the transition from an ending to a new beginning.

Conclusion

Efforts to creatively address intractable conflicts almost invariably involve multiple interventions over an extended period of time. Interveners seeking to design interventions for intractable conflicts, and stakeholders involved in these conflicts, often face considerable difficulty in articulating the desired scope and outcomes of an intervention.

Defining success is particularly difficult in intractable conflicts, where the nature of the conflict, its duration, and its multidimensionality are part of its complexity. Yet, clear expectations about what constitutes success are particularly important in intractable conflicts, because intractability tends to de-legitimize efforts to consensually resolve the conflict. In this context, divergent expectations may undermine interventions. Is a successful intervention one that ends the conflict? Educates Track I (formal) diplomats or Track II (informal citizen-diplomats) about alternatives? Addresses conditions that led to the conflict situation? Promotes incremental change or economic development? And, who defines success before, during, and after an intervention, and how? Action Evaluation breaks new ground in helping practitioners and the field to more broadly define, promote, and evaluate the success of conflict resolution efforts - initiative, by initiative.[6]

[1] The term "action research" has been used widely in the past, and has often been used as a method of program evaluation. While that approach has many similarities to this approach, ours is unique in its use of computer technology to design and assess very complex social interventions.

[1] Rothman, 1997

[2] http://www.socsci.uci.edu/istudies/applied/presentations/site/index.html

[3] Rothman and Friedman, 2002

[4] Rothman and Soderquist, 2002

[5] For more information on Action Evaluation, see http://www.ariagroup.com/

Use the following to cite this article:
Rothman, Jay. "Action Evaluation." Beyond Intractability. Eds. Guy Burgess and Heidi Burgess. Conflict Information Consortium, University of Colorado, Boulder. Posted: October 2003 <http://www.beyondintractability.org/essay/action-evaluation>.

Additional Resources