Learn the Fundamentals of Evaluation and Metaevaluation

To the uninitiated, terms like evaluation and metaevaluation might feel intimidating. But the truth is, informal evaluations are already part of your life. “We conduct evaluations every day when deciding what clothes to wear or what type of breakfast to eat,” explains Dr. Liliana Rodríguez-Campos (who prefers to be addressed as Liliana), an internationally renowned expert in this field and the director of USF’s graduate certification in evaluation. When you become a professional evaluator, you learn how to formalize the process to ensure your evaluation is credible and data driven.

No matter what field you’re in — from nonprofit to business, health to education — understanding how to perform an effective evaluation can benefit your organization and the community you serve. According to the American Evaluation Association (AEA), evaluations are important because:

• They incorporate the experiences and feedback of diverse groups, leading to more equitable policies, programs, and practices.
• They boost public trust in organizations by showing “how funds are being spent, what is being accomplished, and what is not being accomplished.”
• They help organizations make informed decisions — for example, how to allocate limited resources or how to invest in personnel training.
• They show whether a program or policy is working and how it can be improved.

Keep reading this blog post to learn fundamental aspects of evaluations and metaevaluations.

What Is an Evaluation?

To explain evaluations in simple terms, Liliana shows her students a photo of a tiny hedgehog cupped in someone’s hands. She asks, “Do you like it, and why or why not?” Because her students’ hearts aren’t made of stone, their answer is obviously yes, they like it. Why do they like it? Because it’s tiny. With this exercise, Liliana’s students have just performed an evaluation: they made a judgment (they like the tiny hedgehog) based on defensible criteria (the size).

Of course, in the real world, most evaluations aren’t going to be quite this simple (or hedgehog-related). Instead, an evaluation is a systematic process that answers questions about the value (such as merit or worth) of policies, organizations, personnel, products, and programs. For example:

• Policy: How effective is this health department’s vaccine campaign?
• Organization: How well are people with disabilities being served by our business?
• Personnel: How productive is our sales department?
• Product: Is the new product line meeting our quality standards?
• Program: Should a local elementary school expand or close its new literacy program?
  
  

For example, to determine whether the elementary school literacy program should be closed or expanded, an evaluation might explore the following questions (modified from the AEA):

• What learning outcomes are being achieved?
• What are the costs and benefits of the program?
• What is the quality of the program?
• How can the program be improved?
• Are the needs of a diverse student body being equitably met?
  
  

Who Performs Evaluations?

An evaluator is the person who “accepts responsibility for the overall evaluation and its results,” according to Liliana’s book Collaborative Evaluation in Practice: Insights From Business, Nonprofit, and Education (xiii). The evaluator uses “defensible criteria” to judge the value of something (such as a program or policy). Keep in mind that as the evaluator, you aren’t the one making decisions based on the evaluation: the client is the one who takes action after receiving your report.

Anyone can benefit from learning how to improve their evaluative thinking — even if you’re not a professional evaluator. “It will help create a baseline for better decision-making,” explains Liliana.

If you are interested in becoming a professional evaluator, you might work as an internal evaluator within your organization or an external evaluator (through independent consulting or a consulting firm). As the AEA explains, your workplace might include:

• Nonprofit or governmental organizations
• Academic or research settings
• Private industry
  
  

What Are Stakeholder Evaluation Approaches?

In the more traditional approach to evaluations, the evaluator creates their report without having any more contact with stakeholders than necessary. But in recent decades, the stakeholder approach is increasing in popularity, as explained in Liliana’s book Collaborative Evaluations. This approach incorporates stakeholders into the evaluation process.

Stakeholders are people who are affected by the results of the evaluation — in other words, they have a stake in it. Let’s look at some of the people who have a stake in the decision to close or expand a public school literacy program. For example:

• The state Department of Education superintendent is your client because they hired you to do the evaluation. They are the key stakeholder.
• The school board members are the audience — the group of stakeholders who will receive the results of the evaluation.
• The parents and caregivers of the students are stakeholders.
• The faculty and staff at the school are stakeholders.
• The students are stakeholders, even if they’re not aware of the evaluation at all.
  
  

How do you involve stakeholders in the process? Let’s take a closer look at three popular stakeholder approaches:

• Participatory evaluators split control of the evaluation between themselves and stakeholders, who are involved in each step of the process, from defining the evaluation to disseminating the results.
• Empowerment evaluators give control of the evaluation to stakeholder participants. The evaluators are there to guide the participants so the evaluation stays on track and meets rigorous standards.
• Collaborative evaluators are in control of the evaluation, but they work closely with stakeholders. Insight from these stakeholders leads to stronger evaluation designs, more creative problem-solving, and higher-quality data collection and analysis. Plus, because stakeholders have been engaged from the beginning, they can better understand — and use – the results of the evaluation. “I personally favor the collaborative evaluation approach,” says Liliana. “But there is a very important disclaimer: it is not about what the evaluator favors — it is what the client needs.”
  
  

What Are Evaluation Models?

Before undertaking an evaluation, an evaluator chooses the approach that is appropriate for the client and their needs. This helps the client understand what to expect: for example, will there be substantial interaction between the evaluator and the stakeholders? Some approaches also offer models you can follow to guide you throughout the process. For example, Liliana’ Model for Collaborative Evaluations (MCE) can lead to more precise evaluations and useful results. We’ve edited and paraphrased the MCE model for the purposes of this blog post, so if you’d like to learn more, you can check out page 6 of Collaborative Evaluations:

• Identify the situation. Who are the stakeholders? What is the scope of the evaluation? What are your resources?
• Clarify the expectations. What is the role of the evaluator versus the roles of the participants (also called collaboration members)? What process and criteria will you use? What is your budget?
• Establish a collective commitment. What is your shared vision? How will you resolve conflict and make decisions?
• Ensure open communication. How can you make sure that everyone is actively participating and carefully listening? Is your messaging clear? Have you outlined the need for change in your report in a way that can be justified?
• Encourage effective practices. How can you encourage fairness, sincerity, and appreciation for individual differences? What can you do to foster flexibility and creativity?
• Follow specific guidelines. Are you following the guiding principles for evaluators, evaluation standards, and the collaboration model?
  
  

Because stakeholder involvement is central to the success of a collaborative evaluation, you also want to make sure you are choosing your collaboration members wisely. These questions, paraphrased from pages 13-14 of Collaborative Evaluations, can help:

• Which stakeholders are interested in participating?
• Who has the bandwidth to participate actively?
• What characteristics are you looking for?
• What are each candidate’s strengths and weaknesses?
• Do any need training, and if so, in what way?
• Will your selected participants work well together?
• Does your selected group include diverse expertise and experiences?
  
  

What Is a Metaevaluation?

A metaevaluation is an evaluation of evaluation. The purpose of a metaevaluation, as Liliana explains, is to help an evaluation live up to its potential. Metaevaluators are the watchdogs of evaluators — they make sure that evaluations are high-quality. This is important because people make decisions based on evaluations. For example, if the school literacy program is shut down because of a faulty evaluation, its students (whether they know about the evaluation or not) will be adversely affected.

The Metaevaluation Checklist — created by Dr. Daniel Stufflebeam, one of Liliana’ mentors — outlines questions that a metaevaluator should ask to determine the merit of an evaluation. For example, did the evaluator consult stakeholders to identify their information needs? After a thorough process — which is similar to the evaluation process explained previously — the metaevaluator releases their final report, which rates the evaluation on a scale from poor to excellent.

If you are a professional evaluator, becoming a metaevaluator can help you make a bigger impact and take the next step in your career.

Take the Next Step

Are you interested in becoming an evaluator? Do you want to level up your evaluation skills to become a metaevaluator? Do you simply want to improve your decision-making skills? You might be the perfect fit for USF’s Certificate in Evaluation and Metaevaluation. Held for eight half-days in September, this virtual certificate caters to students of all levels and experience and teaches you how to manage the evaluation process and how to facilitate stakeholder involvement.

This certificate offers participants the unmatched opportunity to learn from Dr. Liliana Rodríguez-Campos, who works directly with renowned evaluators such as Drs. David Fetterman, James Sanders, Michael Scriven, and Daniel Stufflebeam. You’ll earn a certificate from the University of South Florida, which has attained top-tier recognition by being a member of the Association of American Universities (AAU), which is the most prestigious invitation-only association in higher education.

Register Today with a Discount Code for 75% Off: NFPSQS