Youth Results Mapping may forever alter the way that prevention programs are evaluated, according to Barry M. Kibel, Ph.D., a senior research Scientist at Pacific Institute for Research and Evaluation, Chapel Hill, NC, and the principal investigator for CSAP-funded High-Risk Youth Grants in Cincinnati and Atlanta. This approach promises to be cheaper, more efficient, and able to develop a comprehensive program evaluation as much as five times sooner than traditional methods.
“There is a major paradigm shift underway in the prevention yield from a problem solving-deficit model to the asset-buiIding model,” said Dr. Kibel, at the 1997 CSAP High-Risk Grantee Conference last summer.
Under a problem-solving-deficit model most kids who had problems are thought to have problems for the same reasons. and programs were designed to keep them out of trouble.
“The new approach emphasizes the uniqueness of every youth, adult and community—and builds on these strengths,” says Dr. Kibel. It requires that a program be “reinvented as you go along.”
“The new asset-building approach can be best evaluated by anecdotes,” says Dr. Kibel. “In using anecdotes and complex stories that describe the specific success of a specific individual in a program, the new model emphasizes that everyone is different.”
Dr. Kibel explains that there are two types of anecdotal stories. The simple and causal story is where a baby gets a flu shot and does not get the flu—cause and effect. In the prevention field, unlike the baby and the flu shot, all programs have complex, synchronistic stories where many outside factors beyond the program factors beyond the program influence the outcome of each individual.
An example of a complex story in the prevention field would be an 18-year-old Hispanic woman with a drinking problem, who has not completed high school, and enters a community center program. As the result of bonding with one of the counselors, and obtaining direction from one of her former teachers who lives in the neighborhood, the young woman has significantly reduced alcohol consumption and is working on a high school equivalency diploma. This is a complex story of a client engaged in “healing and transformation,” in which the community center has provided part of the outcome, along with outside factors such as the teacher and the client has made a contribution to her own positive outcome and future.
This anecdotal story, unlike the simple cause-effect story, emphasizes the uniqueness of each client and the importance of otherwise hard to-measure outside influences on client outcomes
This new paradigm in the evaluation of prevention programs, known as Results Mapping, is only 2 years old. According to Dr. Kibel, Results Mapping is a system for relating anecdotal information in a structured format. It is a scientific process because there are rules and conventions for recording and scoring anecdotal information.
A Growing Approach
Results Mapping is a growing approach to evaluation of programs in the prevention field.
‘At least 25 percent of the alcohol and drug prevention field have made the shift, but evaluation and plan-fling tools have not kept up with the shift,” says Dr. Kibel.
Who is using Results Mapping as an alternative to the old model of evaluation? The States of Connecticut and Colorado now use Results Mapping in evaluating all of their alcohol and drug abuse prevention programs. At the local level, programs in Tennessee, California, Arizona, Florida, Illinois, Texas, and New York are using Results Mapping. CSAP is funding programs in Ohio and Georgia that are being evaluated by Results Mapping.
And why are State and Federal prevention
programs using Results Mapping evaluation? This model for evaluation is much cheaper, according to Dr. Kibel, and it is a form of empowerment evaluation.
Results Mapping places the formulation of program data back into the hands of the program directors and staff. There is less dependency on an outside or consultant evaluator. Unlike other more impersonal forms of evaluation, Results Mapping allows clients to become an active part of the evaluation process. Program staff members are encouraged to sit down with clients and write down what the client has to say about their experiences with a specific prevention program.
“The methodology of Results Mapping is not incompatible with outcome-based funding approaches. In fact, the opposite is true. The way we score the anecdotal information provides the best possible information regarding how well a program is doing in moving its target population toward difficult to reach, long-term outcomes,” says Dr. Kibel.
For a 5-year prevention program, it could take 5 to 6 years to see any comprehensive evaluations. With Results Mapping, results could be seen from the same 5-year program in 6 months.
Dr. Kibel emphasizes that Results Mapping is a very scientific process. When there is concern that a handful of stories provide a distorted or highly exaggerated accounting of the accomplishments of a program, there are answers in making a valid and scientific program evaluation.
“Programs need to provide more stories-enough for a comprehensive picture of program accomplishments to emerge. Programs need to score and rank their stories, much as judges rate athletic performances, based on the contribution these represent to clients,” he added.
According to Dr. Kibel, because Results Mapping can provide timely evaluations, as often as every 6 months, it has the potential of becoming an important new tool and resource for prevention program directors and staff Results Mapping is a new evaluation methodology that may become a cornerstone in future prevention programs throughout the country.
Prevention Pipeline Nov/Dec 1997