ETR Logo ETR Logo ReCAPP logo

Theories & Approaches

All Theories & Approaches

Program Evaluation Basics

For those of you who have had training on developing BDI Logic Models, we believe you will find this edition of ReCAPP a natural companion to that training. This edition focuses on the Basics of Program Evaluation with a particular emphasis on how BDI Logic Models can be used to guide practitioners in developing program evaluation plans.

ETR has conducted various trainings on the Program Evaluation Basics throughout the United States. In February 2004, ETR in partnership with the National Organization on Adolescent Pregnancy, Parenting and Prevention (NOAPPP) designed and delivered a two-day training titled "Designing and Evaluating Program with Intent" in Columbia, SC. We should also mention that some of the content from this training was adapted from ETR's years of training and technical assistance work with the California Office of Family Planning grantees. Much of what is presented in this edition of ReCAPP has been adapted from these trainings.

We would like to take this opportunity to thank the California Office of Family Planning, NOAPPP, the CDC Division of Reproductive Health, and the Annie E. Casey Foundation for their support in developing many of the materials you will find in this edition of ReCAPP.

What You Will Find in this Edition of ReCAPP

Specifically, this edition of ReCAPP includes:

  1. Slide Presentation (in Flash). This presentation was adapted from a face-to-face training on program evaluation. The slides provide information on:
    • impact, outcome and process evaluation
    • writing objectives
    • type of evaluation designs
    • qualitative and quantitative measurement methods
    • constructing good evaluation tools, and
    • evaluation reporting.
  2. "Creating Better Surveys" Worksheet (pdf file). This worksheet provides an example of a survey with several problems in it. After working to find these problems and proposing improvements, review the accompanying answer key to compare your answers.
  3. Sample Focus Group Protocols. Three focus group protocols are provided in this section. Two were designed for teens and the other for parents. One (Wait for Sex) was designed as an evaluation tool and the other two as assessment tools. Specifically, the three protocols are:
    1. Parent-Child Connectedness Focus Group Protocol for Teens
    2. Parent-Child Connectedness Focus Group Protocol for Parents, and
    3. Wait for Sex Focus Group Protocol for Teens
  1. Focus Group Facilitator Guidelines. This list of guidelines was developed as a consequence of facilitating 14 focus groups on parent-child connectedness in support of ETR's Parent-Child Connectedness: Bridging Research and Intervention Design (PCC BRIDGE) project funded by the Annie E. Casey Foundation.
  2. A Mock Board Presentation. The Mock Board Presentation activity is designed to give an authentic practice opportunity for presenting/reporting on program evaluation findings.
  3. Evaluation Planning Matrices(Word documents). These matrices provide the program evaluator with a way of recording impact, outcome and process objectives and their corresponding evaluation designs, measurement methods and questions.
  4. Evaluation Resources

Other useful program evaluation resources on ReCAPP and other sites:
  1. ReCAPP Glossary
    ReCAPP's on-line glossary of research and evaluation terms
  2. The National Center for Health Statistics' research-based glossary

top

Creating Better Surveys

Program practitioners and evaluators often elect to use a written survey to evaluate a program. The survey may be used to obtain pre- and post- data about a group of people receiving an intervention. The hope is that program participants will perform better on the post-test than on the pre-test thereby suggesting that the intervention had a positive effect on knowledge, attitude and/or behaviors of the participants. A survey may also be used to obtain program satisfaction data. (e.g. How well did today's workshop meet your needs?)

There are several resources that practitioners can utilize to develop questions for a survey. First, and in the most ideal of situations, they can review already tested scales and questions available in books like the Handbook of Sexuality-Related Measures. Second, they can adapt these questions or develop their own questions. It is always a good idea to pilot test any survey you develop to make sure questions are understood by participants.

Evaluation slides #29-#40 provide some additional guidance on developing questions for a survey.

ReCAPP has provided you with a short mock survey (Creating Better Surveys Worksheet) for your critique. After completing this exercise you may want to compare your responses to the answer key at the end.

top

Focus Group Protocols and Guidelines

In this section, you will find three examples of focus group protocols.

  • Wait for Sex Focus Group Protocol
    This protocol was developed as part of the evaluation strategy for a program developed by ETR titled Wait for Sex. It was administered at the conclusion of the 20-week program to about 12 youths (aged 11-13 years) who had participated in the program.
  • Two other focus group protocols were designed for the purpose of program assessment (information that helps inform the design of a program) rather than program evaluation (information that tells us about the effectiveness of a program). You will also find other materials that were needed to support this focus group study. ETR conducted 14 focus groups in various regions of the US with low-income parents and teens (aged 11-13 or 13-15) to better understand how the construct of "parent-child connectedness" is understood and what parents and teens do to feel close to each other. This work is in support of ETR's Parent-Child Connectedness: Bridging Research and Intervention Design (PCC BRIDGE) project funded by the Annie E. Casey Foundation. For more information on this project, visit ReCAPP's edition on Parent-Child Connectedness: Bridging Research and Intervention Design.

    The examples of focus group protocols from this project are:

Also, included are the following samples of focus group forms, which are formatted as Microsoft Word documents for easy adaptation:
And finally, during the course of facilitating focus groups for the PCC BRIDGE project, ETR recorded a list of tips for facilitators — Guidelines for Focus Group Facilitation.

top

Evaluation Resources

Measures

  • Davis, C.M., Yarber, W.L., Bauserman, R., Schreer, G., & Davis, S.L. (Eds). (1998). Handbook of sexuality-related measures. Thousand Oaks: SAGE Publications, Inc.
    Description available at: www.sagepub.com
    Telephone: (805) 499-9774
    Email:order@sagepub.com
  • The Kinsey Institute for Research in Sex, Gender and Reproduction. Measurement and Assessment Bibliography.
    www.indiana.edu/~kinsey/services/bib-measures.html
  • Dahlberg, L.L., Toal, S.B., Behrens, C.B. (Eds). (1998). Measuring Violence-Related Attitudes, Beliefs, and Behaviors Among Youths: A Compendium of Assessment Tools. Atlanta: Division of Violence Prevention, National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.

Program Evaluation
Focus Groups
Surveys

top


One may receive training on BDI Logic Models in a variety of ways, including:

  1. taking one of ETR's face-to-face trainings on BDI Logic Models
  2. taking ReCAPP's on-line course titled A Beginner's Course in Developing BDI Logic Models, and/or
  3. reading Dr. Doug Kirby's paper titled BDI Logic Models: A Useful Tool for Designing, Strengthening and Evaluating Programs to Reduce Adolescent Sexual Risk-Taking, Pregnancy, HIV and Other STDs.