Outcome Measurement Resource Network

Outcome Measurement Resource Network

Measuring Program Outcomes: A Practical Approach

Acknowledgements

United Way of America Task Force on Impact: Members Involved in Development of Manual
National Human Service Organization Representatives Meeting to Advise Project Staff and Consultants
Glossary of Selected Outcome Measurement Terms
Introduction to Outcome Measurement
Why Measure Outcomes?
About This Manual: Notes About Terminology; In Conclusion
: Exhibits; A: Summary of Program Outcome Model; B: Measuring Program Outcomes: Eight Steps to Success

Step 1: Get Ready
Task 1: Assemble and Orient an Outcome Measurement Work Group: Work Group Responsibilities; Work Group Membership; Initial Discussions

Definition of Terms
Value of Outcome Measurement
Limitations and Potential Problems of Outcome Measurement
Expectations of Agency Director, Board, Other Key Players
Available Resource Materials

Task 2: Decide Which Program(s) to Start With
Task 3: Develop a Timeline
Task 4: Distribute Your Game Plan to Key Players: Exhibits; 1-A: Responsibilities of Outcome Measurement Work Group; 1-B: The Traditional Service Program Model; 1-C: The Program Outcome Model; 1-D: Hints as to What Are and Are Not Outcomes; 1-E: Examples of Diverse Programs and Possible Outcomes; 1-F: Examples of Potential Benefits of Outcome Measurement; 1-G: Guidelines for Incorporating Outcome Data in Funding Decisions; 1-H: Sample Timeline for Planning and Implementing Outcome Measurement in a Program; Worksheet; 1: Timeline for Planning and Implementing Outcome Measurement

Step 2: Choose the Outcomes You Want to Measure
Levels of Outcomes
Task 1: Gather Ideas for What Your Program's Outcomes Are from a Variety of Sources
Task 2: Construct a Logic Model for Your Program: Issues in Defining Outcomes; Preparing your Logic Model
Task 3: Select the Outcomes That Are Important to Measure
Task 4: Get Feedback on Your Logic Model and the Outcomes Selected for Measurement: Exhibits; 2-A: Inputs Through Outcomes: The Conceptual Chain; 2-B: Conducting Focus Groups; 2-C: Examples of Outcomes Based on Observations of a Neighborhood After-school Program; 2-D: Southside Children's Agency; 2-E: Example Logic Models for Four Programs of Southside Children's Agency (horizontal format); 2-F: Example Logic Models for Four Programs of Southside Children's Agency (vertical format); 2-G: Questions Regarding Proposed Program Logic Model and Outcomes
: Worksheets; 2: Program Logic Model; 3: Assessing Possible Outcomes

Step 3: Specify Indicators for Your Outcomes
Task 1: Specify One or More Indicators for Each Outcome
Task 2: Decide What Factors Could Influence Participant Outcomes
Special Problems with Hard-to-Measure Outcomes: Exhibits; 3-A: Sample Outcome Indicators: Southside Children's Agency; 3-B: Example Outcomes and Outcome Indicators for Various Programs; 3-C: Example of Value of Outcome Data Broken Out by Influencing Factors; 3-D: Participant and Program Characteristics That May Influence Outcomes; 3-E: Factors That May Influence Participant Outcomes: Southside Children's Agency; Worksheets; 4: Outcome Measurement Framework; 5: Outcome Measurement Framework: Factors That May Influence Participant Outcomes

Step 4: Prepare to Collect Data on Your Indicators
Task 1: Identify Data Sources for Your Indicators
Task 2: Design Data Collection Methods: Preparing Data Collection Instruments; Developing Data Collection Procedures

: At What Milestones Should Data Be Collected?
Who Is Considered a Participant?
Include All Participants or Only a Sample?
Who Will Collect the Data?
How Will Confidentiality Be Protected?
How Will Participants Be Informed About the Data Collection Process?

Task 3: Pretest Your Data Collection Instruments and Procedures: Exhibits; 4-A: Advantages and Disadvantages of Data Sources for Outcome Indicators; 4-B: Comparison of Major Data Collection Methods; 4-C: Sample Outcome Measurement Framework: Southside Children's Agency; 4-D: Considering Cultural Issues in Instrument Development; 4-E: Sample Pledge of Confidentiality for Data Collectors; 4-F: Sample Consent Form

Step 5: Try Out Your Outcome Measurement System
Task 1: Develop a Trial Strategy
Task 2: Prepare the Data Collectors
Task 3: Track and Collect Outcome Data
Task 4: Monitor the Outcome Measurement Process: Exhibits; 5-A: Sample Master List of Data Elements Needed for Each Participant; 5-B: Sample Data Collection Log

Step 6: Analyze and Report Your Findings
Task 1: Enter the Data and Check for Errors
Task 2: Tabulate the Data
Task 3: Analyze the Data Broken Out by Key Characteristics
Task 4: Provide Explanatory Information Related to Your Findings
Task 5: Present Your Data in Clear and Understandable Form: Exhibits; 6-A: Sample Spreadsheet for Programs Without Computers; 6-B: Outcome of Delivering Healthy Babies, by Age of Mother; 6-C: Sample Outcome Data Table: Southside Children's Agency Teen Mother Parenting Education Program; 6-D: Percent of Participants with Pre- to Post-Test Increases of Six Points or More; 6-E: Changes in Youths' Conscientiousness About Homework

Step 7: Improve Your Outcome Measurement System
Task 1: Review Your Trial-run Experience, Make Necessary Adjustments, and Start Full-scale Implementation
Task 2: Monitor and Review Your System Periodically
Step 8: Use Your Findings
Internal Uses of Outcome Findings: Provide Direction for Staff; Identify Training and Technical Assistance Needs; Identify Program Improvement Needs and Strategies; Support Annual and Long-range Planning; Guide Budgets and Justify Resource Allocations; Suggest Outcome Targets; Focus Volunteers' Attention on Policy and Programmatic Issues
External Uses of Outcome Findings: Recruit Talented Staff and Volunteers; Promote Your Program to Potential Participants and Referral Sources; Identify Partners for Collaborations; Enhance Your Program's Public Image; Retain and Increase Funding
: Exhibits; 8-A: Outcome Findings by Program Unit and Level of Difficulty of Participants' Problems; 8-B: Comparative Findings for Two Youth Services Approaches: Team vs. Individual Activity Emphasis; 8-C: Percent of Participants with Pre- to Post-Test Increases of Six Points or More, 1995 and 1996; 8-D: Southside Neighborhood Children's Agency Child Abuse Public Awareness and Prevention Program: Actual Outcomes vs. Targets; 8-E: Burglaries in Southside Neighborhood Before and After Implementation of Neighborhood Organizing Program

Appendix: Issues in Developing Data Collection Instruments and Procedures
Record Extraction Forms: Instrument Development; Procedural Issues Specific to Record Extraction
Questionnaires: Instrument Development

: Content
Wording
Format and Style

Procedural Issues Specific to Questionnaires

: Administration
Voluntary Completion
Participant Confidentiality
Anonymity
Times for Contacting Potential Interview Respondents
Response Rate
Costs

Trained Observer Ratings: Instrument Development; Procedural Issues Specific to Trained Observer Ratings
: Exhibits; 1: Example of a Record Extraction Form; 2: Survey Questions on Family Counseling; 3: Examples of Fixed-choice Response Sets; 4: Summary Comparison of Questionnaire Administration Options; 5: Tips for Improving Mail Survey Response Rates; 6: Sample Rating Scale for Trained Observer Rating of a Playground for Preschool Children

Bibliography
Performance Measurement
Data Collection
Performance Indicators

~[back to Measuring Practical Outcomes]