The importance of good evaluation design, and how the design stage can be used to prevent problems that are much harder to fix using statistical methods at the analysis stage; Key problem

  HKS896 

HKS Case Number 2027.0 

�

This case was written by case writer, Anjani Datla for Senior Lecturer in Public Policy, Dan Levy for use at the John F. Kennedy  School of Government (HKS), Harvard University. HKS cases are developed solely as the basis for class discussion.  Cases are not  intended to serve as endorsements, sources of primary data, or illustrations of effective or ineffective management. (January  2014)    

Copyright © 2014 President and Fellows of Harvard College. No part of this publication may be reproduced, revised, translated, 

stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by any means without the express written 

consent of the Case Program. For orders and copyright permission information, please visit our website at 

www.case.hks.harvard.edu or send a written request to Case Program, John F. Kennedy School of Government, Harvard Univer‐

sity, 79 John F. Kennedy Street, Cambridge, MA  02138. 

New York City’s Teen ACTION Program: An Evaluation Gone Awry 

Introduction  

New York City’s Mayor, Michael Bloomberg created the Center for Economic Opportunity (CEO) in 2006. De‐

signed as an innovation fund, CEO tested anti‐poverty programs by applying an evidence‐based approach. "The 

innovation fund will carefully monitor new programs and hold them accountable for producing results—just as a 

business would. And if we find a certain program isn’t making the grade, we will terminate its funding,” Bloomberg 

declared at a press conference announcing the formation of CEO.1  

The Center was located under the Mayor’s office and had an annual budget of $100 million, consisting of pub‐

lic and private funds.2 CEO partnered with more than 20 city agencies to launch 31 programs in its first year of op‐

eration. Programs served three groups: the working poor, disengaged youth, and families with young children. 

Two distinct strategies guided CEO’s collaborations with the rest of city government. First, armed with the 

leadership support of commissioners at city agencies, CEO worked closely with deputy commissioners and senior 

program staff to design and roll out programs. Second, from the outset, CEO made it clear that funding for pro‐

grams would be based on their performance. City agencies “understood that the programs were going to be evalu‐

ated and they understood that certain data would need to be collected,” said Veronica White, CEO’s Executive 

Director. The Center allocated $2 million for evaluations of all programs funded by public finances. One of CEO’s 

programs, Teen ACTION, served an important target population. 

Teen ACTION 

Drugs and sex served as a gateway to pregnancies, substance abuse and school suspensions for teenagers liv‐

ing in poverty in New York City. Such high risk behaviors set the stage for a downward spiral at a critical point in 

1  Press Release December 18, 2006 “Mayor Bloomberg Announces $150 Million Annual Investment for Solutions to Challenges  Raised by the Commission for Economic Opportunity” Available at:  http://www.nyc.gov/portal/site/nycgov/menuitem.c0935b9a57bb4ef3daf2f1c701c789a0/index.jsp?pageID=mayor_press_relea se&catID=1194&doc_name=http%3A%2F%2Fwww.nyc.gov%2Fhtml%2Fom%2Fhtml%2F2006b%2Fpr437‐ 06.html&cc=unused1978&rc=1194&ndi=1 ; accessed May 14, 2012.  2  Mayor Bloomberg made an annual commitment of $75 million city funds to CEO. An additional $25 million was raised annually  from private charitable organizations. 

For the exclusive use of L. ROBINSON, 2024.

This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.

HKS Case Program  2 of 5  Case Number 2027.0 

the lives of these teenagers and perpetuated the cycle of poverty. New York City had seen a steady decline in teen 

births between 1990 and 2000, but the teenagers who continued to get pregnant were overwhelmingly poor and 

unmarried. Average teen pregnancy rates in the United States too had fallen in the same period, yet the U.S. teen 

birth rate remained one of the highest among developed nations.3 

Some proponents of after‐school programs argued that youth oriented after‐school activities could help re‐

duce teen pregnancy rates by keeping teens under adult supervision during weekday afternoons, a time particular‐

ly prone to risky behaviors.4 According to a 1995 study, American 10th graders who did not participate in after 

school extracurricular activities were 37 percent more likely to become teen parents than the 10th graders who 

spent 1 to 4 hours a week in after‐school extracurricular activities.5 The Afterschool Alliance, an advocacy group for 

after school programs claimed that “after school programs help prevent teen pregnancy by encouraging good de‐

cision‐making and providing youth health education and positive role models in a supervised setting after 

school.”6

Teen ACTION (Achieving Change Together in Our Neighborhood) was developed by CEO and the New York City 

Department of Youth and Community Development (DYCD). It was an after‐school service learning program with 

the overarching goal of reducing risky behaviors, promoting youth development and improving community en‐

gagement among teenagers in high poverty neighborhoods. Teen ACTION’s curriculum tried to redirect the atten‐

tion of students to gaining skills they could use in the future. It brought together several service learning elements 

including structured classroom activities, community service, group reflections and opportunities for participants 

to engage in decision‐making. 

Carson Hicks was the deputy Director of Programs and Evaluation at CEO and managed the youth develop‐

ment portfolio. Hicks, a sociologist by training, worked closely on the development of Teen ACTION. The program 

design was a variation on the theme of several successful service learning initiatives including the Teen Outreach 

Program—developed in 1978 and implemented in cities across the United States (see Exhibit A for Teen ACTION’s 

logic model). The Teen Outreach Program was rigorously evaluated and had proven to significantly reduce rates of 

teen pregnancy, course failure and school suspensions among participants when compared to a control group.7 

3 Manlove, Jennifer et al., “Preventing Teenage Pregnancy, Childbearing, and Sexually Transmitted Diseases: What the Research  Shows,” Child Trends Research Brief, May 2002.  4 According to after‐school advocacy group, AfterSchool Alliance, “the conventional wisdom is that by the time youth reach high  school, they are old enough to take care of themselves [after the school day ends] and don’t need adult supervision. However,  if left to their own devices, teens might not make the best use of the free time they have after school,” The Alliance adds that  “adult supervision is closely linked to reduced risky sexual behavior among teens.” For a selection of research on effects of  youth afterschool programs, see Afterschool Alliance “Older Youth Need Afterschool Programs,” Issue Brief No. 20, October  2004. Available at: http://www.afterschoolalliance.org/issue_briefs/issue_older_youth_20.pdf; accessed July 22, 2014.  5 Westat, Inc., "Adolescent Time Use, Risky Behavior and Outcomes: An Analysis of National Data," September 1995.  6 From: Afterschool Alliance, “Afterschool and Pregnancy Prevention,” After School Alert Issue Brief #11, July 2002. For more on  teen birth rates, see ChildTrends “Facts at a Glance: A Fact Sheet Reporting National, State and City Trends in Teen Childbear‐ ing,” 2011. Available at: http://www.childtrends.org/wp‐content/uploads/2011/04/2011‐10FactsAtAGlance2011.pdf; accessed  July 11, 2014.  7 Allen, J. et al “Preventing Teen Pregnancy and Academic Failure: Experimental Evaluation of a Developmentally Based Ap‐ proach,” Child Development, 1997. For summary of the Teen Outreach Program, visit 

For the exclusive use of L. ROBINSON, 2024.

This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.

HKS Case Program  3 of 5  Case Number 2027.0 

CEO officials also found “studies of various types of after‐school programs to have consistently shown that the 

longer participants remained in the program, the greater its impact.”8 

In the classroom component of the Teen ACTION curriculum, students were exposed to topics such as immi‐

gration, the environment and sexual and reproductive health, while also being taught how to work in teams and 

effectively solve problems. As part of the service learning component of the curriculum, students were required to 

research and identify problems in their communities and design projects to serve residents in need. Efforts includ‐

ed helping senior citizens, tutoring younger students and conducting research on nutrition or sex education. “The 

curriculum taught a host of issues like recycling and global warming,” explained Hicks. “After learning about these 

issues, the students looked around in their neighborhoods to see if there was an activity or project that they could 

undertake to help improve their communities.”9

Between fall 2007 and spring 2008, with a budget of $3.8 million, Teen ACTION was deployed in 60 after‐

school sites via DYCD. The program served more than 3,000 young adults between the ages of 13 and 21 years, 

enrolled in grades 6 to 11.  

http://www.blueprintsprograms.com/evaluationAbstracts.php?pid=ec91fc2dc062c0f220b5d7b52ac6446011bf98cd (accessed  July 11, 2014). 8  Center for Economic Opportunity: Teen ACTION http://www.nyc.gov/html/ceo/html/programs/teen_action.shtml (accessed  June 18, 2012)  9  Interviews with Carson Hicks, Director of Programs and Evaluation, Center for Economic Opportunity, April 23 and May 9,  2012 

For the exclusive use of L. ROBINSON, 2024.

This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.

HKS Case Program  4 of 5  Case Number 2027.0 

Target Population

Teen ACTION (Achieving Change Together In Our Neighborhood) Logic Model

Activities Resources Goals

Learning Activities  Guided by curriculum grounded

in youth development & ser- vice-learning principles

 Attention to process issues (team-building, developing leadership skills) and content (thematic units)

Other Activities  Referrals to health care services  Workshops & guest lectures

Outreach, Recruitment & En- rollment  Use of effective outreach &

recruitment strategies

Young adults, 13 to 21 years old, and enrolled in 6th through 12th grade, from high- need neighbor- hoods in NYC’s five boroughs

 CEO Funding ($3.88 M for FY08 and $4.48 M for FY09)

 Management and program oversight, including online data system, pro- vided by DYCD

 Teen ACTION Service-learning Curriculum

 Technical assis- tance and capacity building— provided by DYCD, DOHMH, and external organ- izations (TASC, Global Kids)

 Large number (60+) of sites, with most providers already operating designated DYCD after-school pro- grams (OST or Beacons)

 Provider linkages to schools and oth- er community or- ganizations

 Provider health care linkages

Orientation  Program overview, rules &

expectations

Reflection Activities  Opportunities to reflect and

integrate structured-learning ac- tivities and service experiences

Service Activities  Youth-led decision making with

adult guidance  Activities fulfilling real needs

and valued by intended benefi- ciaries

 Placements of groups, teams, and/or individuals

Exhibit A 

 To reduce risk behav- iors, espe- cially those that might result in teen pregnancy

 To promote positive youth devel- opment

 To promote community engagement

For the exclusive use of L. ROBINSON, 2024.

This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.

HKS Case Program  5 of 5  Case Number 2027.0 

Teen ACTION (Achieving Change Together In Our Neighborhood) Logic Model (continued)

Short-Term Outcomes Long-Term Outcomes Outputs

 For Year One, minimum number of 40 students per site 1. For Round One sites—

completion of 120 minimum hours, with at least 40 hours de- voted to structured learning ac- tivities and at least 40 hours de- voted to service activities in pro- gram year

2. For Round Two sites— completion of 90 minimum hours, with at least 30 hours de- voted to structured learning ac- tivities and at least 30 hours de- voted to service activities in pro- gram year

 For Years Two and Three—

minimum of 40 students per site. Completion of 165 minimum hours, with at least 55 hours devot- ed to structured learning activities and at least 55 hours devoted to service activities in program year

 Number of referrals to health care

services

 Increase in cred- it accumulation and grade pro- motion

 Increase in high school gradua- tion rates

 Reduction in teen pregnancy rates

 Number of individuals enrolled

 Increase in knowledge & attitudes about community needs

 Increase in knowledge & attitudes about health & well-being, HIV/AIDS, and sexual health

 Improvement in school attendance/reduced truancy

 Decrease in school suspen- sion rates

 Improvement in life skills  Improvement in decision-

making skills  Increase in self-confidence  Reduction in risk behavior  Increase in community

engagement

Context

 A large number of live births occur to teenagers in New York City—8,415 in 2004.*  The overwhelming majority occur to people who are unmarried and living in poverty.  Teen pregnancy is one of several risks that young people in poverty face during the transition to adult-

hood.  Older youth are less likely to participate in after-school programs. *The New York City Commission for Economic Opportunity (September 2006). Report to Mayor Michael R. Bloomberg: Increasing opportunity and reducing poverty in New York City.

Source: Center for Economic Opportunity 

For the exclusive use of L. ROBINSON, 2024.

This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.