The importance of good evaluation design, and how the design stage can be used to prevent problems that are much harder to fix using statistical methods at the analysis stage;
Key problems with data collection that can arise when conducting an impact analysis. In particular, low survey response rate and social desirability bias;
Internal and external validity and the extent to which these are threatened by problems with data collection; AND
Statistical power: the consequences of having insufficient power, and the importance of estimating statistical power of evaluation design at the outset.
CASE
New York City's Teen ACTION Program: an Evaluation Gone Awry
ASSIGNMENT: BASED ON WHAT YOU HAVE LEARNED OVER THE COURSE OF THE TERM,
PART 1: Provide a detailed and critical policy assessment of this evaluation gone awry, and what can be learned from this to be better analysts–Be specific about what you would do to do a better analysis. Make sure to draw on readings from previous
HKS896
HKS Case Number 2027.0
�
This case was written by case writer, Anjani Datla for Senior Lecturer in Public Policy, Dan Levy for use at the John F. Kennedy School of Government (HKS), Harvard University. HKS cases are developed solely as the basis for class discussion. Cases are not intended to serve as endorsements, sources of primary data, or illustrations of effective or ineffective management. (January 2014)
Copyright © 2014 President and Fellows of Harvard College. No part of this publication may be reproduced, revised, translated,
stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by any means without the express written
consent of the Case Program. For orders and copyright permission information, please visit our website at
www.case.hks.harvard.edu or send a written request to Case Program, John F. Kennedy School of Government, Harvard Univer‐
sity, 79 John F. Kennedy Street, Cambridge, MA 02138.
New York City’s Teen ACTION Program: An Evaluation Gone Awry
Introduction
New York City’s Mayor, Michael Bloomberg created the Center for Economic Opportunity (CEO) in 2006. De‐
signed as an innovation fund, CEO tested anti‐poverty programs by applying an evidence‐based approach. "The
innovation fund will carefully monitor new programs and hold them accountable for producing results—just as a
business would. And if we find a certain program isn’t making the grade, we will terminate its funding,” Bloomberg
declared at a press conference announcing the formation of CEO.1
The Center was located under the Mayor’s office and had an annual budget of $100 million, consisting of pub‐
lic and private funds.2 CEO partnered with more than 20 city agencies to launch 31 programs in its first year of op‐
eration. Programs served three groups: the working poor, disengaged youth, and families with young children.
Two distinct strategies guided CEO’s collaborations with the rest of city government. First, armed with the
leadership support of commissioners at city agencies, CEO worked closely with deputy commissioners and senior
program staff to design and roll out programs. Second, from the outset, CEO made it clear that funding for pro‐
grams would be based on their performance. City agencies “understood that the programs were going to be evalu‐
ated and they understood that certain data would need to be collected,” said Veronica White, CEO’s Executive
Director. The Center allocated $2 million for evaluations of all programs funded by public finances. One of CEO’s
programs, Teen ACTION, served an important target population.
Teen ACTION
Drugs and sex served as a gateway to pregnancies, substance abuse and school suspensions for teenagers liv‐
ing in poverty in New York City. Such high risk behaviors set the stage for a downward spiral at a critical point in
1 Press Release December 18, 2006 “Mayor Bloomberg Announces $150 Million Annual Investment for Solutions to Challenges Raised by the Commission for Economic Opportunity” Available at: http://www.nyc.gov/portal/site/nycgov/menuitem.c0935b9a57bb4ef3daf2f1c701c789a0/index.jsp?pageID=mayor_press_relea se&catID=1194&doc_name=http%3A%2F%2Fwww.nyc.gov%2Fhtml%2Fom%2Fhtml%2F2006b%2Fpr437‐ 06.html&cc=unused1978&rc=1194&ndi=1 ; accessed May 14, 2012. 2 Mayor Bloomberg made an annual commitment of $75 million city funds to CEO. An additional $25 million was raised annually from private charitable organizations.
For the exclusive use of L. ROBINSON, 2024.
This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.
HKS Case Program 2 of 5 Case Number 2027.0
the lives of these teenagers and perpetuated the cycle of poverty. New York City had seen a steady decline in teen
births between 1990 and 2000, but the teenagers who continued to get pregnant were overwhelmingly poor and
unmarried. Average teen pregnancy rates in the United States too had fallen in the same period, yet the U.S. teen
birth rate remained one of the highest among developed nations.3
Some proponents of after‐school programs argued that youth oriented after‐school activities could help re‐
duce teen pregnancy rates by keeping teens under adult supervision during weekday afternoons, a time particular‐
ly prone to risky behaviors.4 According to a 1995 study, American 10th graders who did not participate in after
school extracurricular activities were 37 percent more likely to become teen parents than the 10th graders who
spent 1 to 4 hours a week in after‐school extracurricular activities.5 The Afterschool Alliance, an advocacy group for
after school programs claimed that “after school programs help prevent teen pregnancy by encouraging good de‐
cision‐making and providing youth health education and positive role models in a supervised setting after
school.”6
Teen ACTION (Achieving Change Together in Our Neighborhood) was developed by CEO and the New York City
Department of Youth and Community Development (DYCD). It was an after‐school service learning program with
the overarching goal of reducing risky behaviors, promoting youth development and improving community en‐
gagement among teenagers in high poverty neighborhoods. Teen ACTION’s curriculum tried to redirect the atten‐
tion of students to gaining skills they could use in the future. It brought together several service learning elements
including structured classroom activities, community service, group reflections and opportunities for participants
to engage in decision‐making.
Carson Hicks was the deputy Director of Programs and Evaluation at CEO and managed the youth develop‐
ment portfolio. Hicks, a sociologist by training, worked closely on the development of Teen ACTION. The program
design was a variation on the theme of several successful service learning initiatives including the Teen Outreach
Program—developed in 1978 and implemented in cities across the United States (see Exhibit A for Teen ACTION’s
logic model). The Teen Outreach Program was rigorously evaluated and had proven to significantly reduce rates of
teen pregnancy, course failure and school suspensions among participants when compared to a control group.7
3 Manlove, Jennifer et al., “Preventing Teenage Pregnancy, Childbearing, and Sexually Transmitted Diseases: What the Research Shows,” Child Trends Research Brief, May 2002. 4 According to after‐school advocacy group, AfterSchool Alliance, “the conventional wisdom is that by the time youth reach high school, they are old enough to take care of themselves [after the school day ends] and don’t need adult supervision. However, if left to their own devices, teens might not make the best use of the free time they have after school,” The Alliance adds that “adult supervision is closely linked to reduced risky sexual behavior among teens.” For a selection of research on effects of youth afterschool programs, see Afterschool Alliance “Older Youth Need Afterschool Programs,” Issue Brief No. 20, October 2004. Available at: http://www.afterschoolalliance.org/issue_briefs/issue_older_youth_20.pdf; accessed July 22, 2014. 5 Westat, Inc., "Adolescent Time Use, Risky Behavior and Outcomes: An Analysis of National Data," September 1995. 6 From: Afterschool Alliance, “Afterschool and Pregnancy Prevention,” After School Alert Issue Brief #11, July 2002. For more on teen birth rates, see ChildTrends “Facts at a Glance: A Fact Sheet Reporting National, State and City Trends in Teen Childbear‐ ing,” 2011. Available at: http://www.childtrends.org/wp‐content/uploads/2011/04/2011‐10FactsAtAGlance2011.pdf; accessed July 11, 2014. 7 Allen, J. et al “Preventing Teen Pregnancy and Academic Failure: Experimental Evaluation of a Developmentally Based Ap‐ proach,” Child Development, 1997. For summary of the Teen Outreach Program, visit
For the exclusive use of L. ROBINSON, 2024.
This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.
HKS Case Program 3 of 5 Case Number 2027.0
CEO officials also found “studies of various types of after‐school programs to have consistently shown that the
longer participants remained in the program, the greater its impact.”8
In the classroom component of the Teen ACTION curriculum, students were exposed to topics such as immi‐
gration, the environment and sexual and reproductive health, while also being taught how to work in teams and
effectively solve problems. As part of the service learning component of the curriculum, students were required to
research and identify problems in their communities and design projects to serve residents in need. Efforts includ‐
ed helping senior citizens, tutoring younger students and conducting research on nutrition or sex education. “The
curriculum taught a host of issues like recycling and global warming,” explained Hicks. “After learning about these
issues, the students looked around in their neighborhoods to see if there was an activity or project that they could
undertake to help improve their communities.”9
Between fall 2007 and spring 2008, with a budget of $3.8 million, Teen ACTION was deployed in 60 after‐
school sites via DYCD. The program served more than 3,000 young adults between the ages of 13 and 21 years,
enrolled in grades 6 to 11.
http://www.blueprintsprograms.com/evaluationAbstracts.php?pid=ec91fc2dc062c0f220b5d7b52ac6446011bf98cd (accessed July 11, 2014). 8 Center for Economic Opportunity: Teen ACTION http://www.nyc.gov/html/ceo/html/programs/teen_action.shtml (accessed June 18, 2012) 9 Interviews with Carson Hicks, Director of Programs and Evaluation, Center for Economic Opportunity, April 23 and May 9, 2012
For the exclusive use of L. ROBINSON, 2024.
This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.
HKS Case Program 4 of 5 Case Number 2027.0
Target Population
Teen ACTION (Achieving Change Together In Our Neighborhood) Logic Model
Activities Resources Goals
Learning Activities Guided by curriculum grounded
in youth development & ser- vice-learning principles
Attention to process issues (team-building, developing leadership skills) and content (thematic units)
Other Activities Referrals to health care services Workshops & guest lectures
Outreach, Recruitment & En- rollment Use of effective outreach &
recruitment strategies
Young adults, 13 to 21 years old, and enrolled in 6th through 12th grade, from high- need neighbor- hoods in NYC’s five boroughs
CEO Funding ($3.88 M for FY08 and $4.48 M for FY09)
Management and program oversight, including online data system, pro- vided by DYCD
Teen ACTION Service-learning Curriculum
Technical assis- tance and capacity building— provided by DYCD, DOHMH, and external organ- izations (TASC, Global Kids)
Large number (60+) of sites, with most providers already operating designated DYCD after-school pro- grams (OST or Beacons)
Provider linkages to schools and oth- er community or- ganizations
Provider health care linkages
Orientation Program overview, rules &
expectations
Reflection Activities Opportunities to reflect and
integrate structured-learning ac- tivities and service experiences
Service Activities Youth-led decision making with
adult guidance Activities fulfilling real needs
and valued by intended benefi- ciaries
Placements of groups, teams, and/or individuals
Exhibit A
To reduce risk behav- iors, espe- cially those that might result in teen pregnancy
To promote positive youth devel- opment
To promote community engagement
For the exclusive use of L. ROBINSON, 2024.
This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.
HKS Case Program 5 of 5 Case Number 2027.0
Teen ACTION (Achieving Change Together In Our Neighborhood) Logic Model (continued)
Short-Term Outcomes Long-Term Outcomes Outputs
For Year One, minimum number of 40 students per site 1. For Round One sites—
completion of 120 minimum hours, with at least 40 hours de- voted to structured learning ac- tivities and at least 40 hours de- voted to service activities in pro- gram year
2. For Round Two sites— completion of 90 minimum hours, with at least 30 hours de- voted to structured learning ac- tivities and at least 30 hours de- voted to service activities in pro- gram year
For Years Two and Three—
minimum of 40 students per site. Completion of 165 minimum hours, with at least 55 hours devot- ed to structured learning activities and at least 55 hours devoted to service activities in program year
Number of referrals to health care
services
Increase in cred- it accumulation and grade pro- motion
Increase in high school gradua- tion rates
Reduction in teen pregnancy rates
Number of individuals enrolled
Increase in knowledge & attitudes about community needs
Increase in knowledge & attitudes about health & well-being, HIV/AIDS, and sexual health
Improvement in school attendance/reduced truancy
Decrease in school suspen- sion rates
Improvement in life skills Improvement in decision-
making skills Increase in self-confidence Reduction in risk behavior Increase in community
engagement
Context
A large number of live births occur to teenagers in New York City—8,415 in 2004.* The overwhelming majority occur to people who are unmarried and living in poverty. Teen pregnancy is one of several risks that young people in poverty face during the transition to adult-
hood. Older youth are less likely to participate in after-school programs. *The New York City Commission for Economic Opportunity (September 2006). Report to Mayor Michael R. Bloomberg: Increasing opportunity and reducing poverty in New York City.
Source: Center for Economic Opportunity
For the exclusive use of L. ROBINSON, 2024.
This document is authorized for use only by LATHESHIA ROBINSON in Copy of PA 565: Advanced Policy Analysis taught by JOHN GRUMMEL, Upper Iowa University from Mar 2024 to Aug 2024.