Program Evaluation

Our process for conducting program evaluation is grounded in a framework of sustainable, systematic change—targeting all aspects of the school environment. Our framework measures school-wide, classroom, and individual support systems across both academic and social/behavioral domains. Using a Multi-Tiered Systems of Support (MTSS) framework to measure across settings (school-wide, classrooms, individuals), requires experience regarding high quality instruction for typically developing, as well as students who receive specialized interventions and supports through special education services. An illustration of this layered approach to evaluation is provided below.

A Program Evaluation of Boys’ & Girls’ Clubs of South Puget Sound Summer Boost: A Whole Child Approach to Closing the Achievement

More than 40 empirical studies since 1906 identified the “summer slide” in learning. It hits low income students very hard. They lose 2 or more months in reading progress compared to their more affluent peers who typically make modest gains. The reading gap slide can be three grade equivalents by 5th grade versus their more affluent peers. Research demonstrates that if a student is not reading well by 3rd grade they are 4 times more likely to drop out of school, and even if they graduate, they are less likely to succeed in postsecondary education. Further, research shows ⅔ of the total achievement gap is attributed to “summer slide”. Yet, traditional summer school is not always a good option for families, particularly low income and/or single parent families, since it is only for a portion of the day.

The Boys & Girls Clubs of South Puget Sound partnered with Tacoma Public Schools to develop an innovative pilot program designed to reverse summer learning loss for students most at risk of falling behind. The project was designed for third grade students who qualified for free/reduced lunches and who were behind in their grade level. The classroom size remained small throughout the 5-week program, with a ratio of 5 to 10 students per teacher. The School District provided certified teachers trained in current curriculum to lead Summer Boost, provided lesson plans, and offered instruction to students for 2 hours each day. The Clubs provided additional enrichment programming and snacks/meals; participating students were able to stay at the Clubs the entire day and were invited to attend the Clubs for the duration of summer at no charge. This project adopted a “whole child” approach and intended to improve incoming 4th grade elementary students’ academic progress and social emotional outcomes.

To evaluate the effectiveness of Summer Boost, the Boys & Girls Clubs of South Puget Sound partnered with the University of Washington Tacoma Center for Strong Schools to conduct an evidence-based program evaluation. The full results are available, but this document serves as the executive summary.

View Results of Boys’ & Girls’ Clubs of South Puget Sound Summer Boost

Surprisingly, the research concludes that at risk students may not only experience the academic, but also social emotional backslides while they are out of school for summer break. Social Emotional Competencies include self-awareness, self-management, social-awareness, relationship, and responsible decision-making skills. Their value includes:

  • providing students the skills to more easily navigate the school environment and improve student attitudes and beliefs about self, others, and school
  • better academic performance: achievement scores an average of 11 percentile points higher than students who did not receive SEL instruction;
  • improved attitudes and behaviors: greater motivation to learn, deeper commitment to school, increased time devoted to schoolwork, and better classroom behavior;
  • fewer negative behaviors: decreased disruptive class behavior, noncompliance, aggression, delinquent acts, and disciplinary referrals; and reduced emotional distress: fewer reports of student depression, anxiety, stress, and social withdrawal.
  • Strong SECs continue to benefit students throughout their lives, supporting career growth, personal relationships and family strength.
  • Students from home environments challenged with poverty, addiction, mental health issues, incarceration, etc. need SEC skills to build resiliency.

There have been numerous studies conducted on the benefits of summer literacy programs. As a remediation strategy, summer school is a key practice, because it offers a way to provide struggling students with extra instructional time that they need to catch up with their peers. Summer reading programs have been shown to be effective in elevating reading levels for students who participate.

What has not been studied is integrating social emotional learning into summer literacy programs. The UW Tacoma was the ideal partner to study the summer program because of the “Whole Child” approach the UWT has been developing, in cooperation with Tacoma Public Schools, as a key strategy for improving K-12 outcomes. The Summer Boost collaboration was the perfect setting for just such an integrated study.

The stated purpose of this study was to examine the effect of an integrated summer learning program on students’ social emotional and literacy outcomes. Specifically, the following questions were addressed:

  1. What was the effect of this summer learning program on students’ literacy outcome?
  2. What was the effect of this summer learning program on student’s social emotional outcome?
  3. What was the social validity of this summer learning program (brief explanation of S.V.)
  4. Control Group received only literacy instruction in a classroom setting for 4 hours per day for 5 weeks. Control Group did not receive the full day enrichment of the Boys & Girls Club.
  5. The treatment group received the same literacy instruction plus the full day of enrichment activities.

The findings suggest that as a result of this program, over 93% of the students did not experience the typical “summer academic backslide”, and over 41% of them made 1 year (or more) of literacy growth. More importantly, it was observed that the treatment group students made significant improvement (increased by 6 points on a standardized measure) on social emotional wellbeing while the control group saw a decline (decreased by 2.5 points). Results also suggest that the proportion of high risk students with emotional or peer relationship problems in the treatment group decreased by 16% and 21% respectively. And without the treatment, the control group’s social emotional wellbeing could be characterized as a “summer social emotional backslide “. This study provides new evidence to support the “whole child” educational approach and call for stakeholders’ attention to address this critical issue.

Graphic divider between sections

Boys & Girls Club of South Puget Sound 21st Century Community Learning Center Evaluation

The 21st Century Community Learning Center (CCLC) is a program facilitated by the Boys’ and Girls’ Clubs of South Puget Sound in partnership with the four neighboring public school systems to include over 500 enrolled CCLC participants. The clubs provide academic enrichment, health and life skills training, sports and recreation, computer-based learning, and a variety of art projects. The goals of the BGCSPS 21st CCLC program are to (1) measurably improve students’ academic performance in reading/language and math; (2) encourage students’ personal growth through high quality enrichment activities such as physical fitness, art and music; (3) positively affect students’ communication, social skills, positive learning behavior and emotional well- being; (4) positively affect the educational development of families of students; (5) establish a sustainable internal organization and workforce to provide engaging learning opportunities; and (6) establish relationships with community organizations and schools that will provide ongoing partnerships of mutual support.

The purposes of this evaluation are to (1) evaluate the program’s effectiveness on meeting the above four goals and the state and federal performance indicators and (2) support continuous improvement at the site and program level. Our evaluation plan aligns the local and site-based evaluation plans with the Washington State mandated reporting requirements, goals, and objectives. A rigorous evaluation plan was developed which employed valid, reliable and psychometrically sound instruments to gather quantitative and qualitative data from multiple sources. Our work will include data analysis, summary results, discussion and strategic suggestions for future program improvement.

Project Leads