Blog written by Femi Vance, Senior Researcher, American Institutes for Research
Blog contributors Celine Provini, Ph.D., Director of Research, Annick Eudes Jean-Baptiste, Ph.D., Research Manager, Suzette Harvey, President/CEO, Prime Time Palm Beach County
Blog published June 13, 2022, on AIR website Making Measurement Meaningful | American Institutes for Research (air.org)

Making Measurement Meaningful

The COVID-19 pandemic has underscored the importance of collecting and using data with intentionality—especially when it comes to the well-being of young people. The Every Hour Counts Measurement Framework can help out-of-school time (OST) system leaders leverage data to improve the lives of young people.

The Measurement Framework, which is accompanied by a guidebook, is the first of its kind in the OST field and we wanted to learn how OST intermediaries are putting this rich resource to use, particularly to promote racial equity. We invited Prime Time Palm Beach County (Prime Time), a member of the Every Hour Counts network, to share how they have used the Measurement Framework. Prime Time is a nonprofit intermediary that provides resources and supports for OST professionals to develop programs that inspire children to thrive. During 2020-2021 Prime Time offered nearly 140,000 learning experiences that impacted roughly 50,000 youth. The Prime Time team shares how they have used the Measurement Framework to collect and analyze data to advance equity-centered practices and policies.
 

What is motivating you to use the Measurement Framework now?

Given all the challenges we and the OST programs we work with in Palm Beach County have experienced during the pandemic, we thought it was a great time to take stock of our processes, including those related to evaluation. We are currently using the Measurement Framework to assess our evaluation processes. Using the Framework, we can apply an equity lens to our work, which is one of our organizational priorities. The dimensions of equity that we are particularly interested in exploring are race/ethnicity, gender, geographic location, and level of community socioeconomic need. In the near future, we plan to use our data to identify any gaps or unmet needs in access to scholarships for staff professional development or access to enrichment opportunities for youth. We are curious to learn if the size of those gaps is related to any community characteristics (geography, socioeconomic need level, racial/ethnic composition, etc.).
 

Many folks find it daunting to use new tools such as the Measurement Framework. How did you start to incorporate the Measurement Framework into your process for continuous improvement?

The Framework has informed our continuous improvement process for years, with the Framework’s system-level outcome 3 (“Afterschool programming community engages in continuous quality improvement”; page 11) serving as the cornerstone of our quality work with OST programs. Prime Time has put this goal into practice by maintaining a strong, voluntary network of OST programs that truly embrace continuous quality improvement. One of the key reasons for the network’s success is that we have cultivated a supportive and nonjudgmental space where programs set their own goals in response to assessment data.

We are currently investing in the first indicator of the system-level outcome 3, which states “System provides high-quality professional development, including workshops, coaching and facilitated peer learning.” We are working on improving the quality of our professional development and better understanding the impact of those opportunities. For example, the recently updated quality assessment tool we use, the Palm Beach County Program Quality Assessment (PBC-PQA), raises expectations around staff members’ social and emotional (SEL) skills, so we are revising the content of our professional development offerings to help staff build that skill set. We also decided to start assessing the quality and impact of professional development using Kirkpatrick’s four-level model, which includes (1) reaction, (2) learning, (3) behavior, and (4) results.

The Measurement Framework was especially helpful when we asked our funders to support increased staffing that would enable Prime Time to expand research and evaluation activities. It represents the best thinking of researchers and experts in the field of OST and offers a robust vision for evaluation as well as a clear path to achieving that ideal. Our funders found that compelling! While the breadth and depth of the Framework can be daunting, we view the tool as presenting a series of options to explore, understanding that it may be feasible to prioritize only a few options at a given time.
 

What has using the Measurement Framework helped your organization to do?

The Framework has reinforced the importance of gathering input from the wider community when (1) planning and evaluating our services as an OST intermediary and (2) helping OST programs ensure high-quality services for youth. One way we gather community feedback by participating in Palm Beach County’s Birth to 22: United for Brighter Futures collective impact initiative, which includes working groups focused on OST and social-emotional learning. The workgroup members include youth-serving agencies, school districts, youth, and parents/caregivers. These workgroups host annual community conversations to solicit input and promote leadership from multiple stakeholders. Outside of Birth to 22, Prime Time has conducted parent/caregiver surveys and focus groups to learn more about how they and their children experience our services, as well as how they perceive the benefits and accessibility of OST programs. We would like to engage in these types of perspective-gathering more frequently.
 

Looking forward, what more do you hope your organization can accomplish with continued use of the Measurement Framework and tools?

Prime Time will use the Framework to guide new data collection efforts and new types of analyses that help us uphold our strong organizational commitment to equity. So far we have:

• Added more gender and race/ethnicity categories to our database to be more inclusive.
• Started using a three-level community need indicator (low, medium, high) for each of our OST programs. The indicator aligns with county-wide rankings of community risk that are based on factors such as poverty and child abuse rates.
• Planned how we will use our data to identify gaps. We will use our community need indicators to see whether our services (e.g., coaching, expanded learning opportunities, scholarships, event participation) are equitably delivered to various adult and youth populations in different geographic locations. We will also determine if access to quality programming is different based on programs’ community need level, geographic location, racial/ethnic makeup of participants, level of youth enrollment, and more.
   

Lessons Learned

The examples that Prime Time shares illuminate how complex it can be to collect and use data with intentionality. The Measurement Framework can be a starting place for any OST program or intermediary. For Prime Time, employing the Measurement Framework has resulted in more effective strategies for advancing organizational priorities, strengthening evaluation capacity, and bringing equity to the forefront of data collection and analyses. To learn more about the available resources, please visit the Wallace Foundation Measurement Framework and Guidebook resource page.