Purpose
The COVID-19 pandemic exacerbated the already common trend of course failure in high schools, especially for students from minoritized populations (Belsha, 2022; Borter & O’Brien, 2021; Gross, 2021; St. George, 2020; Thompson, 2020). High school administrators are increasingly focused on promoting pathways to graduation, including providing opportunities for students to recover credit and graduate on time (Clements, Stafford, et all, 2015; Clements, Zweig, et al., 2015). These credit recovery courses were historically carried out in face-to-face (F2F) settings (Cooper et al., 2000; Lauer et al., 2006) but are now commonly offered virtually through online credit recovery (OCR) options. In North Carolina, for example, OCR was as popular as F2F options by the 2016-2017 school year (Viano, 2021).
Research on this topic is mixed on the impact of OCR courses on student outcomes. Some quasi-experimental research found positive effects on graduation, but negative impacts on test scores, postsecondary enrollment, and future earnings (Hart et al., 2019; Heinrich et al., 2019; Heinrich & Darling-Aduana, 2021; Viano, 2021, 2023; Viano & Henry, 2024). Two additional studies used randomized control trials (RCTs) to assign students to OCR or F2F courses and found no differences in outcomes (Rickles et al., 2018, 2023). Given these mixed findings, it is possible that the differing effects are a result of different approaches in how these courses are implemented, although there is little data on how these decisions are made by school leaders. This study aims to close that gap.
Part of an NSF-funded research project, this mixed-methods study explores how high school administrators currently structure and implement OCR courses, and the impact of each approach on student outcomes. We address the following research questions:
1) How do school leaders structure OCR enrollment, administration, and engagement?
2) To what extent are these different approaches associated with differential outcomes for students who enroll in OCR?
3) To what extent are these different approaches associated with differential outcomes for students who fail courses in general?
Framework
To explore school leaders’ approaches to the administration in a comprehensive way, this study relies on multiple theoretical perspectives. We incorporate the frameworks of both new institutionalism in education and sociocultural theory to construct and analyze the measures used.
From the perspective of new institutionalism in education (Burch, 2007; Meyer & Rowan, 2006), the preferences of the actors that guide the institution are what determines how educational institutions are configured. Beliefs about efficiency, values, and culture are what guide the way administrative decisions are made (Burch, 2007). According to this framework, when designing their OCR offerings, these decisions are influenced by administrator beliefs and incentives that exist in the current educational system. In our study we explore the values and cultural beliefs that guide administrators’ decision-making around OCR.
The second framework we incorporate in our analysis is sociocultural theory, which will help us explore how school leaders structure their OCR programs to support student engagement (Borup et al., 2020; Rogoff, 2003; Vygotsky, 1980; Wertsch, 1998). By conceptualizing these measures, we will be able to explore how school leaders’ beliefs and expectations of OCR students facilitate student engagement in these settings.
Data Sources
We completed our data collection over the course of the 2024-25 school year in a large, diverse, suburban public school district in the mid-Atlantic region. We began with qualitative, semi-structured interviews with school administrators and teachers involved with credit recovery programs, followed by in-person observations of OCR classrooms. We conducted 25 interviews and carried out 33 observations of OCR rooms across nine high schools in our study. We also gathered additional data on student engagement from the platform used by the district in their OCR courses, as well as district administrative data on student outcomes.
For the interviews, we met with staff in charge of OCR programs, including assistant principals, counselors, graduation coaches, and teacher leaders, as well as any teachers or teaching assistants involved in the supervision of rooms in which students worked on their OCR course while in-person. In these interviews, we explored the beliefs and values held by administrators that guided the structuring and implementing of credit recovery programs, including how and for whom they were structured, and what factors drove them to implement OCR versus traditional F2F recovery courses. We also explored OCR teachers’ beliefs about the needs of their students, how they support engagement, and their expectations of student behavior in these courses.
We then conducted in-person observations of OCR classrooms to gather data on student engagement and teacher supports provided in real time. The protocol consists of quantitative measures of student on- and off-task behaviors from sweeps we conducted in fifteen-minute increments, combined with running qualitative notes on the supports provided by teachers that promote affective, cognitive, or behavioral engagement. Finally, we also recorded post-observation notes that captured additional observations of the classroom environment and setup.
These interviews and observation data will be combined with data from the OCR platform itself on student engagement and course outcomes as well as additional data on student outcomes from the partnering school district’s administrative data system. From the platform, we have measures of student engagement including the time spent logged-in, the number of assignments they completed or mastered, as well as their assessment scores and course grades. From the partner district we have attendance data, course completion status, graduation status, exam scores, as well as baseline level data on prior course failures, prior test scores, and past disciplinary referrals. This comprehensive set of data will allow us to use the following methods to address our research questions.
Methods
Research question 1: Credit recovery structure and administration
We use multiple cycles of qualitative analysis to answer our first research question and employ structural and magnitude coding processes to analyze our interviews and qualitative notes. Structural coding is used to break up transcripts of data into categories according to the perspectives identified by our conceptual frameworks, and magnitude coding quantifies these categories into data suitable for quantitative analysis as part of our mixed-methods approach (Guest et al., 2012; Miles et al., 2014; Saldaña, 2013).
We began with structural coding and are currently in the process of developing an initial codebook. This codebook will guide our structural coding of the data and will be continually updated with new codes and definitions as necessary (Guest et al., 2012). We will then use magnitude coding to convert these data codes into numeric indicators that can be quantitatively analyzed in the subsequent part of our analysis.
Research questions 2 and 3: Differences in outcomes across schools
The next part of our study will use the indicators we developed in the first research question to assess whether differences in approaches result in differences in student outcomes across schools. To do so, we will use two multi-level regression models where student data (level 1) is nested within school-level data (level 2) from our sample for each model. Both models will assess the association between our predictors and student outcomes. Predictors include the typologies of approaches to credit recovery we identified earlier along with covariates. Student outcomes include attendance, course credit earned, subsequent course enrollment/completion, standardized exam scores, and graduation.
The first multi-level regression models we use will be restricted to students enrolled in OCR courses. This is to be able to compare outcomes for OCR-enrolled students across schools based on how these programs are implemented. The second model will evaluate if any association exists between the way OCR courses are administered and differential student outcomes for all students who failed classes, regardless of OCR-enrollment. This is to understand if OCR programs with better results are also better for students who failed courses overall.
In addition to these models, we will also include machine learning (ML) techniques that generally result in more robust findings than our traditional models, including k-nearest neighbors, Ridge, Lasso, and random forest regressions. These ML methods can be more accurate than traditional ones when analyzing larger, more complex data and can better identify non-linear relationships (Kyriazos & Poga, 2024).
Results
We are currently in the early stages of data analysis process, which will be completed by Spring 2026. However, we have some initial findings to share regarding OCR structures, including enrollment, administration, and supporting student engagement.
Enrollment
Several themes emerged regarding how school leaders enrolled students in credit recovery. Most offer OCR to upperclassmen, and especially seniors, to help them remediate courses they failed in order to get back on-track as they approach graduation. Some also offered OCR enrollment to underclassmen with substantial missing credits, or in cases where they transferred in with missing credits. School leaders identified the following characteristics in their descriptions of the typical credit recovery student: those with attendance issues due to out-of-school obligations, English learners, and students with behavioral infractions.
Administration
School leaders administered their OCR programs using a variety of approaches. Around half of schools we sampled offered OCR through dedicated courses during the school day, about a third of schools allowed their OCR students to work independently in the program throughout the day, while the rest offered after school or Saturday classes for OCR remediation. Most schools also allowed students to work on the OCR platform outside-of-school.
In schools where OCR was offered as a class during the day, students were assigned to classrooms, computer labs, or library spaces and were supervised by teachers or teachers’ assistants. All courses had teachers-of-record, although only two of the schools sampled had one that was also certified in the content matter. When working independently during school hours, students were often out by administrators or teachers to work on their OCR platform in an available space.
Engagement
An initial review of the data revealed some trends in the way school leaders and staff in charge of credit recovery support student engagement. Some examples of strategies they used to support behavioral engagement include individual check-ins for goal setting and progress monitoring, tracking what students are working on through screen-monitoring software, or accountability measures such as removal from the program or calls home.
School leaders and staff also shared strategies they used to support students’ affective and cognitive engagement. They provided affective support via individual check-ins or praise, or by celebrating progress made with certificates or celebratory messages. Some schools supported cognitive engagement in OCR courses by scheduling content teachers to be present in the OCR space, or at least made them available for student consultations during these times.
This mixed-methods, NSF-funded study contributes substantially to both knowledge on best practices for educational administration and instructional technology. Specifically, it will identify which type of OCR approach yields the most substantial improvement in outcomes for students who fail courses and are at risk of not graduating. The end result of this work will provide school leaders examples of best practices around structuring and administering OCR programming. It will also offer staff in charge of OCR spaces some strategies to improve student engagement in these classes.
This study will also add more empirical evidence to fill gaps in the OCR literature about how credit recovery programs are structured and their effectiveness. There is currently a dearth of knowledge about how school leaders run their OCR programs. This study collects much-needed qualitative data that highlights some common approaches administrators use and uses that to quantitatively assess the impact of each approach on student outcomes. The results of this study has the potential to transform how high school leaders structure their credit recovery programs in ways that improve student outcomes in high school and beyond.
Belsha, K. (2022, May 12). ‘We need to double down’: After a surge in ninth graders held back, schools step up support. Chalkbeat. https://www.chalkbeat.org/2022/5/12/23068627/ninth-grade-retention-credit-recovery-pandemic
Borter, G., & O’Brien, B. (2021, March 29). Another danger for kids in the age of COVID: Failing grades. Reuters. https://www.reuters.com/article/us-health-coronavirus-usa-students-insig-idUSKBN2BL1BF
Borup, J., Graham, C. R., West, R. E., Archambault, L., & Spring, K. J. (2020). Academic Communities of Engagement: An expansive lens for examining support structures in blended and online learning. Educational Technology Research and Development, 68(2), 807–832. https://doi.org/10.1007/s11423-020-09744-x
Burch, P. (2007). Educational Policy and Practice From the Perspective of Institutional Theory: Crafting a Wider Lens. Educational Researcher, 36(2), 84–95. https://doi.org/10.3102/0013189X07299792
Clements, M., Stafford, E., Pazzaglia, A. M., & Jacobs, P. (2015). Online course use in Iowa and Wisconsin public high schools: The results of two statewide surveys (REL 2015–065). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest. https://www.ies.ed.gov/ncee/edlabs/regions/midwest/pdf/REL_2015065.pdf
Clements, M., Zweig, J., & Pazzaglia, A. M. (2015). Online course use in New York high schools: Results from a survey in the Greater Capital Region (REL 2015–075). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast & Islands. http://files.eric.ed.gov/fulltext/ED555633.pdf
Cooper, H., Charlton, K., Valentine, J. C., Muhlenbruck, L., & Borman, G. D. (2000). Making the most of summer school: A meta-analytic and narrative review. Monographs of the Society for Research in Child Development, i–127.
Gross, B. (2021). Credit recovery isn’t enough: How to manage a surge of failing course grades (The Evidence Project). Center for Reinventing Public Education.
Guest, G., MacQueen, K. M., & Namey, E. E. (2012). Applied thematic analysis. SAGE Publications.
Hart, C. M. D., Berger, D., Jacob, B., Loeb, S., & Hill, M. (2019). Online Learning, Offline Outcomes: Online Course Taking and High School Student Performance. AERA Open, 5(1), 1–17. https://doi.org/10.1177/2332858419832852
Heinrich, C. J., & Darling-Aduana, J. (2021). Does online course-taking increase high school completion and open pathways to postsecondary education opportunities? Educational Evaluation and Policy Analysis, 43(3), 367–390. https://doi.org/10.3102/0162373721993485
Heinrich, C. J., Darling-Aduana, J., Good, A., & Cheng, H. (Emily). (2019). A look inside online educational settings in high school: Promise and pitfalls for improving educational opportunities and outcomes. American Educational Research Journal, 56(6), 2147–2188. https://doi.org/10.3102/0002831219838776
Kyriazos, T., & Poga, M. (2024). Application of machine learning models in social sciences: Managing nonlinear relationships. Encyclopedia, 4(4), 1790–1805. https://doi.org/10.3390/encyclopedia4040118
Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school-time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76(2), 275–313.
Meyer, H.-D., & Rowan, B. (2006). Institutional analysis and the study of education. The New Institutionalism in Education, 1–13.
Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (Third edition). SAGE Publications, Inc.
Rickles, J., Clements, M., Brodziak de los Reyes, I., Lachowicz, M., Lin, S., & Heppen, J. (2023). A multisite randomized study of an online learning approach to high school credit recovery: Effects on student experiences and proximal outcomes. Journal of Research on Educational Effectiveness, 17(3), 467–490. https://doi.org/10.1080/19345747.2023.2198524
Rickles, J., Heppen, J. B., Allensworth, E., Sorensen, N., & Walters, K. (2018). Online Credit Recovery and the Path to On-Time High School Graduation. Educational Researcher, 47(8), 481–491. https://doi.org/10.3102/0013189X18788054
Rogoff, B. (2003). The cultural nature of human development. Oxford university press.
Saldaña, J. (2013). The coding manual for qualitative researchers (2nd ed). SAGE.
St. George, D. (2020, December 3). Failing grades double and triple—Some rising sixfold—Amid pandemic learning. Washington Post. https://www.washingtonpost.com/local/education/montgomery-county-failing-grades/2020/12/03/913affd0-34fb-11eb-8d38-6aea1adb3839_story.html
Thompson, C. (2020, December 6). Schools confront “off the rails” numbers of failing grades. AP NEWS. https://apnews.com/article/distance-learning-coronavirus-pandemic-oregon-7fde612c3dbfd2e21fab9673ca49ad89
Viano, S. (2021). A choice between second chances: An analysis of how students address course failure. American Journal of Education, 128(1), 29–58.
Viano, S. (2023). Online credit recovery school-level enrollment: Intended and unintended consequences. Online Learning, 27(2). https://doi.org/10.24059/olj.v27i2.3331
Viano, S., & Henry, G. T. (2024). Online credit recovery as an intervention for high school students who fail courses. Educational Policy, 38(1), 218–253. https://doi.org/10.1177/08959048231153597
Vygotsky, L. S. (1980). Mind in society: The development of higher psychological processes. Harvard university press.
Wertsch, J. V. (1998). Mind as action. Oxford university press.
Back
Trips and Tours
