Event Information
Blended ContentOther presentations in this group:
Mathematics education remains a critical focus for national education improvement efforts, particularly in light of the learning loss due to the COVID-19 pandemic. Research has shown that independent practice and teachers’ use of resulting data to inform instruction are crucial for student learning (Irons & Elkington, 2021). However, traditional, paper-based independent practice has limitations: when students complete their work, they may make errors and practice incorrectly, and teachers have limited time to analyze students’ progress and adjust their subsequent instruction (Mendicino, et al., 2009).
Educational technologies provide opportunities to address these challenges by providing independent practice for students and analytics of formative assessment data for teachers. Established in 2004, ASSISTments, an online math platform, was specifically designed to provide independent practice opportunities for students and formative assessment (Heritage & Popham, 2013) data analytics for teachers.
Purpose/Research Questions
The purpose of this study is to understand the fidelity of implementation and the impact of the virtual professional learning community (vPLC)-augmented ASSISTments platform on middle school students’ math achievement. The following research questions motivate the study:
• What is the effect of using the ASSISTments platform for independent math practice and formative assessment on the math achievement of middle school students (6th - 8th grades)?
• Do participating teachers implement ASSISTments as intended by the developer?
• What are the effects of implementation fidelity on student learning?
Settings / Population
The study is taking place in two cohorts in a diverse sample of 39 schools across 24 states/district in the United States, involving 65 middle school teachers and their math classrooms. Sixteen treatment teachers participated in the first cohort of the study during the 2022-23 school year. A second cohort of 49 teachers is participating during the 2023-24 school year. Participants were selected through a nation-wide recruitment effort. Eligible study participants are 6th, 7th, or 8th grade math teachers who are able and willing to (1) use ASSISTments in their classes three times per week, (2) participate in a virtual Professional Learning Community eight times throughout the school year, (3) administer the NWEA MAP Growth Assessment to students twice, and (4) participate in data collection activities, including surveys, instructional logs, interviews, and classroom observations. Overall, 20 schools are rural, 24 are eligible for Title I. On average, 19% of students are Hispanic and 13% are black. 52% students are eligible for free reduced lunch. Enrolled teachers tend to be from schools that experienced a significant drop in student math performance in the past few years and are interested in boosting math learning using technology.
Intervention
Prior studies have demonstrated the effectiveness of ASSISTments on student learning when using a visiting coach training method that includes face-to-face workshops and in-person classroom visits to support teachers’ implementation (Roschelle et al., 2016; Feng et al., 2023; Sahni et al., 2021). As a part of an Education Innovation Research (EIR) grant, the ASSISTments program is augmented with a vPLC to facilitate training and teacher-to-teacher discussions about math practices and pedagogy. As a key component of the intervention and as a scale-up mechanism, vPLC removes barriers associated with travel and scheduling of coaching.
Teachers will first receive professional development via vPLC that addresses how teachers can use ASSISTments reports to adapt and differentiate instruction, create consistent routines for students, and engage in meaningful discussion about the data. Then, teachers will follow the four steps of formative assessment practices: 1) Create math assignments aligned with grade-level standards, 2) Encourage students to complete assignments independently and use immediate feedback, hints, and explanations to support problem-solving, 3) Obtain reports that summarize individual and class performance and common wrong answers, 4) Analyze the anonymous data with the class, discuss common errors, and customize assignments.
Research Design
This study uses a matched quasi-experimental design: Students using ASSISTments are compared with a virtual comparison group (VCG) drawn from a national testing database. The comparison group will be identified (matched) based on the NWEA MAP Growth Math Assessment administered to the treatment students at the baseline in the Fall of 2022 (for Cohort 1) or the Fall of 2023 (for Cohort 2). For each student in the treatment group, NWEA will identify matched non-treatment group students in their database based on school locale, school level percentage of students eligible for free-or-reduced priced lunch, test-taking dates, student's grade level, and pre-test scores. The mean of the post-test scores of up to 51 random samples of the matched students is considered a “virtual comparison” score that will be used in estimating impact.
Data Collection and Analysis
Outcome measure. The study measures student math achievement with the online MAP Growth Math Assessment provided by NWEA. It generally takes students about 45 minutes - one hour to complete the test. MAP assessments have demonstrated reliability and validity (NWEA, 2019). The test is administered twice a year, once in the fall and once in the spring, to all students in the treatment group.
Fidelity of Implementation. Fidelity of implementation includes four key components: 1) Teacher attendance in vPLC sessions and completion of practice assignments, 2) Teachers making assignments in ASSISTments consistently, 3) Teachers reviewing reports regularly to support data-driven adaptation, and 4) Students completing assigned problems to receive immediate support.
Program Implementation. Teacher interviews and classroom observations provide information on the variation in implementation, the influence of ASSISTments on teachers’ practices in classrooms, and barriers and facilitators. Teacher day-to-day math instruction practices are captured with instructional logs. An end-of-study survey asks teachers to reflect on their overall experience and provide feedback.
Preliminary Findings
We used a partially nested hierarchical linear model with treatment students nested in classrooms to analyze the impact of the intervention on cohort 1 students’ math performance. The results indicated that the first cohort of participants outperformed the virtual comparison group on MAP assessment, although the difference was not statistically significant. Within the treatment group, 60.68% of students maintained or improved their math performance to above the 50th percentile on the MAP assessment.
Descriptive analysis of the implementation fidelity data indicated that the program met the fidelity threshold for components 1, 2, and 3 but the fidelity was relatively low for component 4. Researchers used the percentage of students who completed at least 68% of the problems assigned to them as the class level indicator for component 4. In cohort 1, 72% classes met the threshold for problem completion, which is not slight lower than the program-level threshold (75%) for Component 4.
Analysis for cohort 2 is underway and will be fully competed and shared at the conference.
Abundant visions exist for using technology to improve mathematics outcomes and help students recover from the learning loss caused by the pandemic. Educational technology programs are becoming widely available, and as they go to scale, it is important to measure their impact on teaching and learning. This study investigates the potential effectiveness of an educational technology intervention that attends to both mathematics learning outcomes and the how educator participation in a professional learning community might affect technology usage and outcomes. The findings from the study will add to the evidence base and inform the adoption and classroom practices of implementing similar educational technology programs.
Bennett, R. E. (2011). Formative assessment: a critical review. Assessment in Education: Principles, Policy & Practice, 18, 5–25.
Feng, M., Huang, C., & Collins, K. (2023). Promising Long Term Effects of ASSISTments Online Math Homework Support. In Proceedings of International Conference on Artificial Intelligence in Education, A Late Breaking Result. pp. 212-217. Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-36336-8_32
Heritage, M., & Popham, W. J. (2013). Formative assessment in practice: A process of inquiry and action. Cambridge, MA: Harvard Education Press.
Irons, A., & Elkington, S. (2021). Enhancing learning through formative assessment and feedback. Routledge.
Mendicino, M., Razzaq, L., & Heffernan, N. (2009). A comparison of traditional homework to computer-supported homework. Journal of Research on Technology in Education, 41(3), 331.
NWEA. (2019). MAP® Growth™ technical report. Portland, OR: Author. Retrieved from https://www.nwea.org/uploads/2021/11/MAP-Growth-Technical-Report2019_NWEA.pdf
Roschelle, J., Feng, M., Murphy, R., & Mason, C. (2016). Online mathematics homework increases student achievement. AERA Open Journal. 2(4). https://doi.org/10.1177/2332858416673968
Sahni, S. D., Polanin, J. R., Zhang, Q., Michaelson, L. E., Caverly, S., Polese, M. L., & Yang, J. (2021). A what works clearinghouse rapid evidence review of distance learning programs. US Department of Education.
Back
Trips and Tours