Requested 2020-21 report card recalculation shows little variance using previous method

MADISON — A request from the Assembly Committee on Education for the Wisconsin Department of Public Instruction to recalculate 2020-21 report cards using a previous method showed little change for a majority of school districts when compared to the actual report card calculation.

 

During a public hearing on January 12, 2022, the Assembly Committee on Education requested the DPI generate report card results for the 2020-21 school year by using the previous method, last used for the release of 2018-19 report cards. Actual school and district report cards released in November 2021 reflect statistically sound metrics and are the result of years of technical work and engagement with schools, districts, and other education stakeholders to improve score stability and usability for continuous improvement efforts.

 

Starting in 2019, the DPI worked extensively with an accountability advisory group of diverse educational stakeholders from across the state to make changes to the way report cards are calculated and formatted. Among the changes, the “Closing Gaps” priority area of the report card was replaced with the “Target Group Outcomes” priority area to make the measure of gap closure more reliable, inclusive, and actionable. Prior to receiving assessment data that are on the 2020-21 report cards, the DPI received advice from a Technical Advisory Committee, a group of national experts on accountability systems, to refine the Target Group Outcomes priority area and establish updated cut scores for designating star categories. TAC experts also supported using an equipercentile linking methodology to set new cut scores. This methodology is a strictly mathematical approach that emphasizes preserving the distribution from the previous cut scores (2018-19). The cut score adjustment to the rating category allowed schools, districts, and the public to make more accurate cross-year rating comparisons, although these comparisons were still complicated by the COVID-19 pandemic. See a summary of the changes for the 2020-21 report cards by visiting the DPI’s website here.

 

When results were recalculated using the data from 2020-21 with the previous method used in 2018-19 per the Assembly Committee on Education’s request, results showed similar star rating distributions for districts. The distribution of district- and school-level star ratings is similar, and ratings differed by no more than one star between the actual 2020-21 report cards and the recalculated results. Over 75 percent of schools and over 80 percent of districts had no rating change, and while there were slightly more negative than positive star rating shifts for districts, there were more positive than negative star rating shifts for schools (i.e., more district ratings went down using the 2020-21 business rules compared to the 2018-19 rules). In addition, private schools for both choice and all student report cards had more positive rating shifts and fewer negative rating shifts compared to public schools. Distributions of star ratings in both the 2020-21 report cards and the requested recalculation were similar to the 2018-19 report card star rating distribution, showing the DPI’s efforts to improve the usability of the report cards while maintaining system-wide continuity of the results were successfully achieved.

The recalculated results should not be used to draw conclusions about school and district performance in the 2020-21 school year. The DPI encourages caution when interpreting scores and ratings. Data files produced for this report can be found by visiting the DPI’s website here.