OKLAHOMA CITY (Aug. 20, 2013) – The Oklahoma State Department of Education announced today that the independent study to determine if student tests scores had been compromised by the two days of disruption this past spring is complete.
The HumRRO Statistical Investigation of Oklahoma Disruptions indicated that students appeared to do as well on the test as students not involved in the disruption. Based on the result of this study, State Superintendent Janet Barresi determined the state will retain all scores of impacted students who scored proficient or advanced.
While students as a whole did not experience depressions in scores, it is possible that some individual students did not perform to their highest potential during the period of disruption. Therefore, Superintendent Barresi will not report the scores of impacted students who scored limited knowledge or unsatisfactory.
The full report was provided by Dr. Arthur Thacker, Program Manager for Human Resources Research Organization (HumRRO), Inc. to the department and delivered during Tuesday’s State Board of Education’s meeting by Dr. Maridyth McBee, Assistant State Superintendent of Accountability and Assessment.
“I’d like to thank Dr. Thacker and HumRRO for conducting this study,” said State Superintendent Janet Barresi. “Last year’s testing period was very difficult for our students, teachers and testing coordinators. The difficulties they experienced were unacceptable. It was a high priority for everyone to commission an independent study to determine what effect the disruptions had on the student test scores. Now that the study has concluded, we will work with the districts to take action on behalf of students and schools where the disruptions merit such action.
“Even though this study suggests no systematic impact on test scores, not reporting the scores of students who scored limited knowledge or unsatisfactory will ensure there is no lasting impact on student performance. This is the right thing to do for students and for schools.”
Server capacity problems by testing vendor CTB/McGraw-Hill on April 29 and 30 caused a number of students to be interrupted during their testing experience, making it necessary to determine the extent to which the disruptions impacted students’ test scores. The State Department of Education asked HumRRO to conduct a third-party independent study to investigate the impact.
HumRRO was selected by the SDE because the organization is recognized for their objectivity and independent approach to evaluating the performance of testing vendors. They have also conducted similar work for the State Department of Education during a previous challenge with a different testing coordinator.
A state Technical Advisory Committee of researchers and assessment experts who meet twice per year to advise the State Department of Education on accountability and assessment issues recommended the design for HumRRO’s disruption study. HumRRO conducted the study independently of the SDE and CTB/McGraw-Hill.
Studies conducted for Indiana by another organization found similar results, as did a study conducted for Minnesota by HumRRO.
“The findings of the report in relation to the discrepancies did not merit action in HumRRO’s opinion,” said Superintendent Barresi. “Because the two-day incident resulted in so much stress and chaos of the school day I wanted to assure students, teachers and schools were not held accountable for circumstances beyond their control.”
Additionally the following actions were taken to address discrepant scores:
About 1,400 students received an immediate raw score that differed from the score they received on the two-week preliminary test score report.
The discrepancy occurred for two reasons: 1) There appeared to be a short time period on the second day of the interruptions when some student responses to test questions were not saved and therefore not included on the two-week report. 2) CTB inadvertently left the “winter test system” as an option for download which caused incorrect data to be saved.
Corrective action will include:
Students will receive the higher score between the raw score and the two-week report score.
Students with higher raw scores will receive a letter from CTB indicating their proficiency level. They will be included in the report card, but not in the school or district summary report.
Students with a higher two-week score will receive a traditional score report that includes performance level and performance by content standard.