NCLB’s Statistical Smoke and Mirrors

An interesting article in the July 25th Wall Street Journal by Charles Murray should be on all the minds of those who some how believe that NCLB is making a difference and that there is solid evidence to prove this to be the case. All should review the somewhat laborious reading of Murray’s walk down the laws of statistical analysis. Murray identifies the Civil Right Project of Harvard University which studied NCLB’s impact as proof that the law is not having an impact. Charles Murray states

“The case that NCLB has failed to raise test scores had been made most comprehensively in a report from the Civil Rights Project at Harvard University, released just a few weeks ago. The Civil Rights Project has an openly liberal political agenda, but the author of the report, Jaekyung Lee, lays out the data in graphs that anyone can follow, subjects them to appropriate statistical analyses, and arrives at conclusions that can stand on their scholarly merits: NCLB has not had a significant impact on overall test scores and has not narrowed the racial and socioeconomic achievement gap.” See the link copy of the Report Tracking Achievement Gaps and Assessing the Impact on the Gaps http://www.civilrightsproject.harvard.edu/

Gary Orfield’s forward to the Civil Rights Project report states that “this report concludes that neither a significant rise in achievement, nor closure of the racial achievement gap is being achieved. Small early gains in math have reverted to the preexisting pattern. If that is true, all the pressure and sanctions have, so far, been in vain or even counterproductive.

Orfield known for his concern for racial equity reinforced the conclusions of the Harvard Study “As a leader in a research project concerned about issues of racial equity, I believe that if there were evidence that these things were actually being accomplished it would be very important what ever one thought about some of the means being used to attain them. Unfortunately, these claims rest on misleading interpretations of flawed data as demonstrated in this new report.” …. Since the policy is little more than a theory about how to force change without any grounding in specific educational approaches or targeted resources to ensure that effective programs and supports are put into place ( except for the special early reading programs), then if it does not succeed in improving scores on NAEP, it certainly can not be justified.”

Murray like Orfield would hold out for NCLB if it were making a difference in test scores even in the face of significant evidence that it is fundamentally damaging the structure of public education. But it is not making a difference.

“NCLB takes a giant step toward nationalizing elementary and secondary education, a disaster for federalism. It pushes classrooms toward relentless drilling, not something that inspires able people to become teachers or makes children eager to learn. It holds good students hostage to the performance of the least talented, at a time when the economic future of the country depends more than ever on the performance of the most talented. The one aspect of the act that could have inspired enthusiasm from me, promoting school choice, has fallen far short of its hopes. The only way to justify NCLB is through compelling evidence that test scores are improving.”

In discussing the Department of Education’s smoke and mirrors’ method of data analysis
Charles Murray in his Wall Street Journal opinion piece explains http://www.opinionjournal.com/editorial/feature.html?id=110008701
“Those numbers (The Department of Education’s data on the effectiveness of NCLB) will consist largely of pass percentages, not mean scores. A particular score is deemed to separate "proficient" from "not proficient." Reach that score, and you've passed the test. If 60% of one group--blondes, let's say--pass while only 50% of redheads pass, then the blonde-redhead gap is 10 percentage points.
A pass percentage is a bad standard for educational progress. Conceptually, "proficiency" has no objective meaning that lends itself to a cutoff. Administratively, the NCLB penalties for failure to make adequate progress give the states powerful incentives to make progress as easy to show as possible. A pass percentage throws away valuable information, telling you whether someone got over a bar, but not how high the bar was set or by how much the bar was cleared. Most importantly: If you are trying to measure progress in closing group differences, a comparison of changes in pass percentages is inherently misleading.”

Utilizing what every beginning student in statistics understands about the normal distribution of differences and what most college students learn after their first exam, there is something called the Bell Curve, Murray’s first point is “that using easy tests and discussing results in terms of pass percentages obscures a reality that NCLB seems bent on denying: All the children cannot be above average. They cannot all even be proficient, if "proficient" is defined legitimately. Some children do not have the necessary skills.” There are other distortions created in the use of pass percentages summarized and illustrated in Murray’s piece which any individual interested education policy should know. For an administration that has put so much energy into scientific based methods we find more often then not in the case of NCLB scientific based approaches are not being used and are not documenting success, in the case of this statute it is just plain non-sense. What a the Harvard study and a Wall Street journal opinion writer have concluded, those who testified at the NIEA hearings on NCLB have understood as just plain common sense.
http://www.niea.org/issues/policy_detail.php?id=17
Relevant links
The National Indian Education Association NIEA
http://www.niea.org/
Harvard Civil Rights Project
http://www.civilrightsproject.harvard.edu/
Wall Street Journal http://online.wsj.com/public/us

Kitchizibi-innini