When Comparing School Results Can Be Misleading

Why school comparisons can mislead

Comparing school results seems straightforward: higher average test scores, better graduation rates, or a higher ranking must mean a better school. But research shows that these comparisons can be misleading because they often reflect who attends a school rather than how well the school teaches. As the University of Bristol researchers George Leckie and Harvey Goldstein found, the most high-profile league table measure—the percentage of children getting five A* to C grades at GCSE—’says more about the differences in schools’ intakes than it does about the differences in their quality’ (Leckie & Goldstein, 2009). The same caution applies to many school performance metrics used around the world, including in New York State.

Parents are often overwhelmed by the decision about choosing the right school for their children. The My School website in Australia, for example, provides national comparable data on NAPLAN results and demographic and financial reporting, and it is popular—over 1.4 million site visits in 2016. But as The Conversation noted, it should not be used as the sole guide to the best school for a child (The Conversation, n.d.). The same principle holds for any school data system: a single number or ranking rarely tells the full story.

Lessons from school league tables

League tables are rankings of schools based on one or more performance measures. They are published by governments and media in many countries, and parents are often encouraged to use them when choosing a school. But the Bristol research briefing explains why they can be highly misleading for school choice. First, raw performance measures are heavily influenced by a school’s intake. Schools that serve more advantaged students tend to have higher average scores, regardless of teaching quality. Second, value-added measures—which adjust for students’ prior achievement—are superior in that they attempt to isolate the school’s contribution, but they are derived from limited numbers of students per school and are therefore ‘so noisy that it is hard to statistically separate many schools’ performances from one another.’ Third, just as with stocks and shares, schools’ past value-added performance is not a reliable indicator of their future performance. What parents want to know is how a school will perform in the future when their own child attends, and past results cannot guarantee that (Leckie & Goldstein, 2009).

What test scores can and cannot tell you

In his book Measuring Up, Daniel Koretz warns of the dangers of using test scores to make judgments about schools, students, and teachers. He argues that two common misunderstandings about testing are: that scores on a single test tell us all we need to know about student achievement, and that this information tells us all we need to know about school quality (The Conversation, n.d.). Test scores are a narrow measure. They may be influenced by test preparation, student motivation, language background, and many other factors that are not under the school’s control. A high average score can mask wide variation within a school: some students may be thriving while others are struggling.

Commentators have also pointed to the broader problem with comparisons in education. As Peter Greene wrote in Forbes, comparisons can distort priorities and lead to misleading conclusions about school quality (Greene, 2019). When schools are ranked, the pressure to improve scores can lead to teaching to the test, narrowing the curriculum, and neglecting subjects and skills that are not measured.

How comparisons go wrong: intake, noise, and changing measures

Several specific mechanisms make school comparisons misleading:

• Intake differences. Schools serve different student populations. A school with a high proportion of students from low-income families, students with disabilities, or English language learners may have lower average scores even if it is highly effective. Comparing it to a school with a more advantaged intake is unfair without adjustments.
• Measurement error and noise. Test scores are estimates, not precise measurements. A school’s average can fluctuate from year to year because of random variation, a particularly strong or weak cohort, or changes in the test itself. Small schools are especially prone to large swings.
• Changing tests and standards. When assessments or proficiency cut scores change, comparisons across years become difficult. A school’s performance may appear to rise or fall simply because the test changed, not because teaching improved or declined.
• Suppressed or missing data. To protect student privacy, results for small subgroups are often suppressed. This means that a school’s overall average may hide important gaps or successes. Missing data can also distort comparisons if some students did not take the test.
• Selection effects. Some schools have selective admissions or specialized programs. Their high average scores may reflect who they admit rather than what they teach. Comparing them to open-admission schools is not apples-to-apples.

An illustrative example

The following example is illustrative and uses invented numbers. It does not describe any real school.

Imagine two schools in the same region. School A has an average test score of 80, while School B has an average of 70. At first glance, School A looks better. But suppose School A serves mostly students from affluent families who enter with strong prior achievement, while School B serves students who are more likely to be low-income and who enter with lower scores. A value-added analysis might show that both schools help their students make similar progress over the year. In that case, the raw average difference says more about intake than about school quality. Now imagine that School A’s average of 80 hides a wide gap: many students score above 90, but a significant group scores below 60. School B’s average of 70 might reflect more consistent performance across its students. A single average cannot capture these differences.

What the numbers can and cannot tell you

School performance data can be useful. It can tell you about the student population, overall trends, and how a school compares with others serving similar students. It can highlight gaps between groups of students. It can prompt questions. But it cannot tell you definitively whether a school is ‘good’ or ‘bad,’ whether your child will thrive there, or whether the teachers are effective. It cannot prove that a school caused higher or lower scores. It may be outdated by the time you read it. And it may not reflect the current cohort or recent changes in leadership or programs.

When you look at any school comparison, ask: What is the metric? What is the denominator? Who is included and who is missing? How does this school’s intake compare with others? Is the difference statistically meaningful, or could it be noise? What do multiple years show? What do growth or value-added measures show? What do students, parents, and staff say?

Questions for parents in New York

New York State publishes school report cards and other data through the New York State Education Department (NYSED). These can be a starting point, but they should not be the only factor in your decision. Here are questions to ask when comparing schools:

• What is the student population, and how does it compare with my child’s needs and background?
• How does this school perform compared with schools serving similar students?
• What is the trend over the past three to five years, not just the most recent year?
• Are there growth or value-added measures available, and what do they show?
• How do different groups of students—by income, race, disability, English proficiency—perform in this school?
• What is the school’s approach to teaching and learning? What programs and supports are available?
• How does the school communicate with families? What is the climate like?
• What do current parents and students say about their experience?
• Have I visited the school, observed classes, and met with staff?

Visiting a school and talking to people there remains one of the most reliable ways to find out whether it is a good fit for your child. Data can inform those conversations, but it cannot replace them.

New York context

In New York, as in other states, school accountability systems use multiple measures, including assessment results, graduation rates, and indicators of college and career readiness. The federal Every Student Succeeds Act (ESSA) requires states to identify schools for support and improvement based on these measures. But no single number or designation captures the full quality of a school. The same cautions about intake, noise, and measurement apply. Parents should use NYSED’s official school report cards as a starting point, look for contextual information, and combine data with visits and conversations. For current data and definitions, consult the New York State Education Department directly.

Sources and methodology

This article draws on three sources: The Conversation’s article on the My School website and NAPLAN data in Australia; Peter Greene’s Forbes article on the problem with comparisons in education; and a University of Bristol research briefing by George Leckie and Harvey Goldstein on school league tables. These sources are not specific to New York State; they illustrate general statistical and policy principles that apply to school comparisons in many contexts. The sources span 2009 to 2019. They were retrieved and checked on October 12, 2026. No New York-specific data was analyzed for this article. Readers should verify current New York data and policy with the New York State Education Department.