Counting, measurement, and the Fraser Report

This year’s Fraser Institute report is in.

For those of you unaware of what this means, the Fraser Institute is a private Canadian “think-tank” that concerns itself with education and statistics, attempting to create “a free and prosperous world through choice, markets, and responsibility” by ranking Canadian schools.

Every year, the Fraser Institute releases “report cards” for both elementary and secondary schools, assessing high schools on “diploma examination results, grade-to-grade transition rates, and graduation rates.” All this data is carefully collated and schools are given a score out of ten. My school’s staff room is in a tizzy because our school is not just in the top 20 out of 276 schools in Alberta, but we’ve beat out every other school in our division by 70 points or more.

I looked at what made up this year’s score, and what I’ve discovered has made me laugh, in some cases literally out loud. To put things into perspective, our graduating class last year was 24 students. That’s right: our prestigious score was based on a massive sample size of 24 students. In the top 10 schools, one school had a graduating class of 736, and the smallest had 26. I didn’t study statistics in school, but I do know a thing or two about sample size.

That aside, the reason our school did so well last year was because we had three exceptional students who did extremely well on diploma exams, the final matriculating exam here in Alberta worth 50% of a course’s mark. I went through the data, and the English 30-1 average score went up by – wait for it – three per cent. Three. The average Biology 30 score went up by about five per cent. These gains are repeated through most of the core subjects. Now, in a graduating class of 24, all it takes is for a couple of students to do exceptionally well for these scores to be skewed. If smaller schools can get high results based on small class sizes, how can larger schools compete? If a school is located in a poor neighbourhood in the province, how can they compare with, say, private schools in wealthy suburbs, especially if the differences in these diploma exam results are counted in single percentage points?

The Fraser Institute claims that “[p]arents can use the school report cards to compare the academic performance of schools when choosing a school for their children.” I would contend that the academic performance of our school has remained exactly the same over the past five years, but because our school has so few students, our “report card” fluctuates quite a bit. But a parent seeing these results might wrongly believe that we’re doing something different and special at our school, and remove their students from excellent neighbouring schools and place them at our own.

Can schools really be chosen by diploma exam results? Would any thoughtful parent really take that into account when placing their children at a school? As a parent, I’m more concerned about the atmosphere at a school, whether students are treated with respect and given the freedom to pursue their own learning. I’m more interested in the fine arts programs and the extracurricular opportunities afforded to students. It would seem entirely unfortunate to rate an entire school based on the performance of last year’s graduating class, especially given the tremendous number of variables used to calculate these report card scores. (I did take algebra in high school and as such I’m able to offer the dim recollection that a problem with two variables can’t be conclusively solved; there’s no single answer to the question “x + y + z = 19”.)

At the top of the search page, the following warning is printed in red: “Thomas Lukaszuk, Alberta’s Deputy Premier, has called our school performance reports a misuse of provincial test results. He claims they mislead you and he wants to redesign the tests so we can no longer produce the reports. Our school performance reports are valid and used by many thousands of thoughtful parents each year.” I completely agree with the Hon. Deputy Premier: these reports do mislead parents, and based on what I’ve seen, I don’t know how the Fraser Institute can claim that their score is valid.

Three excellent students were enough to skew our report card upwards some 60 points. That’s all it took. At the top of the report card page, the Fraser Institute opines “If it matters… measure it.” With all due respect to this institution, I completely disagree. William Bruce Cameron’s oft-quoted (and oft misattributed) adage about measurement is applicable here, and I’ll leave it at the end of this musing: “Not everything that can be counted counts, and not everything that counts can be counted.”