Measuring Up

Alumni sometimes ask me where U of T stands compared to other universities in Canada and worldwide. My answer is always, “It depends.” It depends on what dimensions you measure and it depends on how you measure them.

In 2005, for example, Maclean’s ranked the University of Toronto number one among Canadian medical-doctoral institutions for the 12th year in a row. It’s a fabulous record. However, during my academic career in healthcare performance measurement, I learned to be wary of aggregate rankings of institutions. Imagine a hospital that was superb at heart surgery but had a mediocre obstetrics program. The combined rating for those two programs would be useless for heart patients and expectant women alike! It’s much the same when complex universities are reduced to a single score.

For better or worse, the seductive reductionism of institutional rankings still gets attention all over the world. One popular “league table” is published by the Times Higher Education Supplement. The Times ranked U of T 29th on its annual list of the world’s top 200 universities, up eight spots from our 2004 ranking. We were the top Canadian university in biomedicine, science, social science and technology, but stood fourth in arts and humanities and ended up slightly behind McGill overall.

Another popular global ranking is published by the Institute of Higher Education at Shanghai Jiao Tong University. Their system put U of T 24th in 2005, top among Canadian universities, with UBC and McGill respectively second and third in Canada.

Why the differences? The Times creates a composite score by combining reputation ratings, research outputs, proportion of international students and faculty, student-faculty ratios and survey data from employers or recruiters. The Shanghai scoring system relies overwhelmingly on research performance measures.

In January 2006, Alex Usher and Massimo Savino from the independent Educational Policy Institute took a constructively critical look at university league tables worldwide. They noted that within individual countries certain institutions invariably rise to the top, regardless of the ranking scheme: Oxford and Cambridge in the U.K.; Harvard, Yale, Princeton, MIT and Stanford in the U.S.; Peking and Tsinghua in China; and the University of Toronto in Canada. But the authors also cautioned that the basis for this convergence remains mysterious.

Their trenchant observations reminded me of Northrop Frye’s description of human thought – “a process stumbling through emotional entanglements, sudden irrational convictions, involuntary gleams of insight, rationalized prejudices, and blocks of panic and inertia, finally to reach a completely incommunicable intuition.”

I am delighted, of course, that “incommunicable intuition” confirms our top Cana­­dian ranking. That aside, we really need hard data to guide us as we strive to make U of T an even better university. That’s why U of T has worked for years to develop and refine its own performance indicators. And that’s also why we publish an array of indicators that hold our institution up to critical scrutiny.

This year’s report is available at www.provost.utoronto.ca/English/PerfIndic2005.html. It offers both temporal and inter-university comparisons. The report also includes new information from surveys of the student experience at U of T. While our undergrads give U of T high marks for academic standards, they tend to rate their overall experience below that of some of our peers. In contrast, a majority of students in graduate and professional programs rate diverse aspects of their U of T experience from very good to excellent.

When it comes to performance measurement, as I mentioned earlier, how we’re doing depends on what gets measured. We’ve got good reasons to celebrate our overall “top-of-class” average. But there are some lower grades on our report card, and we’re committed to becoming a “straight-A” institution.

One postscript: In healthcare I quickly learned that some of the most important aspects of institutional performance received the least attention. In universities, indexes of alumni engagement and support are among the measures that are often overlooked. U of T’s alumni as a group must surely rank as our most capable ambassadors, our most effective champions, and our most constructive critics. To all of you, your alma mater owes immeasurable thanks.