Weathering the Rankings Storm

Where might you find yourself as the dean of a globally minded business school? If you’re Fuqua’s Bill Boulding, think Dubai. Back in November, he was there for the World Economic Forum and meetings with his regional advisory board. A text message from his office came through—“ We’re no. 1”—leading him to assume that Bloomberg Businessweek was about to place Fuqua at the very top of its business-school rankings. He texted back, “Could you please confirm?” Fifteen minutes went by. Was this for real? “They were just so busy celebrating that they couldn’t get back to me right away.”

“People have asked me whether I was shocked,” Boulding says. “I think that any business-school dean, unless they’re suffering from hubris, would be surprised.” Over the previous four years, Fuqua had been somewhere in the top 6; given Businessweek’s focus on employer satisfaction with a school’s graduates, the students’ satisfaction with their experience, and the research activity of the faculty, he felt Fuqua was in a good position to move up.

As a culture, we’ve become rankings-crazed, and that means Duke plays a delicate balancing act: It celebrates rankings even as it’s alert to their imperfections. And in higher education, you have to be alert to U.S. News & World Report, which began ranking colleges in 1983. Today Duke is number 8; its highest ranking, 3, came in 1999. It’s been up, it’s been down, and it’s been “bunched,” or tied, with other schools in the top tier.

 Now Duke is ranked by all kinds of self-appointed authorities in all kinds of contexts, from campus appearance to study-abroad opportunities. Fuqua’s across-the-street neighbor, the Sanford School of Public Policy, displays a poster in its café showing an apparently well-fed student and a couple of kitchen workers. The text brags that Duke is number 9 for “Best Colleges for Food in America”; a website called the Daily Meal cooked up that ranking, with metrics ranging from healthy food options to a mysterious “X” factor.

Every time you turn around, there's a new ranking,” Boulding says. Still, Businessweek is a source that Fuqua pays attention to, along with The Financial Times and U.S. News. Over ten years or so, no single business school showed an upward trajectory in all three rankings; Fuqua, he says, is the only school that moved up in two of the rankings. “When I talk to our alums and our students, I emphasize that there’s going to be volatility to some degree across these rankings. The important thing is to look over a long time horizon.”

There’s another important thing: recognizing that what weighs heavily in some rankings may not be what, fundamentally, counts. Since they’ve been in the business-school ranking business, both The Financial Times and U.S. News have seen starting salaries as an important metric. “If you really wanted to game those rankings, what would you do? You would try to identify the highest-paying jobs that M.B.A. graduates go into, and then build your business school around attracting and placing people in those high-paying jobs. Now, I’m not opposed to our students getting high-paying jobs in really interesting, exciting areas. But there’s also nothing wrong with getting a low-paying job, if you’re really making a difference in the world,” Boulding says.

Asked if he would like to see business-school rankings disappear, Boulding pauses. “I don’t think you’d want to get rid of rankings. Well, in some ways, it would make life easier. But there’s information in rankings. And I find it valuable to understand the rankings and to see how they connect to our core strategy. Are we producing graduates who are strong in the eyes of their employer?”

It’s no surprise that rankings have a strong presence in the eyes of the consumer, according to one of Boulding’s colleagues, Fuqua marketing professor Gavan Fitzsimons. “You have a classic difference of philosophy between economists and psychologists about how to solve problems. Economists believe we should provide information on all the options to everybody: People will apply the rule of maximum utility and choose the best option. Psychologists have shown over and over again that people hit information overload very quickly. In fact, we’ll often walk away from the problem of too many choices.” Rankings, then, can provide order (if not logic) in a culture with endless inputs.

Think about life in a pre-information age. There were a couple of plumbers who worked your neighborhood. “Now you get your list of plumbers, and 200 names pop up. Well, how are you going to make a choice among 200 plumbers? The easiest way is to outsource our decision-making to some outside agency.”

As rankings systems have proliferated, so have efforts to manipulate those systems. It’s not hard to find accounts of schools that entice applicants who have no chance of being accepted, so that they appear more selective; that accept a fair number of weaker students but award them spring-semester admission, a status overlooked as a measure of student quality; that count coaches and librarians as instructors, thereby enhancing their faculty- to-student ratio; and that even hire their own graduates in temporary jobs, which allows them to elevate their employment numbers. And there’s the tale of the flipping dollars: One school reportedly sent a dollar bill to each of its alumni and asked them to send it back, so the school would register a high percentage of alumni support.

In the higher-ed arena or beyond, a ranking score is questionable cultural currency, says Fitzsimons. When you visit a car dealership, “every salesperson and every service manager tells you, ‘You’re going to be receiving a call, and anything other than a ‘5’ on all of the questions means that I could lose my job.’ The system has become completely ridiculous.” Adding to the ridiculous factor: Companies like Uber and Airbnb are now rating their customers, just as their customers rate them. It’s a feedback loop likely to inspire grade inflation, since no one in search of a service provider wants to be tagged as a complainer.

For Duke’s law school, the number of the moment, via the rankings wisdom of U.S. News, is 10. Its dean, David Levi, points out that rankings are hardly new in the profession: Law firms routinely get ranked on profits per partner, something that, naturally, puts pressure on associates to bring in money. But it’s also the case that rankings may have new significance for future lawyers. Enrollment numbers at law schools nationally (though not at Duke) have been plunging; most observers attribute that to high tuitions and uncertain job prospects. The result is a “flight to quality”: Students flock to the best schools, at least according to their reading of the best schools.

Reputation-oriented rankings are built on the opinions of established legal practitioners. There are obvious drawbacks, according to Levi: low response rates among judges, for example, and the fact that it’s easy to be out of touch with developments at a school you know only vaguely, or knew intimately decades ago, when you were a student there. Other ways of ranking, in places like The American Lawyer and The Princeton Review, ask newly minted lawyers about the quality of their training, or ask about the experience of current students. In such surveys, he says, Duke does “extraordinarily well.”

Back in 1998, a faculty member at the Duke law school, Richard Schmalbeck, wrote an article in the Journal of Legal Education that traced the U.S. News phenomenon to a 1974 “reputational” survey, by two sociologists, of professional schools. (The “reputation” component now accounts for 25 percent of a law school’s overall ranking by U.S. News.) He found—and continues to find—that the same schools occupied the top sixteen spots in every reputational survey. And, typically, the place occupied by each school in that group varied within just a narrow range.

Today, Schmalbeck works in a conversation just before a phone interview with a New York Times reporter on one aspect of his expertise—taxation policies around intercollegiate sports. In the law-journal article he pointed out that most of the schools had, over time, been led by a succession of deans, witnessed considerable faculty turnover, and undergone major changes in buildings, libraries, curricula, and student populations. What explains the reputational stability? As he sees it, law schools may be less influenced by an infusion of resources or by new technology than, say, business schools. So it would take a lot to shake up rankings that hinge, to no small degree, on reputation. Think of the legal principle stari decisis—“stand by that which is decided.”

Such stability aside, law schools have responded to the race for good rankings. Schmalbeck offers an example. A metric for U.S. News is resources spent per student. Traditionally public institutions wouldn’t “tax” their divisions, their law schools included, for janitorial, police, parking, and other services provided campus-wide. Private institutions traditionally have levied such a tax. As a result, a group of (public) law schools looked relatively stingy in their spending patterns. So they asked to be billed for the services they were receiving from their central administration— to be spending more per student, according to some accounting system. Their rankings rose accordingly.

Schmalbeck has also found that associates and partners at large law firms, those with several hundred lawyers, to a huge extent come out of top-ranked law schools—a phenomenon that speaks to how rankings shape recruiting practices.

For his part, Levi says, “You could say in the U.S., we have ten or fifteen of the best law schools in the world. These are just amazing places; they’re uniformly so good. They all are doing many of the same things, they all have terrific faculty, they all have terrific students, and they all have a superb methodology for teaching law.”

All of which suggests the question: How much of a campus reality can any ranking system capture? Or, as Levi asks, what factors make one of those top law schools different from another? “Well, their size. And they have different cultures. It turns out that the culture of an institution is what really matters. It’s the internal rhythm, the spirit of the place, the values that it imparts. It’s what makes us Duke—a culture that is infused with ideals and goals for public service, with entrepreneurial energy and an interest in trying new things, with an interdisciplinary reach. And it’s a very collegial environment. But all those things really are not measurable.”

One quite measurable quality shared by Duke’s professional schools—aside from an array of high rankings—is the growing presence of international students. More organizations outside the U.S., The Financial Times among them, are joining the rush to rank. At the same time, rankings of all kinds may have a particular resonance for non-American students.

Duke physics professor Haiyan Gao says Chinese students and their parents, as a notable example, value rankings highly. In large part that reflects cultural conditioning. China has a long history of relying on an examination system at the local, regional, and national levels. From the ranking he achieved, the exam-taker would be placed in a more or less prestigious government position. “This type of examination system started in 605 and ended in 1905. So it had a history of 1,300 years,” says Gao, the newly named vice chancellor for academic affairs at Duke Kunshan University. “In more recent times, junior and high schools publicly announce who is number 1 in each class each year.” Even with more attention to privacy concerns, “Sometimes they announce that to the entire community.”

Gao notes that many Chinese, and international students in general, do not have a very good understanding of U.S. higher education. And they probably can’t look to a planning a college tour. With little feel for the character of campus, they consider rankings a factor “that they pay a lot of attention to.”

If they’re prospective freshmen, one person paying a lot of attention to them is Duke’s longtime dean of undergraduate admissions, Christoph Guttentag. This past winter he’d be greeting visitors in a very cold office—it has its original, non-weather-resistant windows and a non-functioning fireplace. If the world were more rational, U.S. News would give students access to school-specific data, he says, in a form that would allow them to plug everything into, in effect, their personalized rating system.

That’s a vision for what might be thought of as an educational analog to personalized medical care. For now Guttentag is musing over a couple of hypothetical scenarios. The first: Duke jumps to number 1 in the U.S. News ranking. How would that change the admissions profile? “Less than people think. Some students would consider us who hadn’t considered us before. And some students who had been considering us would figure, ‘I’ll never get in.’ ” In the end, though, he adds, most students would continue to choose a place that felt right to them. “When I’ve gone back and looked, there seems to be no correlation between our rankings and the size and quality of the applicant pool.”

So, the second hypothetical scenario: Duke drops way out of the top 10. “Well, if we dropped to 30 and stayed there, that would have an effect. But it’s hard to imagine an applicant saying, ‘I was going to go to Duke, but now that they’re 11 instead of 9, I don’t think I will.’ I’ve never heard of a student who said, ‘One of my colleges was inside the top 10 and one was just outside the top 10. And that was the compelling reason for my decision.’ ”

No sector of Duke better illustrates the ambiguities around rankings—their limits as well as their uses—than the Pratt School of Engineering. Today Pratt dean Tom Katsouleas can look out on what was, just a decade or so ago, a small footprint on Duke’s campus; it’s now a conspicuous section of the campus, aesthetically and otherwise. In the past decade, the size of the Pratt faculty has grown by 20 percent, and its research productivity similarly has skyrocketed. It also has become home to a host of research centers in areas ranging from nanotechnology to metamaterials.

U.S. News ranks Pratt a top-20 school for undergraduate engineering. Pratt is number 2 in biomedical engineering. (For some forty years, Duke was almost alone with biomedical engineering, as other schools focused on traditional engineering disciplines. More recently, peer schools have been pouring resources into that area, changing the competitive landscape.)

Pratt may have a higher profile, but it’s relatively small within the larger Duke. By contrast, such schools as MIT and Georgia Tech are driven hugely by their engineering divisions. Measuring the total volume of research or the total number of Ph.D.s will put a place like Pratt at an enduring disadvantage. There are other rankings-related challenges: Pratt’s strategy calls for the school to be equal in academic quality to the very best engineering schools, which may be reflected in rankings. But it also calls for Pratt to become a model for engineering education, and to become what Katsouleas calls “a driver for all of Duke.” Those aims may not make a mark on a ranking system.

If you blindly chase a ranking, Katsouleas says, you’ll lose whatever status you were seeking. To build on his point, he reaches for a bit of history with an engineering tie-in. During the Soviet era, Russian factories were given targets involving the production of nails. At one point, the target hinged on the number of nails, and many tiny useless nails were produced. At another point, the target hinged on the combined weight of nails—resulting in a small number of useless giant nails.

The historical lesson? Figure out what you need to be doing in the deepest sense, not just what it takes to look good. It’s idealism with a dose of pragmatism. “I truly believe that if we pursue what we value and become unequivocally the school we want to be, the rankings will follow.”

Numbers-crunching students

Are students sensitive to college rankings? Anecdotally, the answer is yes.

Here’s how a Duke junior described his college-selection process: “I sat down with my dad, we went down the list of the top 10, and I ruled out the ones that I knew I didn’t want to go to because of location.” A sophomore referred to rankings as “a big metric in seeing what colleges interested me.” For a freshman, a college’s score in the U.S. News hierarchy “is not the only criterion, but it’s a major one. My high school sends a lot of students every year to Duke, and people look to see when schools are going up and down.” And a senior remarked that rankings “told me how prestigious a school is. Everyone is prestige-conscious.”

To pursue that question beyond anecdotal insights, Duke Magazine recruited two students to survey undergraduates. Using Duke’s Facebook channels, Megan Lax, a senior majoring in psychology, and Sara Pak, a sophomore majoring in computer science, asked: “How much did the U.S. News college rankings influence your decision to apply to Duke?” The results are presented below.

This wasn’t a scientific survey, but it bolstered the idea that students weigh rankings in their college decisions. As for the students who ran the survey: Both say they were conscious of Duke’s position in U.S. News, but that was only a starting point in the process. They checked out how strongly entrepreneurship is worked into the Duke culture, for example, and how cumbersome it might be to switch majors.

In his own smaller, informal survey of the freshman residents in Giles, resident adviser Ben Brissette, a sophomore, concluded: “They were all aware of the rankings while applying to Duke. They were pleased that Duke was wellranked.” They became more Duke-knowledgeable. And Duke’s position on the scale “seemed like added justification”— a factor, but not the ultimate rationale for ending up where they ended up.