Our Ranking System: How We Rank Schools at TBS

rankingsFinding the school and degree program that are right for you is one of the most important things you’ll ever do. It is a complex process that will affect the rest of your life.

To navigate this process successfully, you need useful and reliable information about the enormous variety of post-secondary schools and programs that exist in this country. That is why TheBestSchools.org provides a wide range of articles on many different aspects of higher education.

However, one of the things uppermost in your mind as you look over our site is bound to be the question of rankings. You want to know how some particular colleges or degree programs that interest you stack up against other similar schools and programs.

Therefore, we know that advice about rankings is one of the most important services we can provide to you. We also know that you have many choices among the numerous other ranking publications and websites that are out there.

So, why should you rely on our rankings, rather than someone else’s? And what is our ranking system, anyway?

To answer these important questions, let’s think for a moment about what a print publication or website really means when they say that a school or a degree program is among, say, the “top 10″ in the country.

While actual rankings may differ considerably, a common method underlies most rankings, namely, “multicriteria optimization.” This method is practiced by the most famous ranking organizations out there, like U.S. News & World Report and The Princeton Review.

Multicriteria optimization works as follows. Having laid out the various criteria one regards as relevant to determining a school’s quality, one evaluates a college or degree program with respect to each of these criteria, and then pools all this information into a single measure, which then yields a ranking.

Because typically there are many such criteria, the information collected from these various criteria will by itself not be sufficient to form a ranking. That’s because different criteria may give conflicting answers. For instance, a school may rank highly for academics but miserably for quality of campus life. Which of these is more important and how should they be weighted in assigning a ranking?

What’s needed, then, once performance on all these criteria is evaluated, is some way of combining all these performance measures into a single number that best reflects how all the various criteria are to be taken into consideration at once. In this way one optimizes across the various criteria—hence “multicriteria optimization.”

What are some of these criteria? Academic excellence figures prominently among them. In this category, ranking organizations often solicit the opinions of a large number of trusted sources, like students, faculty, administration, alumni, and prospective employers. These opinions may be gathered by telephone interviews or by questionnaires. Less subjective ways of assessing academic excellence are also possible, such as scholarly output and impact of a school’s faculty as measured through publications.

What about criteria other than academic excellence? The best-known ranking organizations consider many other factors, such as tuition costs, availability of financial aid, locale, student demographics, student satisfaction, sports opportunities, natural beauty, partying potential, safety, and many others.

This does not mean that all the criteria are treated equally, however. Rather, each ranking organization will not only have its own system of primary criteria and ways of gathering the statistics for each of these, but it will also have its own private formula of how much “weight” to give to each of the criteria when the time comes to crunch the numbers and arrive at an overall estimation of a given college or degree program.

While the criteria themselves and the systems for gathering the statistics behind them may seem highly scientific, often these relative “weights” assigned to the various criteria can be quite subjective. Indeed, the system in place for assigning the weights may be anything but scientific. And yet this relative weighting of the criteria is the most important aspect of the overall calculation from a mathematical point of view.

Nevertheless, concerns about the objectivity of the weighting system aside, it may seem at first glance that the multicriteria approach is still the best way to assign an overall ranking to a set of colleges or degree programs. It is true that it is the only system that attempts to come up with a unique ranking based upon a broad spectrum of criteria considered all together at the same time.

But is that really the best way to go? We believe that, in most cases, it is not. The reason is that different students will have different criteria for what they regard as most important in their education. And this means that different students will differ in the rankings that are meaningful to them.

Some students may be mainly interested in academic excellence. Others may want to be assured of a good return on their very substantial financial investment. And still others may put a premium on congenial surroundings in which to live for the next four years.

The trouble with the multicriteria optimization approach is that all the variously weighted criteria dilute each other in such a way that the resulting ranking may not be optimal for anyone at all.

Now, one obvious way around this difficulty is to limit one’s ranking to a small number of criteria, or even a single criterion. This is what Forbes magazine does, for example. They perform a straightforward calculation of “return on investment,” which can tell you how much extra salary, on average, you can expect to earn from a degree from a given institution, over and above what you could have expected to earn without it, as a function of the cost of the degree.

This type of ranking may be ideal for you—if, for example, return on investment is your overriding concern in choosing a college or degree program. However, if other factors are of equal or greater importance to you, then obviously the Forbes ranking will not answer your needs very well.

We believe the multicriteria approach to ranking practiced by U.S. News and others is too broad. And we also believe the unicriterion approach practiced by Forbes is too narrow, at least for most students. That is why we have chosen to follow a middle path between these two extremes.

Our ranking system is based on three broad master criteria, but the application of each of these criteria will vary from case to case, depending on the kind of college or degree program that is under discussion in a given article.

Here are our three master criteria:

  1. Academic excellence
  2. Return on investment
  3. Incidental benefit

What do these criteria mean? Let us briefly look at each one.

Academic excellence means, above all, the quality and productivity of the faculty. And how are the quality and productivity of the faculty measured? By intellectual reputation and influence within the wider academic community. But what are intellectual reputation and influence based on?

This is where things get a little bit complicated. To some extent, intellectual reputation and influence can be objectively measured, by such indicators as published books and papers, citations, prizes, patents, public visibility, and other similar factors.

Not entirely, however. Unfortunately, some academic disciplines today have become infected with intellectual superficiality and faddishness. Therefore, to arrive at a just evaluation of particular colleges and degree programs, purely quantitative measures of faculty productivity (publications and the like) may sometimes need to be adjusted in favor of work that is likely to be of deep and lasting value. We will make such adjustments wherever necessary.

The second master criterion is return on investment. By this, we mean the kinds of considerations that Forbes takes into account, which were already discussed above—namely, factors like employability and average starting salary that determine how quickly your investment in your college education will be amortized, and then how much better you may expect to do financially over the course of your life as a result of that investment than you could have expected to do without it.

The third master criterion is incidental benefit. By this we mean all those little, and not-so-little, things that have nothing directly to do with your education or future prospects, but may significantly impact your quality of life during your studies. This master criterion will include such factors as locale (urban vs. rural), student demographics (income, geographic, ethnic, etc.), student satisfaction (graduation rates), sports opportunities, natural beauty, campus safety, and many more.

Finally, how we apply the three master criteria will be determined on a case-by-case basis. For some types of ranking articles, academic excellence may predominate. For others, return on investment or incidental benefit may be more important to the type of ranking study we are conducting. In some cases, we may apply all three master criteria to a given ranking.

The important thing for any ranking system is transparency: transparency about the criteria being employed and transparency about how those criteria are weighted in computing the ranking. At TheBestSchools.org, we are committed to such transparency.

Degree Finder