Why Student Outcomes Data Can Be Misleading
Okay, so rankings clearly have some issues. What about something that feels a little more concrete? Money.
Post-graduation salary is another metric families often use to compare colleges, and understandably so. College is an enormous investment, and wanting to know what happens to students after they graduate is completely reasonable.
One of the biggest sources for this information is the U.S. Department of Education's College Scorecard. When it launched in 2015, it was pretty groundbreaking because it gave families much easier access to data about college costs, graduation rates, and earnings after graduation.
And this information can absolutely be useful. The problem comes when we look at a salary number and assume it tells us something definitive about the quality of the college that produced it. Because, once again, the data is a little messier than that.
Caveat one — Scorecard data doesn't include everyone
Here's a pretty important limitation that often gets left out of the conversation: The College Scorecard tracks earnings only for students who received federal financial aid. That means roughly one-third of college students aren't represented in the earnings data at all, and many of those missing students come from higher-income families.
Why does that matter? Because your future earnings are influenced by much more than the college you attend. Family income, professional connections, geography, career field, and the opportunities available to you before you ever step onto a college campus can all play a role.
So if we're comparing the salaries of graduates from two universities using a dataset that excludes a significant (and disproportionately affluent) portion of their student populations, we need to be very careful about the conclusions we draw from those numbers.
The data isn't useless. It just isn't telling us the whole story.
Caveat two — Geography can seriously skew salary data
Where graduates live and work matters too. Imagine College A reports average graduate earnings of $90,000, while College B reports $75,000. At first glance: College A wins!
College A
$90,000
Many graduates working in New York and San Francisco
College B
$75,000
Most graduates heading into lower-cost regions
But what if a large percentage of College A graduates work in New York and San Francisco, while College B sends most of its graduates into lower-cost regions? Suddenly that $15,000 difference means something very different.
Salaries tend to be higher in places where employers have to compensate workers for dramatically higher costs of living. A graduate earning $90,000 in Manhattan may actually have less purchasing power than someone earning $75,000 somewhere else.
So higher graduate salaries don't necessarily mean: “This college gives students a better education and launches them into better careers.” Sometimes they mean: “A lot of this college's graduates live in extremely expensive places.”
Useful distinction.
Caveat three — Outcomes often reflect the students a college admits
And this is probably the biggest caveat of all. Highly selective universities admit students who are already disproportionately academically strong, highly motivated, well-connected, and often from higher-income families. In other words, these students were already pretty likely to be successful.
So when graduates of highly selective colleges go on to earn high salaries, attend prestigious graduate programs, or build impressive careers, we have to ask a surprisingly difficult question: How much of that outcome was created by the college, and how much reflects the students the college selected in the first place?
As Selingo points out, much of the success we associate with elite universities may reflect who those universities admit, rather than something magical that happens to students once they arrive.
Put another way, if you take thousands of extremely intelligent, ambitious, motivated teenagers with strong academic records, impressive accomplishments, and often substantial family resources, put them together for four years, and then discover that many of them become successful adults…we probably shouldn't be completely shocked.
In some cases, as Selingo rather memorably suggests, the university's job over those four years is basically not to derail students who were already on very strong trajectories.
That doesn't mean highly selective colleges aren't providing tremendous value. Many offer extraordinary professors, resources, networks, research, and opportunities. It means we have to be careful about assuming: Successful graduates = the college caused the success.
College outcomes are influenced by the institution. They're also influenced by who the students were before they ever got there. And separating those two things is much harder than a salary number makes it look.