breadcrumbs-homeMethodology

Methodology

Table of Contents

    The top reason students attend college is to get a better-paying job and achieve a financially secure future. But political efforts to hold institutions of higher education to standards of economic accountability have so far failed.

    The college rankings market has not done much better. Ranking systems are not providing prospective students and parents with useful, actionable guidance as to which colleges and programs provide economic value and which do not.

    We fill this gap – detailing which schools and programs of study are most likely to lead students to greater financial security.

    A word on sources

    We use only official government data, namely the U.S. Department of Education’s College Scorecard and the integrated postsecondary education data system (IPEDS). We do this to ensure that the data is factual, unbiased, and based on current real-world earning information. Additionally, the recent relaunch of institutional-level earnings data on College Scorecard after 7-year hiatus signals renewed efforts to hold schools accountable for poor performance. As such, frameworks are established, and the metrics used to set and measure performance may be found in the College Scorecard data sets. It makes sense for us to align our data strategy accordingly.

    Undergraduate rankings

    We rank undergraduate colleges and universities based on how long it takes students to pay back their educational investment after attending. We call this payback.

    Did you know? Did you know?

    Our economic score equation is set so that a 1% advantage in degree premium will offset a 1% disadvantage in payback.

    We then adjust the payback by relative earning potential – the degree premium – between students at different schools and programs in the same state. We call the output of these 2 data points the economic score.

    Payback

    Michael Itzkowitz, a senior education fellow at Third Way and an external consultant on this project, developed the basic payback methodology we use in our ranking system. The idea is that, on average, any worthy postsecondary education will provide additional income above what peers without a college education earn.

    The payback measures the amount time it takes to pay back a student’s educational investment with that additional income.

    Payback

    The number of years it takes a student to pay back their educational investment with new earnings.

    The quicker a student can recoup the total cost of their education, the sooner he or she begins to see a return on their educational investment.

     

    California Universities Earnings beyond California HS graduate Total cost to earn credential Payback
    California State University – Fullerton $30,306 $40,754 1.37
    UCLA $48,924 $72,822 1.49
    Ashford University $12,450 $104,640 8.4

    The above table illustrates how payback is calculated at a few California-based universities.

    The payback rate slaps a glaring red flag on Ashford University, and students may conclude to steer clear of this school. However, Cal State Fullerton outperforms UCLA based on payback rate despite a considerable difference in earnings. Determining which of these 2 schools to attend based on payback rate alone might lead to the wrong choice.

    To account for earning differences, we developed a measurement called the degree premium.

    Degree premium

    The degree premium calculates how much more or less students from a particular school or program earn when compared to the state weighted average.

    Weighted average earnings

    Rather than giving equal weight to different schools, a weighted average weighs the earnings of each school by their number of relevant students. You can read more about how this is calculated in the appendix.

    To calculate the degree premium, we simply deduct the state weighted average earnings from the school’s median earnings.

    School premium Median earnings California weighted average School premium
    California State University – Fullerton $54,586 $50,770 $3,816
    University of California – Los Angeles $73,744 $50,770 $22,974
    Ashford University $37,270 $50,770 -$13,500

     

    The chart above illustrates how degree premiums are calculated.

    The degree premium illustrates the relative earning potential of schools in the same state.

    It is also used to calculate our economic score, the basis for how we rank colleges and degree programs.

    Institution Payback School premium School premium percentage Value score
    California State University – Fullerton 1.37 $3,816 107.5% 1.27
    University of California – Los Angeles 1.49 $22,974 145.3% 1.02
    Ashford University 8.40 -$13,500 73.4% 11.44

     

    The above chart calculates the economic scores of Ashford, UCLA, and Cal State Fullerton.

    Schools that offer earnings of less than 100% of the state average see an economic score higher than their payback rate. Schools that earn more than 100% of the state average will see an economic score lower than their payback rate. The lower the economic score, the better.

    KEY TAKEAWAY

    Our economic score considers both a school’s payback rate and degree premium. Remember that the lower the economic score, the better.

    How program-level analysis is different

    While school-level earnings are based on all students that enrolled at a particular institution of higher education 10 years ago, program-level data looks only at the students in a particular degree program, 3 years after they have graduated.

    • Institutional-level earnings data includes those that have graduated and those that have not graduated.
    • The 10-year, post-enrollment cohort has likely been working longer than the 3-years-post-graduation students, meaning they would be, on average, earning more.

    It’s important to note that different programs at the same school can offer vastly different earning projections. A school might have excellent school-level earning figures but underperform on certain degrees (and vice versa).

    Thus, it is very important for prospective students, particularly those that already know what they want to major in, to research our program-level economic value data.

    Graduate program valuation

    When ranking graduate programs, we start by looking at the debt-to-earnings ratio, which expresses what percentage of a student’s earnings are eaten up by total debt.

    Rule of thumb Rule of thumb

    Don’t take on more debt than you can reasonably expect to earn in annual salary shortly after graduating.

    We then adjust the debt-to-earning ratio by our own graduate degree premium, which uses undergraduate-level earnings in the same program and state as a comparative benchmark. We again call the output of these 2 data points the economic score.

    Debt-to-earnings ratio

    Graduate tuition costs and expenses are not reported by government sources. Instead of using cost to determine a payback, we look at debt as the expression of a graduate student’s investment in their education. This is compared to the student’s earnings by way of a debt-to-earnings ratio.

    School Program Debt Earnings Debt to earnings
    UCLA Education $32,500 $57,318 0.57
    UCLA Film/photographic arts $86,524 $42,682 2.03
    UCLA MBA $93,407 $143,744 0.65
    University of Wisconsin, Madison Social work $36,449 $46,332 0.79
    University of Wisconsin, Madison Curriculum and instruction $26,549 $43,426 0.61
    University of Wisconsin, Madison Marketing $34,068 $111,992 0.30

    The chart above compares the debt and earnings of different programs at the same schools.

    A debt-to-earnings ratio of 1 means the annual debt is the same as annual earnings. Anything higher than 1 means that that debt is higher than annual earnings; anything lower than 1 means earnings are higher than debt. Obviously, the lower the ratio, the better.

    Of course, there are situations in which much higher earnings might excuse a higher debt-to-earnings ratio. To account for earning differences when ranking graduate programs, we have created the graduate degree premium.

    Graduate degree premium

    To calculate the graduate degree premium, we use a weighted average earnings benchmark of the same program, in the same state, at the undergraduate level. This allows students to:

    1. Compare the annual earnings of graduate programs against one another.
    2. Determine whether the earnings from a particular graduate degree make it worth getting from an economic perspective.
    School Premium Earnings beyond bachelor’s Annual debt payment Graduate premium
    UCLA Education $23,192 $4,380 $18,812
    UCLA Film/photographic arts $16,796 $11,676 $5,120
    UCLA MBA $97,128 $12,600 $84,528
    University of Wisconsin, Madison Social work $13,728 $4,920 $8,808
    University of Wisconsin, Madison Curriculum and instruction $2,390 $3,576 -$1,186
    University of Wisconsin, Madison Marketing $67,895 $4,596 $63,299

    The above chart shows how the graduate degree premium is calculated for a few different programs at the same schools.

    Note that we deduct the average annual debt repayment when calculating the graduate degree premium.

    Graduate program economic score

    We then divide the debt to earnings ratio by the graduate premium percentage to get a value score.

    School Program Debt to earnings Graduate premium percentage Value score
    UCLA Education 0.57 155% 0.367
    UCLA Film/photographic arts 2.03 120% 1.694
    UCLA MBA 0.65 281% 0.231
    University of Wisconsin, Madison Social work 0.79 127% 0.621
    University of Wisconsin, Madison Curriculum and instruction 0.61 96% 0.632
    University of Wisconsin, Madison Marketing 0.30 244% 0.123

    Graduate programs offering an economic advantage over undergraduate programs will see an economic score lower than their debt-to-earnings ratio. Schools in which the new earnings + repayment are actually less than undergraduate earnings will have an economic score that is higher than their debt-to-earning ratio. Remember that a lower economic score is better.

    Conclusion

    Our methodology is meant to provide prospective students the tools needed to pick the most economically promising schools and programs.

    While we are not the first in the market to take this approach, we do offer some unique contributions, namely:

    1. Adjusting our payback by the degree premium to more appropriately weight comparative earnings.
    2. Offering a user-friendly product so students are better able to locate, understand, and action the information.

    Some models make assumptions to predict future performance and create long-term ROI projections. We deliberately avoid doing that, limiting our model to using the best available real-life earnings data.

    We will continue to build on this. College Scorecard recently released 3-year post-graduate earnings data, which was immediately reflected in our models. Next year, they will likely release 4-year data, and we will use to update our data sets accordingly.

     

    FAQ

    Our value score equation is set so that a 1% advantage in degree premium will even out a 1% disadvantage in payback.

    Despite what looks like an equal balance, there is some bias towards payback in the model.

    Earnings are forever, while costs are often paid back in a relatively short time frame. Earnings then “last longer”, and so a longer term 1% advantage would seem to more than offset a shorter term 1% disadvantage.

    Students and parents may determine for themselves if they would like to provide additional weight to earning premium amounts between closely ranked schools. From our perspective, cost is currently the most pressing issue. Many parents open education funds at a child’s birth to finance future education. Earnings are subject to change. Much of a student’s future earnings depend on job performance and other immeasurable or unknown factors. While not weighing more in favor of earnings might lead to a few questionable ranking choices, overall we believe it is the right decision.

    The number one indicator of whether a student will graduate or not is income level, and we don’t believe that a school should be penalized for admitting more lower-income demographics. We also don’t think a school should be rewarded for limiting admittance to lower income students.

    We do indicate whether a school falls under or over the state average graduation rate, with red indicating below average and green indicating above average. Additionally, if a school falls into the bottom quarter of the national graduation rate, they are pushed to the end of the rankings list.

    The College Scorecard first released institutional-level earnings data in 2007-8 and continued releasing the data until 2015-6. At that point, earnings data started being measured. The following year, the first of the 1-year degree-level earnings data was released. The 2-year degree level earnings were released the year after that.

    The College Scorecard has stated that they intend to release longer time periods as the data becomes available.

    Appendix: Further explanation of data

    We use weighted averages to create benchmarks used in comparing multiple schools and programs against one another.

    What are weighted average earnings?

    Weighted average earnings adjust the importance of each school’s median earnings in a dataset based on the percentage of students the school accounts for in the dataset.

    We have calculated hundreds of different weighted averages to standardize comparison across different category types – by state, by program, by degree level, by school type, and by different combinations of state, degree, school type. We demonstrate below how weighted averages are calculated.

    Example weighted average calculation Earnings count Median earnings Total earnings
    California State University – Fullerton 7,999 $54,586 $436,633,414
    University of California – Los Angeles 6,327 $73,744 $466,578,288
    Ashford University 13,983 $37,320 $521,845,560
    TOTAL 28,309 $50,339 $1,425,057,262

    How it works

    1. The column earnings count  represents the number of students the National Student Loan Data System has used to calculate a school’s median earnings.
    2. The earning count is then multiplied by the school’s median earnings, which outputs total earnings.
    3. This process is repeated for each school in the comparison category.
    4. The total earnings for all measured schools are then divided by the total earning count, the outcome equaling the weighted average earnings for the category.

    In the above example, earnings from 28,309 students are measured and result in total earnings of $1,425,057,262. This comes out to a weighted average earning of $50,339. When we extend this to include all schools in California, the weighted average earnings of all California bachelor students 10 years after enrollment comes to $50,770.

    Institutional-level undergraduate data

    Cost

    1. We use the average net price, which is the amount that the average student pays out of pocket to earn their credential. This is the total cost of attendance including tuition and fees, books and supplies, and living expenses, minus the average grant/scholarship aid. This includes only students who have received a grant or loan from the federal government.
    2. To determine total cost, we estimate the average number of years it takes students from a particular school to graduate.[i]
      1. If a school makes available the time it takes for students to reach program completion, we use the average time and multiply that by the average annual net price. Therefore, if it takes students, on average, 3 years to complete their degree—and it costs $10,000 per year, their total net price to earn their credential would be $30,000 at that institution.
      2. If a school does not submit completion statistics, we use data from the National Student Clearinghouse. This data indicates that students typically take 5.1 years to earn their bachelor’s degree and 3.3 years for an associate degree. In addition, we use 1 year for certificate.

    Earnings

    1. We use the median earnings of students 10 years after their enrollment.
      1. This includes both, students who’ve completed their degree and those who did not.
      2. The earnings include only students that received some form of federal financial aid.
      3. Earnings are compared to high school earning figures from the Census Bureau’s American Community Surveys, using median earnings from years 24-29.

    Program level data

    Cost

    1. We use the median annual amount of tuition, room and board, books, and fees paid by students that have earned at least $1 in federal loans or grants.
    2. To determine total cost, we attempt to arrive at the average number of total years it takes students from a particular school to graduate.
      1. If a school submits length of graduation KPIs, we take the average length to calculate total cost.
      2. If a school does not submit graduation KPIs, we use a 2016 NCES study which indicates average graduation lengths as 5.1 for bachelor’s level, 3.3 years for associate, and we use 1 year for credential level.
    3. Note that cost data is available only at the institutional level. Certain majors or specialist programs might cost more, or less, than the standard undergraduate program. When this information is made available, we will integrate it into our model.

    Earnings

    1. We use the median earnings of students 3 years after they graduate.
      1. The earnings include only students that received some form of federal financial aid.
      2. Earnings are compared to high school earning figures from the Census Bureau’s American Community Surveys, using median earnings from the ages of 24-29.

    Graduate studies

    Cost: The cost of attending graduate school is not reported by the Department of Education. Instead, debt is used as the metric for measuring relative program costs. Therefore, costing metrics do not take into account the out-of-pocket expenses students may accrue as a result of their advanced studies.

    Total debt amount shows the median Stafford and grad PLUS loan amounts a student is left with after graduation.

    Annual debt amount shows the median annual debt payment.

      1. If a school submits length of graduation KPIs, we take the average length in order to calculate total cost.
      2. If a school does not submit graduation KPIs, we use a 2016 NCES study which indicates average graduation lengths as 5.1 for bachelor’s level, 3.3 years for associate, and we use 1 year for credential level.

    Earnings

    1. We use the median earnings of the students 3 years after graduating.
      1. The earnings include only students that received some form of federal financial aid.

    [i] Bachelor-level years to completion rates are based on the 2014 entering class. We measure the percentage that graduated in 4 years, 5 years, and 6 years. We assume the remaining enrolled students will graduate in 8 years. (4 year grad rate / total grad rate *4)+(5 year grad rate / total grad rate *5)+(6 year grad rate / total grad rate *6)+(8 year grad rate/total grad rate*8)

    Associate-level years to completion rates are based on the 2017 entering class. We measure the percentage of students that graduated in 2 and 3 years. We assume the remaining enrolled students will graduate in 4 years. (2 year grad rate / total grad rate *2)+(3 year grad rate / total grad rate *3)+(4 year grad rate / total grad rate *4)