Weapons of Math Destruction 2/6
Part 1.
2/ Shell Shocked
(Let’s get the obvious out of the way) Math models are based on the past and on the assumptions that past patterns will repeat. In other words, despite any turbulence, the structure of the process will remain substantially the same. The 2020-2021 example, of course, is COVID that has rendered many models obsolete.
The division between the people who build models (quants) and the people who use models in the daily lives (bankers) leads to a surprising consequence: one party doesn’t challenge the actions / outputs of the other party. Bankers sleep well knowing that the subprime loan the model allowed them to provide was provided within the bank lending policy. There’s a responsibility disconnect between the decision making and the decision execution. The executing party relies in good faith (at least, in theory) on the quality of the model that drives their decisions.
This is more of a limitation rather than a flaw, but many models fail to recognize that some independent variables are in fact dependent on the third variable. The traditional 2% mortgage default rate only holds true if the system is stable; the drop in incomes due to crisis, structural shifts in employment, etc. directly affects this rate.
It’s more a psychological flaw, but let’s mention in here, too: even if the model is flawed, but still dollars keep rolling in – the beneficiaries have a clear conflict of interests in trying to keep the status quo and not rocking the boat. The presence of the profit provides very strong motivation to look the other way.
As I’ve mentioned in the previous chapter, even the best model can spit out flawed results if its inputs are biased (for instance, a mortgage broker “massages” customer data so that the system approves the loan). Questioning the model is one thing; questioning the inputs and the motivations of humans is quite another – and it’s the job for other humans.
WMDs are called so because they have a scale effect: the models are so persuasive that their use grows exponentially up to a point where the damage becomes visible.
It’s a known fact that people don’t understand risk; what’s worse is that many people whose compensation is tied to a risk (traders, CEOs, bankers) have ways to capture the upside and dramatically reduce the downside of taking risks. One of the ways is misrepresenting risk (buying convenient reports from risk agencies) making the upside look less achievable than it really is.
The more models are created, the more people believe in them and rely on them, thus widening the gap between the models and the real world these models are supposed to describe.
3/ Arms Race
Scale can be achieved via standards – national or global. As standards dictate what’s acceptable and what’s not – scale shapes industries, creates new and kills existing demand and turns previously mainstream things into niche products for the crazy or privileged (e.g., private schools).
MK: look no further than the nutrition charts promoted by the US Government that were later found misleading; clearly, suppliers for school canteens had no issue with them.
Model outputs (say, university rankings) can reinforce the feedback loop effectively increasing gaps even further. A university that’s not ranked high enough won’t attract enough high-calibre students (who would naturally choose higher-ranked universities), which will lead to drop in ratings even further.
The fallacy here is that certain things should be quantified with care, and the consumer of the single number (or a set of numbers) the model produces must be aware of the model’s limitations. One can’t quantify the qualitative features of, say, universities: the long-standing history of intellectual debates, high-profile alumni, lifetime friendships that started in the lab rooms, etc.
Every manager knows that what doesn’t get counted, gets ignored. Quantifying the things that can be somehow quantified via proxy metrics still leads to ignoring the human aspect of the process.
If a model is built on proxies, it can be gamed because proxies can be manipulated more easily than specific data. I’d say that proxies involve “translation” of one piece of data into another, while manipulating specific data requires either throwing away inconvenient pieces or injecting fake ones. [MK: think of the KPI game as an eternal tennis match between HR and employees.]
Ranking models used at scale are dangerous because they create unnecessary competition (i.e., a rat race). The name of the game is that everyone has to play it or be left behind. What’s even more dangerous is that the “average” universities may accept their fate and (one can also stretch this example to a corporate setting) only accept mediocre students who are likely to graduate (an important factor) and not jump ship for a better university in 1-2 years.
An unintended consequence of all major players playing the same game [MK: look at the ESG race all retailers are part of] is that usually consumers bear the costs of high rankings of the seller/supplier. The causes can be noble, but somehow someone else is paying for the show.
Gaming the system (at least when it comes to university admissions) may not be a cheap exercise, so less privileged (financially or socially) may not be able even to take a bite on this apple. It’s not clear whether the “game” provides any value to the participant other than sunk costs and a shot at something that wouldn’t otherwise be available (elite education in the case of universities).
Part 3.