“If you want to outperform—if you want to do anything not usually done—then you’ll need to conceptually divide our civilization into areas of lower and greater competency. My view is that this is best done from a framework of incentives and the equilibria of those incentives […]”
Eliezer Yudkowsky has a talent for teaching rationality, economics and decision theory concepts in an accessible manner and usually his writing helps transform unformed thoughts into proper labels and names, thus enabling new modes of thinking.
Below are the concepts I’ve taken from this book (might be hard to understand without reading the book):
Efficiency, inexploitability, inadequacy
- Efficiency [as in, efficient market]:
“Microsoft’s stock price is neither too low nor too high, relative to anything you can possibly know about Microsoft’s stock price.” - Inexploitability [There is an inefficiency but we can’t make money off of it]:
“Some houses and housing markets are overpriced, but you can’t make a profit by short-selling them, and you’re unlikely to find any substantially underpriced houses—the market as a whole isn’t rational, but it contains participants who have money and understand housing markets as well as you do." - Inadequacy [a gap where you can outperform the best current human results]:
“Okay, the medical sector is a wildly crazy place where different interventions have orders-of-magnitude differences in cost-effectiveness, but at least there’s no well-known but unused way to save ten thousand lives for just ten dollars each, right? Somebody would have picked up on it! Right?!"
– Modest epistemology [the point of view Eliezer is refuting in this book]: If there’s really something to improve/discover here, someone would’ve already done it! I’m not smarter/better than others so why assume that I know or can do something other don’t?
Moloch’s Toolbox
There’s a toolbox of reusable concepts for analyzing systems I would call “inadequate”—the causes of civilizational failure, some of which correspond to local opportunities to do better yourself. I shall, somewhat arbitrarily, sort these concepts into three larger categories:
- Decisionmakers who are not beneficiaries;
- Asymmetric information;
- and above all, Nash equilibria that aren’t even the best Nash equilibrium, let alone Pareto-optimal.
In other words:
- Cases where the decision lies in the hands of people who would gain little personally, or lose out personally, if they did what was necessary to help someone else;
- Cases where decision-makers can’t reliably learn the information they need to make decisions, even though someone else has that information; and
- Systems that are broken in multiple places so that no one actor can make them better, even though, in principle, some magically coordinated action could move to a new stable state.
After introducing the basic concepts of inadequacy analysis, Eliezer goes into some case studies, sometimes in depth. He starts with the U.S. medical system (“the most broken system that still works”) but then also dissected the failures and inadequacies of Academia, venture capitalism and politics.
As examples of personal contributions in instances where our civilization happens to be inadequate, he cites finding a solution for his wife’s psychological problem which involved installing 65 light bulbs in their house, creating his own ketogenic meal replacement drink recipe, inventing a new decision theory and knowing that the bank of Japan’s monetary policy was harming Japan’s economy.
Overall Eliezer’s writing is as clear and entertaining as ever and it’s obvious he is learning from previous publications such as Rationality: A to Z. He is giving many concrete examples for his abstract writing with practical suggestions on how to use and not misuse the techniques presented. The book is much shorter than previous ones but still too long and could use further editing, especially the last third. Overall, if you’ve ever enjoyed Yudkowsky don’t skip this one.
Finally, some quotes that I thought were worth highlighting:
For our central example, we’ll be using the United States medical system, which is, so far as I know, the most broken system that still works ever recorded in human history. If you were reading about something in 19th-century France which was as broken as US healthcare, you wouldn’t expect to find that it went on working when overloaded with a sufficiently vast amount of money. You would expect it to just not work at all.
In previous years, I would use the case of central-line infections as my go-to example of medical inadequacy. Central-line infections, in the US alone, killed 60,000 patients per year, and infected an additional 200,000 patients at an average treatment cost of $50,000/patient…. So my new example is infants suffering liver damage, brain damage, and death in a way that’s even easier to solve, by changing the lipid distribution of parenteral nutrition to match the proportions in breast milk.
To paraphrase a commenter on Slate Star Codex: suppose that there’s a magical tower that only people with IQs of at least 100 and some amount of conscientiousness can enter, and this magical tower slices four years off your lifespan. The natural next thing that happens is that employers start to prefer prospective employees who have proved they can enter the tower, and employers offer these employees higher salaries, or even make entering the tower a condition of being employed at all.
Visitor: Hold on, I think my cultural translator is broken. You used that word “doctor” and my translator spit out a long sequence of words for Examiner plus Diagnostician plus Treatment Planner plus Surgeon plus Outcome Evaluator plus Student Trainer plus Business Manager. Maybe it’s stuck and spitting out the names of all the professions associated with medicine. CECIE: Your translator wasn’t broken. In our world, “doctors” are supposed to examine patients for symptoms, diagnose especially complicated or obscure ailments using their encyclopedic knowledge and their keen grasp of Bayesian inference, plan the patient’s treatment by weighing the costs and benefits of the latest treatments, execute the treatments using their keen dexterity and reliable stamina, evaluate for themselves how well that went, train students to do it too, and in many cases, also oversee the small business that bills the patients and markets itself. So “doctors” have to be selected for all of those talents simultaneously, and then split their training, experience, and attention between them.
VISITOR: I must still be missing something. I just don’t understand why all of the people with economics training on your planet can’t go off by themselves and establish their own hospitals. Do you literally have people occupying every square mile of land?…
VISITOR: So there’s no way for your planet to try different ways of doing things, anywhere. You literally cannot run experiments about things like this.
The observation stands: there must be, in fact, literally nobody on Earth who can read Wikipedia entries and understand that omega-6 and omega-3 fats are different micronutrients, who also cares and maximizes and can head up new projects, who thinks that saving a few hundred babies per year from death and permanent brain damage is the most important thing they could do with their lives.
Living in an Inadequate World: Whether you’re trying to move past modesty or overcome the Free Energy Fallacy: Step one is to realize that here is a place to build an explicit domain theory—to want to understand the meta-principles of free energy, the principles of Moloch’s toolbox and the converse principles that imply real efficiency, and build up a model of how they apply to various parts of the world. Step two is to adjust your mind’s exploitability detectors until they’re not always answering, “You couldn’t possibly exploit this domain, foolish mortal,” or, “Why trust those hedge-fund managers to price stocks correctly when they have such poor incentives?” And then you can move on to step three: the fine-tuning against reality.
So a realistic lifetime of trying to adapt yourself to a broken civilization looks like: 0-2 lifetime instances of answering “Yes” to “Can I substantially improve on my civilization’s current knowledge if I put years into the attempt?” A few people, but not many, will answer “Yes” to enough instances of this question to count on the fingers of both hands. Moving on to your toes indicates that you are a crackpot. Once per year or thereabouts, an answer of “Yes” to “Can I generate a synthesis of existing correct contrarianism which will beat my current civilization’s next-best alternative, for just myself (i.e., without trying to solve the further problems of widespread adoption), after a few weeks’ research and a bunch of testing and occasionally asking for help?” (See my experiments with ketogenic diets and SAD treatment; also what you would do to generate or judge a startup idea that wasn’t based on a hard science problem.) Many cases of trying to pick a previously existing side in a running dispute between experts, if you think that you can follow the object-level arguments reasonably well and there are strong meta-level cues that you can identify. The accumulation of many judgments of the latter kind is where you get the fuel for many small day-to-day decisions (e.g., about what to eat), and much of your ability to do larger things (like solving a medical problem after going through the medical system has proved fruitless, or executing well on a startup).
Oh, and bet. Bet on everything. Bet real money. It helps a lot with learning.
First published in https://www.goodreads.com/review/show/2392667145.