A pretty key part of my worldview is that individuals and groups are very likely to do bad things if they’re incentivized to do so. A lot of people object to this. They think that it’s too cynical a claim—they say that most humans are fundamentally not evil and aren’t arbitrarily willing to harm others to make their own life more convenient.

For example, imagine we invented some technology which let you save ten minutes on your commute, in return for some other human somewhere getting beaten for that ten minutes. I think that > 80% of Americans would end up taking that deal.

This mostly isn’t because humans don’t care about other humans getting beaten. It’s because strong incentives will find ways around our moral systems when it’s worth it to do so.

I think it’s worth looking at a long list of reasons that humans who are essentially non-evil end up doing terrible things when incentivised to do so:

  • Some part of us is making explicit calculations that we don’t have conscious access to. It chooses whether we should think about the ethics of what we’re doing based on its calculation of whether that’s a good idea; we just notice that we become angry, not the reason that we became angry.
  • Some incentive structures are strong enough that evolution can build in a direct mechanism for it. For example, if there’s extremely strong selection pressure for men to get aroused when they have the opportunity to have sex with women, you expect evolution to build a mechanism so that they get aroused without any explicit calculation of whether it’s a good idea to become aroused.
  • Over time, we learn to pattern match situations and do actions which we were previously rewarded for, without understanding why.
  • We imitate people who are successful, even if we don’t know how particular actions of theirs relate to their success. So if one person is randomly predisposed to behave in a way that turns out to be advantageous, people around them will start behaving in similar ways without understanding why the other person’s behavior is a good idea.
  • If there’s a way of making decisions which lets you callously and unthinkingly harm others for your own benefit if you can get away with it, this decision making process will lead you to more successful decisions. So you’ll notice that this decision making process is a good idea and you’ll try to use it more often, and you’ll imitate people with this decision making process, and so on.

Groups of humans, eg organizations, have even more ways they can end up following incentive structures:

  • There’s a selection process for decision-makers who can follow incentive structures where they lead. This selection process advantages decision makers who are ruthless and who don’t care about hurting people, which can happen in any of the ways I mentioned above.
    • In my experience, founders don’t usually seem ruthless or selfish. They seem more self-deluding; they seem to get really focused on getting a number to go up.
    • Also, there’s a selection pressure for delusional optimism about your product.
  • There’s also a selection process for people who can turn off their morals when working, and so everyone ends up imitating that type of person because that’s what their superior is like.
  • In most cases, middle managers are optimizing metrics set from above. So they get to feel very focused on unthinkingly optimizing a number without thinking about the bigger picture, while their superiors don’t have to think about the details of how the middle managers are hitting their goals.
  • There’s selection processes between organizations–unsuccessful companies go broke.
  • There’s selection processes within organizations–departments which are laid out in more effective ways hit their targets more and are optimized more.

It’s also useful to note that there are certain situations in which it’s advantageous to not understand your motives for doing something, or to not have good introspection. The classic example is head-over-heels love: women want to have partners who are going to support their children and men want partners who are going to be faithful to them, so humans have selection pressure to be able to say with great confidence that they’ll love their partner forever. The easiest way to say things with great confidence is to believe them; and so it is that most humans seem overly convinced of the stability of their relationship.

Of course, I’m not claiming that as a result of this, all companies are ruthlessly efficient. I think that to make companies efficient, you’d have to be way better at creating incentive structures and making people stick to them than we currently are.

I’m also not claiming that humans are entirely rational on an individual level—they’re systematically biased in the ways described by the heuristics and biases literature.

The other useful lesson here is what it feels like from the inside to be someone who’s doing a bad thing. I feel like you almost never have an evil laugh. You feel like you’re selflessly acting for the benefit of your team, or your company. Or you feel like you’re doing what’s right, or getting what you deserve, or playing along with the rules that everyone else follows, or breaking the stupid rules that are unfair anyway. I think that thinking about this is useful for modelling others.