The Banality of Algorithmic Harm: Arendt Applied to Engineering

What does Arendt’s banality of evil reveal about algorithmic harm?

The most dangerous algorithmic harms do not come from malicious actors. They come from competent professionals making reasonable decisions within systems whose cumulative effects no individual examines.

I was part of a team that deployed a content moderation model. The model was trained on English-language data. No one decided to exclude non-English speakers from protection. The training data was English because the previous team had used English data. The previous team had used English data because the original dataset was English. The original dataset was English because the vendor was American. At no point did anyone make a decision. The decision was inherited, layer by layer, default by default, until 340 million non-English-speaking users received measurably worse moderation coverage.

This is Arendt’s insight, transplanted into code. Evil is not always a choice. Sometimes it is the absence of choice. The failure to examine inherited defaults. The assumption that because a threshold was set by someone before you, it must be correct.

How do ordinary engineering decisions produce systemic harm?

Through 5 mechanisms: default inheritance, threshold normalization, metric displacement, abstraction distance, and diffused responsibility. Each mechanism is individually harmless. Together, they produce harm without anyone deciding to cause it.

I have cataloged these patterns across 6 organizations:

• Default inheritance: A threshold set in one context (95% confidence for spam detection) is reused in another context (fraud detection for welfare benefits) without recalibration. The default becomes invisible. No one questions it because no one remembers choosing it.
• Threshold normalization: A false positive rate of 8% becomes “normal” because it has been 8% for 3 quarters. The team stops asking whether 8% is acceptable and starts treating it as a baseline.
• Metric displacement: The team measures model accuracy but not the cost of errors to affected populations. A 92% accurate model sounds good until you calculate that the 8% error rate represents 15,000 incorrectly denied claims per month.
• Abstraction distance: The engineer writing the threshold logic never meets the person denied a benefit. The distance between the code and the consequence is measured in organizational layers, and each layer reduces moral salience.
• Diffused responsibility: No individual owns the end-to-end outcome. The data engineer owns the pipeline. The ML engineer owns the model. The product manager owns the feature. The user owns the complaint. No one owns the harm.

How do you counter banality in your own engineering practice?

You counter it by making the invisible visible: naming defaults, questioning thresholds, closing the distance between code and consequence, and assigning explicit moral ownership.

After the content moderation experience, I adopted 3 practices. First, every inherited default in a system I touch gets documented with its origin, its assumptions, and its last review date. If a threshold cannot be justified by someone currently on the team, it gets reviewed. This is tedious. It is also the minimum standard for avoiding inherited harm.

Second, I insist on what I call “consequence close-ups.” For any model affecting human outcomes, the team reviews 10 individual cases per month where the model’s output changed someone’s experience. Not aggregate metrics. Individual stories. The human-in-the-loop pattern exists precisely for this reason. Abstraction distance is reduced not by better dashboards but by closer contact with the people the system affects.

Third, every pipeline has an explicit “harm owner,” a person whose job is to ask “who could this hurt?” at every stage. This is not a committee. It is an individual with authority to stop a deployment. As I explored in design decisions as moral choices, architectural decisions carry moral weight whether we acknowledge it or not.

Why is this insight uncomfortable for engineers?

Because it means that competence is not enough. You can be skilled, well-intentioned, and compliant with every guideline, and still produce systemic harm through the failure to think about what you are doing.

Arendt’s most disturbing observation about Eichmann was not that he was a monster. It was that he was ordinary. He followed procedures. He fulfilled his role. He did not think about the system he was part of. The application to engineering is direct: following the ticket, shipping the feature, meeting the deadline, these can all be forms of thoughtlessness when the system’s cumulative effect goes unexamined.

I do not say this to paralyze engineers with guilt. I say it to argue that epistemic humility and moral attention are engineering skills, not personality traits. They can be practiced, habituated, and embedded in process. The alternative is to build systems that produce harm we never intended, in patterns we never examined, affecting people we never met. And to do so while believing ourselves to be good people doing good work.

“The sad truth is that most evil is done by people who never make up their minds to be good or evil.” — Hannah Arendt

Algorithmic harm is banal because it emerges from the same routines that produce everything else: sprint planning, code review, deployment pipelines. The harm does not announce itself. It accumulates in defaults, thresholds, and unchallenged assumptions. Combating it requires not heroism but attention. The examined system, like the examined life, is the only one worth building.