Why smart people build bad systems: The curse of local optimization

A highly intelligent, intensely rational, deeply well-compensated database engineer is specifically tasked with urgently improving the sluggish read speeds of a massive database query. She focuses intensely, bringing all her considerable talent to bear on index optimization and the implementation of sleek new caching layers, successfully shaving critical, impressive milliseconds off the response time.

Her specific, individual output is flawless. The ticket is closed.

Yet, within twenty-four hours, the enterprise system as a whole becomes wildly more brittle, because the specific caching layer she just introduced unexpectedly creates a massive race condition with the frontend state management that another, equally brilliant, equally rational engineer is currently building on a different floor.

This is the terrifying curse of local optimization: the incredibly common phenomenon where brilliant individuals, operating entirely rationally within their narrow, siloed purview, collectively architect a disastrous, failing macro-system.

Why do specialized experts frequently design architectures that harm the wider enterprise?

Specialized experts design harmful architectures because human psychology is poorly equipped to track cascading, abstract externalities; we instinctively optimize the small, immediate variable right in front of us, ignoring how our “perfect” solution breaks the assumptions of systems down the line.

Psychologically, human beings are exceptionally, biologically poor at grasping complex, abstract systems dynamics. We are deeply wired by evolution to successfully solve the immediate, physical problem directly in front of us. When presented with a massive, interconnected digital architecture, we instinctively draw a comforting, artificial boundary around a small subset of the variables, brilliantly optimize the hell out of that manageable subset, and disastrously ignore the cascading externalities spreading silently beyond our boundary.

How can engineering leaders protect complex systems from the curse of local optimization?

Engineering leaders must combat local optimization by forcing specialists to cross boundaries, mandating that the “success” of a feature is never judged in isolation, but only by its impact on the system’s global stability.

To successfully build great systems requires the intense psychological discipline to continuously, forcefully zoom out, constantly breaking the comforting, hypnotic trance of the local problem.

• Mandate Cross-Functional Impact Statements: No major optimization or feature can be merged unless the author can explicitly point to, and document, the exact two downstream services their change will most likely disrupt.
• Accept Strategic Inefficiency: We must learn to tolerate a frustrating degree of highly localized inefficiency in explicit service of total systemic harmony. The fastest possible database query is useless if it requires a brittle architecture that triggers a global 502 error during peak traffic.
• Promote the ‘T-Shaped’ Generalist: Do not let backend engineers exist purely in the backend. Force them to spend one sprint a quarter shadowing the frontend team, physically enduring the pain that their “optimized” APIs cause the client-side developers.