Explainability Is Not Optional for Systems That Affect Lives

Why is explainability a moral obligation for consequential systems?

When an automated system determines outcomes that materially affect a person’s life (employment, credit, healthcare, housing, education), the person affected has a moral right to understand the reasoning, and the organization deploying the system has a moral obligation to provide that understanding.

I built a resume screening system for a hiring client. The system evaluated 4,500 applications per month. Every rejection was a consequential decision for a real person. The system produced a relevance score. Without explainability, a rejected candidate saw only “Your application was not selected for further review.” With explainability, they saw: “Your application scored lower primarily because of limited experience with distributed systems (weighted 35%) and no evidence of team leadership (weighted 22%).”

The second message respects the person. It provides actionable information. It treats the candidate as a moral agent who deserves to understand why a system made a decision about them. The first message treats them as an output. The technical difference between these two experiences is approximately $18,000 in explainability infrastructure. The moral difference is the distinction between treating people as ends and treating them as means.

What explainability methods work for consequential systems?

Different consequential systems require different explainability approaches, but all require translating technical explanations into terms meaningful to the affected person, not just the data scientist.

• SHAP (SHapley Additive exPlanations): Provides feature-level attribution for individual predictions. I use SHAP for credit scoring and hiring systems because it produces quantified, consistent explanations that can be translated into plain language. The computation adds 120-340ms per prediction depending on model complexity.
• LIME (Local Interpretable Model-agnostic Explanations): Creates simplified local models that approximate the full model’s behavior around a specific prediction. I use LIME when the model architecture does not support SHAP efficiently. LIME explanations are less consistent than SHAP but more flexible across model types.
• Attention visualization: For transformer-based models, attention weights indicate which input elements the model focused on. I use this for text-based systems (resume screening, medical record analysis) where showing which words or phrases influenced the decision is intuitive for end users.
• Counterfactual explanations: Show what would need to change for a different outcome. “If your experience with distributed systems were 2 years longer, your score would have exceeded the threshold.” I find these most useful for affected individuals because they are inherently actionable.

How do you make explanations meaningful rather than merely available?

Meaningful explanations are designed for the affected person’s context, use their vocabulary, answer their actual questions, and provide actionable information, not just technical feature importance rankings.

I learned this the hard way. The first version of my hiring system’s explanations displayed SHAP values for the top 10 features. Candidates had no idea what “distributed_systems_keywords: -0.23” meant. The explanation was technically accurate and practically useless. I redesigned the interface using the same principles I described in the analysis of dashboard design as information architecture: start with the user’s questions, not the system’s outputs.

The redesigned explanation answered 3 questions: “Why was this decision made?” (top 3 factors in plain language), “What could I change?” (counterfactual suggestions), and “How confident is the system?” (confidence indicator). User comprehension rates increased from 14% to 71% after the redesign.

What happens when organizations choose not to explain?

Organizations that deploy consequential systems without meaningful explainability are making a moral choice to prioritize efficiency over human dignity, and the regulatory environment is increasingly hostile to that choice.

The EU AI Act requires transparency and explainability for high-risk AI systems. The US Equal Employment Opportunity Commission has issued guidance on AI in hiring that emphasizes the need for explainable selection criteria. These regulations are catching up with a moral principle that should have governed these deployments from the start.

I do not build consequential systems without explainability. Not because of regulation, though regulation is coming. Because a system that affects someone’s livelihood, health, or opportunity without explanation fails a basic test of moral respect. The technology to explain exists. The cost is measurable and manageable. The choice not to explain is exactly that: a choice. And it is the wrong one for any system that touches human lives.