Transparency in AI Is a UX Problem, Not Just a Model Problem

Why is AI transparency primarily a UX problem?

The machine learning research community has produced effective interpretability methods (SHAP, LIME, attention visualization), but these outputs are legible only to data scientists, not to the end users who need to act on AI decisions.

I inherited a clinical decision support system that used SHAP values to explain its risk predictions. The system met every technical explainability standard. It generated feature importance rankings, partial dependence plots, and individual prediction explanations. Technically, the system was transparent. In practice, 89% of the clinicians using it reported that they did not understand the explanations and made decisions as if the explanations did not exist.

The SHAP output showed that “hemoglobin_a1c contributed +0.23 to the prediction.” A clinician needed to know: “This patient’s diabetes management history is the primary factor increasing their risk score.” Same information. Entirely different communication. The gap between technical explainability and user-legible transparency is an information design problem, and treating it as a model problem ensures it will never be solved.

What does effective AI explanation design look like?

Effective explanation design starts with the user’s decision context, not the model’s internal mechanics, and translates model outputs into the language and concepts that specific user populations already use.

When I redesigned the clinical explanation interface, I started with 12 hours of observation. I watched clinicians interact with the system. I documented the questions they asked when making decisions. They did not ask “which features contributed most to the prediction.” They asked: “Why is this patient flagged?” “What should I look at?” “How confident should I be in this score?”

I restructured the explanation interface around these questions. Instead of feature importance rankings, the system now displays: a plain-language summary of the top 3 contributing factors using clinical terminology, a confidence indicator (high/medium/low) derived from prediction uncertainty, and a list of similar past patients with known outcomes. The underlying SHAP computations were unchanged. The interface was entirely new. This is the same principle behind dashboard design as information architecture: the value is not in the data but in its presentation to the people who act on it.

How do you build explanation interfaces for different user populations?

Different user populations require different explanation modalities, and a single explainability approach cannot serve data scientists, domain experts, affected individuals, and regulators simultaneously.

• Domain experts (clinicians, loan officers, recruiters): Need explanations in domain-specific language, focused on actionable factors. I design these as short narrative summaries with the top 3 contributing factors translated into domain terms. Maximum 50 words per explanation.
• Affected individuals (patients, applicants, users): Need explanations that answer “why did the system make this decision about me” in plain language. I design these as templated sentences: “Your application was declined primarily because [factor 1] and [factor 2].” No technical jargon. No probabilities. Clear next steps if available.
• Regulators and auditors: Need technical detail, statistical rigor, and reproducibility. I provide full SHAP analysis, demographic performance breakdowns, and model documentation in structured formats. These are the reports that satisfy the EU AI Act’s transparency requirements.
• Engineering teams: Need debugging-oriented explanations, anomaly detection, and performance monitoring. I provide interactive dashboards with feature importance drift tracking and edge case visualization.

What are the implications for the explainability research agenda?

The explainability research community should shift investment from model-centric interpretability methods toward user-centric explanation design, because the bottleneck is no longer computing explanations but communicating them.

According to research published at CHI 2023, only 12% of deployed AI explanation interfaces were designed with end-user input, and user comprehension rates for standard interpretability outputs averaged 23%. We have an abundance of methods for explaining models to data scientists. We have a scarcity of methods for explaining model decisions to the people those decisions affect.

The fix is not more research into model internals. It is investment in information design for technical communication. It is user research. It is prototype testing. It is the same UX discipline that every other software domain has embraced. AI transparency will remain an unsolved problem as long as the field treats it as a machine learning problem. It is a human communication problem. The models are ready to be explained. The interfaces are not ready to do the explaining.