Explainable AI: why a model that's right isn't enough

Picture a model that gets it right nine times out of ten. Sounds good. Now picture having to tell a customer why their financing was turned down, or explaining to your head of operations why one support ticket jumps the queue. “The model said so” is not an answer. Not for the customer, not for an auditor, and not for your own team.

A model that’s right but can’t explain itself is a black box. And a black box only works for you until you need to make an important decision with it.

What explainability means

Explainability (often called XAI, for explainable AI) is a model’s ability to show what drove each result. It isn’t about following every internal calculation. It’s about answering two simple questions: which data moved this decision, and in which direction.

There are two ways to look at it, and it pays to keep them apart.

Global explanation. How the model behaves overall. For example: “in general, the strongest signals that a customer will leave are the number of complaints and the time since their last purchase.” It helps you understand the business and catch a model that has learned something absurd.
Local explanation. Why this result came out for this case. For example: “this customer is at high risk of leaving because they filed three complaints this quarter, even though their spending is still high.” It helps you act on a specific case and justify it.

A few techniques behind these explanations, in plain terms:

Decision trees and linear models. Models that explain themselves: you can read the rules or the weights directly. They’re often enough.
Feature importance. A global measure of how much each input contributes across all predictions.
SHAP. A method that splits a single prediction among its inputs, so you can see how much each one pushed the result up or down in that specific case. It comes from game theory and is described in Lundberg and Lee (2017).

Three places where it matters

Credit. A risk model can be very accurate and still be useless if it can’t say why. With a local explanation, the analyst sees that an application scores high risk because of heavy existing debt, not because of, say, a postcode. That difference is what separates a defensible criterion from one that isn’t. On top of that, the EU AI Act lists creditworthiness assessment of individuals among the high-risk systems in its Annex III. Those obligations were postponed to 2 December 2027 by Regulation (EU) 2026/1744, but the direction is clear: you will need to be able to justify decisions.

Customer churn. Knowing a customer has a 70% chance of leaving doesn’t tell you what to do. Knowing it’s because of late deliveries does. The explanation turns an alert into an action: call them, make it right, or fix the process that’s causing it.

Ticket prioritisation. If a model ranks support tickets, the team needs to see why one sits at the top: a key account, a fault that affects invoicing, a repeat issue. Without that, people stop trusting the list and go back to sorting it by hand.

When not to trust a model

Explainability isn’t only for justifying decisions. Above all, it’s how you find out when a model is fooling you. Be wary when:

The explanation makes no business sense. If the biggest driver is something that shouldn’t matter (the weekday a customer signed up, the sales rep’s name), the model has probably learned a coincidence in your data.
One input “guesses” the answer. If a single variable predicts almost perfectly, be suspicious. Often it’s information that’s only known after the event you’re trying to predict, and it slipped into the training data.
New cases don’t look like the training data. A model trained on a normal year can fail quietly in an unusual season. Reviewing explanations regularly helps you spot it early.
You can’t explain it to the person affected. If you couldn’t tell them why the result came out that way, don’t use it to make a decision about them.

That’s why, before reaching for the most complex model, we ask whether a simple one does the job. A decision tree that’s slightly less accurate but readable at a glance is usually worth more than an opaque model that’s slightly more accurate. When the problem genuinely calls for something more complex, we pair it with local and global explanations and keep a record of every result.

What sits underneath

None of this works without clean data. An explanation built on duplicated or incomplete records only explains the mess. So we start by understanding how you work, and by cleaning and joining your sources. The intelligence comes after, on top.

If you have a model that gets things right but nobody knows why, or a decision you’d like to ground in data without losing control of it, write to us at info@liberionlabs.com and we’ll take a careful look.