Black box vs glass box, what explainable AI actually means for a mortgage recommendation
Two terms get used almost interchangeably in AI marketing at the moment, black box and glass box, and the gap between them matters more in mortgage advice than in most other applications of this technology. A black box system is one whose internal reasoning cannot actually be traced by the people using it, even when the output itself is accurate. A glass box system is one where that reasoning can be traced. Most of what gets called explainable AI does not distinguish clearly between these two things, and in a market where a suitability decision has to remain defensible for years after it was made, that gap is not a minor detail.
What actually makes a model a black box
Complex models, deep neural networks and large ensemble models in particular, map a large number of interacting inputs to an output through so many internal parameters that even the people who built the system often cannot fully trace, in a way a human could read and follow, exactly why one specific case produced one specific result. This is not really a flaw. It is frequently what makes these models accurate in the first place, capable of capturing patterns that a simpler, fully interpretable model would miss entirely. Accuracy and explainability are two separate properties of a system, and it is entirely possible to have a great deal of one and very little of the other.
The trap, an explanation that sounds right is not the same as an explanation that is true
This is where most of the confusion in this market actually sits. A black box model can be wrapped in a layer that generates a plausible sounding narrative for why it produced a particular output, without that narrative being a faithful account of what the underlying calculation actually weighed. Researchers working on this problem draw a specific distinction between an explanation that is plausible and one that is faithful. A plausible explanation reads well and sounds reasonable. A faithful explanation is one that is verifiably tied to what the system actually did.
For a mortgage recommendation, this is the whole question. An affordability tool that returns the sentence recommended because of stable income and low loan to value sounds perfectly reasonable. Whether that sentence is a genuine description of what drove the calculation, or a narrative generated afterward specifically because it sounds reasonable, are two structurally different systems. From the outside, without asking the right question, they can look identical, and most people never ask the question that would tell them apart.
What glass box actually requires, three tests
Three things need to be true for a system to genuinely count as glass box rather than a black box with a friendly explanation bolted on.
The explanation has to be faithful, meaning it is verifiably tied to the actual computation behind the decision, not generated as a separate step designed to sound convincing.
The reasoning has to be traceable for a specific case, so someone looking at one individual recommendation can identify which inputs, income type, credit profile, loan to value, a specific piece of lender criteria, actually moved the outcome, rather than receiving a general description of what the model tends to consider.
And the outcome has to be contestable, meaning a human reviewer can see what would need to be different about the case for the recommendation to change, and can override the system with a documented reason. That last point only means something if the reviewer genuinely understands what the system weighed. An override based on a plausible sounding but unfaithful explanation is not really oversight, it is a signature next to a guess.
Why this matters more here than in most other uses of AI
A mortgage suitability decision is not judged at the moment it is made. It is judged, sometimes years later, by someone who was not in the room when the original recommendation was produced, working only from what was written down at the time. A faithful, traceable explanation is what makes that later review possible at all. A plausible one that turns out not to reflect what actually happened leaves nothing solid to defend.
The regulatory position reflects this directly. The FCA's AI guidance for financial services firms does not set a fixed explainability threshold that every AI application has to meet. What it expects is that firms genuinely know why their own models make the decisions they make, and it treats black box systems as increasingly untenable specifically in retail finance, where the consequences for an individual customer are significant and the products themselves involve advice that has to be justified. There is also a sharper edge to this than compliance paperwork. Without a faithful explanation, a firm cannot actually show why an AI system treated two broadly similar cases differently, which is precisely the kind of question that surfaces in a complaint about unfair or discriminatory treatment under the Equality Act 2010.
What to actually ask about any AI tool used in a mortgage recommendation
A few direct questions separate a genuinely glass box system from one that is marketed that way. For a specific past case, can the system show exactly which inputs drove the output, rather than a general statement of what it usually takes into account. If two similar cases received different outcomes, can the specific difference between them actually be pointed to. Is the explanation generated by the same process that produced the decision, or is it a separate step added afterward to make the output more presentable. And can a reviewer identify precisely what would need to change about the case for the recommendation to be different. A system that cannot answer these clearly, however good its output sounds, is a black box with better manners, not a glass box.
This is the specific meaning behind Mortgage Magic™'s own Glass-Box AI approach, discussed at a higher level in an earlier piece on this blog. The distinction is not a branding choice. It is the difference between a recommendation an adviser can actually stand behind months later, and one that simply sounded right on the day it was made.
The difference is not academic
Explainable AI has become a phrase almost every vendor in this market now uses, which is exactly why it is worth being precise about what it actually requires. A system that produces a reasonable sounding sentence after making a decision is not the same as a system whose decision can genuinely be shown. For a mortgage recommendation specifically, that is not a technical distinction that only matters to engineers. It is the difference between being able to defend a case when it is questioned, and discovering, too late, that there was nothing underneath the explanation to defend.
Comments
Post a Comment