Black box vs glass box, what explainable AI actually means for a mortgage recommendation
Two terms get used almost interchangeably in AI marketing at the moment, black box and glass box, and the gap between them matters more in mortgage advice than in most other applications of this technology. A black box system is one whose internal reasoning cannot actually be traced by the people using it, even when the output itself is accurate. A glass box system is one where that reasoning can be traced. Most of what gets called explainable AI does not distinguish clearly between these two things, and in a market where a suitability decision has to remain defensible for years after it was made, that gap is not a minor detail. What actually makes a model a black box Complex models, deep neural networks and large ensemble models in particular, map a large number of interacting inputs to an output through so many internal parameters that even the people who built the system often cannot fully trace, in a way a human could read and follow, exactly why one specific case produced on...