The LLM Looked Smart. The Metrics Disagreed

This case from early 2025 is an interesting reminder that, even in this brave new world where models are increasingly commoditized through transformers and LLMs, the old concepts of data science still stubbornly refuse to die. Back then, Will Bank had one clear mission: credit. That was about to change. I was hired to help expand the product by offering free digital bank accounts, even to people who wouldn’t qualify for credit. I already touched on part of this story in “An Approval Model That Finally Got Approved”, but this chapter came with an entirely different headache. Among many operational issues, there was one problem loud enough to echo through every metrics dashboard: public brand backlash. People who were approved for a digital account without credit complained, loudly, across every imaginable channel. Social media, review platforms, app stores. The result? Lower brand affinity and, naturally, a bruised App Store rating. Because if there’s one thing users love, it’s…