A large client required frequent changes to their quality assurance standards. The BPO partner handled all QA for this account and engaged FiveLumens to address the challenge. The required number of audits and the grading criteria changed often, which created unpredictable costs and slow response times due to the manual nature of the process.
The objective was to test whether AI could handle the accuracy and consistency required in these audits while providing cost flexibility and faster adaptation to changing requirements.
The Context
The client required thousands of call evaluations per week. Volume fluctuated weekly based on client priorities. Manual quality assurance used offshore analysts. When volume increased, labor had to be added. When volume decreased, the cost per evaluation rose because labor could not scale down smoothly. The pricing model was rigid: labor operated in step functions while demand moved continuously.
Grading criteria also changed periodically. Each change required updating documentation and retraining analysts individually across geographies and shifts. Calibration was needed after every change to ensure consistency across the team. This created a delay between the client requesting a change and full implementation.
The Problem
The manual QA system had three structural constraints.
First, cost did not scale with demand. Labor operated in step functions. When audit volume dropped but not enough to remove an analyst, the cost per evaluation increased. When volume spiked, new analysts had to be hired and trained. The pricing model could not flex with weekly fluctuations.
Second, response time to changes was slow. When the client adjusted grading criteria, documentation had to be updated, and each analyst retrained individually across geos and shifts. Full implementation could take up to a couple of weeks. Consistency suffered during the transition as some analysts operated under old criteria while others used new standards.
Third, calibration was required after every change. Human judgment at volume created variability. Multiple analysts across different locations and shifts needed to be aligned on the same standards. This compounded the delay and increased the risk of inconsistent grading during rollout.
The system could not deliver cost flexibility, speed, or consistency simultaneously. One constraint could be managed, but not all three.
The Change
Leadership worked with the client to define their grading standards during kickoff. This was not surface-level criteria like “did the agent ask probing questions.” This was a deep dive into the definition of the scorecard: understanding nuance, exceptions, and definitions. From this, the FiveLumens scorecard was designed.
The team ran FiveLumens alongside the existing manual process. Several hundred calls were evaluated by both systems. An independent reviewer compared the outputs. The comparison revealed errors in both approaches. The manual process showed a 10% error rate. FiveLumens initially showed inconsistencies in judgment.
The team iterated on the FiveLumens prompts, refining the scoring logic through multiple validation cycles. Each adjustment was tested against manual results until the data proved accurate. FiveLumens reached 97% accuracy before they scaled it. When audit criteria changed, they adjusted the prompts, validated accuracy with 10 to 20 test calls, and deployed the update. Changes that used to take weeks to roll out now took hours. The cost structure shifted from hourly labor to a pay-per-usage model.
The Results
Big Numbers:
Accuracy improved from 90% to 97%. Cost per evaluation dropped 25%. Response time to criteria changes reduced from weeks to hours.
Hidden Wins:
The system eliminated cost rigidity. Audit volume could increase or decrease without adding or removing labor. The pricing model scaled linearly with demand. Consistency improved because the AI applied the same judgment to every call. Human variability was removed from the process. The analysts previously assigned to manual QA were redeployed to higher-value tasks. The cost reduction did not result in headcount cuts but in better resource allocation.