GenAI Chatbot Quality at Scale:gen Z Solutions' QA Framework for a Banking Assistant

GenAI Chatbot Quality at Scale: Gen Z Solutions’ QA Framework for a Banking Assistant

1. Client Background

A mid-sized retail bank in the GCC region (we’ll call them NeoBank for confidentiality) wanted to launch a GenAI-powered customer support assistant across web and mobile. The vision was ambitious:

• 24/7, multilingual support (English + Arabic to start)

• Coverage for 70–80% of Tier-1 queries (balances, statements, card issues, charges, EMI schedules)

• Ability to hand off seamlessly to human agents for complex or high-risk cases

The assistant was built on top of a large language model (LLM) with a RAG (retrieval augmented generation) layer pointing to internal knowledge bases, FAQs, product documentation, and policy docs.

NeoBank’s leadership was clear on one point:

“We don’t just want a cool demo. We want a bank-grade assistant that is safe, compliant, and reliable at scale.”

That’s where Gen Z Solutions came in—owning the end-to-end QA framework for the GenAI assistant.

 

2. Business & Quality Goals

Together with the client’s digital, compliance, and operations teams, we turned the high-level idea into measurable goals:

·       Containment: Achieve 65%+ self-service resolution for Tier-1 queries within six months  of launch.

·         Safety & Compliance: Zero tolerance for responses that:

o   Reveal sensitive PII

o   Give personalised financial advice without disclaimers

o   Contradict regulated policy or local banking rules

·         Experience:

o   ≥ 4.5/5 average customer rating for assistant interactions

o   First meaningful response in < 3 seconds for 90% of sessions

·         Operational stability:

o   No more than 0.1% interaction failures due to model/tool errors

o   Clear observability and audit trails for all conversations

Our QA mandate was not just “find bugs,” but to design a repeatable framework that could keep quality stable as:

·         Models changed

·         New products were launched

·         Languages and channels were added

 

3. Key Challenges

From a QA and quality engineering perspective, three challenges stood out:

1.      Non-determinism

a.   The same question (“Why was my card declined?”) could produce different—but still valid—answers depending on context.

b.     Traditional “expected vs actual string match” tests wouldn’t work.

2.      Regulated environment

a.      Answers had to be consistent with internal policy, regulator guidelines, and legal  wording.

b.  Any hallucinated fee, interest rate, or process step could create compliance  exposure.

3.      Hybrid ecosystem

a.     The assistant talked to multiple systems: core banking APIs, CRM, ticketing, KYC  systems.

b.     Part of the answer came from the LLM; part from hard rules and tools. Testing   had to cover the entire chain.

 

4. Gen Z Solutions’ GenAI QA Framework

We designed a four-layer QA framework specifically for NeoBank’s assistant.

Layer 1 – Prompt & Policy Contract Testing

We treated prompts and system messages as contracts:

·         Version-controlled all prompts in Git alongside application code.

·         Created unit tests to ensure:

o   Mandatory disclaimers and safety instructions were always present.

o   Temperature, max tokens, and safety settings matched environment configs.

·       Built automated checks for “no-go phrases” (e.g., “I guarantee this investment return”,  “Use this advice to avoid paying tax”).

Layer 2 – Component-Level Evaluation

We validated each part of the pipeline separately:

·         RAG retrieval tests

o   Does the retrieval layer pick the right policy document for credit card disputes?

o   Are we pulling the latest circulars and not obsolete PDFs?

·         Tool invocation tests

o When a user asks “Show my last 3 transactions,” does the agent call the transaction-history API with correct parameters?

o  For “Freeze my card,” is the correct secure workflow triggered, including multifactor verification?

We built synthetic and real-data based test suites to exercise these components in isolation before testing full conversations.

Layer 3 – Scenario-Based Golden Datasets

Instead of hundreds of brittle test cases, we curated “golden datasets”—sets of real-world prompts mapped to expected behaviour:

·         400+ English prompts and 250+ Arabic prompts to start

·         Each labelled on multiple axes:

o   Accuracy (Does it match policy and account data?)

o   Completeness (Does it answer the whole question?)

o   Safety (Any PII leakage, hallucination, or advice beyond policy?)

o   Tone & empathy (Does it match NeoBank’s brand voice?)

We used a mix of:

·         Historical chat logs from the human contact centre

·         Inputs from branch staff and relationship managers

·         Edge cases from compliance and legal teams

These datasets became the regression backbone for every model or prompt change.

Layer 4 – End-to-End Conversational & Journey Testing

Finally, we tested full journeys, not just single turns:

·         Card lost → block card → order replacement → track new card

·         EMI query → explain schedule → calculate prepayment impact → generate “next steps”   summary

·         Dispute a transaction → log complaint → share reference number → explain SLA

We measured:

·         Resolution rate (Was the customer’s goal met?)

·         Number of back-and-forth turns needed

·         Escalation reasons (technical error vs policy vs user choice)

We automated large parts of this using scripts that simulate user conversations, with human-in-the-loop evaluation for tricky flows.

 

5. Safety & Compliance Guardrails

Banking meant no compromise on safety. We implemented:

·    Input filters to mask card numbers, CVV, and full account numbers before they ever  reached the LLM.

·         Output scanners that blocked responses containing:

o   Regulated terms used incorrectly (e.g., “assured return”, “guaranteed profit”)

o   Sensitive combinations like full name + account number + address.

·         Policy checks that ensured:

o   Interest rates, fee amounts and timelines were pulled from structured systems, not hallucinated text.

o   The assistant never initiated high-risk actions (like changing contact info) without passing the user through existing KYC and OTP flows.

Any response that failed checks:

·         Was replaced with a safe fallback message

·         Logged as a QA signal into our evaluation dashboards

·         Added as a new sample to the safety golden dataset

 

6. Observability & Feedback Loops

To keep quality stable at scale, we built observability specifically for GenAI behaviour:

·         Logged every conversation with:

o   Prompt, model version, tools used, retrieved documents, and final response

o   Customer rating (thumbs up/down) and escalation events

·         Created dashboards for:

o   Containment rate by intent (e.g., “card issues”, “loan queries”)

o   Safety incident counts and trends

o   Average quality scores from internal reviewers

·         Implemented weekly GenAI quality reviews with:

o   CX head, product owner, compliance lead, and Gen Z Solutions QA lead

o   Top 20 “problem conversations” dissected and fed back into prompts, documents or workflows

This turned QA from a one-time project phase into a continuous discipline.

 

7. Implementation Journey

We rolled out the framework in three phases over 16 weeks.

Phase 1 – Pilot Scope (8 Weeks)

• Covered 10 high-volume intents (balance queries, mini statements, card limits, branch timings, simple FAQs).

• Built the first golden dataset and evaluation rubric.

• Integrated assistant only on web banking for logged-in customers.

Outcomes:

·         52% containment on covered intents

·         0 critical safety incidents in pilot after 2 weeks of tuning

·         Strong buy-in from internal agents who saw fewer repetitive tickets

Phase 2 – Scale to More Intents & Channels (5 Weeks)

·         Expanded to 25 intents, including basic dispute workflows and EMI queries.

·         Added mobile app and WhatsApp as channels.

·         Tightened multilingual coverage for English and Arabic.

Outcomes:

·         63% overall containment for covered intents

·         Average assistant rating: 4.6/5

·         27% reduction in Tier-1 contact centre volume from pilot segments

Phase 3 – Optimisation & Governance (3 Weeks)

• Implemented feature flags for rapid testing of new prompt versions.

• Set up monthly red-teaming: QA, security, and compliance trying to break the assistant with adversarial prompts.

• Documented a GenAI QA playbook for NeoBank’s internal teams to follow going forward.

Outcomes:

·         Stable containment at 68–70% in live cohorts

·         <0.05% conversations triggering safety fallbacks after tuning

·         Clearly defined process for onboarding new banking products into the assistant with QA   sign-off.

 

8. Measurable Impact

Within six months of production launch, NeoBank saw:

·         30–35% reduction in Tier-1 volume routed to human agents in supported segments

·         2x faster first response time, especially during peak hours and weekends

·         Improved NPS among digitally active customers, with many calling out the assistant as “actually useful” compared to older bots

·         Operations team freed up to focus on high-value queries instead of repetitive FAQs

From a risk standpoint:

• No material compliance incidents attributable to the assistant

• All risky behaviours caught within internal QA logs and safety systems, not by customers

For Gen Z Solutions, the most important outcome was intangible:

The bank’s leadership started to trust GenAI as a managed capability, not a black box.

 

9. What Made This Framework Work?

A few principles shaped the result:

1.      Scenario-first, not feature-first
 We started from real customer journeys, not from a list of model capabilities.

2.      Golden datasets as living documentation
 Instead of long Word docs, the golden dataset became the real specification of “what  good looks like”.

3.      QA + Compliance + CX as one team
 We avoided siloed sign-offs. Every major decision had all three lenses in the room.

4.      Continuous evaluation, not just pre-launch testing
 Observability and weekly reviews ensured that quality improved month over month.

 

FAQs: GenAI QA for Banking Assistants

1. How is testing a GenAI banking assistant different from testing a normal chatbot?

Traditional chatbots follow deterministic flows—you test buttons and paths. A GenAI assistant generates variable, language-driven outputs. QA focuses on evaluation scores, safety checks and scenario coverage, not just fixed expected replies.

 

2. How did Gen Z Solutions reduce the risk of hallucinated fees or interest rates?

We never let the LLM invent numbers. All financial values (fees, rates, charges) were pulled from structured systems via tools/APIs, and responses were checked by output validators that compare text against authorised ranges and formats.

 

3. Can this QA framework work for other regulated domains (insurance, fintech, health)?

Yes. The core building blocks—golden datasets, guardrails, observability, risk-based rollout—are domain-agnostic. What changes is the policy layer and evaluation rubric tailored to each industry’s regulations.

 

4. How long does it take to stand up a similar QA framework?

For most banks and fintechs, a focused 10–16 week engagement is enough to:

·         Design the GenAI QA architecture

·         Build an initial golden dataset

·         Integrate evaluations into CI/CD

·         Launch a controlled pilot with clear guardrails

Timelines depend on system integrations and internal alignment.

 

5. What’s the next step for NeoBank’s assistant?

Together with Gen Z Solutions, NeoBank is now:

• Expanding to SME banking use cases (invoice queries, merchant support)

• Adding proactive insights (“Your utility spends are up 20% this month…”) with strict compliance review

• raining internal QA and CX teams to own the GenAI quality playbook long term