CAPR: what to ask, test and sign before an AI agent answers your customers

Record five things for every engaged call

1. 1How it endedresolved · handed over by design (the agent did its part, such as the identity check, then passed the call on) · transferred because the caller asked · transferred because the agent couldn't cope · abandoned (the caller hung up first).
2. 2Human involvementnone, or any. "Any" includes the vendor's own staff or contractors acting on the call before the outcome reached the caller.
3. 3Correct or wrongfrom your audit sample, checked against your system of record for actions and against your approved answer for information.
4. 4Repeat contactthe same customer, about the same thing, on any channel, within 7 days (or the window you set).
5. 5Complaintraised about the call, yes or no.

The rates

Questions for any figure a vendor quotes

• Out of which calls?
• Over what window?
• How are hang-ups and timeouts counted?
• Which call types, which customer, which period?
• Was anyone working behind the scenes?
• Who measured it?

Test in stages

1. 1ShortlistTwelve tests on each vendor's test number, with your own scripts: T01, T02, T04, T05, T12, T13, T16, T18, T24, T25, T34, T37. The aim is to rule vendors out cheaply.
2. 2FinalistsThe critical tests and the tests your tiers call for, on your production phone path (your carrier, your call routing), repeated. The vendor runs a load test at 1.5 times your forecast peak while you watch, and the concurrency limit goes in the contract.
3. 3PilotLive monitoring: every must-escalate call, a monthly audit of "resolved" calls, repeat contact and complaints.
4. 4Every monthCritical tests monthly and after every vendor change, a weekly sample of transcripts read by a person, a monthly audit of "resolved" calls and the full pack every quarter.

The critical tests

Every run must pass. Must-escalate tests (T10, T11, T12, T24, T48, T49 and T51): at least 30 wordings each. Others: at least 5 runs each, with different wording.

The rest of the test pack

Full pass conditions and sources are in the test pack CSV.

Compliance11 questions

1. C1

   For each stage of a call (telephony, recording, speech-to-text, language model, text-to-speech, storage, analytics, human review), which company processes our data and in which country? Include failover and backup locations.

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must provide, and keep current as a contract schedule, a data map naming for each processing stage of a call (telephony, recording, speech-to-text, language model, text-to-speech, storage, analytics and human review) every entity that processes Customer data and every country where it is processed or stored, including failover and backup locations.

   Red-flag answers

   • "Hosted in [country]", with no answer for the speech-to-text and model steps.
   • Failover, support or backup locations left out.
   • The model provider "can't be named".

   Minimum evidence

   Tier 1: DocumentedTier 2: CommittedTier 3: Committed

   Checked by tests

   T44

   Where it goes

   Data processing agreement, Security questionnaire, RFP

   Sources

   Office of the Australian Information Commissioner: Guidance on privacy and the use of commercially available AI products (opens in a new tab)

2. C2

   Which sub-processors touch our call data? How much notice do we get before one is added or replaced, and can we object or leave?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must maintain a list of every sub-processor that accesses Customer data, give the Customer at least [N] days' written notice before adding or replacing one, and allow the Customer to object and, if the objection is not resolved, to terminate without penalty.

   Red-flag answers

   • Notice given only by updating a web page.
   • No right to object or to leave.

   Minimum evidence

   Tier 1: DocumentedTier 2: CommittedTier 3: Committed

   Checked by tests

   T44

   Where it goes

   Data processing agreement, Contract

3. C3

   Is any of our call data (audio, transcripts, summaries, metadata) used to train or improve any model, yours or a third party's? Show us the contract clause and the account setting that prevent it.

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must not use, and must ensure its sub-processors do not use, Customer data (including audio, transcripts, summaries and derived data) to train, fine-tune or otherwise improve any model without the Customer's prior written consent.

   Red-flag answers

   • The promise is in a policy or FAQ, not the contract.
   • Training is on by default and you have to find the setting to turn it off.
   • "De-identified" or "aggregated" data is carved out of the promise.

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   None: check the answer against the contract and the evidence

   Where it goes

   Data processing agreement, Contract

   Sources

   Office of the Australian Information Commissioner: Guidance on privacy and the use of commercially available AI products (opens in a new tab) · Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab) · US Office of Management and Budget: M-25-22: Driving Efficient Acquisition of Artificial Intelligence in Government (opens in a new tab)

4. C4

   How long do you keep audio, transcripts, summaries and logs? Can we set retention for each, and how do we get proof of deletion, including from your sub-processors?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must keep each class of Customer data (audio, transcripts, summaries and logs) no longer than the periods the Customer sets, delete it on request and at the end of the contract, and confirm deletion in writing, including at sub-processors.

   Red-flag answers

   • Retention is fixed and can't be shortened.
   • Deletion excludes backups or the model provider's logs.

   Minimum evidence

   Tier 1: DocumentedTier 2: CommittedTier 3: Committed

   Checked by tests

   T44

   Where it goes

   Data processing agreement, Exit plan

   Sources

   Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab)

5. C5

   How does the agent tell callers they are speaking with an AI, and give our recording notice? What happens when a caller asks whether it's a person, or objects to being recorded?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must state that it is an AI system at the start of every call, before any substantive question, answer truthfully whenever asked, and never claim or imply that it is human. It must deliver the Customer's recording notice and, if a caller objects to recording, stop recording or offer an unrecorded alternative without refusing service.

   Red-flag answers

   • The agent has a human name and a personal backstory.
   • It says it is an AI only if the caller asks.
   • Recording can't be switched off part-way through a call.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T34, T35

   Where it goes

   Acceptance test plan, RFP

   Sources

   Office of the Australian Information Commissioner: Guidance on privacy and the use of commercially available AI products (opens in a new tab) · European Union: EU AI Act (Regulation (EU) 2024/1689), Article 50: transparency obligations (opens in a new tab) · Parliament of Australia: Telecommunications (Interception and Access) Act 1979 (opens in a new tab)

6. C6

   Which independent security certifications cover this product and every component that handles our calls? When was the last penetration test, and did it include attacks on the AI itself, such as those in the OWASP Top 10 for LLM applications?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must hold, and provide on request, current independent security certification (such as ISO/IEC 27001 or a SOC 2 Type II report) covering the service and every sub-processor that handles Customer data, and an independent penetration test completed within the last 12 months that includes the OWASP Top 10 for LLM Applications.

   Red-flag answers

   • The certificate covers the company, not this product.
   • The penetration test left out the AI components.
   • Buyers aren't allowed to test security on a trial.

   Minimum evidence

   Tier 1: DocumentedTier 2: DocumentedTier 3: Documented

   Checked by tests

   T37, T38, T40, T41, T42

   Where it goes

   Security questionnaire

   Sources

   NSW Health: AI Scribe RFP HT25057, Part B statement of requirements (opens in a new tab) · OWASP GenAI Security Project: OWASP Top 10 for LLM Applications 2025 (opens in a new tab)

7. C7

   Does the agent make, or substantially contribute to, decisions that significantly affect callers, such as eligibility, refunds or account changes? What will you give us to describe them in our privacy notice, and how can a caller ask for a person to review one?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must document each decision the agent makes or substantially contributes to that could significantly affect a caller, including the data used and the logic in plain terms, sufficient for the Customer's privacy disclosures, and must support a route for the caller to seek review by a person.

   Red-flag answers

   • "The agent doesn't make decisions", when it approves or declines requests.
   • No way to explain why a request was declined.

   Minimum evidence

   Tier 1: DocumentedTier 2: DocumentedTier 3: Documented

   Checked by tests

   T24

   Where it goes

   Data processing agreement, RFP

   Sources

   Parliament of Australia: Privacy and Other Legislation Amendment Act 2024 (No. 128) (opens in a new tab)

8. C8

   Which of the duties in our country and sector notes can the agent meet today, and how will we test each one?

   +−Requirement wording, red flags and evidence

   Requirement wording

   Before go-live the Supplier must show, by test, how the agent supports the Customer to meet each duty in a dated schedule of the Customer's country and sector requirements (which may draw on the CAPR notes), and must notify the Customer of any change to the service that affects them.

   Red-flag answers

   • "Compliance is the customer's responsibility", with no configuration support.
   • A claim to be "certified" against a law, with no named certification scheme or auditor.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   None: check the answer against the contract and the evidence

   Where it goes

   Acceptance test plan, RFP

9. C9

   For outbound calls: how do you check do-not-call registers, keep to permitted calling hours in the called person's time zone, identify us at the start of the call, and stop straight away when someone says no?

   +−Requirement wording, red flags and evidence

   Requirement wording

   For outbound calls the Supplier must check numbers against every applicable do-not-call register, call only within permitted hours in the called person's local time, identify the Customer and the purpose at the start of the call, provide an opt-out during the call, end the call on any refusal, and present a caller ID that can be called back.

   Red-flag answers

   • Calling hours are checked in the server's time zone.
   • The only opt-out is a later text message.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T47

   Where it goes

   Acceptance test plan, Contract

   Sources

   Parliament of Australia: Do Not Call Register Act 2006 (opens in a new tab)

10. C10

    How are callers served who can't use speech, or can't use it well: relay-service users, people with speech differences or hearing loss, people with limited English?

    +−Requirement wording, red flags and evidence

    Requirement wording

    The service must give callers who cannot use speech, or cannot use it reliably, a working alternative (relay calls handled without timeouts, keypad input, a callback or a person), and the Supplier must test it with such callers before go-live.

    Red-flag answers

    • Silence timeouts that cut off relay calls.
    • "Press 0 for a person" is the only fallback, and nobody checks it works.

    Minimum evidence

    Tier 1: TestedTier 2: TestedTier 3: Tested

    Checked by tests

    T20, T21, T22

    Where it goes

    Acceptance test plan, RFP

    Sources

    University of Illinois Urbana-Champaign: Speech Accessibility Project (opens in a new tab)

11. C11

    Health: for each feature, what is its intended purpose as your labelling, website and sales material state it, and what is its medical-device status in each market where we will use it?

    +−Requirement wording, red flags and evidence

    Requirement wording

    The Supplier warrants that no feature gives a patient-specific recommendation on diagnosis, urgency, care setting or treatment unless it holds the device authorisation stated for that purpose in each market of use (for example ARTG inclusion, FDA clearance, or UKCA or CE marking with its class); that it will not describe the Customer's deployment as triage or clinical assessment; that it will notify the Customer within [N] days of any change to a feature's intended purpose, claims or regulatory status, or any regulator contact about it; and that it can switch off an affected feature within [N] hours.

    Red-flag answers

    • A 'not a medical device' disclaimer next to marketing that describes triage.
    • Won't say which modules are regulated, or in which markets.
    • Relies on FDA enforcement discretion without naming the policy example.
    • The stated intended purpose doesn't match what the agent does in the red-flag test.

    Minimum evidence

    Tier 1: DocumentedTier 2: CommittedTier 3: Tested

    Checked by tests

    T51

    Where it goes

    RFP, Contract

    Sources

    Therapeutic Goods Administration: Understanding the health facility management software exclusion (opens in a new tab) · US Food and Drug Administration: Clinical Decision Support Software: guidance for industry and FDA staff (opens in a new tab) · Medicines and Healthcare products Regulatory Agency: Crafting an intended purpose in the context of Software as a Medical Device (SaMD) (opens in a new tab) · Medical Device Coordination Group: MDCG 2019-11 rev.1: qualification and classification of software under Regulation (EU) 2017/745 and 2017/746 (opens in a new tab)

See what vendors publish on some of these questions: the Vendor Trust Tracker.

Accuracy9 questions

1. A1

   How do we know every action the agent reports (a booking, a cancellation, a refund) exists in our system of record with the right details? What does it do when our system is slow or down?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must confirm an action to a caller only after the Customer's system of record has accepted it, must tell the caller when an action could not be completed and offer a callback or a person, and every confirmed action must be traceable to a system-of-record entry.

   Red-flag answers

   • Success is measured from the agent's own log.
   • It retries in the background after telling the caller it's done.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T01, T02, T03

   Where it goes

   Acceptance test plan

   Sources

   τ-bench: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (opens in a new tab)

2. A2

   How does the agent keep to our approved content and policies? What does it do when the answer isn't there, or when a caller pushes for an exception?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must answer questions about policies, prices, fees, eligibility and rights only from content the Customer has approved, must say when it cannot answer, and must not make commitments outside policy.

   Red-flag answers

   • "Hallucinations are rare", with no test results on your own content.
   • The agent can search the open web for answers.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T04, T05, T06, T07

   Where it goes

   Acceptance test plan, Contract

   Sources

   Civil Resolution Tribunal of British Columbia: Moffatt v. Air Canada, 2024 BCCRT 149 (opens in a new tab) · The Treasury: Review of AI and the Australian Consumer Law: final report (opens in a new tab) · OWASP GenAI Security Project: OWASP Top 10 for LLM Applications 2025 (opens in a new tab)

3. A3

   Which limits stop the agent agreeing to things outside our rules (amounts, quantities, refunds, exceptions)? Are they enforced by our systems, or only by the agent's instructions?

   +−Requirement wording, red flags and evidence

   Requirement wording

   Limits on amounts, quantities, refunds, credits and exceptions must be enforced by the Customer's systems or the Supplier's tool layer, not only by instructions to the model, and actions above thresholds the Customer sets must require approval by a person.

   Red-flag answers

   • The limits live only in the prompt.
   • The agent's system account can do anything a staff member can.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T08, T41

   Where it goes

   Acceptance test plan, Security questionnaire

   Sources

   OWASP GenAI Security Project: OWASP Top 10 for LLM Applications 2025 (opens in a new tab)

4. A4

   How does the agent recognise and route complaints, hardship, vulnerability, family violence and emergencies expressed in everyday words? Show us results across many different wordings.

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must recognise complaints, disclosures of hardship or vulnerability, and emergencies expressed in everyday language, record them, and route them to the Customer's defined pathway, with 100% recall on the Customer's high-severity test set before go-live.

   Red-flag answers

   • Detection depends on keywords such as "complaint".
   • Tested with a handful of scripted phrases.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T10, T11, T12

   Where it goes

   Acceptance test plan

   Sources

   Consumer Financial Protection Bureau: Chatbots in consumer finance (issue spotlight) (opens in a new tab) · Australian Securities and Investments Commission: Regulatory Guide 271: Internal dispute resolution (opens in a new tab)

5. A5

   How quickly does a caller who asks for a person reach one? Are there situations where the agent tries to keep them?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must offer a transfer to a person, or a booked callback where no one is available, at the caller's first clear request, except in situations the Customer has documented in advance, and must not loop or repeat retention offers.

   Red-flag answers

   • "Containment" is a target in the vendor's success plan.
   • The option of a person appears only after several failed attempts.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T24, T53

   Where it goes

   Acceptance test plan, Service level schedule

   Sources

   Consumer Financial Protection Bureau: Chatbots in consumer finance (issue spotlight) (opens in a new tab) · ACXPA: Australian Contact Centre CX Standards (opens in a new tab) · Parliament of Australia: Competition and Consumer Amendment (Unfair Trading Practices) Act 2026 (No. 64) (opens in a new tab)

6. A6

   What does the person receive when a call is transferred, and how quickly?

   +−Requirement wording, red flags and evidence

   Requirement wording

   On transfer, the service must pass the caller's reason for calling, a summary, the identity-check status and any details already captured to the receiving person or queue, so that the caller is not asked to repeat them.

   Red-flag answers

   • Transfers are cold, with no summary.
   • The summary lands in a system the person taking the call doesn't see.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T25

   Where it goes

   Acceptance test plan

   Sources

   ACXPA: 2026 Australian Contact Centre Best Practice Report: summary of key findings (opens in a new tab)

7. A7

   How does the agent check identity before disclosing or changing anything, including when someone calls on another person's behalf? Can a voice, alone or with easily found details, unlock anything?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must not disclose or change personal or account information until the Customer's identity process is complete, must apply the Customer's authority rules for people calling on someone else's behalf, and must not accept a voice match, alone or with easily obtained details, as enough for a high-risk action.

   Red-flag answers

   • Voice ID is the only check for account changes.
   • The agent reads details out to "help" callers verify.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T36, T38, T39

   Where it goes

   Acceptance test plan, Security questionnaire

   Sources

   Guardian Australia: Centrelink voice identification fooled by an AI clone of a journalist's voice (news report) (opens in a new tab) · US Federal Bureau of Investigation, Internet Crime Complaint Center: Public service announcement PSA241203: criminals' use of generative AI in financial fraud (opens in a new tab) · Australian Communications and Media Authority: Telecommunications Service Provider (Customer Identity Authentication) Determination 2022 (opens in a new tab)

8. A8

   How does the agent resist callers trying to override its instructions, and instructions hidden in content it reads? What does it do when one of our systems refuses it access?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must stay within its defined role when callers try to change its instructions, must treat retrieved content as information rather than instructions, and must not try to get round an access refusal. The Supplier must let the Customer test these behaviours before go-live.

   Red-flag answers

   • Red-team results are summarised but not shared.
   • Security testing needs a separate paid engagement.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T37, T40, T42

   Where it goes

   Security questionnaire, Acceptance test plan

   Sources

   OWASP Top 10 for LLM Applications 2025: OWASP Top 10 for LLM Applications 2025 (opens in a new tab) · OWASP Top 10 for Agentic Applications 2026: OWASP Top 10 for Agentic Applications 2026 (opens in a new tab)

9. A9

   Which of our everyday words, product names or clinical terms might the agent's safety filters wrongly refuse?

   +−Requirement wording, red flags and evidence

   Requirement wording

   Before go-live the Supplier must test the agent's content filters against the Customer's vocabulary list (brand, product, clinical and other everyday terms) and fix any wrong refusals.

   Red-flag answers

   • Filters can't be adjusted for one customer.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T09

   Where it goes

   Acceptance test plan

   Sources

   Fortune: Bank's chatbot objects to a customer using the bank's own name (news report) (opens in a new tab)

Performance8 questions

1. P1

   What is the gap between the end of a caller's speech and the start of the agent's reply, measured at the caller's end on our carrier path, at our peak? Give the median and the 95th percentile, and say how you measured it.

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must meet, and report monthly, a response gap (end of caller speech to first agent audio, measured at the telephony edge) of no more than [X] ms at the median and [Y] ms at the 95th percentile at the contracted peak concurrency, and must report silences over 2 seconds per 100 calls.

   Red-flag answers

   • One average figure from the vendor's own lab.
   • Measured at the server, not at the caller's end.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T13

   Where it goes

   Service level schedule, Acceptance test plan

   Sources

   Proceedings of the National Academy of Sciences: Universals and cultural variation in turn-taking in conversation (opens in a new tab)

2. P2

   How does the agent handle interruptions, a caller saying "mm-hm" while it talks, and long pauses from slow speakers?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The agent must stop speaking when a caller interrupts, must not treat brief acknowledgements as interruptions, and must wait for slow speakers, prompting once before offering other help.

   Red-flag answers

   • Interruptions are switched off by default.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T14, T15

   Where it goes

   Acceptance test plan

   Sources

   Full-Duplex-Bench: Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities (opens in a new tab)

3. P3

   Can you show results for each of our caller groups (accent, age, language, device), not only an average?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must report task success and key-detail accuracy for each caller group the Customer specifies, and no group may fall more than [N] percentage points below the overall rate.

   Red-flag answers

   • Results only on clean or US-accented audio.
   • Tested only with simulated callers.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T16, T17, T18, T19

   Where it goes

   Acceptance test plan, Service level schedule

   Sources

   NSW Health: AI Scribe RFP HT25057, Part B statement of requirements (opens in a new tab) · The Edinburgh International Accents of English Corpus: The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR (opens in a new tab) · Lost in Simulation: Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations (opens in a new tab)

4. P4

   Which languages are supported for voice, not only chat? What happens when a caller speaks one you don't support?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must list the languages supported for voice and, for any other language, route the caller under the Customer's language policy (interpreter, in-language team or callback) within [N] turns.

   Red-flag answers

   • The language list is for text, not voice.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T20

   Where it goes

   RFP, Acceptance test plan

   Sources

   Australian Bureau of Statistics: 2021 Census: nearly half of Australians have a parent born overseas (opens in a new tab)

5. P5

   Does keypad input work end to end on our phone path, and is it offered when speech fails?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The service must accept keypad input over the Customer's production telephony path, and offer it after two failed speech attempts to capture any identifier.

   Red-flag answers

   • Keypad input "depends on your carrier" and hasn't been tested.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T21

   Where it goes

   Acceptance test plan

   Sources

   IETF: RFC 4733: RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals (opens in a new tab)

6. P6

   What is your concurrency limit, what happens to calls above it, and what does it cost at our peak?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must support at least [N] concurrent calls without falling below the contracted response gap and accuracy, overflow any excess calls to the Customer's defined queue, and state pricing at peak.

   Red-flag answers

   • Concurrency is "unlimited", with no load test to show it.

   Minimum evidence

   Tier 1: CommittedTier 2: TestedTier 3: Tested

   Checked by tests

   T23

   Where it goes

   Service level schedule, Contract

   Sources

   Finance Sector Union: CBA backflips on customer service job cuts (union statement) (opens in a new tab)

7. P7

   Will you report outcomes on our definitions (verified resolution, containment, transfers by type, time to a person, repeat contact), per call type, with per-call records we can audit, including any human involvement?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must report monthly, per call type, the CAPR outcome measures defined in the contract, and provide per-call records (end reason, transfer reason, human involvement including the Supplier's own staff or contractors, and audit fields) sufficient for the Customer to verify them.

   Red-flag answers

   • Only "containment" or "automation rate" is reported.
   • "Resolved" includes calls where the caller simply stopped responding.
   • Per-call data costs extra, or the vendor chooses the sample.

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   T31, T32

   Where it goes

   Service level schedule, Contract

   Sources

   US Securities and Exchange Commission: Order instituting cease-and-desist proceedings: In the Matter of Presto Automation Inc. (opens in a new tab) · dated vendor documentation

8. P8

   What will each verified resolution cost us in total: platform, usage, telephony, integration and our own oversight time?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier's pricing must state every billable unit and its definition. Where charges are per resolution, a resolution means a verified resolution as defined in the contract, with repeat contacts within the window credited back.

   Red-flag answers

   • Per-minute billing includes silence and hold time, with no cap.
   • "Resolution" means whatever the vendor's dashboard says it means.

   Minimum evidence

   Tier 1: DocumentedTier 2: CommittedTier 3: Committed

   Checked by tests

   T31, T32, T45

   Where it goes

   Contract, RFP

   Sources

   dated vendor documentation

Reliability8 questions

1. R1

   What availability do you commit to, with what service credits, and where can we see your incident history?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must meet a monthly availability of at least [X]% for the service, measured end to end including its speech and model providers, with service credits for any shortfall, and must publish its incident history.

   Red-flag answers

   • Availability excludes the speech or model providers.

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   None: check the answer against the contract and the evidence

   Where it goes

   Service level schedule

2. R2

   Where do calls go when any part fails (telephony, speech, model, voice, our systems), and how quickly?

   +−Requirement wording, red flags and evidence

   Requirement wording

   If any component fails or degrades, the service must route calls to the Customer's fallback (a person, a message or a callback) within [N] seconds, without dead air, and tell the caller what is happening.

   Red-flag answers

   • Failover is "automatic" but has never been tested with a customer.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T26

   Where it goes

   Acceptance test plan, Service level schedule

   Sources

   Anthropic: A postmortem of three recent issues (opens in a new tab)

3. R3

   If we switch the agent off at our busiest hour, where do calls go, who answers them, and how do we switch it back on?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Customer must be able to switch the agent off at any time, for all or some call types, with calls routed to the Customer's fallback, and the Supplier must support a switch-off drill before go-live and at least yearly.

   Red-flag answers

   • Only the vendor can switch the agent off.

   Minimum evidence

   Tier 1: TestedTier 2: TestedTier 3: Tested

   Checked by tests

   T27

   Where it goes

   Acceptance test plan, Contract

   Sources

   Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab) · NSW Government: Artificial intelligence (AI) procurement essentials (opens in a new tab) · Australian Prudential Regulation Authority: Prudential Standard CPS 230 Operational Risk Management (determination No. 1 of 2026) (opens in a new tab)

4. R4

   Which models, voices and providers does the agent run on? What notice do we get before any change, can we re-test first, and can we roll back?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must name the models, voices and providers used, give at least [N] days' written notice of any change to them or to prompts or configuration that affects callers, let the Customer re-run its critical tests before the change reaches callers, and roll back on request.

   Red-flag answers

   • "We continuously improve the model", with no notice.
   • The model provider can't be named.

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   T28, T29

   Where it goes

   Contract

   Sources

   Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab) · NSW Government: ICT Agreement (ICTA) Cloud Module, clause 1.4, in Service NSW RFT SNSW2323 (opens in a new tab) · dated provider documentation · How is ChatGPT's behavior changing over time?: How is ChatGPT's behavior changing over time? (opens in a new tab)

5. R5

   What per-call records do we get (recording, transcript, outcome, systems called, failure reason, any human involvement), in what format, and for how long?

   +−Requirement wording, red flags and evidence

   Requirement wording

   For every call the Supplier must retain and provide to the Customer the recording, transcript, outcome, systems called, failure reasons and any human involvement, exportable in a documented format throughout the contract and at exit.

   Red-flag answers

   • Logs kept for only a few weeks.
   • Export only through the vendor's dashboard.

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   T31

   Where it goes

   Contract, Exit plan

   Sources

   Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab)

6. R6

   What counts as an incident for a voice agent, how fast will you tell us, and how do we stop the agent ourselves?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must notify the Customer within [N] hours of detecting an AI incident, including a wrong answer a caller acted on, personal information given to the wrong caller, a missed emergency or distress escalation, a false confirmation or an outage, and provide a root-cause report within [N] days.

   Red-flag answers

   • Incidents are defined as outages only.
   • Notice "as soon as practicable".

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   T33

   Where it goes

   Contract, Service level schedule

   Sources

   Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab)

7. R7

   Who owns you, how are you funded, where are your support staff and in what hours, and which customer like us will take our call?

   +−Requirement wording, red flags and evidence

   Requirement wording

   The Supplier must disclose its ownership and any change of control, provide support during the Customer's business hours, and give at least one reference customer of similar size and call mix.

   Red-flag answers

   • No reference customer at a similar scale.

   Minimum evidence

   Tier 1: DocumentedTier 2: DocumentedTier 3: Proven

   Checked by tests

   None: check the answer against the contract and the evidence

   Where it goes

   RFP

8. R8

   On exit, what do we get back (call flows, prompts, knowledge content, test sets, recordings, transcripts), in what formats and how fast? Do our phone numbers come with us?

   +−Requirement wording, red flags and evidence

   Requirement wording

   On termination the Supplier must return all Customer data and configuration (call flows, prompts, knowledge content, test sets, recordings, transcripts and reporting data) in documented formats within [N] days, support porting of the Customer's numbers and a transition period of at least [N] months, and confirm deletion.

   Red-flag answers

   • Prompts and call flows are treated as the vendor's intellectual property.
   • Your numbers sit on the vendor's carrier account.

   Minimum evidence

   Tier 1: CommittedTier 2: CommittedTier 3: Committed

   Checked by tests

   T30

   Where it goes

   Exit plan, Contract

   Sources

   Digital Transformation Agency: Artificial Intelligence (AI) Model Clauses, version 2.0 (opens in a new tab) · NSW Health: AI Scribe RFP HT25057, Part B statement of requirements (opens in a new tab)