Exception Desk — AI Delivery Exception Triage Agent

Shipping teams read hundreds of free-text exception reports a week — customer emails, driver notes, carrier scan codes — and have to decide, case by case, whether a parcel is delayed, lost, damaged or stuck in customs, then get it to the right team before the SLA runs out.

Exception Desk automates that first pass. The agent classifies each report and sets severity, retrieves the relevant handling policy through vector search and cites it, suggests the next step, and routes the case with an SLA. Anything below 75% confidence is held for human review instead of being auto-routed, so the automation never runs blind.

Measured, not assumed

The agent is scored on 175 labelled reports it has never seen, against a keyword-rules baseline:

• 98.9% accuracy vs 74.9% for the baseline
• Macro F1 0.988 vs 0.771
• 88.6% grounding — cited a section from the correct policy
• Confidence calibration, so the human-review threshold actually means something
• Every agent call logged for monitoring and governance

Analytics that avoid a trap

The carrier dashboard reports on-time rates with 95% confidence intervals, plus a route-mix adjustment. One carrier ranked 4th on raw numbers but 3rd once route mix was accounted for: half its deliveries go to rural addresses. Judge carriers on raw rates and you penalise the one serving the hardest routes.

The repo also includes a responsible-AI risk assessment and a mapping of each component to Copilot Studio, Microsoft Foundry, Power Automate and Power BI.