Software Engineering / ENGINEERING CASE STUDY
In-App AI Agent in Production
The in-app AI assistant of a UK social-care SaaS, running live. Its tools are serverless TypeScript functions, publishing is a dry run unless told otherwise, and multi-organization users no longer get answers for the wrong organization.
THE PROBLEM
Keep a live AI agent safe to change
The challenge
The startup's in-app AI assistant is an LLM agent running live, with 34 tools and 31 skills. Tool changes ship straight to production, and staff who belong to more than one organization must only get answers about the one they're working in.
My role
I build and ship the agent's tools, and the process for shipping them.
What I did
- Each tool is a serverless TypeScript function the agent calls.
- Publishing goes through a CLI that is a dry run by default and only writes with --apply.
- Wrote a reusable Claude Code skill for shipping agent tools.
- Found and fixed a bug where multi-organization users got answers for the wrong organization: the server took the user's first membership with no ordering.
The tradeoff
A dry-run default adds a step to every publish. That's the point: the command that changes production has to be typed on purpose.
THE RESULT
The agent runs live, and multi-organization users get answers for the organization they are actually in.
LLM agent · TypeScript · Deno · Node (ESM) · Claude Code
HAVE A SIMILAR CHALLENGE?