Applied AI Engineer, Evals

Ciao is a small team of engineers. We build AI apps and agents that draft quotes, read purchase orders and turn job notes into invoices for companies with $50M to $500M in revenue. As an Applied AI Engineer on evals, you make sure those agents are right before anyone relies on them, and you show our clients how we know. This role is remote in the United States, with a trip to a client or to San Francisco about once a month.

What you’ll do

• Build the test sets for each new agent from the client’s own past work, like last quarter’s quotes or last month’s invoices.
• Measure how often an agent gets it right, where it fails, and what a person has to fix.
• Decide with the engineer on the project when an agent is ready for the client’s team to use.
• Watch agents in production and catch drift before the client does.
• Build the tools the rest of the team uses to run evals in minutes, not days.
• Try new models as they come out, and tell us plainly whether they are better for our work.

What we’re looking for

• You’ve built and run evals for LLM systems in production, not only on benchmarks.
• You write Python every day, and you’re comfortable with SQL and messy data.
• You know where LLMs fail: made-up numbers, missed fields, confident wrong answers.
• You can design a test that a client’s operations manager understands and trusts.
• You’d rather ship a simple check that runs every day than a perfect one that never does.

Nice to have

• You’ve worked with documents like purchase orders, invoices, drawings or freight paperwork.
• You’ve published or open-sourced eval work.

How we hire

• An intro call with Paul Sheridan, our founder and CEO.
• A working session where you build an eval with us for a sample agent and a set of real, anonymized documents.
• Meet the team.
• An offer.

Ciao is an equal opportunity employer.

Apply for this role →