Six months ago I wrote that you should ship conversational analytics without perfect architecture. To prove I wasn't blowing smoke, I'll walk you through exactly what we did at Underdog to achieve this.

To date we've moved most of our analytics to be conversational, replacing dashboard development with over ten thousand conversations from more than 200 people across Slack, Claude, and the Hex application.

Your follow up is probably, “how do you trust it?”

Short answer, you can't out of the box. But, by deploying thoughtfully, designing a system that version controls your context, and cataloguing every conversation and analytics run as pieces of information for continuous improvement you can get pretty close.

I'll walk you through that below.

0 /TLCR

too long, claude read. Three things I'd tell someone starting today, none of which are about model choice.

1. Ship before the architecture is perfect. The context layer is what makes questions answerable, and it can be built in parallel with your SQL.
2. Ship four files, not two. SQL and YML say what a table is, and have been around for a while. A semantic model says which measure to trust and how to jump to other tables. A guide is the best place for common agent pitfalls, FAQs, and known gaps.
3. Put the context in your repo, and treat it as code. Analytics has already moved to the repo, so it's time to put the context there too. Set up linting, evaluations, and CI/CD checks, and give it a hub-and-spoke shape with model-level scrutiny the same way you would your data models.

Okay, now if you actually care past that, read on.

1 /What Each PR Ships

is now four files, not two like it was 6 months ago.

A delivered model used to mean a .sql file and a .yml file (and even the .yml file was the first thing to get skipped). Now a table should land with four artifacts, because each one adds the context to a picture that our robot friends need. Taking one real look at a users mart:

semantic_models:
  - name: users
    description: |
      Grain: 1 row per user.
      This table measures users and populations against marketing,
      product, and finance definitions as of current state.
    model: ref('users')
    defaults:
      agg_time_dimension: created_at_date

    entities:
      - name: user_id
        type: primary
      - name: account_id
        type: foreign

    dimensions:
      - name: created_at_date
        type: time
        type_params:
          time_granularity: day
      - name: country
        type: categorical

    measures:
      - name: user_count
        label: User Count
        description: >
          [Aliases: Registrations, Total Users]
          Distinct count of user_id, the registration base.
        agg: count_distinct
        expr: user_id

version: 2

models:
  - name: users
    description: |
      [Endorsement: Gold] Analyst-ready user data. One row per user.
      Refresh cadence: every 15 minutes.
    config:
      meta:
        owner: "{{ var('analytics_engineering_team') }}"
        subscribers: "{{ var('marketing_analytics_team') }}"
        criticality: critical
    columns:
      - name: user_id
        data_type: string
        description: Unique identifier for the user (primary key).
        data_tests:
          - unique
          - not_null

      - name: account_id
        data_type: string
        description: Foreign key to the account the user belongs to.

      - name: created_at_date
        data_type: date
        description: UTC date the account was created.
        data_tests:
          - not_null

      - name: country
        data_type: string
        description: >
          Country on the account's current address. Current state;
          it changes if the user moves.

---
name: Users Table Guide
description: "Load when using the users table, or when a question
  about users or registrations fires."
---

# Table Guide: users

## Employees: always exclude

Internal accounts sit in this table alongside real ones. Almost
every question people ask is about actual customers, so filter
internal accounts out by default. **Agent rule:** apply this
silently; no need to disclose it.

## Known data gaps

This table is current state only. It reflects users as they are
today, not as they were during the period you are reporting on.
For historical analysis use the daily snapshot table, not this one.

2 /Context as a graph

is an attempt to land a piece of information in a single home, in a hub-and-spoke model, with progressive disclosure.

We have over a hundred guides and skills, able to be accessed by all tools that need them. We also work very hard to continuously tune them based on actual usage. Each saves information flagged from previous sessions (a missed definition, a long agent run, user doubt) to minimize the chance it happens again. Every time a new person starts at your job, you have to train them... Every time an agent starts a task, you have to train them.

So, have an issue fire once and if it's important or common update the training manual that gets read.

Treating context as a production system

Silent failure is the real enemy in any context system. It doesn't build tables and it can get tangled in agent thought patterns. This is why you should lint your context like any other piece of code in your repository, and always check, and recheck, its validity.

Linting. Some rules for linting:

• Checking front matter.
• Checking for broken escape hatches.
• Checking for broken links to other guides in the hub-and-spoke model.

Validation. Some rules for validation:

• Checking against other guides and metadata, so things aren't competing.
• Setting up automatic checks that query the warehouse to validate assertions.

3 /Your AGENTS.md

is the file you will rewrite more than any other.

If you didn't know: this file is always read into every session by every agent that can read it. Skills and guides are only read on a pattern match.

When you start, don't overthink where to land context. Just put it in the AGENTS.md. As that bloats, grow the graph above. Over time you'll find what really matters for every question: convert all answers to Eastern time, only report on actual customers, always read the semantic layer first. Things that touch 80% of users go in the AGENTS.md, and don't forget you can link out to your context graph for the heavier topics.

We did not start here

It would be easy to say we designed it this way from the start. Not true. Each guide showed up when retrieval got noisy enough to justify it, which you can only see if you're capturing what gets asked and what gets loaded. So start there: log the questions and the retrievals, read them, and let the shape follow over time.

Don't get bogged down in where to place something. Start with AGENTS.md, move to a guide, move to a sub-guide, rinse and repeat. For us, we had the AGENTS.md file. Then we had a bloated product section, so we created a product-specific guide. Then we had a bloated A/B testing section inside the product guide, so we created an A/B testing guide.

What our AGENTS.md and skills look like today is not what it will look like in three months. So don't sweat it and just start building.

4 /Conversational Analytics Example

An analyst asks:

can you segment accounts by registered user attributes to help me come up with an a/b test for marketing?

Define once, use everywhere

on every tool call, in an optimized fashion. Here is what that one question actually loads:

A note on training users

Like this is new for you, it will also be new for your users, feeling the power of messy tables at their fingertips. We implemented a stoplight system to help train users, which has proven very important and effective. It grades every conversation across three dimensions, each scored between 1–5, with an inspirational quote.

Green needs a 4 as its lowest score and a 4.5 average. Yellow needs a 3 and a 3. Below that it's red. The average is shown for context but never sets the colour.

Yellow  (3.0 min, 4.0 avg)

"All models are wrong, but some are useful." (George Box)

  Question Clarity  4/5   Metric and population explicit; date range
                          defaulted to the last full month
  Data Coverage     3/5   Not in the semantic layer, joined the entries
                          and deposits marts, grain confirmed on user_id
  Thread Health     5/5   No warnings raised

  To earn a 5 on Data Coverage, ask for this as a metric and we'll
  define it in the semantic layer.

5 /You + Claude Code Example

Same machinery, pointed at itself. The question is about the system rather than the data, so what comes back is not a number; it is a pull request.

You ask:

what questions are being asked by marketing related to a/b testing, and where have we had gaps?

A note on development

If you were not in TLCR mode, you may have noticed the lines to a plan and validation file. We baked this into the deliverable for PRs, both in the checklist and in our skill files, so context is always getting built alongside the code as an artifact. The plan saves exactly what your agent set out to do, and the validation confirms that it accomplished the plan with real data and validation queries. A hand-wavy person makes for a hand-wavy agent…

# <entity>: convert to incremental

## Overview
Full rebuilds now dominate the nightly run. Convert to incremental
without changing a single output row.

## Design decisions
- insert_overwrite on the event date, NOT merge. The PK is a random
  UUID, so clustering can't prune and a merge scans the whole target.
- Replace window = 7 days. Later arrivals are rare and the nightly
  full-refresh selector catches them.

## File locations
models/<entity>.sql
models/schemas/<entity>.yml

## Implementation tasks
1. Add the incremental config + the is_incremental() filter
2. Add the PK uniqueness test
3. Write the validation protocol, before building anything

## Deployment & backfill
One --full-refresh in prod after merge. Nightly selector after that.

# <entity> incremental: validation

Plan: docs/plans/<date>-<slug>.md
Env:  built to dbt_<me> in staging, compared against prod.

## Claims under test

1. Full-refresh parity. A --full-refresh build equals prod row for row.
   Metric: row count + full-row fingerprint, per month.
2. The real incremental run is correct. A plain `dbt run` exercises the
   is_incremental() branch and leaves the same content. PK stays unique.
3. Idempotency. Re-running with no new upstream data moves no partition.

## Results, dev build <date>

select
    date_trunc(created_at_date, month) as month,
    count(*) as row_count,
    bit_xor(farm_fingerprint(to_json_string(t))) as fingerprint
from `<project>.<dataset>.<entity>` as t
group by month
order by month

  month     dev rows     prod rows    fingerprint
  2026-06   8,241,006    8,241,006    match
  2026-07   8,902,377    8,902,377    match
  2026-08   9,455,120    9,455,120    match
  2026-09   3,118,904    3,118,904    match

- full refresh ....... 29.7M rows,  84.2 GiB,  51s
- real incremental ...              1.8 GiB,  78s   (~47x less scanned)
- PK unique + not_null: PASS after the incremental run
- idempotency: run 2 = 1.8 GiB, no partition counts moved

Claim 2 initially FAILED: the is_incremental() filter used the load
timestamp, not the event date, so a late-arriving row landed in the
wrong partition. Fixed, re-run, PASS.

## Post-merge, appended once live
- expected partitions present in prod: PASS
- PK still unique in prod: PASS
- nothing written outside the intended window: PASS

6 /Bots and their harness

become powerful, and easy to prove out, once you've set real users loose on the system you built. You can take the goodness humans got out of the LLM and push it toward automation.

If you've never built an agent and the idea is terrifying, know this: it's a skill you schedule to hit the Claude API on a scheduled job. That's it.*

* Oversimplified, but good enough.

Our fleet of scheduled agents runs on GitHub Actions (to be moved to something better over time, #outages) and every one of them uses the same harness:

1. Select: which assets to look at for a particular job or task
2. Fan-out: run targeted agent loops in parallel, not sequentially
3. Tool-Loop: how to read and use tools to completion
4. Verify: hand the work off to a stricter agent to review it
5. Deliver: write a report, like we did with humans above, and open a PR with the report and the edits

Agents are just specific skill abstractions for completing a task, uploaded into that harness. Here are some of ours:

Each agent customizes only the Select and Tool-Loop parts of the harness; everything else is standardized. That's what lets roughly 80% of the code be shared across the harness and the skills; only the agent-specific protocols differ.

Here is one of them end to end, on the same three rows as the two examples above:

7 /Evaluations

the next frontier.

We've used them to test the context we already have, and we're working on productionizing them so that context gets treated like code. Evals can do double duty: they watch the foundational metrics, and they surface the gaps in the context at the same time.

We think about the suite on two axes. First, how we grade:

• E1 · correctness. Is the number right when the ask is well-defined? Deterministic ground truth over a fixed window.
• E2 · consistency. On an ambiguous ask, is the answer consistent and clearly labelled across three attempts, and does it clarify when it should?
• E3 · path. Did it use the right surface? Semantic layer first, the correct table, no wandering into raw or deprecated models.

Second, what is being asked. A ladder of ambition, adapted from George Xing's analytics-question hierarchy, with one rung added underneath it:

Both axes in one view:

8 /Next up for v1.1

Here are the next three things we're going after, in rough order. When accomplished I will write another article you can hand to your robot to read.

Evals scheduled, and hooked into CI. Right now they're an authored suite we run in an ad hoc fashion. They should run like everything else in the fleet: as part of our agent suite and as part of the PR process.

The knowledge graph, exposed as an endpoint. Our data is accessible, and our knowledge graph is really only accessible when paired with data questions. Analytics is the front lines of ontology, taxonomy, and definitions out of necessity… That means what we've built can be incredibly valuable outside the confines of data questions.

State-aware optimization. Dashboards used to cover this by accident: a fixed set of queries, cached and shared by everyone who opened the same tab. Conversational analytics gives every question its own distinct flavour, so almost nothing gets reused. What we want is tooling that caches and shares on the fly; Iceberg and DuckDB are where we're looking.