AI for Product Analytics: Ask in English, Verify in SQL

TL;DR: Most product questions die in a queue. The PM wonders whether the activation drop is concentrated in one signup cohort, the data team is three requests deep, and by the time the answer arrives the decision shipped without it. Natural-language querying, a large language model translating questions into SQL or driving your analytics tool, collapses that queue for the large class of questions that are routine to express and tedious to write. The discipline this requires is different from the other product workflows: the errors aren’t fluent prose, they’re correct-looking numbers computed wrong, and they’re invisible unless you check. This guide covers the setup that prevents most wrong numbers and the verification tiers that catch the rest.

This guide is part of the AI for Product Management hub. It pairs with feedback analysis, numbers say what changed, feedback says why, and it supplies the evidence that prioritization scores against. For general spreadsheet-and-CSV techniques, see using AI for data analysis.

The failure mode to design around

Every other AI workflow in this cluster fails visibly if you look: a fabricated quote isn’t in the transcript, an invented requirement has no source. Analytics fails differently. The model writes a query; the query executes without error; a chart appears; the number is specific. And the number is wrong, because “active users” was computed from the wrong event, or the join fanned out and double-counted, or test accounts and internal traffic are in there, or the date filter used UTC while your reporting week doesn’t, or churned users were silently excluded by an inner join.

Nothing about the output signals the error. This is hallucination’s quieter sibling: not an invented fact, but a real computation of the wrong thing. Two design consequences follow:

1. Most wrong numbers are prevented by context, not checking. The model guesses wrong when it has to guess. Give it your real definitions and it mostly doesn’t have to.
2. The rest are caught by tiered verification. Not every query needs review, that would rebuild the queue you’re removing. Numbers that influence decisions do.

Step 1: write the metric-definition sheet

Before any querying, produce the single most valuable artifact in this workflow, one document containing:

• Exact event and table names for the concepts PMs ask about: what “signup,” “activation,” and “conversion” actually are in the schema, including the deprecated event that still fires and must be ignored.
• The canonical formula for your top ten metrics: what counts as “active” (which events, what window), how retention cohorts are cut, how revenue is recognized in the product data.
• Standard exclusions: test accounts, internal domains, staging traffic, the bot filter, and exactly how to apply them.
• Known traps: the timezone convention, the table that duplicates rows before deduplication, the date the tracking change made pre/post comparisons invalid.

This sheet goes into every analytics prompt, verbatim. If your data team maintains a semantic layer or dbt metric definitions, that’s the industrial-strength version of the same idea, point the model at it. If they don’t, this one-pager is an afternoon of work that eliminates the largest error class in one move, and the exercise reliably surfaces the uncomfortable discovery that your team had three coexisting definitions of “active user” all along.

Step 2: query in plain language, structured

With the sheet in place, the querying prompt:

Using this schema and these metric definitions: {sheet}. Question: {plain-language question}. Return: (1) your interpretation of the question in one sentence, state any ambiguity you resolved and how; (2) the SQL, with comments; (3) the standard exclusions you applied; (4) what could make this number misleading, joins that could fan out, cohort effects, definition edge cases. If the question is ambiguous in a way that changes the answer, ask before writing the query.

The interpretation line is the highest-value component. “I interpreted ‘churned users’ as accounts with no event in 30 days, per your definition sheet” either matches what you meant or it doesn’t, and catching the mismatch before reading the number is how you avoid anchoring on a wrong answer. Many analytics platforms and BI tools now ship this loop natively (question in, chart out); the same rule applies to them, the convenience is real, and so is the need to see what was actually computed.

Step 3: verify by tier

Match verification cost to decision cost:

The sanity checks, which take two minutes and catch a disproportionate share of errors:

1. Check against a known number. If the query says 40k weekly actives and the trusted dashboard says 25k, stop, and make “why do these disagree?” the next question. Disagreement usually means a definition mismatch, and finding it improves the sheet.
2. Recompute one different way. Ask the model for a second query using a different approach (different table, different aggregation path). Two routes agreeing is decent evidence; two routes disagreeing is a definition problem you just caught early.
3. Ask for the failure story. “Give three reasons this number could be wrong, and a check query for each.” Models are notably better at critiquing a query when asked directly than at volunteering doubts.
4. Gut-check magnitude. A 300% week-over-week jump is occasionally real growth and usually a duplicated join or a tracking change. Investigate before celebrating.

And the workflow rule that makes tiers stick: every number that leaves the team carries its query. A figure in a deck footnotes the SQL (or a link to it). This costs nothing at write time and converts “where did this number come from?” from an archaeology project into a click, it’s the analytics version of the no-quote-no-finding rule that runs through this whole cluster.

Step 4: summaries, anomalies, and the “why” boundary

Beyond querying, two adjacent uses earn their keep quickly:

• Dashboard-to-narrative. Paste the weekly metrics (or connect the tool) and have the model draft the metrics-review summary: what moved, by how much, against what baseline, with every claim tied to a specific number. Good prompt constraint: “flag anything that moved more than {your threshold}% and say what segment drove it; do not speculate on causes.”
• Anomaly triage. A model reviewing metric movements produces a decent list of leads: “trial-to-paid dipped for the March cohort; the dip is concentrated in mobile signups.” That’s a real head start on the investigation.

The boundary to hold: the model reports what and where; humans establish why. A model asked “why did activation drop?” will produce a fluent causal story assembled from plausibility, not evidence, and a plausible causal story is exactly the thing that ends investigations prematurely. The strongest “why” answers in product work usually come from joining the numbers with the verified feedback themes: analytics locates the leak, feedback explains it.

Teams that go further, models with live warehouse access running multi-step investigations as an AI agent, should go further on controls too: read-only credentials, query cost limits, and the same decision-grade review before anything reaches a deck. Access patterns where the model writes queries and only aggregates come back also keep row-level user data out of chat logs, which is where it never belonged.

Rollout order

1. Metric-definition sheet, reviewed by whoever owns the data, the afternoon that prevents the most wrong numbers.
2. Exploration tier for PMs, with the “never quote exploration numbers” rule stated explicitly.
3. Sanity-check habit plus query-attached-to-number convention, week two.
4. Decision-grade review path agreed with the data team. Frame it accurately: the queue now only receives verification work, not routine pulls, which is a better week for them too.
5. Baseline and measure per the measuring AI ROI playbook: question-to-answer time, share of decisions with numbers attached, and data-team routine-request load are the honest three.

FAQ

Can AI-generated SQL be trusted? To execute, yes; to be semantically right, only with your definitions in the prompt and tiered verification behind it. The signature failure is a clean-running query that answers a slightly different question than you asked.

Do we need a semantic layer before natural-language querying works? You need the one-page metric-definition sheet at minimum, exact events, canonical formulas, standard exclusions, known traps. A semantic layer is the scaled-up version of the same fix.

Is plain-language querying better than dashboards? It’s for a different job: dashboards monitor the questions you predicted, conversational querying investigates the follow-ups. Promote recurring follow-ups into dashboards.

Can AI find insights in product data on its own? It finds leads, anomalies, segment concentrations, candidate patterns. Each lead needs a verified query and a human judgment before it’s a claim, and causal stories remain human work.

Is product usage data safe to put into AI tools? Schemas and aggregates under business terms, yes. Keep row-level identifiers out of chats, prefer query-out/aggregate-back patterns, and give any warehouse-connected setup read-only credentials.

Next in this cluster: pair the numbers with the why from AI user feedback analysis, and feed both into AI roadmap prioritization. Or return to the AI for Product Management hub.

Not sure where your company stands? Take the free AI-Readiness Assessment.