feat(experiments): add missing experimentation tools. by rodrigoi · Pull Request #123 · PostHog/mcp

rodrigoi · 2025-09-08T05:47:34Z

Problem

The MCP tool is missing some key tooling for retrieve experiment exposures, results and to create and update experiments.

Changes

We've added new tools to the MCP server to complete the loop of experiment manipulation tools:

`experiment-exposure-query`

This tool enables LLMs to access the complete exposure time series using the experimentId.

Very Naive Exposure Chart

`experiment-metric-results-get`

This tool enables LLMs to get the complete set of metric results to provide analysis and interpretation of those results.

Experiment Result Analysis

Naive Credible Interval

`experiment-create`

This tool allows LLMs to create experiments after querying the user for a hypothesis, feature flags and events.

Claude created a new experiment	Experiment Ready for Launch

How did you test this code?

joshsny

Looks great 🙌

Left some small comments to address and then think we're good to merge it in. Do we want to do experiment-update / experiment-delete tools in a different PR?

schema/tool-definitions.json

typescript/src/schema/experiments.ts

joshsny

Oh and we'll need to add integration tests for the changes to the API client and for the new tools :)

rodrigoi · 2025-09-08T20:04:16Z

@joshsny ! Thanks for the review! I'll get your feedback in and let you know when it's ready for another review!

This PR got a bit large, so experiment-update / experiment-delete will come after. I would like to clean up the schemas a bit too.

joshsny · 2025-09-12T11:15:38Z

@rodrigoi FYI there are some issues currently with integration tests, so if those are failing you can ignore that for now, they are having intermittent failure issues

rodrigoi · 2025-09-15T00:47:46Z

@joshsny This is ready for another review. I caved and added the experiment-delete and experiment-update tools. An agent generated tests, and I'll take another look at the experiment schemas in the future because there's some duplication I'd like to remove. Otherwise, I tested with Claude and the MCP Inspector.

joshsny

Super nice, can't wait to get this merged, left some small comments before merging :)

typescript/src/api/client.ts

typescript/src/schema/experiments.ts

typescript/src/schema/tool-inputs.ts

typescript/src/tools/experiments/create.ts

joshsny · 2025-09-15T16:18:58Z

@rodrigoi tested this locally and it works well for me!

joshsny · 2025-09-15T16:28:19Z

@rodrigoi pushed a .nullish() on one of the schemas to get it working for me on team 2 :)

…s tool.

…e and generating python schema.

…ment schema.

…hemas.

…sformations and parsing.

…od schemas, transformations and derived types.

…tterns as the create tool.

…files.

rodrigoi · 2025-09-18T13:47:42Z

@joshsny I did a whole refactor pass over the schema to make more... idiomatic Zod, and removed some of the duplication. Updated integration tests (we need to update the scope of the token we are using today), rebased, and tested extensively, but I would like your final approval before merging 😉

joshsny

Looks great! Let's ship it 🚢 🙌

joshsny · 2025-09-18T14:56:49Z

Updated the API key and the integration tests are passing now, so we should be good to go 🚢

rodrigoi requested review from a team September 8, 2025 05:47

rodrigoi self-assigned this Sep 8, 2025

rodrigoi marked this pull request as ready for review September 8, 2025 07:04

rodrigoi force-pushed the experiments/results-and-create-experiment-tools branch from 18a74ac to 80ab599 Compare September 8, 2025 07:15

joshsny approved these changes Sep 8, 2025

View reviewed changes

schema/tool-definitions.json Outdated Show resolved Hide resolved

schema/tool-definitions.json Outdated Show resolved Hide resolved

typescript/src/schema/experiments.ts Outdated Show resolved Hide resolved

joshsny suggested changes Sep 8, 2025

View reviewed changes

joshsny self-requested a review September 8, 2025 19:56

rodrigoi force-pushed the experiments/results-and-create-experiment-tools branch from 9cc87df to 8d390da Compare September 14, 2025 23:40

joshsny approved these changes Sep 15, 2025

View reviewed changes

rodrigoi added 15 commits September 18, 2025 01:08

feat(experiments): adding missing experiment types to the schema.

0852b76

feat(experiments): adding exposure query and response schema.

1d4519b

feat(experiments): MCP can get the experiment exposure results.

22bb613

feat(experiments): fixing exposure results refresh.

1d0c65d

feat(experiments): scaffolding for the metric results get tool.

64375e1

feat(experiments): fixing the get metric results tool.

5bb7f4b

feat(experiments): adding create experiment tool.

dad4678

feat(experiments): adding a create experiment tool.

93fb7c6

fix(experiments): updating formatted files.

06f88ec

feat(experiments): allow for experiment editing.

0385a0e

fix(experiments): syncronizing python schema.

2d3f159

feat(experiments): adding tool titles.

a750f4d

tests(experiments): adding API integration tests.

b82a8cf

fix(experiments): adding experiment delte API handler

aad9de0

tests(experiments): adding experiment tools integration tests.

a393874

rodrigoi and others added 23 commits September 18, 2025 01:08

fix(experiments): code formatting.

3c789a7

feat(experiments): adding the update tool.

c1fa8dd

fix(experiments): code formatting.

671aec4

fix(experiments): removing exposure query tool in favor of the resutl…

ec10db3

…s tool.

fix(experiments): renaming get metric results tool.

4ec0fa2

fix(experiments): updating create tool description, removing dead cod…

fa80a10

…e and generating python schema.

fix(experiments): replacing some optionals with nullish on the experi…

b5413a1

…ment schema.

fix(experiments): fixing merge conflicts.

fa317fe

fix(experiments): adding missing properties to the tool definitions.

fd4e459

fix(experiments): results return the correct exposure information.

dd799d1

tests(experiments): updating test suite.

10c3f3e

fix(experiments): fixing typecheck errors.

d779a4a

fix(experiments): updating uuid dependencies.

c06b455

nullish on parametrs feature flag variants

aea34dd

feat(experiments): clean up of get all experiments tool client and sc…

b81bf27

…hemas.

refactor(experiments): destructuring parameters.

474109f

refactor(experiments): typing exposures and metrics client api calls.

355bdcb

refactor(experiments): typing metrics response and using Zod for tran…

a26e49f

…sformations and parsing.

refactor(experiments): experiment delete cleanup.

34a0b2c

refactor(experiments): updating the create tool with more idiomatic Z…

20f963a

…od schemas, transformations and derived types.

refactor(experiments): refactoring the update tool to use the same pa…

8624d0b

…tterns as the create tool.

refactor(experiments): updating integration tests and generated tool …

6947130

…files.

fix(experiments): updating typescript lockfile to solve merge conflicts.

ba9076f

rodrigoi force-pushed the experiments/results-and-create-experiment-tools branch from 5213279 to ba9076f Compare September 18, 2025 04:13

joshsny approved these changes Sep 18, 2025

View reviewed changes

rodrigoi merged commit a5ce513 into main Sep 18, 2025
11 of 12 checks passed

rodrigoi deleted the experiments/results-and-create-experiment-tools branch September 18, 2025 15:05

joshsny mentioned this pull request Sep 16, 2025

chore(main): release 1.0.0 #115

Closed

Conversation

rodrigoi commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

experiment-exposure-query

experiment-metric-results-get

experiment-create

How did you test this code?

Uh oh!

joshsny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshsny left a comment

Choose a reason for hiding this comment

Uh oh!

rodrigoi commented Sep 8, 2025

Uh oh!

joshsny commented Sep 12, 2025

Uh oh!

rodrigoi commented Sep 15, 2025

Uh oh!

joshsny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshsny commented Sep 15, 2025

Uh oh!

joshsny commented Sep 15, 2025

Uh oh!

rodrigoi commented Sep 18, 2025

Uh oh!

joshsny left a comment

Choose a reason for hiding this comment

Uh oh!

joshsny commented Sep 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rodrigoi commented Sep 8, 2025 •

edited

Loading

`experiment-exposure-query`

`experiment-metric-results-get`

`experiment-create`