Karpathy Autoresearch: How to Turn It Into a Marketing Experiment Engine

Karpathy autoresearch exposes how slowly most marketing teams test. With AI agents on the scene right now, one agent can now run a full experiment cycle every five minutes while your team sleeps. Obviously, a team running one test a week cannot compete with a system running a hundred a night!

When Andrej Karpathy released his repo last spring, I spent the weekend mapping it to marketing. In my breakdown of Karpathy’s autonomous AI, I argued the same loop works anywhere you have one variable to change and one number to score. Cold email, landing pages, ad creative, and sales scripts all qualify.

So what exactly is Karpathy autosearch and what does it do?

Below, I’ll cover what autoresearch does, how its three files translate into a marketing setup, and where the approach breaks so you can avoid the expensive mistakes.

TL;DR

  • Autoresearch is an agent that runs experiments on its own. It edits code, trains for five minutes, keeps only the winners, and repeats overnight.
  • The loop transfers directly to marketing. Any channel with one variable and one score, such as cold email or landing pages, can run it.
  • Your scoring rule decides everything. Fast metrics like reply rate work well, while slow metrics like SEO need proxy signals and patience.
  • The experiment history becomes your moat. Models are shared by everyone, but a log of what works for your buyers belongs only to you.

What Is Karpathy Autoresearch?

Andrej Karpathy, a founding member of OpenAI and former director of AI at Tesla, published autoresearch on GitHub as a small experiment in letting AI agents do research unattended. You give an agent a real but small language model training setup and walk away.

The agent modifies the training code, trains for a fixed five-minute budget, checks whether the validation score improved, and either keeps or throws away the change. Then it repeats. You wake up to a log of experiments and, hopefully, a better model.

The Three Files Behind the Loop

The repo stays deliberately small. Three files do the real work.

  1. prepare.py holds fixed constants, data prep, and evaluation. Nobody touches it.
  2. train.py contains the model and training loop. The agent edits this file.
  3. program.md gives the agent its instructions. The human edits this file.

Karpathy describes program.md as a lightweight skill. Instead of writing Python, the researcher programs the research organization in plain Markdown, which is the same shift marketers are seeing with AI skills instead of one-off prompts.

Why Five Minutes Per Run Changes the Math

A fixed five-minute budget means about twelve experiments an hour. Over a night, the agent can finish roughly a hundred runs. Run that every night for a year and you pass thirty-six thousand experiments, while a typical marketing team runs a few dozen.

Those numbers belong to machine learning training, where feedback arrives in minutes. Feedback in marketing is slower, which shapes how you design every loop below.

How Autoresearch Marketing Loops Work

Every growth channel already follows the same structure. You modify one variable, deploy it, measure one metric, keep or discard the change, and repeat. Most teams run that cycle by hand, a few times a quarter.

Video thumbnail

In the Leveling Up video on autoresearch, I walk through how this applies to cold email, landing pages, and ad creative. The pattern holds anywhere you have a feedback signal and a variable to optimize.

Mapping Karpathy’s Files to Marketing

My marketing version swaps each file for a business equivalent.

  1. baseline.md replaces prepare.py. It stores your ICP, positioning, brand rules, and guardrails, the material that never changes during a test.
  2. template.json replaces train.py. It’s the asset being optimized, such as an email, an ad, a landing page, or a call script.
  3. program.md stays program.md. It defines the campaign goal and the scoring rule.

Once those three files exist, the agent can run the loop without a person pushing buttons. The quality of baseline.md decides whether the variants stay on brand.

Where Marketers Can Point the Loop

In a follow-up post on business uses, I listed the areas where this pattern fits: landing pages, creatives, AEO, SEO, pricing, email, cold outreach, warm outreach, accounts payable and receivable, and procurement. Each one has a variable and a score, even if the score arrives slowly.

A hamster engineer turns a glowing loop wheel labeled modify, deploy, measure, and keep while email, ad, and landing page cards orbit around it.

With the setup clear, a real example shows how the pieces behave under pressure.

A Cold Email Example for AI Marketing Experiments

Cold outreach was the first loop my team built, because reply data arrives within days instead of months. Here’s the setup I described in the original breakdown.

The Setup

The loop used fifteen inboxes sending about three hundred emails per day. The agent changes one variable per experiment, sends one hundred emails, waits seventy-two hours, and scores the positive reply rate. Then it keeps or discards the change and starts again.

Variables included the subject line, opener, call to action, personalization depth, and framing angle. One variable at a time keeps the result readable.

The Target and Why Small Gains Compound

The starting baseline was a 1% positive reply rate, with a target of 2% or better after thirty days of autonomous testing. A single point sounds small. At nine thousand sends a month, it’s the difference between ninety interested replies and one hundred eighty.

I estimated that gap at roughly $500,000 in added pipeline at my team’s close rates. Treat that as one agency’s projection, since your deal size and close rate will move the number. If you want to go deeper on the outreach side, this breakdown of AI outreach agents and cold email covers the infrastructure.

Pro-Tip
Start with one loop that scores in under a week, such as cold email reply rate or paid ad click-through rate. Add slower loops only after the first one runs cleanly for a month.

That first loop teaches you more about scoring than any planning document, and it sets up the bigger question of running many loops at once.

Running Many Loops at Once: What to Expect

The same engine can run across every growth surface. In my case, seventeen loops were running at the same time, each with its own scoring speed.

Fast Loops and Slow Loops

Email and paid social score in days. A landing page needs enough traffic to reach significance, which can take weeks. SEO and AEO tests can take months before rankings settle.

Group your loops by feedback speed. Fast loops run continuously, medium loops run on a weekly cadence, and slow loops get fewer, bigger bets with proxy metrics like impressions or AI citations as early signals. For landing page tests, keep this checklist for continuous experimentation without breaking tracking close.

How Learning Spreads Across Channels

Each loop produces learning, and the learning travels. A subject line angle that lifts replies can inform an ad hook, which can inform a landing page headline.

Over time, the system builds a map of which hooks, numbers, and objections resonate with your ICP. Most teams run two or three experiments per channel each month. These loops run two or three per week, and across seventeen channels that adds up to roughly two thousand experiments a year.

Where Autoresearch Breaks in Marketing

Autonomous testing has real failure modes. The replies to my posts kept circling back to the same three concerns.

1. Your Scoring Rule Is the Bottleneck

The agent only optimizes what you measure. If you score opens instead of positive replies, you’ll get clickbait subject lines. Spend more time on program.md than on any other file, and write the scoring rule so a stranger could apply it.

2. Slow Metrics Need Proxy Signals

SEO results take months, so a nightly loop can’t wait for rankings. Use leading indicators like click-through rate from Search Console, impressions on new queries, or citations in AI answers. Then confirm the winners with a slower quarterly review. My team uses a similar approach for SEO A/B testing.

3. Guardrails Before Autonomy

An agent sending emails or changing live pages needs limits. Decide which actions it can take alone and which need human approval, then log every change. This framework for AI agent guardrails is a good starting point, and a weekly agent monitoring report catches drift early.

A robot gardener trims test plants in a fenced garden beside a gate marked human approval and a scoreboard on the wall.

Those guardrails sound like overhead, but they’re what let the loop run for months without surprises.

How to Start Your First Autoresearch Loop

You don’t need a GPU or a research team to begin. The marketing version runs on APIs, a sending tool, and a scoring script.

Pick One Channel and One Metric

Choose a channel where you already have volume and a metric that arrives within a week. Write baseline.md with your ICP, offer, and banned claims. Keep it short enough for a new hire to read in ten minutes.

Write the Scoring Rule First

Define exactly what counts as a win before the agent generates a single variant. Set a minimum sample size and a threshold for keeping a change. My team’s open-source autoresearch skill shows one way to structure variant generation and expert scoring for conversion copy.

Keep the Costs in Perspective

I estimated these loops at roughly $100 to $150 a month for compute, sending infrastructure, and scoring scripts. A junior marketer costs far more and runs fewer tests. People still own strategy and judgment, while the agent owns repetition.

Once the first loop is stable, add the next one. The compounding starts when several loops share what they learn.

Why the Experiment Graph Becomes the Moat

Agents will likely become commodities, since everyone can access the same models. The asset nobody else can copy is your experiment history.

A system running thousands of tests a year for one ICP builds a private record of which hooks work, which numbers convert, and which objections show up. A vendor or tool switch wipes out that record. Teams used to optimize campaigns, and the next generation will optimize learning speed.

If your team runs a few tests a quarter, start with one fast loop this month and let the system teach you how to scale it. Want experienced operators to build those loops for you? Single Grain’s AI transformation services team designs and manages AI marketing systems for growth-stage companies. Contact Single Grain to map your first experiment loop.

Frequently Asked Questions About Karpathy Autoresearch

If you were unable to find the answer you’ve been looking for, do not hesitate to get in touch and ask us directly.