Autotest
Run a pilot
Running in production at Academia.edu

Your funnels should be testing themselves

Run A/B tests 24/7 with no humans in the loop. Get alerts in Slack when something wins.

Put a funnel on Autotest
Academia ▾
Channels
general
growth
autotest-alerts
engineering
design
☆ # autotest-alerts 25
Autotest APP 12:21 AM
During the night I got a 34% CTR win on the citation alert email (citation_alert). 7 minutes later I released a new batch of 10 variants building on this win.
control vs quoted_name_card ▾
Control email: plain name card with a green button
Winning email: the name shown as a pull quote on a parchment card
Autotest APP 3:07 AM
I got a 2.4% trial-start win on the Premium funnel headline (premium_funnel_headline), and 7 minutes later I released a new batch of 10 variants building on this win.
control vs benefit_first_v2 ▾
Message #autotest-alerts➤

A real alert from Academia.edu’s Autotest channel.

The problem

You are leaving money on the table by not testing 24/7

Testing by handOn Autotest
Variants launched per week~20~2,000*
People needed per test40
Time from idea to resultWeeksDays
Ideas in the backlog that get testedA fewAll of them
Tests running at 3am0All of them

* Based on the case study from Academia.edu.

How it works

Set it up once. Then do something else.

  1. 01

    Pick a funnel

    A signup page, an upgrade flow, a checkout, a high-volume email. It needs enough traffic for A/B tests to reach significance.

  2. 02

    Pick a metric

    Trial starts, upgrades, purchases, click-through. One number for Autotest to improve.

  3. 03

    Choose how much to review

    Approve every variant, approve only winners, or let it run on its own.

After that, Autotest does the rest. It proposes a change, writes the code, runs the test against control, ships the winner as the new control, and proposes the next change, based on the win history. It runs around the clock and deploys to production on its own. You get a log of what it tried and what won.

ProductionDeploys · academia.eduLast 24 hours
WhenChangeDeployed byStatus
  • Tue 03:07Promote winner to controlpremium_funnel_headline · benefit_first_v2Autotest✓ Deployed
  • Tue 03:07Release batch of 10 variantspremium_funnel_headline · v3 – v12Autotest✓ Deployed
  • Tue 00:28Promote winner to controlcitation_alert · quoted_name_cardAutotest✓ Deployed
  • Tue 00:21Release batch of 10 variantscitation_alert · v8 – v17Autotest✓ Deployed
  • Mon 22:40Retire losing variantbulk_download_button · download_nowAutotest✓ Deployed

Every deploy was made by Autotest. No person opened a pull request, reviewed a diff, or pressed deploy.

A CASE STUDY · ACADEMIA.EDU

Academia.edu implemented Autotest and increased A/B testing velocity by 100x

A/B test variants launched per week, 2026 1,000 / week 2,000 / week Jan 2026FebMarAprMayJunJulAug 2026 Autotest goes live → 1,950 →

Before Autotest, Academia.edu’s growth team launched about 20 variants a week. Autotest went live in April 2026. By August it was launching about 2,000 a week.

The results

Trial starts doubled between January and August 2026.

Academia.edu is a research platform with a Premium subscription business. The number it cares most about is trial starts per month. In January 2026 Academia.edu was doing 24,000 trials a month. Autotest went live in the spring. In August, Academia.edu did 48,000 trial starts. Autotest accounts for about 12,000 of the increase. Autotest has driven about 8% a month growth in trial starts.

Trial starts per month · 2026

From 24k to 48k

24kJan 24kFeb 33kMar 33kApr 32kMay 30kJun 36kJul 48kAug

Growth came from several initiatives; Autotest accounts for about 12k of the 24k increase.

Highest-volume email · paper recommendations

Click-through rate doubled by optimizing each part separately

  • From name +36%
  • Visual design +21%
  • Subject line +19%
  • Preview text +13%
  • Body copy +6%

This is a free product email, not a marketing email. It now drives as many trial starts as Academia.edu's best marketing email. Its subject lines are on autoloop.

Two largest funnels
+40%

In the bulk download upsell on autoloop, and again in the top Premium funnel in hybrid mode

Bulk download: headlines, buttons, checklist copy and modal layout, shipped by the system with no review step, on top of a 65% lift the team had found by hand in Q1. Premium funnel: a designer's redesign, built and tested by Autotest, worth 6k trial starts a month.

What the changes look like

Three examples from Academia.edu

Autoloop · bulk download funnel

Small changes, shipped one at a time

Two of the changes Autotest made to Academia.edu's largest trial-start funnel. Each one was proposed, built, tested against control and shipped by the system, with no review step.

+25% and +15% on this screen. +40% in total.
Bulk download modal, with the two changes marked
The bulk download modal on Academia.edu, showing the 'Your paper' label and the 'Cancel anytime' line that Autotest added
  1. Labelled the reader’s own paper “Your paper” and highlighted it+25%
  2. Added “Cancel anytime · Full refund within 30 days” under the button+15%
Autoloop · paper recommendation email

Each part of the email, optimized separately

The old email showed a paper card with two buttons. The new one shows the first page of the paper, says "Not an excerpt, the whole paper", and leads with the title as a link. The from name, subject line and preview text were changed in the same way.

Click-through rate 2x
Before
The old paper recommendation email: a paper card with View PDF and Download PDF buttons
After
The new paper recommendation email: a page preview of the paper, 'page 1 of 28, full text unlocked', and the title as a link
Hybrid mode · top Premium funnel

A designer's redesign, built and tested by Autotest

Academia.edu's head of design mocked a new paper page with the PDF pack offer built into it. Autotest implemented it and ran it against the old page. It won.

+40% trial starts, about 6k a month
Before
The old paper page on Academia.edu with Download PDF, Download Full PDF Package and Translate PDF buttons
After
The redesigned paper page with the PDF pack offer built in

"Autotest has increased our A/B testing by 100x this year, and has been generating 1–2% a month growth in new trial starts. Initially we had humans review every test before it went out. We built enough guardrails that it now runs without humans in the loop."

Richard PriceCEO and co-founder, Academia.edu
Guardrails

Each funnel chooses how much autonomy it gets.

Academia.edu ran Autotest with a human reviewing every variant for its first few months. That harness is still there, and every funnel can use it.

Autonomy

Three settings per funnel

Review every variant, review winners only, or full autoloop. You can change the setting at any time.

Rollback

Anyone can pull a variant

Any variant can be stopped instantly by anyone on your team. Any shipped winner can be rolled back to the previous control.

Off limits

Some things need a human

Pricing disclosures, legal and consent text, and anything that changes what you ask a user for are not touched unless a human approves. A variant only ships if it beats control at significance.

Where this is going

Autotest is already starting to test algorithm changes. Next it will build and test features on its own.

Copy and design changes were the first thing Autotest could test. At Academia.edu it is now testing changes to the paper recommendation algorithm and to SEO. The next stages are research, so it generates its own ideas, and new features. Each stage can run with your team reviewing, or on its own.

  1. Stage 1

    Copy, design, layout

    Headlines, buttons, email components, page layouts, and designer-led redesigns in hybrid mode.

    Running at Academia
  2. Stage 2

    Functionality and algorithms

    New funnel steps, pricing pages, ranking and recommendation algorithms, SEO changes.

    Rolling out at Academia
  3. Stage 3

    Research

    Automated user interviews, competitor analysis, and data queries, so the system pulls ideas from a wider data-set than just the previous win history.

    Next
  4. Stage 4

    New features

    Autotest will use its research capabilities to propose a feature, build it, and then run an A/B test, all autonomously.

    Later

Stage 1 has been running in production since spring 2026. Stage 2 is working on the algorithm tests run so far at Academia.edu. Stages 3 and 4 are in development.

A pilot

Two engineers, one or two weeks, one funnel.

Find out in a couple of weeks whether Autotest can find compounding wins in one of your funnels.

  1. You pick one funnel. Account signup, an upgrade flow, a checkout, or a high-volume email, with enough traffic to support A/B tests.
  2. We wire it in. Two of our engineers work with your team for a week or two to deploy Autotest on that funnel.
  3. Your team is in the loop. You review each test before it goes out, and you decide whether a winning variant is deployed to production or we run another batch against control.
  4. You look at the numbers. After a few weeks you know whether it found wins in your funnel. If it did, expand it to more funnels. If it did not, it cost you two weeks of light attention.

A good fit if

  • You have a web funnel with real traffic, tens of thousands of visitors a month or more
  • You already run A/B tests and would like to run many more of them
  • The constraint on testing is people's time, not engineering capacity
  • You can give one funnel two weeks of light attention

We are taking a small number of pilots in Q4 2026.

Put a funnel on Autotest

pilot@getautotest.ai

Questions

Questions people have asked.

Does it replace product managers?

Human product managers will have more leverage. Ultimately they will oversee a network of AI product managers, and thereby build a lot more product. We see a similar process playing out with engineers shipping more code with Claude Code and Devin.

Does it touch our codebase?

Yes. Variants are real code changes behind a flag, not an overlay injected by a visual editor. That is why it can change layouts and flows, and eventually functionality, and why your engineers can read every variant in a pull request.

How is this different from Optimizely with an AI copy generator?

Optimizely with a copy generator is a tool. A person still decides what to test, sets up the test, and reads the result; the generator saves them some typing. Autotest is an agent, in the way Devin is an agent. You give it a funnel and a metric and it does the work: it decides what to test, builds the variant, runs the test, ships the winner, and starts the next one. The difference is the same as the difference between autocomplete and Devin.

What if it ships something bad?

A variant only ships if it beats control at significance on the metric you chose. Anything can be rolled back instantly. Pricing, legal and consent text need a human, and any funnel can stay in review mode for as long as you like.

What does it cost?

Autotest is sold in experiment credits. A credit is one A/B test, proposed, built, run and resolved. Bigger changes, such as a page redesign or an algorithm test, use more credits than a headline change. A human-run experiment costs a product manager, a designer, an engineer and an analyst a few days each; a credit costs a small fraction of that. Pilots are priced to be an easy yes.

Who built it?

Autotest was built inside Academia.edu by the team that runs its growth and AI products. It has been running in production there since spring 2026.