skip to content

← back to the plays

independence4 unconnected · 1 more author in those circles
standing

contested · 5 claims against 1

applies to

b2b and consumer, no audience needed

confidence

high

evidence

7 claims · 6 articles · 4 receipts

window

2011-10-03 → 2023-09-10

4 builders who never spoke to each other · contested

Do not split test below readable traffic

Below roughly a thousand conversions per variant a result is noise, and treating a few hundred clicks either way as an insight is how teams talk themselves into changes that do nothing. The practical substitute at small scale is to make the change, watch it, and revert quickly if it flops, and to reserve testing for questions you can genuinely run twice.

What both sides agree on

  • Reserve split testing for product-level questions you can actually run twice, like which button colour converts, and stop expecting it to settle career or life choices.

Where this splits

SparkToro, Gimenez and Cohen all report optimisation being worthless at low traffic, yet Cohen also argues you should keep running split tests knowing most will find nothing, because a rejected idea is still a result.

5 claims against 1

Dru Riley · Jason Cohen · Jordi Giménez · SparkToro

Skip split testing until the traffic is large

  1. 01Do not run an A/B test unless you can reach roughly a thousand conversions per variant, because below that you are reading noise.
  2. 02Split testing your site copy needs many thousands of visitors to say anything. Below that the results are inconclusive or the difference is too small to matter.
the objection · 1 claimJason Cohen

Keep testing anyway, a rejected idea is a result

Keep running split tests even though most of them find nothing: a test that rejects your idea is a result, not a wasted effort.

  1. 03Conversion optimisation is wasted at low traffic: lifting a rate from 0.9% to 1.1% on a hundred visitors a day is worth less than getting to a thousand visitors a day.
  2. 04Do not act on a test result until enough traffic has passed through it, because thin samples give a reading that will not hold.
  3. 05Make the change, watch it, and change it back if it flops, instead of running a test for weeks.

What it returned

  • 845

    SparkToro's example of the problem is treating 845 clicks on one button colour against 692 on another as an insight, and argues most small and medium businesses never have the volume to clear the bar they are citing as best practice.

  • Bugfender tried A/B testing its website content and found it not useful at their traffic level.

  • He states plainly that most of the time an A/B test will not turn up an improvement, and still calls it good practice.

  • SparkToro's marketing team does not A/B test its homepage copy at all: they talk to the audience, write what they believe will land, and fix it fast if it does not.

Sources

the second source

Three plays a week, for the phase you are in

No roundup of links, no news. Three tactics more than one builder arrived at separately, with the numbers each one returned and the disagreements left in.

One email a week. Unsubscribe in one click, and the address is used for this and nothing else.

More in landing page

all landing page plays

the second source

Three plays a week, for the phase you are in

No roundup of links, no news. Three tactics more than one builder arrived at separately, with the numbers each one returned and the disagreements left in.

963 corroborated plays to draw from · 3 a week

One email a week. Unsubscribe in one click, and the address is used for this and nothing else.