skip to content

← back to the plays

independence5 unconnected
standing

contested · 4 claims against 1

applies to

b2b, no audience needed

outcome not stated

confidence

high

evidence

8 claims · 7 articles · 4 receipts

window

2020-10-30 → 2023-08-08

5 builders who never spoke to each other · contested

Publish the method, not just the finding

Where buyers judge credibility the way researchers do, release the data sources, the code and the paper behind the claim, including the sources you dropped, and invite outside experts to try to break it. Test any new data against something you already know well before trusting it, and change what your own product reports the moment the evidence says it measures the wrong thing.

What both sides agree on

  • Trust a new data source only after you run it on something you already know well, such as your own site or your own social account, and see whether the answer matches reality.
  • Change the metric your own product reports the moment your data shows it is measuring the wrong thing.
  • Swap an absolute figure for a relative score when customers need to compare across cases.

Where this splits

Publishing the method and the data is presented as what earns credibility, and separately as no substitute for distribution, since buyers would not adopt an open sourced technology while no company stood behind it.

4 claims against 1

Jorn Rigter · Josh Nicholson · Julia Angwin · Rand Fishkin

Publish the method, data and code openly

  1. 01Publish your method, your data and your code, and invite outside experts to try to break your conclusions.
  2. 02When your buyers judge credibility the way scientists do, publish your method: a paper, a preprint, and the data behind it.
the objection · 1 claimAli Ghodsi

Publishing is not distribution, someone must sell it

Open sourcing your technology and publishing the research is not distribution. Buyers will not adopt it while no company stands behind it.

  1. 03Measure the outcome your product actually produces, then publish the number and the method you used to get it.
  2. 04Publish the actual data sources behind your product, including the ones you dropped.

What it returned

  • 35

    Screen time fell by an average of 35 minutes a day for free users and 69 minutes for Key owners, calculated by storing the seven days of per-app usage before install and comparing it weeks later, for the roughly two thirds of users who grant access.

  • 100

    Rand Fishkin lists opt-in clickstream from millions of devices via Datos, roughly 100 million English-language search results from MarketMuse refreshed quarterly, and profile data moved away from Twitter toward LinkedIn, Facebook, YouTube, Reddit and Instagram, plus crawls of podcasts, channels and subreddits.

  • The Markup runs half engineers and half journalists on every investigation from start to finish, and releases research methods, datasets and code with each story.

  • Scite published the details of its citation extraction and classification system, and a preprint plus public data on how it detects retractions.

Sources

the second source

Three plays a week, for the phase you are in

No roundup of links, no news. Three tactics more than one builder arrived at separately, with the numbers each one returned and the disagreements left in.

One email a week. Unsubscribe in one click, and the address is used for this and nothing else.

More in build in public

all build in public plays

the second source

Three plays a week, for the phase you are in

No roundup of links, no news. Three tactics more than one builder arrived at separately, with the numbers each one returned and the disagreements left in.

963 corroborated plays to draw from · 3 a week

One email a week. Unsubscribe in one click, and the address is used for this and nothing else.