# Methodology

**Version 1.6** — corresponds to extractor version `1.0.0`.

This document states exactly how each signal is measured, what counts as a
change, and what the index will refuse to claim. It is versioned because the
measurement is part of the data: a value recorded under v1.0 and a value
recorded under v2.0 are not necessarily comparable, and every stored
observation carries the extractor version that produced it.

If you disagree with a definition here, you can check it. Every rule below
corresponds to code in `src/` and to a test in `tests/`.

---

## 1. What is observed

Two pages per company, both public, both linked from the company's own
navigation:

| Page | Purpose |
|---|---|
| Homepage | the positioning claim as presented to a first-time visitor |
| Pricing page | the commercial model as published |

Nothing behind a login. No app subdomains. No APIs. No search engines. No
third-party data. The full list of URLs is `seed/companies.json`, and it is
part of the repository, so the input set is auditable.

Each page is fetched at most once per day. Observation is **on demand**: a crawl
happens when a person or a manually triggered workflow asks for one, and then
the process exits. There is no hosted scheduler and nothing runs continuously.
That has one consequence a reader has to know about, and §3.2 states it in full:
the record advances only when somebody advances it, and the archive says
explicitly when it last did.

### 1.1 How the page is requested

Content negotiation is a variable, so it is held still. Every request from every
machine sends exactly the same language preference:

```
Accept-Language: en-US,en;q=0.9
```

`en-US` rather than a bare `en` because it is explicit about the region as well
as the language, and because it matches the origin this index treats as canonical
(§1.2). `en;q=0.9` keeps any English variant acceptable rather than risking a 406
or a fallback locale from a site that only publishes `en-GB`.

**This reduces one source of variance. It does not eliminate geo-routing.** Sites
that choose a currency or a locale from the client's IP address ignore
`Accept-Language` entirely — `notion.com/pricing` is one of them, and finding
that out cost this index two false change events (see `CORRECTIONS.md`). What is
left after pinning the header is handled by §1.2 and §4.9.

### 1.2 Where the page is requested from

**Every observation and every run records the crawl origin.** Two fields:

| Field | How it is obtained | Can it be unknown? |
|---|---|---|
| `environment` | `local` or `github-actions`, from the variables GitHub Actions sets | no |
| `country` / `region` | one Cloudflare edge trace per run — no key, no account, no quota — reporting the ISO 3166-1 country the request egressed from | yes |

The country lookup is best-effort by design. It has its own short timeout, it
never throws, and it never delays or cancels a crawl. When it fails, the origin
is recorded as `unknown`, and `unknown` is treated downstream as *cannot rule out
a shift*, never as *no shift*.

The country is **never** inferred from a system timezone, a system locale or an
environment variable naming a region. A laptop configured in one place and
connected through another would then produce a confident lie, and this index
prefers a stated gap to an unstated guess.

**GitHub Actions is the canonical origin.** It is reproducible, it is documented
in `.github/workflows/crawl.yml`, and anybody can re-run it. A laptop is none of
those things. Local runs remain fully supported and are the right way to develop
against live pages, but a local run against a page last read from CI produces an
`origin-shift` record (§4.9) rather than a change event — which is the correct,
visible outcome rather than a silent one.

Observations recorded before 2026-08-07 carry no origin field at all, because the
field did not exist. They read as `unknown` and are **not** backfilled with a
guess. §4.10 is what protects that stretch of the archive, because it needs no
origin.

---

## 2. Signals

Twelve signals. Each is extracted by a chain of strategies tried in order; the
first that produces a plausible value wins, and the winning strategy is stored
alongside the value as `method`, together with a `confidence` between 0 and 1.

A low `confidence` is not a hedge for presentation. The diff engine uses it: a
value that changes at the same time as the method degrades is suppressed rather
than published (§4.4).

### 2.1 `headline` — hero headline

**Definition.** The primary heading a first-time visitor sees, as a screen
reader would announce it.

**Strategies.**

| Order | Method | Confidence | Rule |
|---|---|---|---|
| 1 | `h1` | 1.00 | first visible `<h1>` in the first 120kB of markup |
| 2 | `h1-below-fold` | 0.75 | first visible `<h1>` anywhere in the document |
| 3 | `aria-heading` | 0.70 | first `role="heading" aria-level="1"` element |
| 4 | `og:title-fallback` | 0.40 | `og:title`, only if it contains a space and is not the bare brand name |

**Normalisation, in order.**

1. Subtrees marked `aria-hidden="true"` are removed. This is not cosmetic.
   Linear's live `<h1>` contains three visually duplicated copies of the
   headline — a mobile variant, a desktop variant, and per-word animation
   spans — all inside `aria-hidden="true"`, plus one canonical copy in a
   visually-hidden span. Naive tag-stripping returns the headline three times.
2. Tags stripped, HTML entities decoded, whitespace collapsed.
3. A string that is one phrase repeated is collapsed to that phrase, which
   catches responsive markup that renders the hero twice without ARIA.

**Rejected as headlines.** Anything shorter than 8 or longer than 200
characters. A string exactly equal to the company name. Navigation strings
(`Home`, `Menu`, `Skip to main content`, `Search`, `Log in`). Elements hidden by
`class` (`sr-only`, `visually-hidden`, `screen-reader`), by `style`
(`display:none`, `visibility:hidden`), or by the `hidden` attribute.

### 2.2 `subhead` — hero subhead

The first `<p>`, `<h2>` or `<h3>` following the winning headline within 6,000
characters of markup, whose text is between 20 and 400 characters. Cookie and
consent boilerplate (`By clicking…`, `We use cookies…`) is skipped.

Falls back to `og:description` at confidence 0.35. Null if nothing qualifies.

### 2.3 `category_label` — the category noun

**Definition.** The noun phrase a company uses for the thing it sells:
*platform*, *system of record*, *AI agent for X*. This is the most compressed
statement of positioning a company publishes, and the signal this index exists
for.

**How it is derived.** A candidate is generated from each of four sources —
`<h1>`, meta title, subhead, meta description — plus any free-text JSON-LD
`applicationCategory`. Each candidate is parsed into three parts:

```
[ up to 3 modifiers ] [ category noun ] [ "for" + up to 5 words ]
        AI                 workspace
   product development       system            for teams and agents
```

The category noun is matched against a fixed vocabulary of ~90 nouns
(`CATEGORY_NOUNS` in `src/extract/hero.js`), ordered longest-first so
*system of record* beats *system*. Modifier collection walks backwards from the
noun and stops at an article, a preposition, a verb, or any clause boundary.

**Candidates are scored, not ranked by source.**

```
score = source weight + 0.12 x min(modifiers, 2) + 0.20 if an object is present
```

Source weights: JSON-LD 0.95, `<h1>` 0.85, meta title 0.70, subhead 0.60, meta
description 0.50. The highest score wins.

This matters. Notion's `<h1>` is *"Where teams and agents Think together."*,
whose only category noun is a bare *agents* — technically a match, and
meaningless. Its meta title is *"The AI workspace that works for you."* The
scoring picks **ai workspace**, which is right.

**Two exclusions.**

- A bare noun from the weak set (*agent, tool, app, software, solution, service,
  system, suite, engine, layer, stack, hub, network, framework, builder, editor,
  inbox, api, sdk, os, automation*) with no modifier and no object is rejected.
  It is a word in a sentence, not a claim.
- schema.org's `applicationCategory` vocabulary (`BusinessApplication`,
  `DeveloperApplication`, `WebApplication`) is rejected outright. It is a fixed
  SEO taxonomy that never moves when positioning does, so accepting it would
  flatten the most interesting signal in the index into a constant.

The result is lowercased. Case carries no positioning information at this level
of abstraction, and normalising it prevents title-case CSS changes reading as
category changes.

### 2.4 `meta_title` and `meta_description`

`meta_title`: `<title>` (1.00), else `og:title` (0.70).
`meta_description`: `<meta name="description">` (1.00), else `og:description`
(0.80), else `twitter:description` (0.60).

Included because they are frequently rewritten *before* the visible page is, and
because they are the version of the positioning that reaches search results and
AI answers.

### 2.5 `customer_logos` — logo wall

**Definition.** The set of customer names displayed as logos on the homepage.

**Location.** A "proof region" is any 14,000-character window of markup
following a lead phrase: *trusted by*, *powering*, *used by*, *loved by*,
*join N*, *our customers*, *customers include*, *built for teams at*, *works
with*, *the best teams*, *teams at*, *from startups to*, *backed by*.

**Names come from three places.**

1. `alt` text of `<img>` elements, with `Logo of X` / `X logo` / `X's logo`
   reduced to `X`.
2. The basename of the image `src`, with separators normalised, colour and size
   variants stripped (`-white`, `-2x`, `@3x`, `-mono`, content hashes), and the
   result title-cased. Title-casing matters: without it,
   `/logos/vercel.svg` yields `vercel` while `alt="Vercel"` yields `Vercel`, and
   a site simply adding alt text to an unchanged wall would read as replacing
   every customer at once.
3. `<title>` inside inline `<svg>`, which is how a large share of logo walls are
   built. This is why the extraction pipeline strips scripts and styles in one
   pass but keeps SVG until after logo extraction.

**Rejections.** Names under 2 or over 30 characters, or longer than four words.
Generic terms as the whole string (`icon`, `logo`, `hero`, `avatar`, …) or as
any word within a multi-word candidate (so `/hero-background.png` →
`hero background` is rejected). Call-to-action alt text — anything beginning
*read, learn, see, view, watch, get, try, start, download, explore, book,
request, join…* The company's own name.

**Threshold.** Fewer than three names is treated as noise and the signal is
null. A wall found via a lead phrase scores 0.85; names found only by the
generic logo heuristic score 0.50.

**Ordering.** The list is sorted alphabetically before storage, so shuffling a
logo carousel is not a change.

### 2.6 `proof_points` — quantified claims

Quantified marketing claims in the page's visible text, matched by seven pattern
families:

| Family | Example |
|---|---|
| multiplier | `10x faster` |
| percentage, leading | `40% fewer escalations` |
| percentage, trailing | `cut onboarding by 60%` |
| count | `20,000 teams`, `5M developers` |
| money | `$2.4M saved` |
| duration | `in under 5 minutes` |
| rank | `Fortune 500`, `#1 rated` |

A bare four-digit number in the range 1800–2199 with no separator or `+` is
treated as a year, not a quantity. Copyright and privacy boilerplate is
excluded. Claims are lowercased, de-duplicated, sorted, and capped at 25, again
so that reordering is not a change.

### 2.7 `pricing_tiers` — published plans

**Preferred source.** JSON-LD `Offer` / `priceSpecification`, confidence 0.95.
Exact when present; uncommon on marketing pricing pages.

**Heuristic source**, confidence 0.70:

1. Collect anchors: `<h2>`–`<h5>` and `<dt>` elements, plus any element whose
   class matches `(plan|tier|package|pricing)[-_]?(name|title|heading|label)`.
2. Find every price token. Both `$12` and `12 €` are matched, because a European
   locale will be served the latter.
3. Attach each price to the nearest anchor above it within 2,500 characters.
   The first price under an anchor is taken as that plan's headline price;
   monthly/annual toggles usually render both.
4. Anchors of more than three words or 28 characters are rejected. Anchors that
   do not contain a recognised plan word are rejected unless they were found by
   class.
5. Anchors with no price are still recorded when the surrounding copy indicates
   a free plan (amount `0`) or an enterprise plan (amount `null`, "contact
   sales").

**Decimal handling.** `1.234,56` and `1,234.56` are both parsed correctly by
treating the last separator as the decimal point, with a lone comma followed by
exactly three digits treated as a thousands separator.

**Badges** (`Most Popular`, `Recommended`, `Best value`, `New`) are stripped from
plan names.

**Threshold.** Fewer than two tiers is not a pricing table — it is a stray
currency symbol near a heading — and the signal is null.

### 2.8 Derived pricing signals

`pricing_entry_price`, `pricing_free_tier` and the tier list all come from the
same extraction, and **when tier extraction fails, all of them are null.**

This is the most important definition in the document. `pricing_free_tier: "no"`
is a *value*, not an absence. If the parser failed and we emitted `no`, the
index would announce that a company removed its free plan on the strength of our
own bug. So the derivation is enforced in code, not left to the caller, and
tested directly in `tests/pricing.test.js`.

- `pricing_entry_price` — the cheapest tier with an amount greater than zero,
  recorded with its currency, billing period and seat unit.
- `pricing_free_tier` — `yes` if any tier has amount 0, or if the page text
  contains an explicit free-plan phrase; `no` if tiers were extracted and none
  qualify; **null if tiers were not extracted.**

### 2.9 `pricing_seat_minimum`

Matched from pricing-page copy by five phrasings: *minimum of N seats*, *N seats
minimum*, *starts at N seats*, *billed for a minimum of N*, *requires a minimum
of N*. Values outside 1–5,000 are rejected. Independent of tier extraction,
because the phrasing lives in body copy or a footnote rather than in a plan card.

Null means "no minimum found", not "no minimum exists". Most companies publish
none, so this signal is null for most of the index most of the time. The removal
rules in §4.3 apply before the index would ever claim a minimum was dropped, and
§4.11 applies before it would claim one was introduced.

### 2.10 `pricing_meta_title`

`<title>` of the pricing page. Tracked separately from the homepage title
because pricing-page titles are rewritten when the packaging changes, often
before the tiers themselves are updated.

---

## 3. Storage

### 3.1 Three append-only files

Everything is newline-delimited JSON in `data/`, versioned by git. Nothing is
stored as a binary, ever, because a binary makes `git diff` meaningless and the
diff is the point.

| File | Contents |
|---|---|
| `data/companies/<slug>.ndjson` | the series: one line per observation of one page |
| `data/events.ndjson` | the feed: one line per published change event, plus one line per retraction of an earlier event |
| `data/runs.ndjson` | the ledger: one line per crawl run, always |

Each observation line records every declared signal for that page **including
the null ones**, together with the value's structured form where applicable, the
method that produced it, the confidence, the extractor version, the document
facts (language, canonical URL, content variant), the **crawl origin** (§1.2),
and the parser-health state the diff engine derived from it. A gap in the data
and a null measurement are different facts and are stored differently.

No code path rewrites an existing line. The only write operation is append.

### 3.1.1 How a wrong claim is withdrawn

By **appending**, never by deleting and never by editing. A retraction is a line
in `data/events.ndjson` naming the `(slug, signal, detected_at)` of the event it
withdraws, with a stated reason and a link to the entry in `CORRECTIONS.md`.

The wrong claim stays in the file exactly as it was published. The public feed
excludes retracted events; `docs/api/retractions.json` and the site list them
separately, struck through. Deleting the line would be the one thing an archive
whose whole premise is *"check this against a history nobody can quietly
rewrite"* must never do — and a reader who acted on a false event deserves to
find out that it was withdrawn, not to find that it never happened.

`npm run retract` is the only supported way to write one. It refuses to retract
an event that was never published, and refuses to run without a reason.

Nothing mutable is stored separately, because there is nothing mutable to store:
the crawl queue (when each page is next due, its ETag, its content hash, its
consecutive failure count) is a fold over `data/runs.ndjson`, and the current
state of each signal is the last line of the company's own file. There is
therefore no file that can disagree with the history, because there is no file
besides the history.

### 3.2 What a run ledger is for

**A run record is written on every run, unconditionally** — including a run that
found nothing due, crawled nothing and changed nothing.

This is the single most important integrity property in the system. An archive
whose value is "nothing moved last month" is worthless unless it can also prove
it looked. "We ran and nothing had changed" and "nobody ran the crawler for six
weeks" produce identical silence in the series, and the only thing that tells
them apart is a receipt written every single time. A gap in `data/runs.ndjson`
therefore means exactly one thing, and it is never ambiguous.

The public health page reads the same ledger, which is why it can say "no
successful read in nine days" rather than showing a calm, plausible, stale
index.

### 3.3 Why an unchanged observation is not appended

An observation is appended only when it differs from the previous observation of
the same page. A company that has not touched its homepage in four months would
otherwise contribute a hundred and twenty byte-identical lines, and `git log -p`
on its file — which is how a reader is meant to inspect the series — would be a
hundred and twenty repetitions with the signal buried in them.

Nothing is lost. "We looked and it was the same" is recorded in the run ledger,
which names every target it touched and what happened to it. **The series says
what was true; the ledger says when we checked.** When a value does change, the
event's `previous_seen_at` is taken from the ledger, so it reports when the old
value was last confirmed rather than when it first appeared.

The comparison ignores timestamps and includes the parser-health counters, so an
advancing null counter is itself new information and does get its own line. That
matters: the removal rule in §4.3 counts consecutive nulls, and a
de-duplication that swallowed them would silently disable it. The one exception
is a signal that has *never* produced a value — `linear.app` publishes no logo
wall this extractor can read — where the counter can never mean anything,
because a removal cannot be confirmed for a value we never had.

---

## 4. What counts as a change

A change event is a public claim that a company changed something on a date. The
bar is deliberately high, and asymmetric: **missing a real change costs one day,
because the next run catches it. Publishing a false one costs credibility
permanently.** Every rule below resolves ambiguity toward silence.

### 4.1 Page-level gates

If any of these hold, the page's observations are still recorded but **no change
events are produced for any signal on it**:

| Gate | Recorded status | Why |
|---|---|---|
| Fetch failed | `error` | nothing was read |
| Non-HTML variant returned | `blocked` | not comparable with the HTML everyone else serves |
| `<html lang>` changed | `ok`, re-baselined | a German page is not a repositioning |
| Canonical URL changed | `ok`, re-baselined | a different page is not a changed page |
| Signal yield fell below 50% of its previous level | `changed-structure` | the page was redesigned; our selectors need review |
| Extractor version changed | `ok`, re-baselined | *we* changed, and that is not their news |

One gate does **not** suppress the whole page:

| Gate | Recorded status | Effect |
|---|---|---|
| Crawl origin changed (§1.2) | `origin-shift` | only locale-sensitive signals are withheld — see §4.9 |

A hero headline read from Virginia is still comparable with one read from
Frankfurt, and muting the whole page would discard real signal in order to
protect the price fields. It is the price fields that get protected.

The language gate is not theoretical. Fetched from Germany during development,
`klaviyo.com`, `stripe.com`, `zendesk.com` and `snowflake.com` all redirected to
localised pages. A crawler that ignores this reports that Klaviyo rewrote its
homepage in German.

### 4.2 First sighting is a baseline

The first time a signal is observed for a company, the value is recorded and
**no event is emitted**. The first observation of a *target* therefore emits
nothing at all, for any of its signals: it is a recording, not news. Otherwise
day one of the index would emit roughly 700 events into an empty feed and mean
nothing.

Nor is a signal that acquires a value *later* an addition. That case is §4.11,
and it is the reason no rule in this document emits an `added` event any more.

### 4.3 A null is a parser failure, not a removal

**If extraction returns null where it previously returned a value, that is a
parser failure and it is never reported as a change.** This is the single most
important correctness property in the project.

What happens instead:

- A null counter increments. The last known-good value is retained, so the next
  successful run diffs against what we actually believed rather than against
  the gap.
- After **2** consecutive nulls the signal is marked `suspect`, shown on the
  public health page, and excluded from change detection.
- After **5** consecutive nulls, *and* only if the rest of the page kept
  extracting normally throughout, a single `removed` event is emitted. It is not
  re-emitted afterwards.

If the page itself is unhealthy — fewer than half its signals extracting — no
removal is ever confirmed, however long the absence lasts.

The mirror of this rule is §4.11, and it took until v1.3 to write down: if a null
is not evidence that a value was removed, a value is not evidence that one was
added.

### 4.4 A confidence downgrade is our problem, not theirs

If the extraction method changed *and* confidence dropped by 0.30 or more *and*
the value differs, the difference is attributed to our fallback rather than to
the page. No event is emitted, the signal is marked suspect, and — importantly —
the weaker value does **not** become the new baseline.

Concretely: `h1` (1.00) degrading to `og:title-fallback` (0.40) usually means the
hero markup changed, not that the company rewrote its headline.

### 4.5 A collapsing list is a broken selector

For list signals, if the item count falls below 40% of its previous value (and
the previous list had at least four items), no event is emitted and the signal is
marked suspect.

A logo wall going from 14 names to 3 is our selector missing the wall. It is not
eleven customers churning overnight.

### 4.6 Typography is not positioning

Before comparison, values are normalised: curly quotes to straight, all dash
variants to hyphen-minus, non-breaking and zero-width spaces removed, whitespace
runs collapsed.

**Capitalisation is not normalised.** "The AI workspace" and "The AI Workspace"
are a real editorial decision and count as a change.

### 4.7 A value the page recently held is probably an experiment

Each signal remembers its last six value hashes. A change *back* to a value the
page has recently held is still recorded — it did happen — but is flagged
`oscillating` and labelled in the public feed as a likely A/B test rather than
presented as news.

During development `airtable.com` served two different `<h1>` strings to two
requests minutes apart. Split-tested heroes are the most likely source of false
positives in this index, and they are the one thing that is definitely not a
repositioning.

### 4.8 Change magnitude

For text signals, normalised Levenshtein distance over the first 200 characters,
0 to 1. For list signals, Jaccard distance over the item sets. Magnitude is
descriptive only; it never gates whether an event is emitted.

### 4.9 A value that depends on where we stood is not their news

**If the crawl origin (§1.2) differs from the origin of the previous observation
of the same page, no change event is emitted for any locale-sensitive signal on
it.** This is the sibling of §4.3: §4.3 says a value that went missing is our
parser breaking, and §4.9 says a value that moved can be our vantage point
moving.

A signal is locale-sensitive if either of these holds:

- it is one of the published-price signals — `pricing_tiers`,
  `pricing_entry_price`, `pricing_free_tier`, `pricing_seat_minimum`; or
- the value it holds, before or after, **quotes a currency at all**. This catches
  a proof point reading *"$2.4M saved"* or a headline quoting a price, without
  every such signal having to be enumerated.

What happens instead: the observation is recorded in full, the page is recorded
with status `origin-shift`, the signal is marked suspect, and **the last
known-good value is not overwritten** — exactly as in §4.4, so that the next
reading from the origin we baselined against is compared with what we actually
believed rather than with a value we only saw because we were standing somewhere
else.

Origins are compared conservatively. A difference is only asserted when it can be
proved: a different `environment`, or two known and different countries. If either
side's country is unknown, the comparison returns *indeterminate* and this rule
does not fire — muting the index every time a probe timed out would be its own
kind of dishonesty. §4.10 is what covers the indeterminate case.

This rule was added in v1.2, after `notion.com/pricing` produced two false change
events. `CORRECTIONS.md` has the full account.

### 4.10 A currency that moves while the numbers stay proportionate is routing

**If the currency changes and every comparable amount changes by the same factor,
that is a converted price list, not a repricing, and no event is emitted.**

Concretely: `EUR 9.5 → USD 10` is a ratio of 1.053; `EUR 19.5 → USD 20` is 1.026.
Nobody reprices by five per cent and switches currency in the same release. A
site that geo-routes does exactly that on every request.

The test is applied per tier, matched by tier name so a reordered pricing table
does not defeat it, and it holds only when:

- every ratio lies within 0.5–2.0 of 1, and
- the largest ratio is at most 1.35 times the smallest.

A tier that is free in one currency and free in the other carries no rate and is
ignored. A tier that is free in one and priced in the other fails the test.

**This rule needs no origin at all**, which is the point of it: it covers the two
cases where the origin is unknown — an observation recorded before v1.2, and a
probe that failed.

It requires corroboration before the new value is adopted: the same currency,
from the same origin, for **three** consecutive observations. Any change of
origin restarts the count. Even then the new value is adopted **silently**.

The consequence, stated plainly because it is a real limitation and not a
detail: **this index will never report a currency-only price change.** It cannot
distinguish one from locale routing, and the asymmetry in §4 resolves that toward
saying nothing. A currency change accompanied by a *disproportionate* price move
is a genuine repricing and is published normally.

### 4.11 A value appearing where there was none is an acquisition, not an addition

**If a signal that had no value produces one, that is recorded as a signal
acquisition and no change event is emitted.** This is the mirror image of §4.3,
and it exists for the same reason: a null means *we have no value*, which is a
statement about our reading and not about their page.

*"Our extractor finally succeeded"* and *"the company finally added it"* are one
transition wearing two hats. A logo wall rendered from CSS sprites that later
ships as plain `<img>` tags produces exactly the same null-then-value pair as a
logo wall that genuinely did not exist last week, and nothing in the pair
separates them.

What happens instead:

- The observation is recorded in full, including the value.
- The outcome is classified `acquisition`, beside `parser-fault` and
  `origin-shift`, and counted in the run ledger and the commit subject. A
  suppression nobody can see is indistinguishable from a crawler that found
  nothing.
- The value does **not** become the signal's known-good baseline yet.

**Corroboration.** The same value must be read **three** consecutive times, from
a healthy page, before it is adopted — the same number as §4.10, because the two
rules ask the same question with the same evidence, which is repetition. A read
that returns byte-identical content counts, since identical bytes cannot hold a
different value. A read of a page we understood poorly does not count and resets
the window, and a value that differs from the previous reading restarts it: a
signal whose first value never settles never acquires a baseline, and therefore
never publishes anything at all.

Adoption is **silent**. No event is emitted when it happens.

**Where there is evidence, the record says so.** If the previous read of the page
was classified `changed-structure`, had another signal in a parser fault, yielded
materially less than this one (the same 50% threshold as §4.1, inverted), or ran
a different extractor version, the acquisition is recorded as extractor recovery
with high confidence and the reason string names the evidence. None of this
changes what is published — nothing is, either way. It changes what an auditor
reading the archive in a year has to work with, because *"we could not tell"* and
*"we could tell, and it was us"* are different findings.

The consequence, stated plainly because it is a real limitation and not a detail:
**this index will not report the moment a company first adds a logo wall, a proof
point or a seat minimum.** It cannot distinguish that moment from the moment our
extractor first managed to read one. A false *"company X added Y"* costs more
than a missed one, and the asymmetry in §4 resolves that toward silence.

A second consequence, smaller and worth knowing: for the three readings the
corroboration window lasts, the site shows the signal as having no established
value while the raw observation lines plainly contain one. The observation is
right and the summary lags it.

This rule was added in v1.3, after `airtable.com` produced a false
`customer_logos` addition in the first full sweep. `CORRECTIONS.md` has the
account.

---

## 4A. Derived cross-sectional measures

Everything above is about one company over time. The published site also reports
the whole set at one moment — the words in 59 hero headlines, the noun 52
companies claim, how many use AI language. Those are the numbers a reader sees
first, so the rules that produce them belong here rather than in a comment.

They are computed in `src/insights.js`, tested in `tests/insights.test.js`, and
published as `docs/api/positioning.json` so any figure on the page can be
checked against the file that produced it.

### 4A.1 The denominator is always stated, and it is never 60

Every aggregate carries a coverage block: how many companies are tracked, how
many the number was actually computed over, and how many were unreadable. The
site prints it under the chart. `pricing_free_tier` is readable for 32 of 60
companies; the other 28 are **not** companies without a free tier, they are
companies whose pricing page could not be read, and the chart says so in those
words.

Nothing is imputed. There is no mean over missing values, no "assume no", no
filling a gap with the mode. §4.3's rule that a null is a parser failure rather
than a removal is the same rule here: a null is excluded from the count and
named, never converted to a zero.

### 4A.2 Last known-good, with the freshness declared

Aggregates count `last_good_value`, which is what §4.3 retains when extraction
fails. That is the consistent choice — a broken selector is not a company that
stopped saying something — but it means a count can include a value that did not
extract this morning.

So the coverage block carries two more numbers:

| | meaning |
|---|---|
| `held` | the value is counted, but the most recent read of that page produced nothing and this is the last value seen |
| `suspect` | the null has repeated enough that the signal is flagged and change detection on it is paused (§4.3) |

This is why the site reports 52 readable category labels where the raw latest
observations hold 51. Airtable's did not extract on 2026-08-07 and its previous
value is being held; the chart counts it and the coverage line says one of the 52
is held.

### 4A.3 Word frequency

Hero headlines are lowercased and split on every character that is not a letter
or a digit. An apostrophe is a separator like any other, so "you're" yields
"you" and a two-letter tail.

- **Tokens shorter than three characters are dropped.** This is what excludes
  "ai" from the word count, deliberately: AI language is measured separately in
  §4A.5 rather than sitting in a word count as one row that dominates and
  explains nothing.
- **Stopwords are the published NLTK English list**, minus the fragments this
  split cannot produce. A stopword list assembled by hand while looking at the
  results is a way of choosing the answer.
- **Nothing is stemmed.** "agent" and "agents" are counted apart, because they
  are different claims: a plural noun about a plural thing, and a singular
  product.
- **One vote per company.** A headline that repeats a word contributes one. The
  unit of this index is the company, and a company that repeats itself has not
  doubled its opinion.

### 4A.4 Category noun grouping

Each `category_label` (§2.3) is grouped by the **first** noun in it that appears
in a fixed vocabulary, reading left to right. Left to right because English puts
the head noun of a marketing category before its qualifiers: "AI platform for
marketers" is a platform, not a marketer.

The vocabulary is a literal list in `src/insights.js` and is not extended to make
a chart tidier. Singular and plural map to one group here — unlike §4A.3 —
because as a self-description they are the same claim. "agentic" is not "agent";
nothing is stemmed or fuzzy-matched.

A label containing no noun in the vocabulary is **not** forced into a bucket. It
is counted as readable, reported separately, and shown verbatim, and its bar on
the site is drawn in the neutral tone because it measures the vocabulary rather
than the market.

This grouping inherits §2.3's weakness in full. Category label extraction is a
scored guess over a fixed noun vocabulary and is the least reliable of the twelve
signals; a distribution built on it is no better.

### 4A.5 AI language

A company counts as using AI language if any of `headline`, `subhead` or
`category_label` contains one of four term families, matched as whole tokens:

| family | tokens |
|---|---|
| ai | `ai` |
| agent | `agent`, `agents`, `agentic` |
| copilot | `copilot`, `copilots` |
| autonomous | `autonomous`, `autonomy` |

Whole tokens, so "AI-powered" counts and "said" does not. The list is the
definition: a company selling AI without using any of these four words counts as
not using them, which is the honest limit of a word count and is stated on the
page.

Three buckets, and they account for every tracked company exactly once: uses the
language, does not, and could not be read. A company we could not read is neither
a mention nor a non-mention. A company appears once however many of the three
fields mention it, so the per-field counts sum to more than the company count.

### 4A.6 Proof point kinds

Companies, not claims. A homepage with eleven percentage claims has one opinion
about how to prove things, and counting claims would let one verbose page outvote
ten others. Total claim counts are published alongside.

The extractor's `percent` and `percent-trailing` kinds (§2.6) are reported as one
category. The difference between "40% faster" and "faster by 40%" is a property
of our regexes, not of the market, and publishing it as two bars would report the
first as if it were the second.

### 4A.7 Logo counts

Counted per company citing, not per appearance, and case-folded first, because
the same customer is "OpenAI" on one page and "Openai" on the next. The spelling
displayed is the most common original; ties are broken by `en` collation so the
choice does not depend on which file was read first.

These numbers are a **floor and not a count**. §2.5 reads logo names from `alt`
text, image filenames and inline SVG titles, so a wall built from CSS sprites or
a single flat image reads as no logos at all. Fourteen of the sixty companies are
missing from this measure rather than empty.

### 4A.8 Price distribution

Amounts are **not currency-converted**. Twenty-two of the twenty-three readable
entry prices are USD and one is EUR; converting would require a rate this project
does not have and would put a number in the archive that no page ever published.
The currency mix is published with the chart.

Buckets are half-open (`[min, max)`) so each price lands in exactly one. The
lowest bucket is real rather than a rounding artefact: for a usage-priced product
the cheapest published number is a per-unit rate, not a seat price, and §2.8's
entry-price rule takes the cheapest paid number on the page.

A median is published for the readable entry prices, labelled with its n. It
describes twenty-three numbers. It is not a market price and the site does not
call it one.

### 4A.9 Segment grouping

`seed/companies.json` labels every company with one of **twenty-nine** segments.
Ten of them hold fewer than six companies and one holds a single company:

```
marketing 13 · fintech-ops 13 · dev-infra 13 · industrial 12 · data 12
work-mgmt 11 · grc 11 · security 10 · product-dev 10 · hr-ops 9 · gtm 8
analytics 8 · erp 7 · ccaas 7 · banking-tech 6 · support 5 · payments 5
identity 5 · healthcare-it 5 · bpm 5 · observability 4 · clm-esign 4
automation 4 · localization 3 · itsm 3 · procurement 2 · legal-tech 2
design 2 · process-mining 1
```

Charting those directly would put a bar over one company next to a bar over
thirteen and invite a conclusion about "process mining companies" from one
homepage. The twenty-nine are therefore folded into **ten groups** of 11 to 27:

| Group | n | Seed segments |
|---|---|---|
| Developer & infrastructure | 27 | `dev-infra`, `observability`, `security` |
| Work & product | 27 | `product-dev`, `work-mgmt`, `automation`, `design` |
| Go-to-market | 24 | `gtm`, `marketing`, `localization` |
| Risk, compliance & identity | 22 | `grc`, `identity`, `clm-esign`, `legal-tech` |
| Finance & people ops | 22 | `fintech-ops`, `hr-ops` |
| Data & analytics | 20 | `data`, `analytics` |
| Enterprise applications | 18 | `erp`, `bpm`, `process-mining`, `procurement`, `itsm` |
| Industry & vertical software | 17 | `industrial`, `healthcare-it` |
| Customer service | 12 | `support`, `ccaas` |
| Financial infrastructure | 11 | `banking-tech`, `payments` |

Two of the original five groups changed when the seed grew to 200 on 2026-08-08,
and both changes are stated because a reader comparing an old chart to a new one
will see them. `support` left go-to-market: the old rationale was that a helpdesk
is sold to the revenue side of the house, which held when support was three
companies and stopped holding when the seed gained seven contact-centre vendors
whose buyer is a VP of Service. And the new banking and payments vendors did not
join the back office, because selling core banking to a bank is not the same
business as selling expense management to everybody.

The fold is a judgement and is published as one. Three properties keep it
auditable rather than convenient:

- **A seed segment is never split.** Every company carrying a given `segment`
  moves as one. The mapping is twenty-nine whole segments moved into ten boxes, so
  the only way to disagree with it is to disagree with a box — not to discover
  that two companies were placed individually to make a number come out.
- **The mapping is on the page**, with the rationale for each fold and every
  company in it, in a table the reader opens.
- **The groups partition the seed.** No company is dropped and none is counted
  twice; a company whose segment the mapping does not know is reported by name,
  and there is a test that fails if one appears.

**The minimum cell.** A group cell computed over fewer than **six** readable
companies is not drawn. At six, one company is 17 percentage points; at four it
is 25, and a bar that one homepage edit moves a quarter of its length is a
decoration. A suppressed cell is not rounded, pooled or omitted — it keeps its
row and reads "too few to say" with its own n, because a group that vanishes
from a chart reads as a group that scored zero.

**The width of a difference.** For two cells, the site computes the smallest
number of companies in the lower cell that would have to change their answer for
its share to reach the higher one's — the smallest `k` satisfying

```
(low.yes + k) × high.readable  ≥  high.yes × low.readable
```

solved in integers. This number is printed next to every comparison. With cells
of 6 to 16 it is usually one to four, and *"go-to-market leads finance by 50
points"* and *"three companies separate them"* are the same fact told
dishonestly and honestly.

**What gets drawn.** Nine cuts are computed. Two rules, applied in order, decide
which are published, and both are computed from the numbers rather than chosen
by looking at them:

1. **Coverage.** A cut where fewer than eight of the ten groups clear the
   minimum cell is withheld. A comparison across two or three groups of a
   two-hundred-company set is not a segment breakdown. The threshold was four of
   five when there were five groups; it moved to eight of ten so that the
   guarantee would not weaken when the number of groups doubled.
2. **Fragility.** A cut whose best and worst group are within **two companies**
   of each other is withheld. A reader looks at the bars, not at the caveat.

A withheld cut is listed on the page with its rule and its numbers. Which cuts
those are on any given day is computed, published on the site, and deliberately
not restated here: a worked example typed into this document is a figure that
stops being true at the next crawl and that nobody re-reads. See the segment
section of the published page for the current list.

Selecting the cuts after seeing which came out interesting would be the ordinary
way this analysis goes wrong, and the rules above exist to make it impossible:
the list of nine is fixed in the source, and the site publishes the ones that
survive plus the reasons the others did not.

**Percentages never travel alone.** Every share on this section of the site is
rendered in the same string as the counts that produced it (`4 of 6 · 67%`), on
the bar itself rather than in a tooltip or a footnote.

---

## 5. What this index does not claim

- **Not intent.** It records what was published and when. Whether a change was
  strategic, a copy test, a CMS migration or an intern is outside what a crawler
  can know.
- **Not the moment something appeared.** A change event needs two values this
  index actually read. It cannot report a first appearance, because a signal that
  was null and is now not null is at least as likely to be our extractor
  recovering as their page gaining something (§4.11). Where a value shows up, it
  is adopted quietly and reported only once it moves again.
- **Not completeness.** Pricing tables rendered entirely client-side are
  invisible to us; `vercel.com/pricing` is one such page and reports null tiers
  rather than a guess. Sites that refuse identified automated clients are
  recorded as `blocked` and contribute no data.
- **Not a single global view.** Pages are fetched from one place at a time,
  because the crawler makes one request per page. Companies that geo-route serve
  different content to different addresses and the index sees one of them. From
  v1.2 the canonical origin is a GitHub Actions runner (§1.2), so the series is
  internally consistent — but where a company geo-routes, what is recorded is
  what a US-based client is shown, and that is not what a European buyer sees.
  Anyone who needs the European figure has to fetch it from Europe; this is not
  a multi-region crawler and is not going to become one. A crawl from a GitHub
  runner is also refused outright more often than one from a laptop — see the
  README's note on block rates.
- **Not continuous.** The record advances when a crawl is triggered, not on a
  clock. A quiet stretch in the data is a quiet stretch in the crawling until
  `data/runs.ndjson` says otherwise, and the site says which it was.
- **Not causal.** Two companies adopting the same category noun in the same
  month is an observation, not an influence.
- **Not a segment effect.** §4A.9 groups sixty companies into five buckets and
  reports where they differ. A difference between two buckets of 6 to 16
  companies is a description of those companies, not an effect of the market
  they sell into: nothing here is a significance test, no group is a sample of
  anything, and the seed is sixty companies one person chose. The number the
  site prints next to every such comparison is how many companies would have to
  change their homepage to erase it, and on most cuts that number is two or
  three.

---

## 6. Version history

**v1.6** — 2026-08-08. The seed grows from 60 companies to 200, weighted toward
enterprise-tier and EU-headquartered vendors, and §4A.9 is rewritten around it.
Twenty-nine segments now fold into ten groups of 11 to 27; the coverage rule
moves from four-of-five groups to eight-of-ten so that doubling the number of
groups does not quietly halve the standard. Two folds changed and both are stated
in §4A.9: `support` joined the new customer-service group, and the new banking
and payments vendors were kept out of the back office.

Two consequences of the weighting are measurement facts and not opinions, and
they belong here rather than in a launch note. First, **the block rate rises with
enterprise tier**: verifying the 140 new URLs on 2026-08-08 found 12 companies
refusing an identified crawler, against 2 in the original 60. That is not random
noise, it is correlated with exactly the tier the expansion was meant to measure,
so enterprise cells will be thinner than their seed counts suggest and the
per-cell coverage block is the only honest place to read them. Second, **fewer of
these companies publish a price at all**, and six of them were confirmed by
observation to route `/pricing` to a contact-sales page, a product page or the
homepage. A missing pricing page is now sometimes a finding rather than a gap,
and §4.6's rules for reading one are unchanged.

No signal definition changed and the extractor version stays `1.0.0`. No stored
value is affected: §4A.9 defines how existing seed metadata is grouped.

**v1.5** — 2026-08-07. The `segment` field in the seed becomes a published cut.
New §4A.9: the fourteen seed segments are folded into five groups of 7 to 16, a
cell under six readable companies is not drawn, and every comparison carries the
number of companies that would have to change to erase it. §5 gains a
corresponding refusal.

The section is as much about what cannot be said as what can. Nine cuts are
computed and four are withheld by rule — three because the spread between the
best and worst group is within two companies, and free-tier prevalence because
pricing is readable for only 32 of 60 and three of the five groups fall below the
minimum cell. The withheld cuts are published with their reasons, because a cut
this data cannot support is a finding about the data.

**No signal definition changed** and the extractor version stays `1.0.0`. §1
through §4 are untouched and no stored value is affected; §4A.9 defines how the
existing seed metadata is *grouped and counted*, and the grouping is a judgement
published in full rather than derived from the data.

**v1.4** — 2026-08-07. The breadth axis gets a definition. New §4A, covering the
cross-sectional measures the published site now leads with: word frequency,
category noun grouping, AI language, proof point kinds, logo counts and the price
distribution. They existed as data from the first sweep and were on no page; the
site was built entirely around change over time, so a one-day-old archive showed
three events across sixty companies and read as an empty crawler dashboard.

None of this is a new measurement. §4A defines how the twelve signals in §2 are
*counted*, and every rule in it is a rule about refusing to overstate: the
denominator is always published and is never sixty, a null is excluded and named
rather than zeroed, held and suspect values are declared, and small counts are
reported as counts.

**No signal definition changed** and the extractor version stays `1.0.0`. §1
through §4 are untouched; values recorded before and after this version are
comparable. Two bugs in the new aggregation code were found and fixed before
publication: a two-character token floor that silently removed "os" from the
category vocabulary, and an apostrophe-stripping step that let contraction
fragments through the stopword list.

**v1.3** — 2026-08-07. The asymmetry is closed in the other direction. New §4.11
(a value appearing where there was none is an acquisition, not an addition);
§4.2 and §5 updated accordingly. §4.3 has said since v1.0 that a value going
missing is our parser breaking; nothing said the same about a value arriving, and
the first full sweep of 120 targets published `airtable/home customer_logos` as
an addition on the strength of that gap. It was retracted the same day;
`CORRECTIONS.md` is the account.

**No signal definition changed** and the extractor version stays `1.0.0`: §2
describes the same measurements it did under v1.2, and values recorded before and
after this change are comparable. §4 gained one rule and it only ever suppresses.
Nothing that was previously suppressed is now published; one class of claim that
was previously permitted — `added` events — is now permitted nowhere, and no code
path emits one.

**v1.2** — 2026-08-07. Crawl origin becomes part of the measurement. New §1.1
(pinned `Accept-Language`), §1.2 (how the origin is resolved and which one is
canonical), §3.1.1 (how a wrong claim is withdrawn), §4.9 (an origin shift is a
context fault) and §4.10 (a proportionate currency move is routing, not
repricing). §3.1, §4.1 and §5 updated accordingly.

Prompted by a real failure: two false change events about `notion.com/pricing`,
published on 2026-08-07 and retracted the same day. `CORRECTIONS.md` is the full
account.

**No signal definition changed** and the extractor version stays `1.0.0`: §2
describes the same measurements it did under v1.1, and values recorded before and
after this change are comparable *within one origin*. §4 gained two rules, both of
which only ever suppress. Nothing that was previously suppressed is now published,
so no claim this index has made becomes newly permissible under v1.2 — only fewer
claims are permissible than before.

**v1.1** — 2026-08-07. Storage moved from a hosted SQL database to append-only
NDJSON in git, and observation from a hosted cron to on-demand runs. §1, §3 and
§5 changed accordingly. **No signal definition and no change rule changed**, and
the extractor version stays `1.0.0`: §2 and §4 describe the same measurements
they did under v1.0, so values recorded before and after this change are
comparable.

**v1.0** — 2026-08-07. Initial release. 12 signals, 60 companies, extractor
`1.0.0`.

Any future change to extraction that could alter the value produced for an
unchanged page bumps the extractor version. When that version changes, the first
run against each page re-baselines and emits nothing, so that our own revision is
never attributed to a company.