The Pretty dataset: our own web crawl and link index
Pretty is a web crawl we built and run ourselves.
Moz and Majestic we license. Pretty we crawl. It is a directed graph of pages and the links between them, parsed by our own workers and served out of ClickHouse — and it is the reason we can return raw backlink rows at a price the licensed feeds cannot reach.
Index size as measured on 1 August 2026. These are a snapshot, not a live counter — the index grows continuously between measurements.
From crawl file to API response.
Most providers will not tell you how their index is produced. Here is ours, because knowing how a number was made is part of knowing whether to trust it.
-
01
Discovery
A coordinator service tracks which crawl datasets exist and lists every file in them, holding the state for each one.
-
02
Leasing
Linux workers ask for the next viable file and are leased exactly one at a time. Stuck leases expire so no file is lost if a worker dies mid-run.
-
03
Parsing
Each worker streams its file, extracts pages and links, resolves relative URLs against their source, and records nofollow, sponsored and UGC separately rather than collapsing them into one flag.
-
04
Identity
Every URL gets both an exact form and a normalised identity key. Tracking parameters are stripped for the key but kept in the stored URL, so displayed links stay real while metrics stay stable.
-
05
Deduplication
Within one source page, only the first link to a given target page counts. Links to different pages on the same target domain each count separately, because they are different endorsements.
-
06
Ingest
Workers write compressed TSV plus a manifest with counts and checksums. The manifest is validated before anything enters ClickHouse, so a truncated file cannot silently corrupt the index.
-
07
Rollup
Serving tables are rebuilt from the raw observations - target-sorted for inbound queries, source-sorted for outbound, with wide latest-metric tables for fast stats lookups.
What our words mean.
Link data providers use the same terms for different things. These are ours, stated precisely enough to argue with.
| Term | Our definition | Commonly confused with |
|---|---|---|
links_in |
External inbound links into the element from other registrable domains. | Referring domains, which counts unique sources rather than links. |
links_out |
External outbound links from the element to other registrable domains. | Total outbound links, which would include internal navigation. |
ref_domains |
Unique external registrable domains pointing at the element. | Referring IPs or subnets, which measure network diversity instead. |
linked_domains |
Unique external registrable domains the element links out to. | links_out, which counts links rather than distinct destinations. |
element |
A page, a subdomain, or a registrable domain. Domain-level queries include everything beneath the domain. | Treating a subdomain as a separate site, which some providers do. |
internal links |
Stored, but excluded from public links_in and links_out unless an endpoint asks for them. | Being silently folded into inbound counts, which inflates them. |
a lost link |
A link whose source page has been observed again without it. | A link absent from the next crawl, which usually just means coverage moved. |
Pretty V2.
The index is being rebuilt as a proper directed graph rather than a set of precomputed totals. It is going live endpoint by endpoint behind the existing API, so nothing you have integrated changes underneath you.
Everything below is in progress rather than shipped. We will say so on this page when each one lands.
Element-level queries
In buildAsk about a page, a subdomain or a whole domain from the same graph, rather than only at domain level.
Separate link attributes
In buildnofollow, sponsored and ugc recorded and queryable individually instead of collapsed into a single flag.
First seen and last seen
In buildPer-link history, so new and lost links become answerable questions rather than guesses.
Outbound as a first-class view
In buildlinked_domains alongside links_out, so you can profile who a site endorses as easily as who endorses it.
Top anchors and pages
In buildAggregate views over the graph: most common anchor text, most linked pages, referring domain lists.
Paginated link lists
In buildWorking past the current 5,000-row ceiling for domains with very large profiles.
Access the Pretty dataset
Pretty data is available through paid, authenticated API requests. Retrieve per-domain counts with the domain metrics API, or inspect anchor text, source pages and follow status with the backlink data API. Responses use JSON; the browser tools can export the results for your research workflow.
Access is subject to the service terms and third-party data conditions. API access does not itself grant redistribution rights. Compare DomDetailer credit packs or contact DomDetailer about data access.
Where does Pretty data come from?
Common Crawl WAT records are the current input. A fleet of Linux workers leases one file at a time from a coordinator, parses the pages and links, normalises the URLs and writes the results into ClickHouse. Our own live crawler is being added as a second source feeding the same graph.
How is Pretty different from Moz and Majestic?
Moz and Majestic return scores derived from their indexes. Pretty returns counts and the individual links behind them, so you can check the reasoning rather than trusting a number. It is also what lets us price backlink rows the way we do - we are not paying anyone else for them.
How often is the index refreshed?
Continuously, as crawl data is processed. There is no fixed release cycle. Higher-authority and faster-changing domains are revisited more often than static ones.
Do you delete links that disappear?
No, and deliberately. A link is only treated as lost once the source page has been observed again without it. A page missing from one crawl is usually a gap in coverage rather than a removed link, and treating those as losses would make the data confidently wrong.
Start at 25,000 lookups for $34.
One balance covers the browser tools, the desktop app and the API. Nothing renews, nothing expires, and you can stop using it for six months without losing anything.