How Link Previewers Identify Themselves

Pasting a link into a chat or social application triggers a fetch from that platform's servers. These fetchers behave unlike browsers and unlike search crawlers, and they identify themselves accordingly.

The fetch happens once, for metadata

The fetcher requests the page to extract a title, description and image reference, then stops. It is not indexing and usually does not follow links.

Because only the document head matters, many fetchers stop reading after a limited amount of content. Metadata placed late in a document may never be seen.

Sites that render metadata through script frequently produce empty previews, since most fetchers do not execute script at all.

They identify themselves distinctly

Preview fetchers use their own tokens rather than imitating browsers, because platforms want site operators to permit them.

The strings usually name the platform and often include a documentation address, following the same convention as crawlers.

This makes them straightforward to identify in logs, provided detection rules are updated as platforms change their tokens.

Volume is bursty and follows sharing

Unlike crawlers, which pace themselves, preview traffic arrives when people share a link. A widely shared page can attract many simultaneous fetches.

Platforms cache previews, so the burst is bounded, but the cache is per platform and sometimes per region, which multiplies the initial fetches.

Rate limits tuned for browsers can catch these bursts and produce failed previews, which looks to users like a broken link.

Blocking them has visible consequences

A blocked fetcher produces a bare link with no title or image wherever the page is shared. The effect appears in the sharing application, not on the site.

This is a common accidental outcome of aggressive filtering, and it is hard to diagnose because nothing about the site appears wrong.

Testing what a preview looks like after any change to filtering rules catches it quickly, and most platforms provide a tool for exactly that.

Serving them well is cheap

Placing metadata early in the document, serving it without requiring script, and exempting known fetchers from aggressive limits covers nearly all of it.

Their requests are few and inexpensive relative to the traffic a good preview generates, which makes the exemption easy to justify.

Treating them as a recognised category rather than as unclassified automated traffic is the main thing, and it needs a detection rule set that stays current.