Transparent by design

A small crawler built to respect the web.

Crawlandbot discovers public pages for a searchable web index. It checks each site’s rules before requesting pages, moves slowly on each host, and gives site owners clear control.

Manage crawler access View technical details

User-agent: Crawlandbot
Default pace: 1 request / second / host
Contact: Email us

How it works

Rules first. Pages second.

Every crawl starts with a fresh check rather than assuming old permission still applies.

01
Read robots.txt
We fetch the host’s robots file and stop if Crawlandbot is not allowed.
02
Inspect sitemaps
Declared sitemaps help us find public URLs without guessing unnecessary paths.
03
Check each URL
Every sitemap URL and discovered link is compared with the active robots policy.
04
Respect page controls
Pages marked noindex are not indexed; nofollow links are not followed.

Technical details

Easy to identify.

Requests identify themselves with the following HTTP user-agent:

Crawlandbot/1.0 (+https://crawlandbot.klcdev.com/)

Bounded crawling

Runs use explicit page, host, depth, and download-size limits.

Host-friendly concurrency

Different hosts may run in parallel, but each host stays sequential and rate-limited.

Public pages only

Private and local network addresses are rejected before requests are made.

No restricted content

Robots blocks, page-level exclusions, and manual suppression requests are enforced.

Site-owner controls

How to opt out

Choose the control that matches what you want. Changes are checked at the beginning of every new crawl.

Block all crawling

Add this to the /robots.txt file at the root of your hostname.

User-agent: Crawlandbot
Disallow: /

Block selected paths

List only the areas you do not want requested.

User-agent: Crawlandbot
Disallow: /private/
Disallow: /archive/

Prevent indexing

Allow a page to be fetched but keep it out of the index.

<meta name="Crawlandbot"
      content="noindex, nofollow">