Configuring and Governing Crawl Jobs

A crawl job is a governed plan to fetch a set of pages. Creating one produces a draft record — it never starts a crawl by itself. This article covers how to configure a job's targets, scope, and safety limits, and how it moves through governance to the point where execution can be enabled.

What a crawl job contains

• Crawl mode — the intent: owned_site_deep_crawl, client_authorized-style work via seo_content_crawl, competitor_crawl, brand_audit_crawl, or stretch_owned_crawl.
• Seed URLs — one to 250 starting URLs.
• Scope — seed_urls_only, same_domain_shallow, same_domain_standard, or approved_deep_crawl.
• Safety limits — max_depth (0–10), max_pages (1–5,000), and rate_limit_ms (250–60,000 ms).
• Robots respect — respect_robots on or off.
• Governance state — governance_status, authorization_status, risk_level, approval_required, approved_by, and notes.
• Readiness state — readiness_status, driven by a checklist.

Create a crawl job

1. Open your project and go to Crawl jobs → New crawl job.
2. Choose a crawl mode that matches the work.
3. Add your seed URLs (at least one valid https:// URL).
4. Set the scope. Begin with seed_urls_only for a first pass.
5. Set max depth, max pages, and a rate limit. Conservative values are safer and kinder to the target.
6. Confirm respect robots is on unless you have documented authorization to override it.
7. Set the authorization status and risk level to match the target.
8. Save. The job is created as a draft — the response confirms no crawler execution was started.

Move a job through governance

A job has to earn the right to run:

1. Draft — you're still configuring. Governance status starts at draft.
2. Readiness — complete the readiness checklist (see Using the Crawl Job Readiness Checklist). Passing the required items moves readiness to ready.
3. Review & approval — set governance to review_required; a reviewer confirms authorization and scope and approves. approved_by records who signed off.
4. Execution — only after approval can a platform operator enable execution. Until then, and any time after, the master kill switch can halt activity.

Setting either governance or readiness to blocked stops the job cold until the blocker is resolved.

Adjust safety limits

Edit the job to tighten or widen limits at any time. Lowering max_pages, raising rate_limit_ms, or narrowing scope are all good moves if a first pass was noisy. Every update reasserts that no crawler execution was started.

Pause or cancel a job

Job status can be draft, queued, paused, cancelled, completed, or failed. Pause a running-eligible job to hold it, or Cancel to stop it — cancelling never triggers execution and simply retires the plan.

A realistic example

For the Northwind Coffee engagement, Maya creates an seo_content_crawl job seeded with https://northwindcoffee.com/blog/ and the shop landing page. She sets scope same_domain_shallow, max depth 2, max pages 300, and a 1,000 ms rate limit, leaving robots respect on. She completes the readiness checklist, sets governance to review_required, and her lead approves — recording their name in approved_by. Only then does the platform operator enable execution.

Tips

• Prefer more, well-chosen seed URLs over a broad scope. Precise seeds plus seed_urls_only produce the cleanest first index.
• A slower rate_limit_ms (1,000–2,000 ms) is polite and reduces the chance of being throttled or blocked.
• Keep max_pages realistic. A 5,000-page cap on a first pass is rarely what you want.
• Record scope decisions in the governance/readiness notes so reviewers understand your intent.

Troubleshooting

"Invalid StretchSearch crawl job payload." You likely have zero seed URLs, an invalid URL, or a limit out of range (depth 0–10, pages 1–5,000, rate 250–60,000 ms). Fix and resave.

The job is stuck in draft/needs_review. Required readiness items aren't passed, or it hasn't been approved. Complete the checklist and get sign-off.

I cancelled a job — did it stop crawling? Cancelling retires the plan and never starts execution. If a crawl was live, the kill switch and central controls govern that; cancelling the record is safe either way.