The Authorization, Governance & Safety Model

StretchSearch is built so that you can always prove what you crawled and why you were allowed to. This article explains the governance and safety model that sits in front of every crawl.

Why governance comes first

Crawling other people's sites carries legal, contractual, and technical risk. StretchSearch treats crawling as a privileged action: crawler execution is disabled by default, and a target or crawl job is a governed record until it has passed authorization, readiness, and approval. This is what makes the platform safe for owned properties, client-authorized sites, and careful competitive research.

Authorization status

Every target and crawl job carries an authorization status that answers "are we allowed to fetch this?":

• owned_property — a site your organization owns.
• client_authorized — a site a client has authorized you to crawl.
• public_research_only — public reference material for research only.
• blocked — explicitly not permitted; execution is refused.

Set this honestly. It is the field auditors and teammates look at first.

Robots policy review

StretchSearch records where each target stands in a robots-policy review so robots.txt is a conscious decision, not an accident:

• not_reviewed → pending_review → reviewed_allowed or reviewed_limited, or blocked.

Crawl jobs also carry a respect_robots flag. Keep robots respect on unless you have a documented reason and the authorization to override it.

Risk level

Targets and jobs are tagged low, medium, or high risk. Risk reflects sensitivity — a public owned blog is low; a login-gated or competitor property is higher. Higher risk should mean tighter scope, lower page caps, and closer review.

Crawl scope

The crawl scope bounds how far a job may travel:

• seed_urls_only — fetch only the URLs you listed.
• same_domain_shallow — the seeds plus shallow same-domain links.
• same_domain_standard — standard same-domain crawl.
• approved_deep_crawl — a deep crawl that requires explicit approval.

Start narrow and widen only after a clean pass.

Governance and readiness state

Each crawl job carries two workflow states:

• Governance status: draft → review_required → approved, or blocked.
• Readiness status: planning → needs_review → ready, or blocked.

Readiness is driven by a checklist (see Using the Crawl Job Readiness Checklist). A job is only fit to run when required checklist items pass and it has been approved. Jobs also track approval_required and approved_by so the sign-off is recorded.

The safety layer

Under the hood, StretchSearch enforces technical safeguards whenever execution is enabled: an SSRF guard and safe-fetch layer block requests to internal or dangerous addresses, a controlled crawler policy enforces rate limits and allowed domains/prefixes, and a robots policy engine honors your review decision. Rate limit (rate_limit_ms, 250–60,000 ms), max depth (0–10), and max pages (1–5,000) are hard bounds you set per job. A master kill switch can halt crawler activity instantly.

Who can do what

• Workspace users create and edit projects, targets, and crawl jobs, complete readiness items, and review findings, rankings, and search.
• Platform operators hold the governed controls: enabling crawler execution, generating SEO findings and ranking previews, and toggling the kill switch. These actions require scoped action tokens and are audited.

A realistic example

An agency wins a competitive-research engagement. The analyst registers three competitor domains as competitor_research targets, marks robots policy reviewed_limited, sets risk high, and drafts jobs scoped to seed_urls_only with a 2,000 ms rate limit and a 150-page cap. The engagement lead reviews the readiness checklist, confirms the research is contractually permitted, and the platform operator approves and enables execution. Every step is recorded.

Tips

• Treat authorization_status and robots_policy_status as compliance fields, not busywork — they are your audit trail.
• Match risk level to scope: high risk → narrow scope, low page cap, slow rate.
• Never set respect_robots off casually. Document the reason and the authorization.
• If in doubt about permission, use public_research_only and keep scope to seed_urls_only.

Troubleshooting

A crawl job won't move to approved. Its readiness checklist has unmet required items or it's marked blocked. Resolve the checklist first.

Execution seems disabled everywhere. That's expected. crawler_enabled: false is the default state; a platform operator enables execution per approved job.