Registering and Authorizing Crawl Targets

A target is a domain you have declared StretchSearch may work with. Registering a target is how you record — before anything is ever fetched — that you're authorized to crawl a site and under what limits. Targets never start a crawl on their own.

What a target records

Each target belongs to a project and captures:

• Root URL — a full https:// entry point.
• Domain — the canonical host (stored lowercased).
• Target type — owned_property, client_authorized, competitor_research, or public_reference.
• Authorization status — owned_property, client_authorized, public_research_only, or blocked.
• Robots policy status — the review state for robots.txt.
• Allowed domains and allowed URL prefixes — the boundaries a crawl may stay within.
• Risk level — low, medium, or high.
• Notes — free-text context (permission reference, contacts, caveats).

Register a target

1. Open your project and go to Targets → Register target.
2. Enter the root URL (must be a valid https:// URL, up to 2,000 characters) and the domain.
3. Choose the target type that matches your relationship to the site.
4. Set the authorization status to reflect your actual permission.
5. Set the robots policy status — start at not_reviewed or pending_review, and update it to reviewed_allowed, reviewed_limited, or blocked after you check robots.txt.
6. Add allowed domains and allowed URL prefixes to fence the crawl (for example, only https://site.com/blog/). You can list up to 50 domains and 100 prefixes.
7. Set the risk level and add notes documenting your authorization.
8. Save. The target is registered with crawler execution explicitly disabled.

Edit or re-scope a target

1. Open the target from the Targets list.
2. Choose Edit.
3. Update authorization, robots status, allowed domains/prefixes, risk, or notes. Only changed fields are applied; crawler execution stays disabled on save.
4. Save.

Pause, block, or archive a target

Targets support active, paused, archived, and blocked status. Set blocked to hard-stop any future use of the domain, paused to set it aside, or Archive to retire it while keeping the record.

A realistic example

An agency registers northwindcoffee.com for a client engagement. They set target type client_authorized, authorization status client_authorized, and paste the client's written-permission reference into notes. They add allowed prefix https://northwindcoffee.com/blog/ so the crawl can't wander into the checkout flow, review robots.txt and mark it reviewed_allowed, and set risk low. The target is now ready to attach to a crawl job.

Tips

• Fence aggressively with allowed URL prefixes — it's the cleanest way to keep a crawl on the pages you actually care about.
• Keep the domain field to the bare host (no scheme, no path). It's stored lowercased and used for matching.
• Put your permission evidence in notes. Future-you (and auditors) will thank you.
• Mark a domain blocked the moment permission is withdrawn — it prevents accidental reuse.

Troubleshooting

"Invalid StretchSearch crawl target payload." The root URL isn't a valid https:// URL, the domain is empty, or an allowed prefix isn't a valid URL. Fix the offending field and resave.

My target won't attach to a job in another project. Targets are project-scoped. Register the target in the same project as the crawl job, or move the job.

I set robots policy to reviewed_allowed but nothing crawls. Registering and authorizing a target is separate from running a job. You still need an approved, ready crawl job and centrally enabled execution.