Crawl Audit Monitoring and the Kill Switch

Two controls give StretchSearch its safety backbone: the crawl-audit feed for watching what the crawler does, and the master kill switch for stopping it instantly. Both live on the platform command center and are central to operating StretchSearch responsibly.

The crawl-audit feed

The crawl-audit feed is a read-only monitoring surface that records crawler decisions and events. It's how operators answer "what did the crawler do, and did it stay within policy?"

Each audit event captures details such as:

• Event type — what happened (a fetch decision, a policy check, a block).
• Decision — whether the action was allowed or refused.
• Resolved IP class — the classification of the resolved address (a key SSRF-safety signal, distinguishing public targets from internal/dangerous ones).
• Crawl job — which job the event belongs to.

Read the audit feed

1. Open the Crawl audit view in the command center. The overview summarizes recent activity and decision counts.
2. Open the event list and filter by crawl job, decision, event type, or resolved IP class.
3. Page through with limit and offset.
4. Investigate refusals: a blocked decision or a non-public resolved IP class is exactly what the safety layer is meant to catch.

The master kill switch

The kill switch is StretchSearch's emergency brake. When engaged, it halts crawler execution across the platform. Its status is readable by everyone (the command center reflects it), but toggling it is a governed, audited action performed by a platform operator with a scoped action token — and it records the reason and who flipped it.

Read the kill-switch status

1. Open the command center. The kill-switch status is shown alongside crawler state.
2. When the switch is engaged, no crawler execution runs regardless of individual job state — this is intentional and overrides everything.

Toggle the kill switch (operators)

1. Confirm you have authority and a reason to engage or release it.
2. In the command center, set the kill switch on (to halt) or off (to permit), and enter a reason.
3. Submit. The change is recorded with the actor and reason for the audit trail.

The safety layers behind the scenes

Even with execution enabled, every fetch passes through guards: an SSRF guard and safe-fetch layer that refuse internal/dangerous addresses, a controlled crawler policy enforcing rate limits and allowed domains/prefixes, and a robots policy honoring your review decision. The crawl-audit feed is where those guards become visible.

A realistic example

During a large crawl, an operator notices audit events showing repeated refusals with an unexpected resolved IP class on one target. They engage the kill switch with the reason "investigating unexpected IP resolution on target X," pausing all crawler execution. After confirming the target's DNS was misconfigured (not a policy breach), they correct the target's allowed domains, then release the kill switch — every step recorded.

Tips

• Filter the audit feed by decision = blocked to review exactly what the safety layer refused and why.
• Watch resolved IP class — anything non-public is a signal worth investigating before continuing.
• Always enter a meaningful reason when toggling the kill switch; it's part of the audit record.
• Remember the kill switch overrides individual job state — a ready, approved job still won't run while it's engaged.

Troubleshooting

Nothing is crawling and jobs look approved. Check the kill-switch status first — if it's engaged, execution is globally halted until it's released.

I see blocked events I didn't expect. That's the safety layer working. Review the event's resolved IP class and the target's allowed domains/prefixes; a block usually means the request fell outside policy.

I can see the kill-switch status but can't change it. Toggling requires a governed action token. Status is readable by all; the toggle is operator-only and audited.