Current approach
Crawl scope is inferred from includes and excluded without a deterministic precedence. Edge cases can lead to unexpected discoveries.
Proposed v1.3
Explicit controls, rule previews and deterministic resolution would make a scope predictable and auditable.
boundaries:
includes:
- https://example.com/
- https://example.com/blog/**
excludes:
- https://example.com/admin/**
limits:
max_urls: 50000
max_depth: 6Boundary controls
Precise includes, excludes, and limits.
Rule previews
See resolved scope before a run.
Deterministic scope
Stable precedence. Repeatable results.
Why boundaries matter.
Unbounded discovery can turn one question into a noisy list of URLs. This sample describes a deliberate perimeter: name the surfaces that matter, then leave the rest outside the run.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
What changes for agents.
An agent receives a compact scope statement instead of an implied target. It can explain which pages match, which do not, and where a human decision is still needed.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
Boundary controls.
Includes, excludes, depth and page limits are separate controls. Keeping them visible makes a proposed crawl easier to inspect before it consumes resources.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
Rule previews.
A preview is a small list of representative matches, not a promise of exhaustive coverage. It gives reviewers a useful place to catch an accidental wildcard.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
Deterministic scope resolution.
When two rules overlap, a declared order makes the outcome repeatable. The point is clarity for a review, not a claim that every site behaves the same way.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
Example: same input, stable scope.
For a documentation area, include /guides/ and exclude previews or account paths. Run the preview twice and compare the same small set before asking for a larger crawl.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
Migration guide.
Start by recording today’s intended target in plain language. Convert one rule at a time, retain the old notes, and have an owner approve changes to sensitive paths.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.
What’s next.
Use this as a discussion prompt, not release documentation. A real rollout would need tested behavior, versioned configuration and an approved operating owner.
The useful next move is to record the assumption, make the boundary visible, and ask for a review before expanding scope.