Hister can fetch individual pages, crawl links recursively, or process a saved list of URLs from the terminal. Persistent crawl jobs keep their queue in the database, so a long crawl can be stopped and continued later without starting over.
The two relevant commands have different responsibilities:
| Command | Purpose |
|---|---|
hister index | Fetch pages, extract their content, send documents to Hister, and run jobs. |
hister crawl | Inspect or delete persistent job records. It does not fetch or resume pages. |
The Hister server must be running while hister index adds documents. Global connection options
such as --server-url, --token, and --client-timeout are covered in the Terminal Client documentation.
Choose an Index Mode
The arguments passed to hister index select one of four modes:
| Invocation | Persistent | Follows links | Result |
|---|---|---|---|
hister index URL... | No | No | Fetch each positional URL once. |
hister index --recursive [--job-id NAME] URL | Yes | Yes | Create or resume a recursive crawl job. |
hister index --input FILE | Yes | No | Queue every URL in the input without following links. |
hister index --job-id NAME | Yes | Saved setting | Resume the pending queue of an existing job. |
Add --recursive to --input when links discovered on the listed pages should also be queued.
Index Individual URLs
Pass one or more URLs without --recursive, --job-id, or --input for a direct indexing run:
hister index https://example.com/a https://example.com/bThis mode does not create a crawl job. Each URL is fetched once and discovered links are ignored.
Already indexed URLs are skipped unless --force is present:
hister index --force https://example.com/aUse --ignore-rules to explicitly save submitted documents despite URL allow and skip rules.
The saved override also protects them from allow and skip rules during hister reindex.
Combine it with --force to fetch and mark a URL that is already indexed:
hister index --ignore-rules --force https://example.com/aThe override applies to every document submitted by the invocation, including recursive
crawls and URL lists. Pass it again when resuming a job with --job-id; it is not stored
as a crawl job setting. Robots rules, crawl filters, and sensitive content checks still apply.
Direct indexing supports the same backend, header, cookie, timeout, ownership, and labeling flags as persistent crawling.
Start a Persistent Recursive Crawl
Use --recursive, or its short form -r, with a starting URL:
hister index --recursive https://example.comHister creates a job with a random eight character ID and prints it before fetching begins:
Starting crawl job: a1b2c3d4Keep that ID to inspect or resume the crawl later. The queue is processed breadth first. Every discovered URL is recorded with its depth and status before the crawl moves on.
By default, recursive crawling is not restricted to the starting domain. Add --allowed-domain when the crawler should remain on one site:
hister index --recursive
--allowed-domain example.com
https://example.comThe allowed domain also includes its subdomains. Repeat the flag to permit more than one domain.
Give a Job a Custom Name
The job ID is also its human readable name. Supply one with --job-id when creating the crawl:
hister index --recursive
--job-id example-docs
--allowed-domain docs.example.com
https://docs.example.comIf example-docs does not exist, this command creates it. If it already exists, the same command
resumes it. This makes named jobs convenient in scripts because the creation and continuation
command can be identical:
hister index --recursive
--job-id example-docs
--allowed-domain docs.example.com
https://docs.example.comWhen the ID already exists, Hister uses the starting URL and traversal rules saved in that job. Positional URLs and traversal flags from the new invocation do not replace the saved values.
Creating a new named job requires all three of these pieces: --recursive, an unused --job-id,
and at least one URL. Passing an unknown job ID without --recursive produces an error instead of
silently creating a job.
Stop and Resume a Job
It is safe to stop a persistent crawl. To continue it with the shortest form, pass only its ID:
hister index --job-id example-docsThere is no hister crawl resume subcommand. Resuming is always done through hister index.
On resume, Hister restores the saved starting URL, traversal rules, label, and URL queue. Any URL
left with in_progress status is moved back to pending. URLs already marked done, failed, or skipped are not fetched again. Pending URLs continue in their original queue order.
A process stopped abruptly can leave the job status as running. This does not mean its queue was
lost. Use hister crawl show NAME to check the queue counts, then resume it normally.
Saved Settings and Runtime Settings
The settings that define crawl scope are stored when a job is created:
| Saved with the job | Examples |
|---|---|
| Starting URL | https://docs.example.com |
| Traversal rules | --max-depth, --max-links, domain flags, and pattern flags |
| Recursive behavior | Whether discovered links are added to the queue |
| Initial document label | --label docs |
| Queue and URL statuses | Pending, in progress, done, failed, and skipped URL records |
Fetch and destination settings are not restored from the job. Keep them in crawler configuration, repeat them when resuming, or rerun the
same complete command. These include --backend, --backend-option, --proxy, --header, --cookie, --delay, --timeout, --user-agent, --no-robots, --force, --global, and --user-id.
For example, resume a job with Chromium and a one second request delay:
hister index --job-id example-docs
--backend chromedp
--delay 1An explicit --label on a resume changes the label for documents indexed during that invocation.
Without it, Hister uses the original stored label.
Queue URLs From Input
Use --input with a text file containing one URL per line:
https://example.com/guide
https://example.com/reference
https://example.org/manualhister index --input documentation-urls.txtBlank lines and surrounding whitespace are ignored. Positional URLs are ignored when input is present. Empty input produces an error.
Before fetching starts, Hister creates a persistent job and inserts the complete list into its
queue in one database transaction. The file base name becomes the job ID. In this example, the ID
is documentation-urls.txt. If that ID exists, Hister tries documentation-urls.txt-2, then documentation-urls.txt-3, and continues increasing the suffix until it finds a free name.
To choose a recognizable name for this kind of job, give the file that name before running the
command. --input and --job-id cannot be used together.
Running the --input command again creates a new suffixed job. It does not continue the earlier
one. Resume the printed job ID instead:
hister index --job-id documentation-urls.txtOnly the listed URLs are fetched by default. Add --recursive to follow links from every seed URL:
hister index --recursive
--input documentation-urls.txt
--allowed-domain example.comUse --input - to read URLs from standard input:
printf '%s\n' https://example.com/guide https://example.com/reference | hister index --input -Standard input jobs use stdin as their base ID. Later jobs use stdin-2, stdin-3, and so on.
Control Crawl Scope
Traversal flags apply when a persistent job is created and are then stored with it:
| Flag | Behavior |
|---|---|
--recursive, -r | Follow discovered links. Without it, only explicitly queued URLs are fetched. |
--max-depth N | Visit links up to depth N. The starting URL has depth zero. Zero means unlimited. |
--max-links N | Stop after allowing N pages. Zero means unlimited. |
--allowed-domain DOMAIN | Allow this domain and its subdomains. Repeatable. An empty list allows every domain. |
--exclude-domain DOMAIN | Skip this domain and its subdomains. Repeatable. |
--allowed-pattern REGEXP | Fetch URLs matching at least one allowed regular expression. Repeatable. |
--exclude-pattern REGEXP | Skip URLs matching any excluded regular expression. Repeatable. |
Domain and pattern checks apply to the starting URL too. Make sure an allowed pattern includes the seed URL.
This example crawls at most 500 pages from two documentation domains while excluding sign in pages:
hister index --recursive
--job-id product-docs
--allowed-domain docs.example.com
--allowed-domain api.example.com
--exclude-pattern '/(login|logout)(/|$)'
--max-depth 5
--max-links 500
https://docs.example.comSelect a Fetch Backend
The backend controls how the hister index process fetches pages. The browser must be available
to that process, which can run on a different machine from hister listen. Changing crawler
settings on the server does not start a browser or change how the extension captures pages.
| Backend | Browser setup | Connection |
|---|---|---|
http | No browser needed. The default for server rendered pages. | Direct HTTP requests. |
chromedp | Chrome or Chromium installed where hister index runs. | Hister launches and controls a local browser using CDP. |
bidi | Start Firefox before running hister index. | Hister connects to its WebDriver BiDi WebSocket endpoint. |
Chrome or Chromium
Install Chrome or Chromium on the machine running the indexing command. Hister starts the
browser itself. Set exec_path to its executable if it is not found automatically:
hister index --recursive
--job-id dynamic-docs
--backend chromedp
--backend-option exec_path=/usr/bin/chromium
--allowed-domain docs.example.com
https://docs.example.comHister’s chromedp backend currently supports local browser launching only. It has no remote
WebSocket option. A Chromium --remote-debugging-port socket speaks CDP and cannot be used with --backend bidi.
Firefox with BiDi
Start Firefox with a dedicated profile so the command creates a separate browser instance. This Linux example creates the profile directory and runs without a graphical display:
mkdir -p "$HOME/.local/share/hister-firefox"
firefox --headless --no-remote
--profile "$HOME/.local/share/hister-firefox"
--remote-debugging-port 9222Keep Firefox running and use a second terminal for Hister. Omit --headless if you want to see
the browser window. Firefox’s Remote Agent enables BiDi through --remote-debugging-port. This setup was verified with Firefox 156.0 without --marionette; Hister connects to the BiDi endpoint rather than the separate Marionette interface.
With the Hister server running, fetch a page through Firefox:
hister index --force
--backend bidi
--backend-option socket=ws://127.0.0.1:9222/session
https://example.comThe /session path is required for this direct Firefox connection. --force makes the check
fetch the page even if it is already indexed; omit it for normal runs that should skip existing
documents. Look for an indexed count rather than a skipped count when verifying the setup.
Use one indexing process per Firefox instance. Firefox accepts one active WebDriver session at a time, so another automation client can prevent Hister from creating a session.
If the server runs in Docker or LXC, see browser backends with Docker for placement and network requirements.
Fetch Options
The request related flags are:
| Flag | Behavior |
|---|---|
--backend NAME | Select http, chromedp, or bidi. |
--backend-option KEY=VALUE | Set a backend option such as exec_path. Repeatable. |
--proxy URL | Route crawler and robots.txt requests through an HTTP or SOCKS5 proxy. |
--header KEY=VALUE | Add or replace an HTTP header. Repeatable. |
--cookie VALUE | Add a cookie in Set-Cookie format. Repeatable and requires a Domain attribute. |
--delay N | Wait N seconds between requests. |
--timeout N | Set the request timeout in seconds. |
--user-agent VALUE | Override the crawler user agent. |
--no-robots | Disable robots.txt checks. By default Hister follows robots.txt rules. |
For all backend options and configuration examples, see the Crawler Configuration reference.
For example, route a crawl through a local SOCKS5 proxy:
hister index --recursive
--proxy socks5://127.0.0.1:1080
https://docs.example.comBrowser Connection Troubleshooting
| Symptom | What to check |
|---|---|
| Chrome or Chromium executable not found | Install the browser where hister index runs and check exec_path there. A browser on the Docker host is not an executable inside the container. |
| Connection refused | Keep Firefox running with --remote-debugging-port and check the address from the machine or container running hister index. |
| WebSocket handshake fails | Check the port and /session path. Firefox restricts accepted Host headers; use a loopback IP for a local connection. |
session.new was not found, or an error mentions cannot unmarshal object | Check the endpoint protocol. A CDP endpoint, including Browserless’s Chromium endpoint, cannot handle Hister’s BiDi commands. |
| Maximum number of active sessions | Stop the other automation session or use a separate Firefox instance with its own profile and port. |
| URL already indexed, skipping | Repeat a single URL check with --force to exercise fetching. A skipped URL does not verify browser navigation. |
Browserless’s Chromium endpoint uses CDP.
Pointing crawler.backend_options.socket at that endpoint does not make it compatible with
Hister’s bidi backend. Use local Chrome or Chromium with chromedp, or Firefox with BiDi as
described above.
Indexing and Ownership Options
These flags control how fetched documents are added to Hister:
| Flag | Behavior |
|---|---|
--label VALUE | Attach a label to documents indexed by this run. |
--force | Fetch and reindex URLs even when they already exist. |
--ignore-rules | Bypass URL allow and skip rules and preserve submitted documents during index rebuilds. |
--allow-sensitive | Bypass sensitive content checks for the indexed documents. |
--global | Make documents available to all users. Requires an administrator in multiuser mode. |
--user-id ID | Index documents for a specific user. Requires an administrator in multiuser mode. |
--format FORMAT | Print the indexing summary as text, json, jsonl, or csv. |
--failed-urls PATH | Save failed URLs, one per line, replacing the file’s contents. |
--global and --user-id cannot be used together.
Without --force, a persistent crawl marks already indexed pending URLs as skipped before
fetching them. Since skipped pages are not fetched, Hister cannot discover new links from them.
Use --force when starting a fresh recursive job over pages already present in the index.
Indexing prints indexed, skipped, and failed counts. Persistent jobs also report their job ID and pending count. These are cumulative stored job counts, including previous runs and redirect tracking URLs. Job progress messages go to stderr so stdout contains only the selected summary.
An indexing run exits with status 2 if it finishes with failed URLs, including failures already
stored in a resumed job. Setup, output, and background crawl errors return status 1. A configured
crawl limit can leave pending URLs without causing an error. See Indexing Results And Exit Status for details.
Save a retry list directly from plain indexing, an input job, or a recursive job:
hister index --input urls.txt --failed-urls failed-urls.txt --format json
hister index --force --input failed-urls.txt --failed-urls still-failed.txtThe output file is created before indexing and is empty when no URLs failed. For persistent jobs, it lists every URL currently marked failed, including failures from previous runs.
Inspect Jobs With hister crawl
The list, show, urls, queue, and errors commands accept --format / -f with text (the default), json, jsonl, or csv:
hister crawl list --format json
hister crawl show example-docs --format json
hister crawl urls example-docs --status failed --format jsonl
hister crawl errors example-docs --format csv > failures.csv
hister crawl urls example-docs --status pending --count --format jsonJSON output is always an array, including a single job or count. JSONL emits one object per line.
CSV includes a header. With --count, structured output contains a single count field.
Job records contain id, status, start_url, label, created_at, updated_at, and counts named pending, in_progress, done, failed, and skipped. Timestamps include the time zone. The show command also includes the saved rules object. If stored rules cannot be decoded, that
field contains the original string and a warning is logged.
URL records contain status, depth, and url. Error records contain error_code, url, and error. Structured error messages preserve their original tabs and newlines; text output replaces
those characters with spaces to keep each failure on one line.
List Jobs
hister crawl listEach entry includes its ID, stored status, starting URL, creation time, and counts for pending,
done, failed, and skipped URLs. The displayed job status values are unfinished, completed, and interrupted. unfinished means the job was started but has not recorded completion. It does not
mean that a crawler process is currently executing it.
Show One Job
hister crawl show example-docsThis prints job metadata, counts for every URL state, and the saved traversal rules. It is the best command for confirming the limits and filters that will be restored on resume.
Inspect Job URLs
hister crawl urls example-docsThe output contains tab separated status, depth, and URL fields for every row in queue order. The
possible URL states are pending, in_progress, done, failed, and skipped.
Filter the output to one status with --status. The accepted values are pending, failed, done, and skipped:
hister crawl urls example-docs --status failedPrint only the number of matching rows with --count or -c:
hister crawl urls example-docs --status pending --countWithout --status, the count includes every status. The older hister crawl queue command remains
available and lists every row without filtering.
Inspect Failures
hister crawl errors example-docsThis prints the stored error code, URL, and error message as three tab separated fields for each failed row. Failed rows are not retried when the same job resumes. A simple retry workflow is to extract the URLs into a new input job:
hister crawl errors example-docs | cut -f2 > failed-urls.txt
hister index --force --input failed-urls.txtDelete a Job
hister crawl delete example-docsDeleting a job removes its metadata and URL queue. Documents already added to the search index are not deleted.
To create a completely new crawl with the same name, delete the old job, then start it again. Add --force if its previously indexed pages must be fetched again so links can be rediscovered:
hister crawl delete example-docs
hister index --recursive
--force
--job-id example-docs
--allowed-domain docs.example.com
https://docs.example.comA Complete Resumable Workflow
Start a named and bounded crawl:
hister index --recursive
--job-id example-docs
--allowed-domain docs.example.com
--max-depth 4
--delay 1
https://docs.example.comStop it when needed, inspect its durable state, and continue it later:
hister crawl show example-docs
hister index --job-id example-docs --delay 1When it finishes, review the queue and any failed pages:
hister crawl queue example-docs --count
hister crawl errors example-docs