Data Storage and Lifecycle

Hister keeps current searchable documents until you replace or delete them. It does not apply an automatic expiry period, a maximum document count, or a total disk quota. Some related SQL records can remain after current document deletion. The server operator is responsible for monitoring storage, creating backups, and applying any required deletion schedule.

Lifecycle at a Glance

Event	Result
A new URL is submitted	Hister creates a searchable document for that URL and owner
The same normalized URL is submitted again	Hister replaces the current document content and increases its submission count
A versioning rule matches a changed page	Hister also stores a difference record in the SQL database
A watched file changes	Hister updates its searchable document
A watched file is removed	Hister keeps the document unless delete_on_remove: true is configured
A source bookmark or browser history entry is removed	No change occurs in Hister
A document is deleted in Hister	The current index record and associated current assets are removed
hister cleanup is run	Unreferenced HTML and favicon files are removed, but documents are not deleted

Where Data Lives

The app.directory setting is the main Hister data directory. A default SQLite deployment stores these items there:

Location	Contents
index.db and index_LANG.db	Searchable document fields and full text indexes
data/html/	Compressed HTML previews, addressed by content hash
data/favicon/	Compressed favicons, addressed by content hash
db.sqlite3	Users, search result history, crawl jobs, version differences, and internal job state
vectors.sqlite3	Semantic search chunks and vectors when semantic search uses SQLite
rules.json	Rules and aliases in single user mode
.secret_key	The secret used to sign sessions

Language detection can create more than one index_LANG.db directory. Identical HTML or favicons share the same content addressed file, which reduces duplicate storage.

When server.database is a PostgreSQL connection string, SQL data is stored in PostgreSQL instead of db.sqlite3. Semantic vectors are also stored in PostgreSQL. The Bleve search indexes and HTML and favicon data remain under app.directory.

Configuration files can live outside app.directory. Source files in watched directories are not copied into the SQL database, but their extracted content is placed in the search index.

Storage Limits

Hister does not enforce a total storage limit. Available disk space and the selected database backend are the practical limits.

Two settings are easy to mistake for quotas:

• indexer.max_file_size_mb limits an individual watched local file. Its default is 1 MiB. It does not limit browser pages, imported documents, or total index size.
• server.max_batch_body_size limits one batch API request. Its default is 40 MiB. A document submitted in a batch must fit within that request. The setting is not a total storage quota and does not apply to other submission routes.

Full HTML previews, large document text, version differences, and semantic vectors all increase storage use. Browser history size alone is not a reliable estimate because page content varies greatly.

To reduce growth:

• Set app.disable_previews: true if saved HTML previews are not needed.
• Use skip rules to exclude noisy, private, or low value URLs before capture.
• Apply versioning rules only to pages whose earlier content matters.
• Delete stale documents with a date query after reviewing the matches.
• Remove completed persistent crawl jobs when their resume data is no longer useful.
• Monitor the data directory, PostgreSQL database, and backup storage with the tools used for your deployment.

Updates and Versioning

The document identity is its normalized URL plus its owner. Hister removes URL fragments and common utm tracking parameters during normalization. Several submissions can therefore resolve to one current document.

On a normal revisit, the new title, text, HTML preview, metadata, and update time replace the prior current values. If the previous HTML or favicon file is no longer referenced by another document, Hister removes that file.

Versioning is separate. When a URL matches a versioning rule and content changes, Hister stores text and HTML differences in the SQL database. These records let the preview interface reconstruct earlier content. Version records have no automatic age or count limit.

Disabling a versioning rule stops future differences from being recorded. It does not remove differences already stored.

Preview Retention

Full HTML previews are enabled by default. To stop storing new HTML:

app:
  disable_previews: true

Restart Hister after changing the setting. Running hister reindex while previews are disabled removes previously stored HTML files and rebuilds the search indexes without preview content. Extracted plain text remains searchable. Favicons are not affected.

See Disable Previews for the user interface effects.

Deleting Documents

Delete one result from the web or terminal interface, or delete every document matching a query with the command line client:

hister delete 'domain:example.com'

Always preview a broad deletion query first:

hister delete --dry --verbose 'updated:>90d'

Relative time filters describe elapsed age, so updated:>90d matches documents not updated for more than 90 days. The command asks for confirmation unless --yes is supplied.

For each matching current document, deletion removes:

• The searchable document from the Bleve indexes.
• Its HTML preview and favicon when no other indexed document references the same content.
• Its semantic chunks and pending embedding job.
• Matching opened result history associations for the same URL and owner. These associations use soft deletion in the SQL database.

Document deletion does not remove:

• The source browser history entry, bookmark, local file, or remote page.
• Stored version differences in the SQL database.
• Persistent crawl job records.
• Server logs, exported JSON files, filesystem snapshots, or backups.
• Search history records that are unrelated to the deleted URL.

If a complete erasure is required, the server operator must also address version records, job metadata, logs, exports, and backups according to the deployment. Hister does not currently provide one command that erases every related copy.

Avoiding Automatic Reappearance

A deleted document can return if an active collector submits it again. Before deletion, pause the browser extension, add a skip rule, remove the URL from a crawl queue, or change the watched directory configuration as appropriate.

For watched files, delete_on_remove: true makes future file removal delete the corresponding indexed document automatically. This option is disabled by default.

Cleanup Is Not Retention

hister cleanup scans stored HTML and favicon files and removes files that no current indexed document references:

hister cleanup

Normal updates and deletions already attempt to remove unreferenced assets. Cleanup handles leftovers, such as files left by an interrupted operation. It does not delete searchable documents, version differences, crawl jobs, search history, or semantic vectors that still belong to a document.

Crawl Job Lifecycle

Persistent crawls and browser imports keep queue state in the SQL database so they can resume after interruption. Completed and failed job records remain available for inspection.

List and remove them explicitly:

hister crawl list
hister crawl delete JOB_ID

Deleting a crawl job removes its queue and status records. It does not delete documents that the job already indexed.

Multi User Ownership

All users share the server storage and search index files, but each current document has an owner ID. Normal searches for an authenticated user include that user’s documents and global documents owned by user ID 0.

A regular user can delete only their own current documents. An administrator deletion request has no automatic owner restriction and can affect other users. The command line --dry check uses normal administrator search scope, so it does not reveal matching documents owned by other users. On a shared server, constrain an administrator deletion query with user_id:ID and never infer its full impact from the dry run count alone.

Deleting a user with hister delete-user USERNAME --purge asks the server to remove current indexed documents detected by its preflight, then soft deletes the account. Verify the result because this is not complete data erasure and records or current documents missed by the preflight can remain. See delete-user for details.

Backups and Exports

For a complete server backup, stop Hister and back up the entire app.directory, plus any configuration and tui.yaml files stored elsewhere. PostgreSQL deployments also require a database backup. Watched source files need their own backup plan. Remember that exports, filesystem snapshots, container volume backups, and remote database backups create additional retained copies.

hister export exports current indexed documents and any current preview content available to the caller. It is useful for migration, but it is not a complete server backup. It does not preserve user accounts, search result history, crawl jobs, stored version differences, rules, sessions, or all server configuration.

Protect backups as carefully as the live server because they can contain full page text, private previews, local file content, and credentials accidentally captured from pages.