Skip to content
Development documentation for unreleased code on main. Behavior can change before a release. Latest release docs

Operating Luxirdev

Luxir runs as one process on one machine and hosts many isolated collections. Capacity comes from the size of that machine. Replication, sharding across nodes, failover orchestration, and snapshots are not built into the server yet; the production boundary at the end of this page lists what is missing.

The defaults are good for evaluation. Before you keep real data, decide on persistence and network isolation. The other settings on this page can stay at their defaults until you have a reason to change them.

The default backend is in-memory and loses every collection when the process exits:

Terminal window
./luxir

Use the filesystem backend on local storage for durable data:

Terminal window
./luxir --store.backend=fs --store.data-dir=/srv/luxir/data

Each collection is an independent index below the data directory. Collections present there are discovered at startup; a write to a new collection creates its storage by default.

A commit makes updates durable and visible to searches. After a crash, Luxir reopens the last durable commit. Updates accepted since that commit can be lost.

Use luxir pull for an online copy, then luxir promote to restore it as a writer under new collection incarnations. Do not restore a raw directory copy over an existing writer identity.

The portable tree is LAYOUT and t/. Copy each file’s bytes under the same relative key; do not copy node/, which contains the local write.lock, replication state and other node-private state. The destination needs a compatible Luxir build and the plugins referenced by the stored schemas. Hard links can become independent objects: link counts, inode numbers, timestamps, permissions and empty directories carry no index identity. Publication versions are computed by the destination backend, not stored in STATE or CATALOG.

For a stopped node, copy into an empty destination in any order, then open the destination only after the entire copy finishes. A filesystem snapshot of a stopped node has the same rule. The format can be served by a backend that reads these keys directly, including a cache over an object store; this does not require rewriting manifests or catalog entries.

For an online copy, dependency order is necessary but is not sufficient by itself. Capture the catalog and selected STATE bodies first and hold their snapshots and dependencies against retirement while copying. Install data files first, then their manifests, then the captured STATE bodies, and finally the captured CATALOG bodies. Install LAYOUT before exposing the destination. Never read a newer STATE or CATALOG midway through the copy: it could name files that were created after the data pass. Keep the captured catalogs stable against collection deletion or replacement as well. A coherent filesystem snapshot can provide the frozen source; otherwise use luxir pull, which pins snapshots. Plain recursive copying of a running writer provides none of those protections.

Readers ignore publication scratch files (.luxir-publish-*), unfinished data files (*.tmp) and unrelated names, even if a raw copy included them. These are not dependencies and may be omitted. Corruption markers are portable diagnostic metadata and should be preserved. A new writer over the copied namespace claims a new epoch there; it does not fence the original namespace. Do not combine the subsequent histories of independently written copies. Use promotion when a restore should start a new lineage.

--store.backend names the storage backend: ram and fs are built in, and a plugin build may add more (an unknown name fails startup with the list of registered ones). --store.layer NAME (repeatable, innermost first) wraps the backend in layers a plugin provides, such as a cache. --store.option COMPONENT.KEY=VALUE (repeatable) passes settings to the named backend or layer; the plugin documents its keys. Unknown keys and components that are not configured fail startup. The built-in ram and fs take no options.

Terminal window
./luxir --store.backend=s3 --store.option s3.bucket=search-data --store.layer=cache --store.option cache.bytes=10G

Luxir’s own --store.checked-dir.sync and --read-only wrappers always sit outside the layers. The effective stack is logged at startup as storage: .... See plugins.

A data directory has exactly one writer, which holds node/write.lock from startup until exit. This is node admission policy. Conditional file publication uses a separate directory flock, so Match is an atomic compare-and-swap among processes on a local filesystem, including processes that bypass write.lock. All publish modes take that lock only for the check and install; file and directory fsyncs run outside it. The kernel releases the lock when the process dies. This guarantee does not extend to network filesystems, whose server may revoke a stalled client’s lock.

--read-only opens an existing data directory without that lock:

Terminal window
./luxir --read-only --store.backend=fs --store.data-dir=/srv/luxir/data

A read-only node requires an existing data directory and writes no files. It serves searches, schema reads, and _stats.

Use it to query a directory another instance is writing, or to inspect one offline without risking a stray write. Current limitations:

  • The view does not advance. A read-only node opens its index view at startup and never reopens, so later commits are invisible until it restarts.
  • _stats describes the same snapshot as searches. Directory byte totals (files=true) still reflect the files present when stats are requested.
  • Pinned files are not reclaimed. Segment files are held open by mmap, so files the writer deletes stay on disk until the read-only node exits. A long-lived read-only node against a busy writer holds disk space that du attributes to no visible file.

Restart the read-only node to pick up newer commits and release pinned files. For a continuously refreshed copy with its own storage, use --replicate-from http://writer:9400. Followers own their local data directory and install verified snapshots without constructing an index writer.

Create and delete collections with POST /collections/_create and POST /collections/_delete (or unary luxir.Admin/CreateCollection / DeleteCollection). Deleting a collection also deletes its stored data; there is no undo. GET /collections (canonically /collections/_list) returns the sorted collection names, including any that failed to load; /_stats?collections=true has the per-collection detail.

Rename with POST /collections/_rename and {"from":"old","to":"new"}. Swap with POST /collections/_swap and {"a":"live","b":"staging"}. Both operate within one tenant; use /tenants/TENANT/collections/... to select it. The gRPC calls are RenameCollection and SwapCollections. Rename requires a free destination and swap requires two active collections. Read-only nodes and followers reject both operations.

Each operation changes the catalog once, then changes each name’s lookup binding. No files move and commit ids remain unchanged. A request keeps the collection it resolved at admission. During a swap, each lookup may see the old or new binding for that name; both names may briefly resolve to the same collection. Indexing into a name during a swap is undefined. Collection metrics, storage charges and cached data follow the incarnation. Session cache-size overrides follow names and apply to their new selections. Followers apply ready bindings independently, reusing incarnations they already have. A stuck download does not delay unrelated names. A lagging name keeps its old data only if the source has not assigned that data to another name; otherwise it is unavailable until its replacement is ready.

Deletion is synchronous and wins over concurrent use. When the call returns, the name resolves to nothing, the on-disk data is gone, and the name can be recreated as a fresh empty collection. New requests receive collection_unavailable while deletion is in progress. Searches already executing keep their index view and complete normally. Deletion waits for indexing work already accepted, including a running merge, so deleting a collection mid-merge can take as long as that merge.

On the filesystem backend, incarnations live at t/<tenant>/i/<id>/. A tenant’s CATALOG maps collection names to active incarnations and records reservations and retirements. Single-tenant deployments use tenant t0. Incarnation ids are millisecond timestamps in 11 lowercase hex digits, allocated monotonically by the catalog. The root LAYOUT marker identifies this format; older layouts are refused at startup and must be reindexed. Node-private state, including write.lock and replication.json, lives under node/. Each commit records its index format version. A collection written with another version is listed as failed to load, with both versions in the error, and must be deleted and reindexed.

Deletion marks the incarnation retiring in the catalog before removing its directory, then removes its entry. Writer startup finishes interrupted retirements and discards abandoned reservations; followers preserve reserved downloads. If deletion fails partway (for example an I/O error), the name stays unavailable with the recorded error and the delete can simply be retried. Deleting a collection that failed to load at startup is also the supported way to clear its broken on-disk state.

Updates become visible to searches at commit. An immediate "commit": {} waits until a new index view is published. commit_within_ms can coalesce publication and return before the update is visible while still bounding staleness.

Search freshness_ms expresses how stale a reader may be; 0 requires the latest commit. Decide these together:

  • Interactive writes that must be read immediately should use an immediate commit.
  • Sustained feeds should use a positive commit interval rather than publishing a new view for every small update.
  • Forced merges are a maintenance action, not part of a routine commit.

max_segments and wait_for_merges are explicit maintenance controls. A normal commit lets background merging choose the layout while ingest and search continue.

Defaults:

Surface Port Control
HTTP/JSON 9400 --server.http.port or -p
gRPC 9401 --server.grpc.port (defaults to HTTP port plus one)

A configured nonzero port binds 0.0.0.0; there is no listen-address option yet. Disable HTTP with --no-http. The gRPC server always starts.

Luxir does not provide TLS, authentication, authorization, per-tenant quotas, or a permission model. Do not expose either port directly to an untrusted network. Network policy is mandatory: place the process in a private network namespace, firewall both ports, or front them with an authenticated TLS proxy/service mesh.

HTTP liveness:

Terminal window
curl http://localhost:9400/health
{"status":"ok"}

gRPC registers the standard health service. The current HTTP check is process liveness, not a deep read/write check of every collection or storage device.

Operational statistics and Prometheus metrics are described in Stats and monitoring.

Filesystem publication syncs data files and their directory entries, then writes an immutable commit_<index_gen>.<epoch> candidate with a length and xxh3 checksum footer. It acknowledges the commit only after a durable conditional update of the incarnation’s STATE tip. Each writer open claims a new epoch; a superseded writer cannot advance that tip. Segment files carry their creator epoch, and later live-doc and overlay files also carry their writer’s epoch.

Startup follows STATE, not the highest candidate filename. If its manifest is invalid, recovery scans older candidates whose referenced files are present with matching sizes, and copies the recovered snapshot into a new incarnation. By default startup does not hash data files (see file integrity checks).

Once the new tip is durable, obsolete manifests and unreferenced data are removed. Writer startup sweeps unreferenced index files from fenced, older epochs; it leaves files of newer epochs alone. These local cleanup rules and process-local pins are not a retention protocol for shared remote readers. Catalog reservation/orphan cleanup also assumes startup has exclusive authority to decide what was abandoned, and collection transitions use process-local leases. Cross-process CAS protects catalog updates and writer fencing; it does not turn those cleanup policies into a multi-node retention coordinator. Removal durably deletes STATE first, fencing existing writers, then removes the incarnation directory and its catalog entry. Only catalog reservation can create STATE; claiming a writer or installing a snapshot requires it to exist. Writer incarnation ids are never reused. A follower can reserve a deleted source id again; its fresh STATE owner distinguishes that lifetime from earlier publications.

Replication pin counters are described in the replication reference. Replication file transfers have a fixed 60 s idle deadline per socket write, reset as writes complete. Ordinary HTTP responses have no write deadline.

Publication scratch files use the reserved .luxir-publish- prefix and are hidden from index listings. An owned filesystem view removes scratch files from processes confirmed dead on the same host when it opens. Live or inaccessible processes’ files are left alone. Copied scratch names are harmless even when a PID cannot be checked or has been reused. Unfinished *.tmp data files are also ignored by readers.

Retirement removals are not directory-synced: a crash may restore obsolete names, but the durable tip does not reference them. A failed candidate is removed only if STATE definitely does not name it. An error response can still have an unknown outcome; a visible candidate alone does not prove a commit. Clients continue to use incarnation:index_gen tokens. Candidate generations are decimal and epochs are lowercase base36; tools resolve the candidate from STATE rather than sorting the directory listing.

Set log verbosity with:

Terminal window
./luxir --log-level=info

Valid spdlog levels include trace, debug, info, warn, error, and critical. Startup logs include the loaded time-zone database version.

_stats returns one typed response, rendered as JSON by default or as Prometheus text format 0.0.4 with format=prometheus. GET /metrics is the node’s Prometheus scrape: the same response and rendering, with collections=true and system=true unless the URL says otherwise (it takes the same parameters, so format=json returns that response as JSON). Both renderings come from the same response. Prometheus carries every monitoring value; generations, segment and aux index layout, and the pre-summed totals are JSON-only (Prometheus sums the per-index series itself). A test fails when a new field is in neither list. The same response is available through unary luxir.Admin/Stats (structured protobuf; there is no text rendering on gRPC). The endpoint shares the administration API’s network access boundary and includes host information and tenant/collection names.

Route Scope
GET /_stats The node.
GET /metrics The node, as Prometheus text with collections and system statistics.
GET /tenants/{tenant}/_stats Every collection of one tenant.
GET /collections/{collection}/_stats, GET /tenants/{tenant}/collections/{collection}/_stats One collection.

The node scope returns the cheap counters by default; anything that walks collections, lists directories, or reads the operating system is opt-in. A tenant or collection scope always walks its collections and returns no node blocks.

Parameter Adds Cost
(default, node scope) service: request, search, ingest and background activity and latency; memory: indexing RAM budget, draining, pending-merge and retained delete bytes, RAM storage, field metadata, jemalloc and its arenas, configured budgets. Atomic loads and one allocator statistics read.
collections=true totals and per-collection collections: documents, segments, committed bytes, commit time, generations, merges, inverters, retained delete bytes, snapshot pins and drops, query cache, availability. Implied on tenant and collection routes. Briefly takes each writer’s index lock and decodes each reader-only manifest. No file I/O.
files=true bytes from an index directory listing; aux index and segment bytes. Implies collections. Lists every index directory.
segments=true Per-segment records. Implies files. Iterates segments.
system=true system: this process, the data filesystem and the process’s cgroup v2. Node scope only. About 15 procfs/cgroupfs reads and a scan of /proc/self/fd.
host=true host: host-wide CPU, memory, network, TCP, disk and pressure statistics, which node_exporter also reports. Node scope only. About 10 procfs reads.
format=prometheus Text exposition instead of JSON (text/plain; version=0.0.4). Rendering only.

Flags accept exactly true or false; format accepts json or prometheus. Any other value is a 400.

Terminal window
curl 'http://localhost:9400/_stats' # cheap counters
curl 'http://localhost:9400/_stats?collections=true&system=true' # everything without directory listings
curl 'http://localhost:9400/collections/docs/_stats?segments=true' # one collection's layout

Counts follow the JSON dialect: a zero-valued field is omitted rather than emitted, so a reader must treat an absent field as zero. Totals appear at the node, collection, and index levels, and each level sets only the counts that mean something there: collections only on the node total, shards only on node and collection totals. A node-wide walk includes load-failure tombstones with their error and excludes them from totals. A collection deleted during the walk is reported the same way, as unavailable; one whose statistics cannot be read for another reason carries that failure in its error, and the failure is logged. Each collection index is sampled coherently, but a node-wide response is not one transaction across collections.

Each serving collection also reports its activity since it was opened: searches and index batches by outcome with their summed seconds, and the documents and delete IDs its writer accepted or rejected. These count only work that reached the collection. service has the node totals, which also count requests that never resolved a collection and keep their counts after a collection is deleted.

Two byte counts measure different things and will disagree:

  • committed_bytes sums the file sizes recorded in the current published snapshot. It needs no filesystem access, so it is always reported. It excludes in-flight files and files kept only by snapshot pins.
  • bytes (present only with files=true) is a directory listing of the whole index directory: manifests, in-flight flush and merge output, and pin-retained files. It is normally larger than committed_bytes, and can exceed the sum of segment bytes. With segments=true each segment reports its own bytes (data + deletes + overlays) and each aux entry reports the bytes of its listed files.

service counters are process-wide and reset on restart. Each activity (operations per transport, search, index_batches, flush, commit, delete_pass, merge, vector_build, reader_refresh) reports in_flight, completions by outcome (ok, error, cancelled) and a duration histogram. Histogram counts are cumulative, aligned with service.request_duration_bounds (requests, search execution and index batches, from 10 microseconds) or service.background_duration_bounds (the background stages, up to an hour), in seconds, plus a final +Inf entry, and are omitted while count is zero. A request or call fails when its response carries a whole-request error (a failed search, an update that applied nothing), when a stream ends in failure, or when the server fails it; a client that goes away first cancels it. errors counts failed requests and calls by the kind of their first error. An update that applies some of its documents succeeds; its rejected documents are counted in documents_rejected, and those with an error of their own in document_errors by kind. An NDJSON stream fails as a whole: when it ends in failure, when its URL commit fails, or when it applied none of its input. Messages on a gRPC Search or UpdateStream call count as searches or index batches; the call fails only when it ends in failure. _stats requests themselves are counted under the stats operation; replication routes are not counted.

In memory, jemalloc is present only when Luxir runs on jemalloc. jemalloc.allocated_bytes counts live bytes allocated through jemalloc; resident_bytes is jemalloc’s upper bound including fragmentation and freed pages awaiting decay; the process RssAnon also counts memory jemalloc does not manage.

memory.indexing_budget.retained_delete_bytes reports resident delete packets across all collections and owners, including commit and merge catch-up references. It is included in reserved_bytes, not an additional allocation. Each collections[].shards[].index.retained_delete_bytes reports that writer’s pending and merge catch-up packets, counted once per packet. Packets remain charged until their last owner releases them; flushing an inverter transfers their reservation rather than freeing it.

Ids and generations that appear in filenames use their filesystem spelling so the response correlates directly with a directory listing: each segment’s seg is its data-file prefix (e.g. s0a), while live_gen and aux gen are the sortable strings embedded in filenames (segment s0a with live_gen 01 has its deletes in s0a__L01). These strings sort in generation order, and an absent field means none. schema_gen, index_gen, core_gen, and update_version are numeric. The commit_<index_gen>.<epoch> manifest selected by STATE references an immutable schema_0.0.<local>.<epoch> file in the incarnation. Schema files contain the authored definition and field introduction history, with the same length/xxh3 footer as manifests. They participate in the ordinary file inventory, transfers, verification and snapshot retention. Copy them with the data files before copying manifests.

Each node shares parsed schema versions across its snapshots and readers. memory.schema_cache_bytes reports an estimated owned payload, charged once per cached version to its schema tenant and released with its last user. This is separate from RAM backing-file bytes and allocator RSS; it excludes allocator overhead and analysis factories’ private resources.

Prometheus scrapes /metrics by default, so a job needs only its targets. Parameters choose the cost: collections=false on nodes with very many collections, and host=true only where node_exporter does not cover the host.

scrape_configs:
- job_name: luxir
scrape_interval: 15s
static_configs:
- targets: ["localhost:9400"]

Prometheus supplies up, job, and instance. Aggregate service traffic over instances; retain instance when diagnosing a specific node. Scrapes are counted as operation="stats" traffic, so exclude that label when charting client load.

Response took_us measures one complete request message through readiness to serialize its final response; luxir_request_duration_seconds also covers HTTP body reception and response delivery, and counts an entire gRPC stream as one call. luxir_search_duration_seconds starts later, at engine submission, and shares the search’s final-response endpoint with took_us.

Metric or family Meaning
luxir_requests_total, luxir_requests_in_flight, luxir_request_duration_seconds HTTP requests or gRPC calls, by transport, bounded operation, and completion outcome. Duration includes body reception after HTTP headers, execution, waits, and response delivery. A gRPC stream is one call, regardless of its message count.
luxir_request_errors_total Failed requests/calls, by transport and the fixed kind of their first error. An update that applies some documents is not a failed request.
luxir_request_body_bytes_total, luxir_response_body_bytes_total Application payload throughput; excludes transport framing. Response bytes are counted when submitted for sending, even if delivery subsequently fails.
luxir_searches_total, luxir_searches_in_flight, luxir_search_duration_seconds Searches the engine accepted, by outcome, from submission to the final response. A gRPC Search call can carry many searches, so this is its per-search view; cancelled means the search ended without a final response.
luxir_search_execution_duration_seconds Search-body execution, excluding scheduler/visibility waits and later streaming drains.
luxir_search_resource_exhausted_total, luxir_search_warnings_total Resource-limit failures (including the request memory breaker), and searches with declared degradation warnings.
luxir_index_batches_total, luxir_index_batches_in_flight, luxir_index_batch_duration_seconds Index-batch processing inside the engine, excluding queueing and commit work. HTTP NDJSON can produce many batches per request. A batch fails only when none of it applied.
luxir_ingest_documents_total, luxir_ingest_delete_ids_total Documents accepted into indexing or rejected/rolled back, and accepted delete IDs. Acceptance does not mean durability; commits can still fail. Deletes count submitted IDs, not matched documents.
luxir_ingest_document_errors_total Documents rejected with an error of their own, by kind: invalid_request is bad input, internal an engine fault on that document. Documents rolled back with a failed batch have no error of their own.
luxir_stream_pauses_total Response backpressure events by transport.
luxir_http_connections, luxir_http_io_threads Live HTTP sessions and running connection-shard threads; the thread gauge excludes the accept thread. gRPC active calls are in luxir_requests_in_flight.
luxir_background_total, luxir_background_in_flight, luxir_background_duration_seconds Flush, commit-finish, merge, vector-build, and reader-refresh stages by kind. These are stage attempts; a commit handing off to a forced merge can have multiple finish stages.
luxir_node_memory_budget_bytes, luxir_search_request_memory_limit_bytes Configured managed-memory budget and tracked per-search ceiling. Neither is a process RSS limit.
luxir_index_ram_*_bytes Indexing reservations, budget, draining reservations, pending merge demand, and retained delete packets (part of the reservations, never freed by a flush). The budget is soft; pending demand is capped at that budget when limited.
luxir_storage_ram_bytes, luxir_storage_ram_limit_bytes RAM storage consumption and its cap, separate from filesystem usage.
luxir_field_metadata_objects, luxir_field_metadata_bytes Cached field metadata by part (directory, page, entry, name).
luxir_allocator_bytes, luxir_allocator_arena_*_bytes Jemalloc allocated, active, resident, mapped, retained, dirty, and metadata memory, plus indexing/big-buffer arenas. Resident is an allocator estimate, not process RSS. Absent without jemalloc; check luxir_allocator_stats_available.
luxir_scrub_*_total, luxir_scrub_last_pass_end_timestamp_seconds Checksum scrub passes, files, read bytes, mismatches, and other check errors, and when the last pass ended. Present only while scrubbing is enabled; the timestamp is absent before the first pass.
luxir_multiterm_expansion_refills_total Process-wide multi-term expansion refills.
luxir_build_info, luxir_node_read_only Build version, CPU target and source commit, and read-only configuration.
luxir_stats_duration_seconds Time gathering the response, excluding serialization.

With collections=true:

Metric or family Meaning
luxir_collections, luxir_collection_available Collection availability, including collections deleted or unreadable during the walk. An available HTTP endpoint does not imply every collection is usable.
luxir_collection_storage_ram_bytes RAM storage owned by the collection.
luxir_index_documents, luxir_index_segments, luxir_index_uncommitted_segments Flushed segment document/segment state; excludes documents still buffered in inverters.
luxir_index_committed_bytes, luxir_index_last_commit_timestamp_seconds Current snapshot descriptor bytes and publication timestamp. An old timestamp alone does not indicate a problem on an idle collection.
luxir_index_directory_bytes Directory-listing bytes; only with files=true. See the two byte counts above.
luxir_index_commits_total Commits since the writer opened.
luxir_index_retained_delete_bytes Delete packets the writer retains for its pending set and merge catch-up, counted once per packet; part of luxir_index_ram_reserved_bytes.
luxir_collection_searches_total, luxir_collection_search_seconds_total Searches that reached the collection, by outcome, and their summed duration from submission to the final response. The mean is the ratio of the two rates.
luxir_collection_index_batches_total, luxir_collection_index_batch_seconds_total The collection’s index batches by outcome, and their summed processing time.
luxir_collection_documents_total, luxir_collection_delete_ids_total Documents the collection accepted or rejected, and its accepted delete IDs.
luxir_inverters, luxir_merge_tasks Busy, idle, and flushing inverters; outstanding merge work includes queued tasks.
luxir_filter_cache_* Hit/miss counters, build attempts, builds, dead builds, thrash skips, admissions, rejections, evictions, purges, bypasses, reader-stable counters, payload/metadata memory, and payload budget.
luxir_snapshot_pins, luxir_snapshot_retained_bytes, luxir_snapshot_pin_drops_total Current pins, retired files held only by pins, and pins dropped by reason (idle, budget). A stats request never expires pins or removes files.

Counters reset when their owner restarts. Collection cache counters also reset on recreation or explicit cache-control reset_counters; use rate() or increase() rather than subtracting raw samples. Histograms use seconds and cumulative buckets, including +Inf; compute fleet percentiles from aggregated buckets, not averages of node percentiles.

Collection metrics use tenant and collection labels. There are no labels for raw URLs, query text, field names, document IDs, commit IDs, or segment IDs. Each collection adds about 50 series; on a node with very many collections, scrape /metrics?collections=false and read per-collection detail from _stats on demand. Generations, aux indexes, and per-segment layout are JSON only.

With system=true (Linux only), the response describes this process, its data filesystem and its cgroup:

Family Coverage
process_* CPU time, thread count, resident/virtual/anonymous/file/shared/swap/peak memory, open descriptors and soft descriptor limit, start time, voluntary/involuntary context switches, minor/major page faults, storage bytes, and I/O syscall bytes/counts. Context switches and CPU cover all process threads.
luxir_data_filesystem_* The configured filesystem data directory’s capacity, free/available bytes, total/free inodes, and read-only state. Only for filesystem storage. Arbitrary remote mounts are not probed.
luxir_cgroup_* Current cgroup v2 memory usage/high/max, task count/limit, memory pressure/OOM events, CPU time/quota, and CPU throttling time/periods. Memory includes page cache and descendants.

Cgroup statistics locate this process’s membership using procfs and the cgroup mount. They report local limits; ancestor cgroups can impose stricter limits. A cgroup limit of -1 means unlimited. Cgroup v1 is not supported.

With host=true (Linux only), the response adds host-wide statistics, the ones node_exporter reports. They are named luxir_host_* after their JSON fields, so they never collide with node_exporter’s series:

Family Coverage
luxir_host_cpu_seconds_total, luxir_host_load*, luxir_host_procs_*, luxir_host_forks_total, luxir_host_context_switches_total, luxir_host_boot_time_seconds CPU modes per logical CPU, load averages, runnable/blocked tasks, task creation, context switches, and boot time.
luxir_host_memory_* Total/available/free memory, page cache, swap, dirty/writeback pages, kernel/slab memory, faults, swap events, and OOM kills where supported.
luxir_host_filefd_* Host-wide allocated file handles and their limit.
luxir_host_network_*_total, luxir_host_tcp_* Per-interface bytes/packets/errors/drops and TCP established connections, retransmissions, resets, input errors, and failed attempts. These describe the network namespace, not just Luxir traffic.
luxir_host_disk_* Every visible block device’s read/write bytes, operations, merges, cumulative latency, I/O in progress, busy time, and weighted I/O time. Device and partition counters overlap; do not sum them indiscriminately.
luxir_host_pressure_stall_seconds_total CPU, memory, and I/O pressure stall time, with some/full labels where supported.

Inside a container, the host CPU, memory and disk figures describe the host, not the container; network counters follow the container’s network namespace. The cgroup statistics under system are the container’s own view.

Each section lists the collectors that read their sources in available and the others in unavailable (JSON), reported as luxir_stats_collector_success{collector="..."} 1 or 0. An unavailable collector’s measurements are omitted, not reported as zero. Restricted procfs, disabled PSI, and absent cgroup controllers are common reasons.

Useful initial alerts are: scrape failures, unavailable collections, rising internal/unavailable errors, p99 latency above the service target, sustained indexing-memory pressure, filesystem byte/inode exhaustion, descriptor usage near its limit, cgroup OOM events/throttling, and failed background stages. For example, aggregate HTTP search p99 across nodes with:

histogram_quantile(0.99,
sum by (le) (rate(luxir_request_duration_seconds_bucket{
transport="http", operation="search"
}[5m]))
)

There are no replication metrics in _stats; see replication status. Search admission queues and deadline-driven partial results are not implemented on this revision, so no statistics exist for those features.

HTTP defaults to one connection-I/O shard per available logical CPU, with a minimum of one, plus one dedicated accept thread. gRPC defaults to half the available logical CPUs, also with a minimum of one; its completion-queue threads only move bytes and hand requests off, so they need not match the query concurrency. Automatic sizing respects the process’s CPU affinity, so a server restricted to 28 logical CPUs defaults to 28 HTTP shards and 14 gRPC threads. Set explicit counts with:

Terminal window
./luxir --server.http.threads=8 --server.grpc.threads=8

--server.http.threads sets the number of shards, not the total HTTP thread count; each shard owns one io_context and one runner. Connections are assigned round-robin at accept and remain pinned to their shard.

Indexing, merges, and search work moved off a transport thread run on a shared pool of worker threads. --server.workers sets its size. The default (0) is one worker per available logical CPU less one, which leaves a CPU for the transport threads; on machines with 4 or fewer logical CPUs it is one worker per CPU, because holding one back would idle a large share of a small node (half of a 2-vCPU one). Transport threads are not counted: they share the CPUs with the workers.

Terminal window
./luxir --server.workers=2

HTTP search requests default to serial execution directly on the io shard that received them. This avoids a scheduler handoff and idle-worker wakeups but occupies that connection’s thread for the query’s duration. The request-level max_parallel changes that: 1 moves the request onto the shared work-stealing scheduler, still serial (use this for requests expected to be expensive, so they do not delay other traffic on the same connection thread); -1 keeps the request on its transport thread but lets the scheduler’s workers join in with unlimited intra-request parallelism; values above 1 are reserved for a bounded parallelism budget and not implemented yet. A request with min_commit continues on a shared worker once its commit wait ends, whatever the setting.

gRPC searches always run on the shared workers, never on the completion-queue thread: a client such as a gateway multiplexes many calls over one HTTP/2 connection, and gRPC offers a connection’s new calls to its preferred completion queue first, so inline execution would run them one at a time. max_parallel keeps its parallelism meaning there: 0 and 1 run serially on a worker, and -1 fans out from one.

--server.stream_buffer_bytes sets the per-HTTP-connection and per-gRPC-call high-water mark for serialized responses (default 1 MiB). Producers pause above it and resume after the buffer drains below half. Increase it only when more per-connection buffering is a measured win; total memory scales with the number of simultaneously slow streams.

--max-ram-mb is the node-wide budget (MiB) for memory Luxir manages explicitly. It defaults to 25% of system RAM - or of the cgroup memory limit when the process runs under one, so a container is sized by what the kernel actually enforces rather than by the host’s RAM. Subsystem budgets are derived from it, so raising or lowering this one value moves them together. 0 leaves the node unlimited. The resolved values are logged at startup.

Terminal window
./luxir --max-ram-mb=16384 # node budget; indexing gets 8192
./luxir --indexing.max-ram-mb=4096 # override just the indexing share
Terminal window
./luxir \
--indexing.max-ram-mb=8192 \
--indexing.max-inverter-ram-mb=2048
  • indexing.max-ram-mb is the shared MiB budget for all of a node’s indexing work: merges reserve against it before they start, and inverters count their RAM against it, so a release that finds the budget over its cap flushes the idle inverter offering the most relief above the pressure flush floor. Retained delete packets count against the budget and are not flush relief. It defaults to half of --max-ram-mb, and to none at all on a --read-only node, which never indexes. 0 leaves it unlimited.
  • indexing.max-inverter-ram-mb flushes one inverter to a segment when it grows past that size, checked at the end of an update batch. Total indexing RAM is bounded by indexing.max-ram-mb, so this cap defaults to it: one inverter may hold the whole indexing budget (with a single stream there is nothing else holding it) but no more. It is clamped to 3814 MiB - an inverter’s memory pool can address at most 4 GiB, and the clamp leaves headroom for one batch’s overshoot - so an explicit larger value is clamped with a startup warning.
  • Lower indexing.max-inverter-ram-mb to trade merge work for flush latency and peak RSS: smaller flushes land sooner and cost less at once, at the price of more segments to merge.
  • A merge that cannot reserve its memory waits and asks indexing to give some back, but never more than half of indexing.max-ram-mb: idle inverters are flushed for it only while indexing holds more than that half, and only an inverter holding at least its share of the half (and at least indexing.pressure-flush-floor-mb, default 4). Without a waiting merge, inverters are flushed only once the whole budget is in use.

--indexing.huge-pages is enabled by default in jemalloc builds. It advises transparent huge pages for whole-huge-page indexing allocations; disable it with --no-indexing.huge-pages (or --indexing.huge-pages=false). This is a best-effort hint: a host with THP set to never still works using ordinary pages.

The HTTP request limits distinguish bounded material from streams:

Terminal window
./luxir \
--indexing.max-request-body=32MB \
--indexing.max-record=32MB \
--indexing.stream-batch-size=1MB \
--indexing.stream-batch-docs=10000 \
--indexing.max-inflight-batches=0
  • max-request-body caps buffered JSON requests and atomic NDJSON groups.
  • max-record caps one streamed NDJSON record so a missing newline cannot grow memory without bound. It inherits max-request-body when unset.
  • stream-batch-size and stream-batch-docs are internal handoff thresholds, not limits on the number of documents in one HTTP stream.
  • max-inflight-batches bounds the number of internal batches one connection has decoding or indexing at once. 0 (the default) uses the worker count (--server.workers) plus two; 1 indexes one batch at a time. Reads pause when the limit is reached and resume as batches are indexed, without waiting for earlier batches to complete. A larger limit uses more staging memory.

gRPC has one request limit, on the size of each received message (a unary request or one message of a stream):

Terminal window
./luxir --server.grpc.max-receive-message-size=64MB

It defaults to --indexing.max-request-body, so the buffered HTTP and gRPC ceilings move together. A larger message fails with RESOURCE_EXHAUSTED before it is decoded; see gRPC API. Replies have no size limit; search results are streamed in bounded batches.

Search has its own limits:

Terminal window
./luxir \
--search.request-memory-max-bytes=0 \
--search.max-op-depth=8 \
--query-cache-bytes=64MB
  • search.request-memory-max-bytes is a per-request breaker for query memory such as facet aggregate state and prefix top-term selection; 0 (the default) is unlimited.
  • search.max-op-depth bounds how deeply operations may nest in one request.
  • Multi-term expansion limits are operator opt-ins, all off by default (0). With limits off, broad patterns on high-cardinality fields simply run. Operators exposing public endpoints can bound this work using the following options. A request cannot raise or disable an operator limit. Exceeding one fails with RESOURCE_EXHAUSTED naming the query and limit, never a truncated match set. See Expansion limits.
    • search.multiterm-max-expansions: terms one query may expand per segment, counting selected terms for a capped prefix. A merge can change whether a query passes.
    • search.multiterm-max-terms-examined: dictionary terms visited across all multi-term and fuzzy scans, including refills, whether matching or not.
    • search.multiterm-max-postings: request-wide estimate, the summed docFreq of selected terms, not postings decoded. Fuzzy’s scored rewrite is excluded.
    • search.multiterm-min-prefix-length: pattern-only minimum literal prefix in bytes for unbounded prefix, wildcard, and regex queries. Finite languages are exempt. It never consults the index or cache. A type-ahead endpoint using N rejects keystrokes shorter than N. Default 0 disables the rule.
  • multi_match field lists containing patterns may resolve to at most 64 distinct physical targets after resolving search bindings and deduplicating, including explicit fields. See Multi-match.
  • query-cache-bytes caps each collection’s filter cache (default 64MB; 0 disables it). Cached filters are evicted by rebuild cost per byte, so under pressure the cache keeps expensive compound filters and drops cheap ones first. Cache counters appear in _stats?collections=true (and on the collection and tenant _stats routes). Date ranges with an unrounded NOW bound are never cached; see Dates. The gRPC Admin/CacheControl call’s max_bytes (administration only) changes the budget at runtime: a shrink evicts down to the new budget, a grow evicts nothing, and 0 disables the cache and drops its entries. Without a tenant or collection it sets every collection, including ones created later; with a tenant or collection it sets the existing ones selected. The setting is kept when a collection is replaced (for example by a follower’s re-download), dropped when the collection is deleted, and not persisted: a restart returns to query-cache-bytes.

IANA time-zone names require the system zoneinfo database (tzdata), read from $TZDIR or /usr/share/zoneinfo at startup. Without it, date queries retain UTC and fixed-offset support but named zones are unavailable. Keep the zoneinfo data identical across machines if requests must produce identical civil-time boundaries.

On SIGTERM or SIGINT the server stops accepting connections, waits for in-flight gRPC and HTTP requests to finish, stops background work (scrubbing, replication), releases the data directory and exits with status 0. Updates that were not yet committed are not committed by shutdown: only acknowledged commits are durable, so before a planned stop quiesce producers and publish an immediate commit. A second signal during the drain exits immediately with status 128 + signal. After an unplanned stop, the filesystem backend reopens the last durable commit. If recovery selects an older valid root, a writer publishes it under a new incarnation so followers cannot confuse it with an older generation of their current source. File retirement does not fsync removals: reappearing obsolete files are harmless and a later sweep reclaims them.

At startup, a collection whose top-level index metadata cannot be parsed is kept as a tombstone rather than preventing healthy collections from loading. Queries and updates to that collection return the recorded load error as an unavailable error (HTTP 503, code collection_unavailable) and the server will not silently recreate storage over it. Other segment corruption is detected when the affected reader is opened, or earlier with the file integrity checks below.

Every committed manifest records each file it references with its size and xxh3-64 digest, computed while the file is written. Checks read a file with its own descriptor (never through the mapping queries use), so a bad sector fails the check instead of the process, and a check never touches the request threads’ address space.

  • Transfers always verify: a replicated file is hashed while it streams in and is installed only if it matches, and a follower hashes a leftover local candidate before reusing it. Reuse by hard link (a repointed follower, luxir promote, writer recovery from an older root) compares manifest descriptors only; whether the linked bytes still match is the at-rest question the checks below answer.
  • --store.verify-on-open=full reads every file of every collection’s committed snapshot at startup, one collection at a time on the node’s indexing threads. Within a collection, files hash in parallel, largest first; checksum descriptors are held for at most one collection at a time. Checked files are verified before any reader maps or parses them. Readers map those same descriptors, so a file retired during hashing remains readable. The default, none, trusts the manifest footer and file sizes. A mismatching, missing or unreadable file fails only that collection: it loads as unavailable (HTTP 503, code collection_unavailable) with an error naming the file (checksum mismatch: <file>: expected size=.. xxh3=.., actual .., missing file: <file> or unreadable file: <file>: .. for EIO), and its data is kept. Other verification errors also fail the load, with could not verify file: <file>: ..; verification never creates corruption markers. A follower then fetches the damaged files from its source. luxir promote --store.verify-on-open=full checks the files it links the same way. Startup reads the whole index, so expect it to take as long as reading the data directory once.
  • --store.scrub-interval-s=N (default 0, off) re-reads the files of every published snapshot every N seconds on one thread, with O_DIRECT where the filesystem allows so the pass reads the media and leaves the page cache to queries, paced to --store.scrub-max-bytes-per-sec (default 64MB, 0 for unthrottled; the rate is the throttle, the thread’s low CPU priority only yields the CPU). A size/digest mismatch, EIO, or a held file missing on a writer or follower is logged at error level and fails the collection: it is unavailable with the same error, a writer publishes nothing more over the damaged bytes (no further update, commit or merge), and the suspect file is recorded in a corrupted_<id> marker in the incarnation directory. GET /_stats reports node-wide scrub counters (passes, files, bytes, mismatches, errors, last_pass_end in Unix milliseconds) while scrubbing is enabled. mismatches includes EIO and unexpectedly missing files. Other errors (such as descriptor exhaustion, allocation failures or permission errors) are warnings counted in errors; they leave the collection available and the next pass retries. On a read-only node a live writer can retire files still served through mappings: the scrub skips missing files.

On a writable node a marker survives restarts: at startup the files it names are re-verified with O_DIRECT where supported, whatever --store.verify-on-open says. Under full other files use buffered reads. The collection stays unavailable while a suspect file still fails; the marker is removed once its files pass or are replaced. Read-only nodes honor existing markers but cannot persist new ones or clear them, so their scrub failures do not survive a restart unless another process recorded a marker.

Markers travel with luxir promote and writer recovery into a new incarnation, restricted to files in the copied manifest. A promoted or recovered writer therefore re-checks known suspects even under the default policy. A follower repairs a scrub hit by itself: the next sync re-downloads every marker-named file even if its buffered hash matches, checks other local candidates as usual, and installs the copy again. Suspect files are never linked from the serving copy, and a failed serving collection supplies no hard-link reuse. An advance cannot replace a suspect serving file in place; repair first unloads the failed collection and re-stages it. Repair a writer by restoring from a follower (luxir pull and luxir promote) or by replacing the damaged file from a backup and restarting.

Before treating Luxir as a production service, account explicitly for the features it does not yet supply:

  • no automatic failover, leader election, or distributed query execution;
  • no built-in TLS/authentication/authorization;
  • no scheduled backups or automatic restore (use luxir pull and luxir promote);
  • pre-1.0 wire and schema interfaces that may change.

None of these can be enabled by configuration. The pieces those layers will build on are already in place: immutable commits, isolated collections, transport-independent request messages, and a node-wide scheduler and memory model.