Skip to content
Development documentation for unreleased code on main. Behavior can change before a release. Latest release docs

Luxir Featuresdev

Luxir is a native-code search engine with full-text relevance, vector similarity, faceting, analytics, and geo search in one index. Queries, filters, facets, statistics, and rank fusion go in one request, over a JSON/HTTP API that is easy to write by hand and a gRPC API for programs. It is native code written for modern hardware.

The simplest search is a URL:

GET /collections/main/_search?query=title_t:kings
{
"docs": [
{
"id": "1",
"author_name": "Brandon Sanderson",
"price_f": 12.5,
"series_s": "Stormlight",
"title_t": "The Way of Kings",
"year_i": 2010
}
]
}

A JSON body takes the same fields and adds ops. This one returns documents, counts by series, and the average price of all matches in one request:

POST /collections/main/_search
{
"query": {
"match": {
"author_name": "sanderson"
}
},
"fields": ["id", "title_t", "price_f"],
"limit": 2,
"count_limit": -1,
"ops": {
"series": {
"field_facet": {
"field": "series_s"
}
},
"average_price": "avg(price_f)"
}
}
{
"found": 3,
"docs": [
{
"id": "1",
"title_t": "The Way of Kings",
"price_f": 12.5
},
{
"id": "2",
"title_t": "Words of Radiance",
"price_f": 15.0
}
],
"ops": {
"average_price": 12.0,
"series": {
"buckets": [
{
"val": "Stormlight",
"count": 2
},
{
"val": "Mistborn",
"count": 1
}
]
}
}
}

That works with no schema or collection created ahead of time and no client library. The Quickstart gets you here in a few commands with these same documents, and the architecture page explains why the engine is built the way it is. This page lists what ships today.

Native code with no garbage collector and no heap ceiling, built to use a whole machine. Designed from the start for parallelism on modern hardware.

  • Work-stealing parallelism: A shared scheduler balances indexing, merging, and search across cores. Idle workers steal tasks from busy workers.
  • Pipelined indexing: A pipeline of independent stages, so ingest never waits behind a merge and a commit does not stop the world.
  • Automatic parallel indexing: Send one NDJSON stream (or one gRPC stream); Luxir spreads the indexing work across available cores.
  • Multicore merges: Even a single merge runs across multiple cores, with concurrency governed by the indexing RAM budget.
  • Zero-copy reads: The on-disk format is the in-memory format. Queries consume postings, columns, and vector indexes directly from memory-mapped files, with no deserialization or extra copy from the OS page cache.
  • Asynchronous IO: HTTP/JSON and gRPC handle many connections without requiring a thread per connection. Cheap queries execute inline with no scheduler handoff; requests can opt onto the shared worker pool for parallel execution or to keep long computations off the IO threads.
  • SIMD codecs for postings and numeric data (FastPFOR-based bit-packing, StreamVByte), roaring-style two-level bitsets for document sets, and BM25 hot loops written to auto-vectorize.
  • Block-max pruning: Skips whole blocks of documents that cannot enter the top k. Pareto frontiers provide score bounds that eliminate scoring work while preserving exact top-k results.
  • Sort pruning: Field-sorted top-k queries skip whole blocks of values that cannot be competitive. Per-block min/max zone maps enable pruning even without a points (BKD) index.
  • Range pruning: Numeric ranges and value sets skip irrelevant column blocks, accept fully covered blocks, and inspect only the values that need it: crossing blocks are split by their 512-value leaf zones, and linear-fit blocks (sorted ramps such as time-ordered timestamps) by the rank intervals their fit bounds, leaving a narrow band at each range edge to decode. The same zone maps enable this without a points (BKD) index.
  • Adaptive facet execution: Dynamically selects counting strategies and counter representations to suit the matching documents and field cardinality, balancing speed and memory use.
  • Cost-aware filter cache: Ranks filters by recent charged and credited work per effective residency byte.
  • Arena allocation: Indexing runs on rollback-capable memory pools, and requests decode into arena-backed message objects with no per-field heap allocation on the way in.
  • Field templates: built-in suffix rules (title_t, year_i, tags_ss, …) let you start without defining your own schema. Templates also handle fields you cannot enumerate ahead of time, such as new product attributes in an ecommerce catalog. Define the rule once; new matching fields work automatically. Templates can be customized and combined with explicit field definitions, which take precedence.
  • Field types: analyzed text, string, int, float, double, date (ISO-8601 in, epoch-millis storage), id, vector, geo point.
  • Explicit schema API over HTTP/JSON and gRPC. GET returns the authored schema, and the output is itself a valid write body; POST sets the named definitions (mode=set) or replaces the whole schema (mode=replace_all; the destructive form has to be spelled out rather than implied by an HTTP verb).
  • Fields and templates can inherit settings with parent. The built-in _name and _names templates support people and titles: word search plus a whole-value variant for facets and sorting.
  • Nested documents: lists of structured children (items, variants, chunks) kept as their own rows, to any depth. items:(color:red AND size:L) requires one item with both, a document scores by its best child, and facets and other operations can run over the child rows of the matching documents.
  • Field variants: one input value indexed under several representations (author for word search, author__s for exact facets and sorts), with per-operation default bindings so a bare field name does the right thing.
  • Per-field choices: index mode (match, or range acceleration), multi-valued modes, column-stored (sorting, faceting, analytics), stored source (retrieval, in LZ4- or ZSTD-compressed chunks, configurable per stored resource).
  • Numeric, date, and geo fields with index: "range" build a points index; without it the same queries run off the column with the same results.
  • Floats and doubles are stored order-preserving, so numeric sorting and ranges over columns need no decode step.
  • Ordered character filters before tokenization: html_strip for HTML text extraction, mapping for character replacements/removal, and nfkc_cf for Unicode NFKC case folding.
  • Tokenizers: Unicode word segmentation (UAX#29), whitespace, keyword.
  • Token filters: NFKC case folding, Unicode lowercase, accent and diacritic folding, stop-word removal, English possessive removal, and KStem English stemming (the dictionary-based Krovetz stemmer).
  • The default _t field applies Unicode words, case folding, accent folding, possessive removal, and KStem. _un is Unicode text without stemming, _u keeps case and accents, and _w/_wl split on whitespace.
  • Analyzer components carry typed parameters ({"name": "kstem", "params": {"possessive": false}}); string fields take a normalizer (filters only, applied to the whole value).
  • Query-time analysis matches index-time analysis, so match finds what was indexed.
  • Streaming ingest on both surfaces: HTTP NDJSON with no stream-size limit, and a gRPC bidirectional update stream. Individual records and explicit atomic groups are bounded; the stream is not.
  • Stream grammar: _update_ opens a group with options (allow_dups, all_or_none, return_ids, a different collection), _end_ closes it and can commit. One connection can feed several collections.
  • Field mapping at ingest (field_map, drop_unmapped): index a foreign dump as-is, renaming or dropping keys per request, with no file editing and no schema change.
  • Update and overwrite by id, delete by id, a duplicate-allowed mode, and optional per-request atomicity (all-or-none with rollback).
  • Per-document failures are reported by id and index with the same error object used everywhere else; the other documents in the request are still indexed.
  • Collections auto-create on first write (can be disabled).
  • Commits publish a new view: immediate, commit_within_ms, ?commit=true on the URL, or at the end of a stream. Indexing memory is budgeted and flushes automatically; merges run in the background and never stall ingest or search.

Every query type is a node in one tree and composes freely under boolean clauses, as non-scoring filters, as facet domains, and as fusion sources. Anywhere a query goes, it can be a structured object or an expression string.

  • match (analyzed, AND/OR operator, minimum-match; numeric and date equality through columns), phrase (positional, with slop), boolean (required, optional, prohibited, and filter clauses with minimum-match), any_of (exact value-set membership), exists, and match-all.
  • range over numeric, date, string, ID, and text fields, either side open. Date bounds accept ISO-8601, epoch millis, partial dates that mean the window they name (2024-06 is all of June), and date math (NOW/DAY-30DAYS, 2024-06-25||+2d/d) with a request-stable NOW, evaluated in a request time zone (IANA name or fixed offset).
  • prefix, wildcard (* and ?), regex (anchored whole-term), and fuzzy (edit distance up to 2, with blended scoring so a rare misspelling never outranks the exact term; fuzzy clauses take part in block-max pruning like any other clause). Prefix, wildcard, regex, and TEXT/STRING/ID match take an opt-in case_insensitive flag. Match scores case variants as one word, with the exact case of the query preferred slightly, weighted term frequency summed before BM25 saturation, and shared statistics. The others keep constant scores. Expansion follows query-time analysis, so case-preserving stemmers can be lopsided (KStem stems lowercase words only). For frequent case-blind TEXT matching, prefer a folding variant (_t, _un, _wl); it is faster.
  • constant_score, boost, and rescore: keep a query’s matches and replace each score with a value expression over columns and the original score (score * def(popularity_i, 1)). Expressions cover arithmetic, exp, pow, comparisons, ?: conditionals, seeded random, geo_distance, and Elasticsearch-style gauss_decay, exp_decay, and linear_decay decay functions over numeric, date, and geo values. The same language has bitwise operators and %, trigonometry and rounding, calendar parts of dates, coordinates and distances between places, geohash and geotile keys, hash functions for consistent sampling, let bindings, $name request variables, _score as an alias of score, and typed field components such as loc.lat and ts.year (value expressions). rescore takes an optional min_score cut-off (the function_score / script_score counterpart).
  • Request min_score (Elasticsearch-style): documents scoring below it are dropped from the returned list, the exact count, and facet and metric domains alike.
  • knn vector search, geo_box, geo_distance, and geo_polygon.
  • multi_match: search one text across weighted fields. combined (default) scores TEXT fields sharing an analyzer and long-term policy as one field with BM25F; weights scale term frequency and length at every group size. STRING/ID/numeric/date fields form singleton groups with score multipliers. per_field multiplies each field’s match score, with optional phrase matching on TEXT fields. tie combines field/group scores (default 1, sum); 0 keeps the best. Fields whose text cannot coerce are skipped. * patterns select indexed TEXT/STRING/ID targets, limited to 64 physical targets after resolving bindings and deduplicating. See Multi-match.
  • dis_max: the best matching query score plus tie times the other matching scores. Default tie: 0 keeps the best; tie: 1 adds scores like a boolean disjunction. Duplicate queries contribute separately.
  • simple_query: a syntax for end-user input that never fails to parse (+/-, |, quoted phrases with ~N slop, grouping, trailing-* prefix, ~N fuzzy terms, and field:value terms against an allowlist).
  • expr, the query language for developers: a bare string anywhere the JSON API takes a query. Fielded terms and phrases, AND/OR/NOT with real precedence, +/-, ranges (year_i:[1960 TO 1970}) and comparisons (year_i:>=1960), exact values (category_s:=(classic, fiction)), field groups (title_t:(a OR b)), per-clause boosts, and function forms for most structured query types (fuzzy(smith, field=name_s, max_edits=2); vector and geo queries stay structured). Special characters act only where they mean something, so most values need no escaping. Strict grammar with byte-offset errors; $vars substitute values without re-parsing them, so user input cannot inject operators.
  • Both forms build the same query tree, so ?explain=request returns the structured form of any request, expression strings included.
  • Non-scoring filters on top-docs and fusion sources, routable past named sub-operations (except_ops). Expression strings are convenient as filters: "filter": ["status_s:active AND year_i:>=1960"].
  • BM25 relevance with block-max pruning (per-block score bounds plus MaxScore skipping) whenever the request does not consume an exact count; exhaustive counting when it does.
  • Sorting by column values, query score, value expressions (popularity_i + score * $weight), and reader order; ascending or descending; lexicographic over several keys; offset and limit paging.
  • Field projection: named fields, * wildcard patterns, or every retrievable field when fields is omitted. Row- or column-oriented documents per request (document_format): HTTP defaults to rows (a missing field is an absent key), gRPC to dense columns; adaptive picks per batch and field whichever encodes more compactly.
  • Any number of named operations in one request over the same index view: ranked lists, facets, metrics, and fusion, nested to any depth and executed in parallel. HTTP responses preserve their names and nesting under ops. Root query shorthand returns found and docs directly in the HTTP envelope.
  • Count-only and analytics-only requests: limit: 0 with count_limit: -1 loads no document fields.
  • Stream every match: limit: -1 with ?format=docs returns one document per line over one connection, with no cursor, scroll state, or page size. The output pipes straight back into NDJSON ingest.
  • Hybrid fusion: weighted reciprocal rank fusion, or weighted linear fusion of min-max normalized scores, over named sources with shared and per-source filters, limit and offset over the fused list, and facets and metrics over the fused candidate set.
  • A warnings channel: declared degradations (a clamped fuzzy distance, a skipped calendar bucket) carry a code and a message in the response instead of silently changing the query.
  • Opt-in execution profiles (profile: true) with per-segment strategy, inputs, and timing for instrumented operations (string facets today), and max_parallel to force serial execution when isolating scheduler effects.
  • Field facets over string, ID, int, date, and text fields; range facets over int, float, double, and date fields with fixed or calendar gaps; query facets with arbitrary named query buckets.
  • Date histograms: calendar gaps (day, week, month, quarter, year) stepped in a request or per-facet time zone, date-math bounds, every bucket returned in order with zero counts included, daylight-saving changes handled.
  • Nesting: sub-facets under buckets, metrics per bucket, and a ranked top_docs (or fusion) list per bucket, all in one request.
  • Expression metrics: avg, sum, min, max, distinct counts (cardinality), percentile, and exact percentile_rank over value expressions (sum(price_f * qty_i) / sum(qty_i)), at query level or per bucket; facets can sort by a named metric.
  • Easy multi-select navigation: put choices in each facet’s selected array. Luxir builds the filters and automatically counts alternatives with the other facets’ selections applied. domain overrides and except_ops routing allow custom filtering behavior.
  • Facet controls: limit, mincount, and a missing bucket. Counts are exact by default.
  • Dense float32 vector fields, single- or multi-valued; the column store is the source of truth, so there is no separate vector store to keep in sync and deletes, updates, and merges apply to vectors as they do to text. Metrics: L2, inner product, cosine (normalize-on-write by default).
  • Exact kNN (full-precision column scan) and approximate search through per-segment IVF indexes (PQ codes by default, or opt-in RaBitQ, configured per field), mixed per segment by a size gate. Approximate candidates are rescored at full precision straight from the column, so the recall recovery costs no extra storage.
  • Filtered kNN applies filters inside the vector search and adaptively deepens ANN breadth instead of post-filtering. Each segment chooses between its index and an exact scan of the filtered vectors by modeled cost, and switches to exact mid-query when the index turns out more expensive. exact: true gives the exact filtered top-k, which also serves as the ground truth for measuring recall.
  • Multi-valued vector fields collapse to one hit per document at its best similarity, with adaptive over-fetch so k documents still come back.
  • ANN indexes are segment overlays: built at commit from the vector column and carried through merges automatically, without reindexing documents.
  • Effort knobs: k, refine_candidates, exact, and per-engine knobs under ivf (nprobe, min_scan_fraction).
  • similarity threshold in the metric’s own space (maximum L2 distance, minimum dot product or cosine), with Elasticsearch knn.similarity semantics: below-threshold neighbors leave results, counts, and facets.
  • Deterministic: parallel vector search returns bit-identical results to a serial run.
  • Geo point fields, quantized to about a centimeter, single- or multi-valued, with an optional two-dimensional points index. Points are [lon, lat] arrays (GeoJSON order), {"lat", "lon"} objects, GeoJSON Point objects, "lat,lon" strings, WKT POINT strings, or geohashes. _p and _ps suffix templates; per-field ignore_malformed and ignore_z_value.
  • Bounding-box queries (dateline-aware), great-circle distance queries, and polygon queries over a union of polygons with holes, usable as the main query, as a non-scoring filter, or as a boolean clause.
  • Requested geo fields come back in their ingest shape, [lon, lat], or [[lon, lat], ...] for a multi-valued field, so results index back unchanged.
  • HTTP/JSON: search (GET with URL parameters for a bookmarkable search, or POST), update (JSON and NDJSON), schema, collection create, delete, and list, stats, and health. ?pretty for humans; every body ends with a newline.
  • gRPC: streaming search, unary and streaming update, schema and collection admin, stats, the standard health service, and known-symbol reflection. Search results are natively columnar.
  • One vocabulary: both surfaces share the protobuf message model, so a shape learned over HTTP is the message a generated client sends.
  • Strict validation everywhere: unknown keys are errors, nesting depth and sizes are bounded, and the public port is treated as hostile input.
  • One error shape everywhere: {kind, code, message}, in-band and in HTTP error bodies and gRPC status details alike. kind fixes the transport status; code is the stable key.
  • Request and response correlation ids, and a per-request freshness bound (freshness_ms) on reads.
  • Storage: filesystem (memory-mapped reads) or in-memory. Collections are independent indexes below one data directory.
  • Crash-safe commits: after a crash, Luxir reopens the last durable commit.
  • One writer per data directory, enforced by a lock; --read-only nodes serve a directory that another process is writing.
  • Background merging that parallelizes inside a single merge, admitted against a node-wide indexing-RAM budget. --max-ram-mb sizes the whole node from one number (25% of system or cgroup RAM by default).
  • Operational statistics per node and per collection (/_stats, scraped as /metrics), as JSON or Prometheus text from one response: request, search, ingest and background rates and latencies, memory budgets and allocator state, and opt-in collection state (documents, segments, bytes, merges, inverters, cache counters, per-collection search and ingest activity) and process, host and cgroup statistics.
  • No built-in authentication or TLS yet: run the server inside your own network boundary.

Luxir is pre-1.0 and moving fast; interfaces can change without back-compat. It is a single-node engine with no replication, distributed query execution, authentication, or TLS. Benchmark results are not yet published. This page lists shipped capabilities; the operations guide states the deployment boundary and the current limitations.