Skip to content
Development documentation for unreleased code on main. Behavior can change before a release. Latest release docs

Schemadev

The schema defines field types, text analysis, storage, and indexes. You can define fields individually, use field templates for names that share a suffix, or combine both. Built-in templates let you start without writing your own schema. Templates also handle new fields as your data grows.

Choose a type based on how you want to search and use the values:

Type Description and typical uses
nested Structural object path; see Nested documents. Accepts an object, an array of objects, null or an empty array.
text Analyzed text for full-text search: titles, descriptions, and article bodies. An analyzer splits each value into searchable tokens for word and phrase queries.
string A whole value for exact matching, filtering, sorting, and faceting: categories, tags, and product codes. Values are not split into words.
int Signed 64-bit integer: counts, quantities, and years.
float 32-bit floating-point number: measurements and other numeric values where single precision is sufficient.
double 64-bit floating-point number for greater precision than float.
decimal Fixed-point number with a required scale (0 to 18 digits after the point). See Decimals.
ip IPv4 or IPv6 address, stored as a 16-byte term. See IP addresses.
boolean true or false: flags and on/off attributes.
date Timestamp with ms (default) or ns storage precision. Ingest accepts ISO-8601 text or numeric epoch values in the field’s unit. See Dates.
vector Dense vector of floating-point values, such as an embedding. Set a similarity metric to enable nearest-neighbor search. See Vector search.
geo_point Geographic location supplied as [longitude, latitude], an object, a string, or a geohash, for distance and bounding-box queries. See Geo search.
long_range, double_range, date_range An interval of int, double, or date endpoints, such as a validity window, a price band, or an event’s duration. See Range types.
id Unique document identifier, reserved for the built-in id field. Used to identify documents for replacement and deletion.

decimal stores value * 10^scale as a signed 64-bit integer. Set the required scale from 0 to 18; it is inherited from parent like other properties. A direct scale is invalid on other types; an inherited scale is ignored when a child changes type. There is no default decimal suffix. For example:

{"fields":{"price":{"type":"decimal","scale":2,"index":"range"}}}

JSON integers, floating-point numbers and numeric strings are accepted. Excess fractional digits round half away from zero: 1.005 and "1.005" become 1.01 at scale 2, and -1.005 becomes -1.01. JSON floating-point inputs use their shortest round-trip text, rather than their binary expansion. Exponents and long fractions are accepted; overflow after rounding rejects an ingest value. Out-of-domain exact-match literals match nothing, and range bounds saturate to the field’s value domain. Scale is persisted with the field type; reading or merging data under a different type or scale is rejected.

Match and any_of literals round like ingest. Range bounds round in the requested direction: gte: 10.005 starts at 10.01, while lt: 10.005 includes values through 10.00. Integer field bounds also round directionally (gte: 9.5 starts at 10); integer ingest and exact-match literals still reject non-integral values.

Results are doubles by default, computed as raw divided by 10^scale. This applies to documents, facet buckets and range bounds, sort values, collapse keys, and decimal statistics. The int64 storage is exact; doubles can lose distinctions at large magnitudes.

The request or operation flag decimal_scaled instead returns raw integers and interprets decimal field literals as raw integers. Integral numeric strings, including "1250.0", mean raw 1250 at every scale; non-integral match literals are rejected. Bounds still round directionally in raw units. Ingest always uses decimal values and is unaffected by the flag. Combine decimal_scaled with int64_str to round-trip large raw integers through JSON strings. Averages and percentiles remain doubles (in raw units under the flag).

Range facet gaps and origins must be exact multiples of 10^-scale; gaps must be positive. With decimal_scaled they use raw units. Selection, min, max, sum, and avg retain the scale when possible, aggregating raw values exactly (INT128 sums). Arithmetic and mixed-scale expressions decode to numeric values. Expression constants always mean values, independent of decimal_scaled. Same-scale comparisons use raw values; per-document comparisons against numeric constants use raw integer bounds when representable. A decimal sum wider than int64 renders as a double by default; scaled output emits null with an aggregate overflow warning, as integer sums do.

ip accepts IPv4 and IPv6 text. IPv4 uses the IPv6 v4-mapped encoding, so 10.0.0.1 and ::ffff:10.0.0.1 are equal. Results use dotted IPv4 for mapped addresses and compressed lowercase IPv6 otherwise. Sorting follows address byte order. The default suffix templates are _ip and multi-valued _ips.

IP fields require index: "match" (the default), and reject stored: true. There is no original source text to retain. Columns support retrieval, field facets, sorting, collapse, and column-only value expressions such as count. Facet selection and collapse identity use the encoded address.

match accepts an address or CIDR: 10.1.2.3/8 matches the entire 10.0.0.0/8 network because host bits are ignored. IPv4 /0 covers all mapped IPv4 addresses; IPv6 ::/0 covers all addresses. /32 for IPv4 and /128 for IPv6 match one address. Range bounds and any_of members must be addresses; CIDR is rejected there. Numbers, zone ids, and text containing NUL reject. Prefix, wildcard/regexp, case-insensitive match, fuzzy matching, and facet term_filter are unsupported. IP fields cannot be bare-word fields in simple_query, but addr:10.0.0.1 is an exact match. Wildcard multi_match field expansion probes each IP field and skips text it cannot parse.

boolean accepts JSON true/false and the strings "true"/"false"; other values, including numbers 0 and 1, reject the document. Query literals use the same rules: flag:true, {"match": {"flag": false}}.

Values are stored as 0/1 in the integer column. Sorting puts false before true; field facet bucket values and collapse keys are JSON booleans. Range queries accept boolean bounds, but range facets are not supported. The default is index: "none", as for int; index: "range" and index: "match" add the corresponding numeric indexes. For filtering on a boolean, range is the faster of the two, and the _b/_bs templates use it.

Retrieved values are JSON booleans. Document columns have no boolean kind, so gRPC returns a col_val column holding one boolean value (or array) per document. Value expressions preserve booleans through selection (min, max, and choices between booleans); arithmetic, sum, and avg produce numbers. Summing a boolean counts true values.

A range value is an object with an optional lower bound (gte or gt) and an optional upper bound (lte or lt); a missing side is an open end:

{"id": "promo-7", "valid": {"gte": "2024-06-01", "lt": "2024-07-01"}, "price": {"gte": 10.5, "lte": 20}}

Ingest is strict about the endpoint type: long_range rejects fractional endpoints, and long or date endpoints outside the supported scalar domain fail the document. An exclusive bound moves one value inward ({"gt": 5} on long_range starts at 6). A lower bound above the upper bound, including an empty interval after exclusive bounds, fails the document.

Luxir treats a partial date as its whole granule at ingest: {"gte": "2024-06"} starts at the first millisecond of June and {"lte": "2024-06"} ends at its last millisecond. gt excludes the whole named granule, as does lt. Elasticsearch and OpenSearch differ: when they ingest a range field, a partial-date lte such as "2024-06" does not extend to the end of the month. Dates without an explicit offset are evaluated in UTC at ingest. date_range uses millisecond precision; ISO strings, epoch milliseconds, now, and date math follow the scalar date rules. Query bounds follow the scalar range-query coercion rules.

For double_range, an open lower end is -Infinity and an open upper end is Infinity; explicit infinities at those ends have the same meaning. NaN is rejected at ingest and in queries. gt: "Infinity" or lt: "-Infinity" makes an empty interval. Negative and positive zero remain distinct, with -0.0 below +0.0. Long and date open ends extend to the int64 limits.

Search results return {"gte": lower, "lte": upper}, omitting open ends. Date endpoints follow the request’s date output mode: epoch milliseconds by default, ISO strings with date_iso, milliseconds with date_ms, or nanoseconds with date_ns. This applies to both ROWS and COLUMNS responses.

Range fields require column (the default) and support index: "none" (the default) or index: "range" for an optional query index. They do not support stored or index: "match". Multi-valued range fields are not supported yet.

The range query supports intersects, contains, and within. match finds intervals containing a point value; a partial date matches intervals overlapping its granule. multi_match accepts explicitly named range fields and field weights; wildcard field selectors do not select range fields. Sort, facets, collapse, value expressions, and any_of are not supported on range fields.

Both accept text values, but they make different things searchable. Suppose you index "Red Bicycle" using the built-in templates:

  • title_t (text) indexes the words red and bicycle. title_t:bicycle matches, as does title_t:"red bicycle".
  • title_s (string) indexes the whole value Red Bicycle. title_s:"Red Bicycle" matches; title_s:bicycle does not.

Sorting text values requires a whole-value column, which string provides. Faceting can use analyzed terms from a text field, but whole values are often more appropriate, such as complete author names or categories.

The analyzer controls how text is split and normalized. Bare "type": "text" defaults to splitting on whitespace with no filters; the _t template adds Unicode word segmentation, case and accent folding, and English stemming. See Text analysis for the available components.

string keeps values verbatim by default. You can configure a normalizer to transform them, for example by folding case or accents, without splitting them into words. When one value needs both full-text search and whole-value sorting or faceting, use a text field with a string variant. The built-in _name template provides this combination for names.

The schema API uses the same JSON in both directions: what GET returns is a valid POST body, and posting it back keeps the same definitions.

GET /collections/main/_schema
{
"fields": {
"id": {
"type": "id"
},
"_version_": {
"type": "int"
}
},
"templates": {
"_i": {
"type": "int"
},
"_s": {
"type": "string"
},
"_t": {
"type": "text",
"analyzer": {
"tokenizer": "unicode_word",
"filters": ["nfkc_cf", "fold", "kstem"]
}
},
"_name": {
"type": "text",
"analyzer": {
"tokenizer": "unicode_word",
"filters": ["nfkc_cf", "fold"]
},
"variants": {
"s": {
"parent": "_s"
}
},
"defaults": {
"value": "s"
}
}
}
}

(Abbreviated: the default schema defines a template for every suffix.)

Two sections:

  • fields define individual fields. A document field named title uses fields.title. Explicit definitions take precedence over templates.
  • templates define shared settings for fields whose names end in a matching suffix. A field named anything_t with no explicit definition uses the _t template. Matching uses the suffix from the final underscore. Templates are never usable as document fields themselves, and any definition can name one as parent to inherit its properties.

The built-in templates are:

Suffix Type and default behavior
_nested Nested object path; structural, with no scalar value.
_s Indexed exact string with a column.
_sc Column-only exact string.
_ss, _ssc Multi-valued forms of _s and _sc.
_i, _is Integer column, single- or multi-valued.
_b, _bs Boolean column with a points index (index: "range"), single- or multi-valued.
_ip, _ips IP address terms, single- or multi-valued.
_f, _fs Float column, single- or multi-valued.
_d, _ds Double column, single- or multi-valued.
_dt, _dts Date column with a points index (index: "range"), single- or multi-valued.
_p, _ps Geo point column with a points index (index: "range"), single- or multi-valued.
_t Unicode-word text with NFKC case folding, accent folding, English possessive removal, and KStem English stemming. The general-purpose text field.
_u Unicode-word text, case- and accent-sensitive.
_un Unicode-word text with NFKC case folding; accents preserved, no stemming.
_w Text split on whitespace, case- and accent-sensitive.
_wl Text split on whitespace, Unicode-lowercased; no normalization or accent folding.
_name Unicode-word text with case and accent folding and no stemming, plus a whole-value string variant for facets and sorting.
_names Multi-valued form of _name.
_v, _vs Single- or multi-valued vector column; storage-only until a metric is set on a concrete field.

Text fields are stored in the default compressed stored_resource. Numeric suffixes are column-backed but do not build a points index by default; range and exact-match queries still work by quickly scanning the column. Define a concrete field with index: "range" when those operations need a points index, or index: "match" for a term dictionary that serves exact matches and lets field facets on low- and medium-cardinality values count without scanning the column (see Faceting). The geo suffixes _p and _ps build one by default, because geo queries are their only use.

Templates are useful when you do not know every field name ahead of time. An ecommerce catalog can gain new product classes, each with its own attributes. Define their shared settings once, and new matching fields work automatically.

This schema combines an explicit productname field with a custom _attr template for string attributes. The built-in templates remain available:

POST /collections/catalog/_schema
{
"fields": {"productname": {"parent": "_t"}},
"templates": {"_attr": "string"}
}

Index a product with a shoe_width_attr field:

POST /collections/catalog/_update
{"docs": [{"id": "shoe1", "productname": "Trail shoes", "shoe_width_attr": "wide"}], "commit": {}}

Later, a new product class introduces socket_attr. No schema update is needed:

POST /collections/catalog/_update
{"docs": [{"id": "cpu1", "productname": "Desktop processor", "socket_attr": "AM5"}], "commit": {}}

Both attribute fields use _attr’s string settings, so you can filter, sort, and facet on them immediately after the commit:

POST /collections/catalog/_search
{"query": "socket_attr:AM5", "fields": ["id", "productname", "socket_attr"]}
{
"docs": [
{
"id": "cpu1",
"productname": "Desktop processor",
"socket_attr": "AM5"
}
]
}

An explicit definition overrides a template for that field. You can also define every field individually if the field names are known in advance.

Use POST to define fields. The default, mode=set, sets each named definition and leaves everything else alone:

POST /collections/main/_schema
{
"fields": {
"title": {
"type": "text",
"stored": true,
"analyzer": {
"tokenizer": "unicode_word",
"filters": ["nfkc_cf", "fold"]
}
},
"year": {
"type": "int",
"index": "range"
},
"vec": {
"type": "vector",
"dims": 768,
"metric": "cosine"
}
}
}

The response is the full resulting schema, the same shape GET returns.

When a definition only needs a type, a bare string works:

POST /collections/main/_schema
{"fields": {"year": "int", "author": "string"}}

Setting an existing field replaces its entire definition.

Each tokenizer or filter is {"name": ..., "params": {...}}. Components without parameters can use a bare name, as in the title example above. Parameters are typed JSON values (a string, number, bool, or list).

mode=replace_all replaces the whole schema:

POST /collections/main/_schema?mode=replace_all
{"fields": {"title": "text", "year": "int"}}

The reserved id and _version_ fields are added automatically if omitted. _version_ is a fixed, retrieval-only reserved field with no index, column or stored capability. It returns the update version of the request that indexed the document. Its capabilities cannot be configured. Setting it in document input is rejected. Optimistic concurrency is not supported. The resolved schema view reports null search and value bindings for this field. Omitting templates removes them, leaving a schema with only explicit fields. To edit the existing schema, start from GET output. Each successful publication advances the schema generation.

Key Meaning
type The field type: text, string, int, float, double, decimal, boolean, ip, date, vector, geo_point, long_range, double_range, date_range, id
scale decimal only, required from 0 to 18; inherited from parent.
index match, range, or none; absent = the type’s default (text/string index for match, numerics don’t)
column store values in a per-field column (sorting, faceting, analytics); supported and default-on for non-text types
multi false, true, "set", or "array"; see Multi-valued modes
stored keep canonical source text for TEXT/STRING/ID retrieval, before analysis/normalization; default on for text only; ignored for numerics
stored_resource stored-field group for TEXT/STRING/ID; empty or absent inherits, falling back to _stored_; see Stored resources
analyzer text only: {"char_filters": [...], "tokenizer": <component>, "filters": [...]}, or the name of a registered analyzer such as "english"; see Text analysis
normalizer string only: a list of filter components applied to each whole value, with no tokenizer; see STRING normalization and length
long_terms string, text (per token), and id: hash128 (default), truncate, or reject for terms over 255 bytes after normalization/analysis; see Long terms
unit date only: ms (default) or ns. Inherits from the parent and is part of the field signature. An existing date definition cannot change its resolved unit.
variants Map from label to another field definition receiving the same input value; bare type strings work here too; see Field variants
defaults For fields with variants, choose self or a variant label to override the inferred search and value bindings; see Default bindings for variants
parent inherit any unset properties from a field or template
dims, truncate_dims, metric, normalized, normalize_on_write vector only; metric: l2, ip, cosine, none; see Vector search
ignore_malformed, ignore_z_value geo_point only; skip malformed points instead of failing the document, and accept and drop a third (altitude) coordinate. ignore_malformed defaults to false; ignore_z_value defaults to true. See Malformed points and altitude.
reverse_map nested only; defaults to true. Store each child row’s owner in segments written from now on; see Declaring nested paths.
ann vector only; approximate index codec and parameters. RaBitQ is the default; PQ requires explicit codec:"pq". See Index settings.

Every property is optional. Absent means “inherit from parent, else the type’s default”, and the schema you read back stays as sparse as the one you wrote.

An empty analyzer: {} inherits the parent’s analyzer. Naming a tokenizer, listing any token filter, or naming a registered analyzer replaces the whole chain, including character filters (token filters alone get the whitespace tokenizer). Setting only char_filters keeps the parent’s tokenizer and token filters, replacing its character filters. Without a parent analyzer, the base is whitespace with no token filters.

column and stored solve different problems. A column is a typed, per-field structure used by sorting, faceting, analytics, numeric/geo queries, and vector search. Stored fields keep canonical source text for retrieval in compressed chunks. Analyzed text has postings and stored retrieval but no per-document value column; use a string variant when the same source value must also sort or facet. Scalar types return their values from columns without a second stored copy.

For PQ, positive ann.pq.m requires positive effective (inherited) dims and must divide the stored dimension: truncate_dims when set, else dims. Omitted or zero m selects automatically and permits inferred dimensions. Validation includes inherited ANN settings and variants.

multi is inherited when omitted and defaults to false (one value). true selects the type’s default: set for INT, FLOAT, DOUBLE, DECIMAL, BOOLEAN, IP, DATE, and STRING; array for TEXT, VECTOR, and GEO_POINT. Explicit "set" and "array" select a mode. Array-only primary types reject "set"; ID is reserved for the single-valued id field.

  • Set: each document’s column values are sorted and deduplicated after coercion and normalization. Numbers and dates use ascending numeric order; strings use byte order after normalization and any long-term fitting, whether indexed or column-only.
  • Array: columns preserve source order and repeats after coercion and normalization. Stored source, when enabled, keeps source order and repeats independently of the column mode. Numeric fields have no stored-source copy.

Bare multi-valued numeric, date, and STRING fields sort by their minimum ascending and maximum descending in both modes. Per-document numeric reducers such as min(f), max(f), count(f), sum(f), and avg(f) consume the stored values; count(f) also counts STRING, ID and GEO_POINT values, so a set counts a repeated value once and an array counts each repeat. Default facets count each document once per bucket in either mode, even when several values fall in that bucket. Indexed STRING postings also contain each document once per term in both modes.

The declaration resolves against each representation’s own type. With true, a TEXT primary uses an array and its STRING variant uses a set. With "set", set-capable representations use sets and array-only variants use arrays. Explicit "array" applies to every representation.

The authored schema view echoes true, false, "set", or "array", preserving inheritance through type-changing templates. The resolved view reports the effective mode: true for the type’s default, "array" for an array on a set-capable type, and false for a single-valued representation. Changing the effective mode starts a new introduction generation and affects newly indexed data; reindex original values to convert existing documents.

Stored values are grouped by stored_resource, and each group is compressed in chunks of whole documents. The top-level stored_resources map declares the groups; the default _stored_ always exists, and without an entry it uses LZ4 with 16KB chunks of at most 128 documents. A field may name only _stored_ or a declared group, and a group name starts and ends with _ (for example _stored_logs_), so it never collides with a field or template name.

POST /collections/logs/_schema
{
"fields": {"message": {"type": "text", "stored_resource": "_stored_logs_"}},
"stored_resources": {
"_stored_logs_": {"codec": "zstd"}
}
}
Key Meaning
codec lz4 (default): fast to write and read. zstd: much smaller for repetitive text such as logs, at more CPU per chunk
zstd_level zstd only: 1 (fastest) to 22 (smallest); default 3
chunk_size_bytes target uncompressed bytes per chunk, 1024 to 4194304; default 16384, or 65536 with zstd
max_docs_per_chunk most documents per chunk, 1 to 65536; default 128, or 1024 with zstd
zstd_dictionary zstd only: enable one dictionary per resource, shared by its segments; trained by a merge
zstd_dictionary_bytes dictionary capacity upper bound, 1024 to 262144; default 65536. Requires zstd_dictionary

Larger chunks compress better, but returning one stored document decompresses its whole chunk. Chunks are compressed as documents are indexed, so the codec and level also set indexing cost; levels above the default compress much more slowly for little further gain. Each segment records its codec and chunk limits, so a change applies to segments written afterwards. A merge copies compressed chunks from inputs that already match its output codec, chunk limits and dictionary and recompresses the others; the level alone does not trigger recompression. Inputs where more than 1% of chunks were sealed before filling (each small flush ends in one) are also recompressed, so documents indexed a few at a time still end up in full-size chunks. mode=set replaces the named entries and keeps the others.

A dictionary holds text shared by documents of a resource, such as common values and log templates, so chunks can refer to it. That matters most for small chunks, which compress poorly on their own. For retrieval-heavy workloads (many single-document fetches), prefer small chunks plus a dictionary over large chunks:

"stored_resources": {
"_stored_": {"codec": "zstd", "zstd_dictionary": true,
"chunk_size_bytes": 4096, "max_docs_per_chunk": 32}
}

The first merge with at least 100 times the configured capacity (at most 8 MiB) of live stored data trains the resource’s dictionary. Earlier segments have none; later segments share the dictionary and each stores its bytes once. Capacity is an upper bound: training can produce a smaller dictionary. The benefit is greatest with small chunks; with large chunks, decoding can be slower.

There is no automatic retraining. Changing zstd_dictionary, zstd_dictionary_bytes or the codec clears the current dictionary so a later merge can train another. After a restart, the dictionary from the largest live segment carrying one is used.

A text analyzer applies an ordered list of char_filters to the whole value, then one tokenizer, then an ordered list of token filters. Both lists are optional.

{
"fields": {
"body": {
"type": "text",
"analyzer": {
"char_filters": [
"html_strip",
"nfkc_cf",
{"name": "mapping", "params": {"mappings": ["- =>", "\u2014 =>"]}}
],
"tokenizer": "unicode_word",
"filters": ["fold", "kstem"]
}
}
}
}

Character filters run in the written order, separately for each field value. For example, HTML stripping followed by character removal can decode &#45; and then delete the resulting hyphen. Reversing those filters leaves that hyphen in the tokenizer’s input. Removing - from foo-bar produces one foobar token, instead of two words separated by punctuation.

Character filter Behavior
nfkc_cf Unicode NFKC case folding, including default-ignorable removal, before segmentation.
mapping Apply case-sensitive, leftmost-longest rules from params.mappings. Map to an empty replacement to delete text.
html_strip Extract text, remove tags/comments/declarations and script/style contents, and decode named and numeric HTML character references once.

All except mapping take no parameters. Lowercase, accent folding, stemming, and possessive removal are token filters only. nfkc_cf is also available as a char filter because compatibility normalization before segmentation can change word boundaries.

mapping takes a list such as ["ph => f", "\u2014 => "]. The longest source at each input position wins; rule order does not break ties, and duplicate decoded sources are rejected. Replacements are not rescanned by the same filter. Use multiple mapping filters to apply successive rewrites. An empty list is a no-op; empty sources are invalid.

Each rule has exactly one unescaped =>; ASCII whitespace at either edge of each side is trimmed. Use ordinary JSON escapes for characters, as in "\u2014 =>" to delete an em dash. Mapping-level escapes are needed only to preserve edge whitespace or include a literal =>: "x => \\u0020" maps x to a space, and "\\u003d> => arrow" maps => to arrow. Double the backslash in JSON for these mapping-level escapes. Mapping escapes accept \\, \n, \r, \t, \b, \f, and \uXXXX, including surrogate pairs. Invalid UTF-8, malformed escapes, unknown parameters, and non-string rules are rejected. For folded matching, put nfkc_cf before mapping and write folded sources in the rules.

html_strip removes inline tags without separating text: H<sub>2</sub>O becomes H2O. Its inline list follows Lucene, including img, input, select, and textarea; other tags, including br and unknown names, emit newlines. Comments/declarations disappear; script/style regions become separators. Quotes protect attribute values only immediately after = and optional whitespace, so title=O'Reilly works. Unterminated ordinary tags preserve <; unterminated comments and script/style regions consume the remainder. CDATA delimiters disappear but contents stay literal, including < and &. Named/numeric references decode once, including legacy semicolonless names; unknown references stay literal and invalid numeric scalars become U+FFFD. Decoded &lt;b&gt; stays text. It never executes scripts, fetches resources, or expands document-defined entities. This extracts indexing text; it is neither a browser rendering model nor an HTML sanitizer for safe display.

Cleanup runs for analyzed queries. Prefix/fuzzy terms and TEXT range bounds use normalizing character filters before token normalization; html_strip is excluded. Wildcard/regex literals retain per-codepoint normalization, while character classes remain exact. Multi-character mapping sources do not span adjacent pattern literals, so ph => f applies to a prefix term but not the ph in ph*. Literal normalization currently rejects expansions beyond 32 codepoints per input codepoint. HTML markup and entities are not decoded inside patterns. Already-analyzed phrase terms remain verbatim. Stored source values retain the submitted text.

An analyzer specifying only char_filters keeps the inherited tokenizer and token filters. Schema changes affect future indexing; reindex existing documents to apply new cleanup rules to them.

A tokenizer is followed by the configured token filters:

Component Meaning
unicode_word Tokenizer: Unicode word segmentation (UAX#29).
whitespace Tokenizer: split on whitespace. The default when an analyzer lists filters but no tokenizer.
keyword Tokenizer: the whole value as one token.
nfkc_cf Filter: Unicode NFKC_Casefold, including default-ignorable removal.
lowercase Filter: Unicode lowercasing only.
fold Filter: accent and diacritic folding.
english_possessive Filter: remove a trailing 's.
kstem Filter: KStem English stemming; optional possessive parameter.
stop Filter: remove tokens in the required non-empty params.words list.

Prefer nfkc_cf to lowercase on Unicode-segmented text: bare lowercasing leaves equal-looking strings unequal (composed versus decomposed forms, final sigma), while NFKC case folding makes them compare equal. It also removes invisible default-ignorable characters such as soft hyphens, ZWJ/ZWNJ, and variation selectors: co<U+00AD>operate matches cooperate. Tokens that become empty are dropped without a position gap. Normalization follows Unicode 15.1.0 and stream-safe NFC; runs exceeding 30 non-starters in NFKD receive U+034F (CGJ), so those adversarial inputs can differ from ICU.

stop takes authored words, for example {"name":"stop","params":{"words":["THE","Cafe\u0301"]}}. Stop words are analyzed like field text up to the stop filter, including character filters, tokenization, and preceding token filters. Each must produce exactly one token; analyzed duplicates collapse. A schema set rejects words producing zero or several tokens. Loading a stored schema logs a warning and drops those words, so schemas saved under an older analysis version still load. There is no ignore_case parameter: preceding filters determine how words and tokens meet. For example, stop after kstem analyzes ponies to pony. Later filters do not affect the stop set.

Removed tokens leave position holes before the next emitted token, preserving phrase distances. Trailing holes are preserved between indexed values, in addition to the fixed 100-position gap, and between phrase.words entries. Stop does not remove prefix/fuzzy/range terms or wildcard/regex literals during normalization, and cannot appear in STRING normalizers.

kstem implements Bob Krovetz’s dictionary-based English stemming algorithm: ponies becomes pony, while recognized dictionary words are preserved or replaced by their dictionary mapping. It runs at both index and query time for _t fields. In custom analyzers, put it after lowercase or nfkc_cf, and after fold if used. Tokens containing anything outside lowercase ASCII letters, or with lengths outside 3-49 letters, pass through unstemmed.

english_possessive removes one trailing apostrophe followed by s or S, matching Lucene’s EnglishPossessiveFilter. It accepts ASCII apostrophes, right single quotation marks (U+2019), and fullwidth apostrophes (U+FF07). Other apostrophes and a bare trailing apostrophe are unchanged: Winter's becomes Winter; don't and dogs' are left alone. The tokenizer may already discard a trailing apostrophe before filtering.

Bare "kstem" includes equivalent possessive removal by default, stripping the suffix before stemming within one filter stage. _t uses this default, so title_t:winter matches Winter's Tale. To request stemming alone, disable possessive removal explicitly:

{"name": "kstem", "params": {"possessive": false}}

The standalone filter is useful without stemming. When placing it before kstem, set possessive: false on KStem so removal happens only once. Possessive removal applies even when the remaining word is non-ASCII or outside KStem’s length range. Source spelling, offsets, and positions are preserved.

Prefix, wildcard, regex, and fuzzy query terms receive the field’s case and accent folding without possessive removal or stemming. Neither english_possessive nor kstem is available in STRING normalizers.

For Unicode text without English stemming, use _un, or configure unicode_word with ["nfkc_cf", "fold"] to retain accent folding as well.

An analyzer can be a registered name instead of a definition. A bare string is shorthand for the object form:

{"fields": {"body": {"type": "text", "analyzer": "english"}}}
{"fields": {"body": {"type": "text", "analyzer": {"name": "english"}}}}

A named analyzer takes no tokenizer or filters alongside the name. char_filters next to the name replace the analyzer’s own character filters, just as setting only char_filters does under an inherited analyzer. Built in:

Name Expands to
standard unicode_word + ["nfkc_cf"]
english unicode_word + ["nfkc_cf", "fold", "kstem"], the same chain as _t, with kstem’s default possessive=true

Plugins can register more (a language bundle, for example). The schema stores the name, and expands it through the registry each time the collection loads, so a changed registration applies at the next load, to queries and new documents alike. Reindex after such a change so existing segments match. The resolved view reports the current expansion in analyzer and the name in analyzer_preset. A schema naming an analyzer or component that this build does not have is rejected on write; a collection already holding such a schema fails to load with that message while the node keeps serving its other collections.

A document supplies one value per logical field, and the schema can index that value several ways. The primary keeps the field’s name; each variant indexes the same submitted value under <field>__<label>.

See Documents and values for the worked example: searching author names by word and faceting on their whole values with the built-in _name template. Its schema definition is shown below, followed by the rules for inheritance and customization.

Labels are ASCII: a letter first, then letters, digits, or single underscores. self is reserved case-insensitively, and labels differing only by case collide. Label lookup otherwise uses the authored case. __ is reserved in authored field names, template names, labels, and final document keys after field mapping. It is only used in requests to select a representation. author_name__self selects the primary; it does not create another physical field.

Physical names are limited to 127 bytes. With variants, a template’s maximum instance name length is 127 - 2 - longest label length. The __self selector is not a physical name and does not consume that budget.

multi, stored, and stored_resource belong to the logical field. A variant may not set them or declare nested variants or defaults. There is one stored source on the primary when storage is enabled. Neither id nor _version_ may have variants, including inherited ones, and a variant cannot have type id.

The primary and every variant must use types from the same group:

Group Allowed types
Scalar text, string, int, float, double, decimal, boolean, ip, date.
Vector vector; every branch must resolve to the same positive dims.
Geo geo_point.
Range long_range, double_range, date_range.

For a field with variants, default bindings choose which representation to use when a request names the field without a variant selector, such as author_name. Fields without variants use the field itself; there is nothing to configure.

With variants, defaults.search selects the representation for text search, and defaults.value selects it for operations such as sorting and faceting. Explicit bindings, including inherited ones, take precedence. An omitted binding is inferred from the resolved field types:

Primary type Search binding Value binding
text self Its sole string variant, or self if none exists.
string Its sole text variant, or self if none exists. self
Other types self self

A text field with one string variant therefore searches words but sorts and facets on whole values without a defaults block. A string field with one text variant searches words through that variant and uses the primary for whole-value operations.

If a text primary has multiple string variants, set defaults.value; if a string primary has multiple text variants, set defaults.search. Otherwise, the schema is rejected as ambiguous. Use "self" to keep an operation class on the primary, or a variant label to select a particular representation. The _name template explicitly sets defaults.value to s, preserving that choice even if another string variant is added.

Inference uses types after inheritance. Each operation still checks the chosen representation’s index and column capabilities; it does not look for an alternative if those checks fail.

Operation Bare field name uses
Match, phrase, simple query, expression terms and phrases, prefix, fuzzy, wildcard, regex defaults.search, including a match inside a filter.
any_of / :=, ranges, field and range facets, sort, column expressions, metrics defaults.value, then the operation’s normal type and capability checks.
Exists (f:*), kNN, geo The primary physical field.
Retrieval The primary’s source store, else its own typed column; never the value default.

An explicit selector bypasses both bindings: author_name__s selects that variant and author_name__self selects the primary. Exists on a bare name tests primary presence; an explicit selector tests that representation, which can differ for documents indexed before it was added.

A binding names self or a label in the field’s effective variant map. See Searching for query and projection examples, and use ?explain=resolved to see which physical field a request used.

Templates can carry variants and defaults. The default schema includes _name and its multi-valued form _names:

{
"templates": {
"_name": {
"type": "text",
"analyzer": {"tokenizer": "unicode_word", "filters": ["nfkc_cf", "fold"]},
"variants": {"s": {"parent": "_s"}},
"defaults": {"value": "s"}
},
"_names": {"parent": "_name", "multi": true}
}
}

author_name uses _name; author_name__s selects its original whole-name string variant. The root is resolved before the label, so the tail _s never picks the default string template independently.

_name uses word search with case and accent folding, without English stemming or possessive removal. Its string variant preserves the supplied spelling for facets and sorting, so normalize names before indexing if you need to. A bare type: text would use the whitespace analyzer.

variants and defaults each inherit atomically: absent inherits, present replaces the whole object, and {} clears it. There is no per-label merge. For example, add fields to the authors collection using the default _name template:

POST /collections/authors/_schema
{
"fields": {
"inherited": {
"parent": "_name"
},
"replaced": {
"parent": "_name",
"variants": {
"raw": "string"
},
"defaults": {}
},
"cleared": {
"parent": "_name",
"variants": {},
"defaults": {}
}
}
}

inherited keeps s and the value binding. replaced has only raw; clearing the inherited defaults lets its value binding infer raw. cleared clears both the variants and their inherited defaults, leaving only the primary. Within a present defaults, an omitted binding follows the inference rules above. An empty defaults: {} restores inference; use an explicit "self" binding to pin the primary.

A variant’s own parent borrows physical settings only: type, index, column, analyzer, normalizer, long_terms, and vector settings. It does not inherit that parent’s shape, storage, variants, or defaults. Parents name authored fields or templates, never derived selectors.

A new suffix can inherit _t’s word analysis and add a whole-value variant. Templates can be supplied when the collection is created:

POST /collections/_create
{
"name": "templates",
"schema": {
"templates": {
"_title": {
"parent": "_t",
"variants": {
"s": "string"
}
}
}
}
}

A field named book_title now indexes both book_title and book_title__s. The _title template inherits _t’s case folding, accent folding, and English stemming; its string variant keeps the whole value for sorting, faceting, and exact lookup through the inferred value binding. See field retrieval for retrieving a particular variant.

normalizer takes filters only: lowercase, nfkc_cf, and fold, using the same component syntax as analyzer filters. Each input element stays one whole value. Normalization applies at ingest and to query literals, facet selected values, and range bounds, so a literal accepted at ingest finds its document later. Absent normalizer inherits; [] clears it.

IDs, indexed STRING values after normalization, and analyzed TEXT tokens share a 255-byte term space. Terms of 255 bytes or less stay unchanged. The long_terms policy says what happens to longer ones; the default, hash128, keeps long values distinct with no limit on value length:

Policy Over-limit term becomes
hash128 (default) The first 230 bytes, backed off to a UTF-8 boundary, followed immediately by 25 base36 characters hashing the whole term.
truncate The first 255 bytes, backed off to a UTF-8 boundary. Values sharing that prefix merge, including IDs for overwrite and delete.
reject Reject documents and requests containing over-limit terms.

The hash128 suffix is unseeded XXH3_128 of the whole term, taken as one 128-bit number from its canonical 16 bytes (high 64 bits then low 64 bits, both big-endian) and written in 0-9 and a-z, most significant digit first, zero padded to 25 digits. A 300-byte value in a _s field therefore facets as its first 230 bytes plus a 25-character tail such as ...131pru8lxm5t1cohchv0tgjtg.

What the policy means at query time:

  • Exact queries (any_of, :=), range bounds, facet selected values, overwrite, and delete apply the same transform as ingest, so full values round-trip. TEXT exact membership applies it to the single analyzed term; match and phrase apply it per analyzed token.
  • Different long values keep distinct IDs, exact matches, and facet buckets except for hash collisions.
  • A prefix query longer than the kept prefix falls back to that prefix and returns a superset. Wildcard and regex queries do the same when their common leading literal prefix exceeds the limit; other patterns operate on the stored term bytes, hash suffix included, so they can miss a long value or match its hash characters. Fuzzy distance also compares transformed bytes, so similarity between discarded tails is not measured. When a wildcard, regex, or fuzzy query can reach a term of 252 bytes or more and a segment holds one for that field, the response carries a fitted_terms warning.
  • The stored source, when enabled, keeps the full bytes before normalization or hashing and is used to return the exact value if its column representation was truncated.
  • Column-only strings (index: "none") have no term-space limit, but cannot serve field facets or exact queries.

Absent long_terms inherits through the parent chain, otherwise defaults to hash128; an explicit policy overrides inheritance. Variants may set the policy or borrow it through their own parent; they do not inherit it from their logical primary. The policy applies only to ID, TEXT, and indexed STRING representations.

Request Meaning
GET /collections/{c}/_schema the authored schema (valid write body)
GET /collections/{c}/_schema?view=resolved effective representations, bindings, and schema coverage (HTTP only)
POST /collections/{c}/_schema mode=set (default): set the named definitions, keep the rest
POST /collections/{c}/_schema?mode=replace_all replace the whole schema
POST /collections/_create create a collection, optionally with a schema installed before it becomes visible

The same operations are available over gRPC as luxir.Admin/GetSchema, SetSchema, and CreateCollection; see the gRPC API.

The default GET, also available as ?view=authored, stays sparse: explicit parents, variants, and defaults are returned without inheritance expansion. The resolved view expands everything, per physical representation. After adding the fields in Templates and inheritance, inspect the authors collection:

GET /collections/authors/_schema?view=resolved
{
"generation": 1,
"fields": {
"inherited": {
"bindings": {
"search": "inherited",
"value": "inherited__s"
},
"representations": {
"s": {
"analyzer": null,
"column": true,
"coverage_complete": false,
"index": "match",
"introduced_generation": 1,
"long_terms": "hash128",
"multi": false,
"name": "inherited__s",
"normalizer": [],
"oldest_generation": 0,
"stored": false,
"type": "string"
},
"self": {
"analyzer": {
"filters": ["nfkc_cf", "fold"],
"tokenizer": "unicode_word"
},
"column": false,
"coverage_complete": false,
"index": "match",
"introduced_generation": 1,
"long_terms": "hash128",
"multi": false,
"name": "inherited",
"normalizer": null,
"oldest_generation": 0,
"stored": true,
"stored_resource": "_stored_",
"type": "text"
}
}
}
}
}

(Abbreviated to the inherited field; the real response lists every declared field.) The resolved view is diagnostic, not a schema write body. It contains:

Property Meaning
generation Current schema generation; the initial persisted schema is generation 1.
fields Declared concrete fields and their variants; suffix-template instances are not listed.
fields.<f>.bindings search and value, each naming the effective physical field.
fields.<f>.representations Definitions keyed by self or variant label.
Representation settings name, type, multi, index, column, stored, analyzer (always the expanded definition; analyzer_preset adds the registered name when one was used), normalizer, effective long_terms for term-backed representations, plus stored_resource and vector settings when applicable. These describe the structures available to operations.
ann (vector representations) Effective ANN settings in the authored vocabulary, including defaults and inherited values. Contains codec, nlist, and only the selected codec’s pq or rabitq object. Zero nlist, pq.m and rabitq.bits retain their automatic policies. ANN-only changes preserve introduction and coverage generations.
introduced_generation Start of this representation’s current continuous availability. Redefining a representation or removing and re-adding its label starts a new introduction.
oldest_generation Minimum schema generation across all committed live segments, even those lacking this field; null for an empty index.
coverage_complete True for an empty index or when that minimum reaches the introduction generation.

Coverage is derived from segment schema generations, not from counting documents that contain the field. Merges keep the minimum input generation and cannot establish backfill. Deleting some documents from an old segment does not refine its generation.

Schema changes apply to new requests and do not rewrite existing documents.

Adding a field or variant leaves older documents without its values. Adding a variant can also change an inferred binding immediately. For a staged rollout, explicitly pin the affected binding to its current representation (for example, "defaults": {"value": "self"}) when adding the variant. Reindex from your source data, then select the new variant explicitly or set defaults: {} to restore inference for both bindings. To keep one binding pinned, retain it in the defaults object and omit only the other. Older documents without a new author__s variant will not match its exact queries or appear in its facet buckets.

Schema edits are not checked for compatibility with existing data. To change a field’s type or analysis, use a new field or variant label and reindex. Removing and reusing a name does not erase its old indexed values.

  • Schema edits do not rewrite existing documents; see Changes to a live collection.
  • Analyzed text has no value column; use a string variant to sort text values or facet on whole values.
  • english_possessive and kstem are analyzer filters only; STRING normalizers accept lowercase, nfkc_cf, and fold.

Declare nested paths and their fields by flat qualified names, for example "items": "nested" and "items.color": "string" in fields. Each intermediate component must be a nested path. Global suffix templates also work under paths: lines_nested.color_s needs no explicit declaration in the default schema. A nested path cannot declare scalar value settings, variants, indexing, columns or storage. Objects may contain nested paths recursively, up to 32 levels.

A nested path’s one setting is reverse_map (default true), described in Declaring nested paths. It can be changed on a live collection without reindexing.

Dots separate path components. Each component starts with an ASCII letter and contains letters, digits or underscores. A physical name, including a generated __variant suffix, must fit in 127 bytes.

Vector ANN overlays can be built on child fields using their qualified physical names, for example vec.lines_nested.embedding_v.

Request field names are relative to their view: in the items view, color resolves to items.color. See Nested documents for indexing, queries, operations on child rows, and limits.