Schemadev
The schema defines field types, text analysis, storage, and indexes. You can define fields individually, use field templates for names that share a suffix, or combine both. Built-in templates let you start without writing your own schema. Templates also handle new fields as your data grows.
Field types
Section titled “Field types”Choose a type based on how you want to search and use the values:
| Type | Description and typical uses |
|---|---|
nested |
Structural object path; see Nested documents. Accepts an object, an array of objects, null or an empty array. |
text |
Analyzed text for full-text search: titles, descriptions, and article bodies. An analyzer splits each value into searchable tokens for word and phrase queries. |
string |
A whole value for exact matching, filtering, sorting, and faceting: categories, tags, and product codes. Values are not split into words. |
int |
Signed 64-bit integer: counts, quantities, and years. |
float |
32-bit floating-point number: measurements and other numeric values where single precision is sufficient. |
double |
64-bit floating-point number for greater precision than float. |
decimal |
Fixed-point number with a required scale (0 to 18 digits after the point). See Decimals. |
ip |
IPv4 or IPv6 address, stored as a 16-byte term. See IP addresses. |
boolean |
true or false: flags and on/off attributes. |
date |
Timestamp with ms (default) or ns storage precision. Ingest accepts ISO-8601 text or numeric epoch values in the field’s unit. See Dates. |
vector |
Dense vector of floating-point values, such as an embedding. Set a similarity metric to enable nearest-neighbor search. See Vector search. |
geo_point |
Geographic location supplied as [longitude, latitude], an object, a string, or a geohash, for distance and bounding-box queries. See Geo search. |
long_range, double_range, date_range |
An interval of int, double, or date endpoints, such as a validity window, a price band, or an event’s duration. See Range types. |
id |
Unique document identifier, reserved for the built-in id field. Used to identify documents for replacement and deletion. |
Decimals
Section titled “Decimals”decimal stores value * 10^scale as a signed 64-bit integer. Set the
required scale from 0 to 18; it is inherited from parent like other
properties. A direct scale is invalid on other types; an inherited scale
is ignored when a child changes type. There is no default decimal
suffix. For example:
{"fields":{"price":{"type":"decimal","scale":2,"index":"range"}}}JSON integers, floating-point numbers and numeric strings are accepted.
Excess fractional digits round half away from zero: 1.005 and "1.005"
become 1.01 at scale 2, and -1.005 becomes -1.01. JSON floating-point inputs
use their shortest round-trip text, rather than their binary expansion.
Exponents and long fractions are accepted; overflow after rounding rejects
an ingest value. Out-of-domain exact-match literals match nothing, and
range bounds saturate to the field’s value domain. Scale is persisted with the
field type; reading or merging data under a different type or scale is rejected.
Match and any_of literals round like ingest. Range bounds round in the
requested direction: gte: 10.005 starts at 10.01, while lt: 10.005
includes values through 10.00. Integer field bounds also round directionally
(gte: 9.5 starts at 10); integer ingest and exact-match literals still
reject non-integral values.
Results are doubles by default, computed as raw divided by 10^scale.
This applies to documents, facet buckets and range bounds, sort values,
collapse keys, and decimal statistics. The int64 storage is exact; doubles
can lose distinctions at large magnitudes.
The request or operation flag decimal_scaled instead returns raw integers
and interprets decimal field literals as raw integers. Integral numeric
strings, including "1250.0", mean raw 1250 at every scale; non-integral
match literals are rejected. Bounds still round directionally in raw units.
Ingest always uses decimal values and is unaffected by the flag. Combine
decimal_scaled with int64_str to round-trip large raw integers through
JSON strings. Averages and percentiles remain doubles (in raw units under the flag).
Range facet gaps and origins must be exact multiples of 10^-scale; gaps
must be positive. With decimal_scaled they use raw units. Selection,
min, max, sum, and avg retain the scale when possible, aggregating
raw values exactly (INT128 sums). Arithmetic and mixed-scale expressions
decode to numeric values. Expression constants always mean values, independent
of decimal_scaled. Same-scale comparisons use raw values; per-document
comparisons against numeric constants use raw integer bounds when representable.
A decimal sum wider than int64 renders as a double by default; scaled
output emits null with an aggregate overflow warning, as integer sums do.
IP addresses
Section titled “IP addresses”ip accepts IPv4 and IPv6 text. IPv4 uses the IPv6 v4-mapped encoding, so
10.0.0.1 and ::ffff:10.0.0.1 are equal. Results use dotted IPv4 for mapped
addresses and compressed lowercase IPv6 otherwise. Sorting follows address
byte order. The default suffix templates are _ip and multi-valued _ips.
IP fields require index: "match" (the default), and reject stored: true.
There is no original source text to retain. Columns support retrieval, field
facets, sorting, collapse, and column-only value expressions such as count.
Facet selection and collapse identity use the encoded address.
match accepts an address or CIDR: 10.1.2.3/8 matches the entire 10.0.0.0/8
network because host bits are ignored. IPv4 /0 covers all mapped IPv4
addresses; IPv6 ::/0 covers all addresses. /32 for IPv4 and /128 for
IPv6 match one address. Range bounds and any_of members must be addresses;
CIDR is rejected there. Numbers, zone ids, and text containing NUL reject.
Prefix, wildcard/regexp, case-insensitive match, fuzzy matching, and facet
term_filter are unsupported. IP fields cannot be bare-word fields in
simple_query, but addr:10.0.0.1 is an exact match. Wildcard multi_match
field expansion probes each IP field and skips text it cannot parse.
Booleans
Section titled “Booleans”boolean accepts JSON true/false and the strings "true"/"false";
other values, including numbers 0 and 1, reject the document. Query literals
use the same rules: flag:true, {"match": {"flag": false}}.
Values are stored as 0/1 in the integer column. Sorting puts false before
true; field facet bucket values and collapse keys are JSON booleans.
Range queries accept boolean bounds, but range facets are not supported.
The default is index: "none", as for int; index: "range" and
index: "match" add the corresponding numeric indexes. For filtering on a
boolean, range is the faster of the two, and the _b/_bs templates use it.
Retrieved values are JSON booleans. Document columns have no boolean kind,
so gRPC returns a col_val column holding one boolean value (or array) per
document. Value expressions preserve booleans through selection (min,
max, and choices between booleans); arithmetic, sum, and avg produce
numbers. Summing a boolean counts true values.
Range types
Section titled “Range types”A range value is an object with an optional lower bound (gte or gt) and an
optional upper bound (lte or lt); a missing side is an open end:
{"id": "promo-7", "valid": {"gte": "2024-06-01", "lt": "2024-07-01"}, "price": {"gte": 10.5, "lte": 20}}Ingest is strict about the endpoint type: long_range rejects fractional
endpoints, and long or date endpoints outside the supported scalar domain
fail the document. An exclusive bound moves one value inward ({"gt": 5} on
long_range starts at 6). A lower bound above the upper bound, including an
empty interval after exclusive bounds, fails the document.
Luxir treats a partial date as its whole granule at ingest: {"gte": "2024-06"}
starts at the first millisecond of June and {"lte": "2024-06"} ends at its
last millisecond. gt excludes the whole named granule, as does lt.
Elasticsearch and OpenSearch differ: when they ingest a range field, a
partial-date lte such as "2024-06" does not extend to the end of the month. Dates without an explicit
offset are evaluated in UTC at ingest. date_range uses millisecond precision;
ISO strings, epoch milliseconds, now, and date math follow the scalar
date rules. Query bounds follow the scalar range-query coercion rules.
For double_range, an open lower end is -Infinity and an open upper end is
Infinity; explicit infinities at those ends have the same meaning. NaN is
rejected at ingest and in queries. gt: "Infinity" or lt: "-Infinity" makes
an empty interval. Negative and positive zero remain distinct, with -0.0
below +0.0. Long and date open ends extend to the int64 limits.
Search results return {"gte": lower, "lte": upper}, omitting open ends.
Date endpoints follow the request’s date output mode: epoch milliseconds by
default, ISO strings with date_iso, milliseconds with date_ms, or
nanoseconds with date_ns. This applies to both ROWS and COLUMNS responses.
Range fields require column (the default) and support index: "none"
(the default) or index: "range" for an optional query index. They do not
support stored or index: "match". Multi-valued range fields are not
supported yet.
The range query supports intersects,
contains, and within. match finds intervals containing a point value;
a partial date matches intervals overlapping its granule. multi_match
accepts explicitly named range fields and field weights;
wildcard field selectors do not select range fields. Sort, facets, collapse,
value expressions, and any_of are not supported on range fields.
text versus string
Section titled “text versus string”Both accept text values, but they make different things searchable. Suppose
you index "Red Bicycle" using the built-in templates:
title_t(text) indexes the wordsredandbicycle.title_t:bicyclematches, as doestitle_t:"red bicycle".title_s(string) indexes the whole valueRed Bicycle.title_s:"Red Bicycle"matches;title_s:bicycledoes not.
Sorting text values requires a whole-value column, which string provides.
Faceting can use analyzed terms from a text field, but whole values are
often more appropriate, such as complete author names or categories.
The analyzer controls how text is split and normalized. Bare "type": "text"
defaults to splitting on whitespace with no filters; the _t template adds
Unicode word segmentation, case and accent folding, and English stemming.
See Text analysis for the available components.
string keeps values verbatim by default. You can configure a
normalizer to transform them, for example
by folding case or accents, without splitting them into words.
When one value needs both full-text search and whole-value
sorting or faceting, use a text field with a string
variant. The built-in _name template provides this
combination for names.
Read the schema
Section titled “Read the schema”The schema API uses the same JSON in both directions: what GET returns is
a valid POST body, and posting it back keeps the same definitions.
{ "fields": { "id": { "type": "id" }, "_version_": { "type": "int" } }, "templates": { "_i": { "type": "int" }, "_s": { "type": "string" }, "_t": { "type": "text", "analyzer": { "tokenizer": "unicode_word", "filters": ["nfkc_cf", "fold", "kstem"] } }, "_name": { "type": "text", "analyzer": { "tokenizer": "unicode_word", "filters": ["nfkc_cf", "fold"] }, "variants": { "s": { "parent": "_s" } }, "defaults": { "value": "s" } } }}(Abbreviated: the default schema defines a template for every suffix.)
Two sections:
fieldsdefine individual fields. A document field namedtitleusesfields.title. Explicit definitions take precedence over templates.templatesdefine shared settings for fields whose names end in a matching suffix. A field namedanything_twith no explicit definition uses the_ttemplate. Matching uses the suffix from the final underscore. Templates are never usable as document fields themselves, and any definition can name one asparentto inherit its properties.
The built-in templates are:
| Suffix | Type and default behavior |
|---|---|
_nested |
Nested object path; structural, with no scalar value. |
_s |
Indexed exact string with a column. |
_sc |
Column-only exact string. |
_ss, _ssc |
Multi-valued forms of _s and _sc. |
_i, _is |
Integer column, single- or multi-valued. |
_b, _bs |
Boolean column with a points index (index: "range"), single- or multi-valued. |
_ip, _ips |
IP address terms, single- or multi-valued. |
_f, _fs |
Float column, single- or multi-valued. |
_d, _ds |
Double column, single- or multi-valued. |
_dt, _dts |
Date column with a points index (index: "range"), single- or multi-valued. |
_p, _ps |
Geo point column with a points index (index: "range"), single- or multi-valued. |
_t |
Unicode-word text with NFKC case folding, accent folding, English possessive removal, and KStem English stemming. The general-purpose text field. |
_u |
Unicode-word text, case- and accent-sensitive. |
_un |
Unicode-word text with NFKC case folding; accents preserved, no stemming. |
_w |
Text split on whitespace, case- and accent-sensitive. |
_wl |
Text split on whitespace, Unicode-lowercased; no normalization or accent folding. |
_name |
Unicode-word text with case and accent folding and no stemming, plus a whole-value string variant for facets and sorting. |
_names |
Multi-valued form of _name. |
_v, _vs |
Single- or multi-valued vector column; storage-only until a metric is set on a concrete field. |
Text fields are stored in the default compressed stored_resource.
Numeric suffixes are column-backed but do not build a points index by default;
range and exact-match queries still work by quickly scanning the column. Define a
concrete field with index: "range" when those operations need a points index,
or index: "match" for a term dictionary that serves exact matches and lets
field facets on low- and medium-cardinality values count without scanning the
column (see Faceting).
The geo suffixes _p and _ps build one by default, because geo queries are
their only use.
Field templates for dynamic fields
Section titled “Field templates for dynamic fields”Templates are useful when you do not know every field name ahead of time. An ecommerce catalog can gain new product classes, each with its own attributes. Define their shared settings once, and new matching fields work automatically.
This schema combines an explicit productname field with a custom _attr template
for string attributes. The built-in templates remain available:
{ "fields": {"productname": {"parent": "_t"}}, "templates": {"_attr": "string"}}Index a product with a shoe_width_attr field:
{"docs": [{"id": "shoe1", "productname": "Trail shoes", "shoe_width_attr": "wide"}], "commit": {}}Later, a new product class introduces socket_attr. No schema update is
needed:
{"docs": [{"id": "cpu1", "productname": "Desktop processor", "socket_attr": "AM5"}], "commit": {}}Both attribute fields use _attr’s string settings, so you can filter, sort,
and facet on them immediately after the commit:
{"query": "socket_attr:AM5", "fields": ["id", "productname", "socket_attr"]}{ "docs": [ { "id": "cpu1", "productname": "Desktop processor", "socket_attr": "AM5" } ]}An explicit definition overrides a template for that field. You can also define every field individually if the field names are known in advance.
Define fields
Section titled “Define fields”Use POST to define fields. The default, mode=set, sets each named
definition and leaves everything else alone:
{ "fields": { "title": { "type": "text", "stored": true, "analyzer": { "tokenizer": "unicode_word", "filters": ["nfkc_cf", "fold"] } }, "year": { "type": "int", "index": "range" }, "vec": { "type": "vector", "dims": 768, "metric": "cosine" } }}The response is the full resulting schema, the same shape GET returns.
When a definition only needs a type, a bare string works:
{"fields": {"year": "int", "author": "string"}}Setting an existing field replaces its entire definition.
Each tokenizer or filter is {"name": ..., "params": {...}}. Components
without parameters can use a bare name, as in the title example above.
Parameters are typed JSON values (a string, number, bool, or list).
mode=replace_all replaces the whole schema:
{"fields": {"title": "text", "year": "int"}}The reserved id and _version_ fields are added automatically if omitted.
_version_ is a fixed, retrieval-only reserved field with no index, column or
stored capability. It returns the update version of the request that indexed
the document. Its capabilities cannot be configured. Setting it in document
input is rejected. Optimistic concurrency is not supported. The resolved schema
view reports null search and value bindings for this field.
Omitting templates removes them, leaving a schema with only explicit fields.
To edit the existing schema, start from GET output. Each successful
publication advances the schema generation.
Field properties
Section titled “Field properties”| Key | Meaning |
|---|---|
type |
The field type: text, string, int, float, double, decimal, boolean, ip, date, vector, geo_point, long_range, double_range, date_range, id |
scale |
decimal only, required from 0 to 18; inherited from parent. |
index |
match, range, or none; absent = the type’s default (text/string index for match, numerics don’t) |
column |
store values in a per-field column (sorting, faceting, analytics); supported and default-on for non-text types |
multi |
false, true, "set", or "array"; see Multi-valued modes |
stored |
keep canonical source text for TEXT/STRING/ID retrieval, before analysis/normalization; default on for text only; ignored for numerics |
stored_resource |
stored-field group for TEXT/STRING/ID; empty or absent inherits, falling back to _stored_; see Stored resources |
analyzer |
text only: {"char_filters": [...], "tokenizer": <component>, "filters": [...]}, or the name of a registered analyzer such as "english"; see Text analysis |
normalizer |
string only: a list of filter components applied to each whole value, with no tokenizer; see STRING normalization and length |
long_terms |
string, text (per token), and id: hash128 (default), truncate, or reject for terms over 255 bytes after normalization/analysis; see Long terms |
unit |
date only: ms (default) or ns. Inherits from the parent and is part of the field signature. An existing date definition cannot change its resolved unit. |
variants |
Map from label to another field definition receiving the same input value; bare type strings work here too; see Field variants |
defaults |
For fields with variants, choose self or a variant label to override the inferred search and value bindings; see Default bindings for variants |
parent |
inherit any unset properties from a field or template |
dims, truncate_dims, metric, normalized, normalize_on_write |
vector only; metric: l2, ip, cosine, none; see Vector search |
ignore_malformed, ignore_z_value |
geo_point only; skip malformed points instead of failing the document, and accept and drop a third (altitude) coordinate. ignore_malformed defaults to false; ignore_z_value defaults to true. See Malformed points and altitude. |
reverse_map |
nested only; defaults to true. Store each child row’s owner in segments written from now on; see Declaring nested paths. |
ann |
vector only; approximate index codec and parameters. RaBitQ is the default; PQ requires explicit codec:"pq". See Index settings. |
Every property is optional. Absent means “inherit from parent, else the
type’s default”, and the schema you read back stays as sparse as the one you
wrote.
An empty analyzer: {} inherits the parent’s analyzer. Naming a tokenizer,
listing any token filter, or naming a registered analyzer replaces the whole
chain, including character filters (token filters alone get the whitespace
tokenizer). Setting only char_filters keeps the parent’s tokenizer and token
filters, replacing its character filters. Without a parent analyzer, the base
is whitespace with no token filters.
column and stored solve different problems. A column is a typed,
per-field structure used by sorting, faceting, analytics, numeric/geo queries,
and vector search. Stored fields keep canonical source text for retrieval in
compressed chunks. Analyzed text has postings and stored retrieval but no
per-document value column; use a string variant when the same source value
must also sort or facet. Scalar types return their values from columns without
a second stored copy.
For PQ, positive ann.pq.m requires positive effective (inherited) dims
and must divide the stored dimension: truncate_dims when set, else dims.
Omitted or zero m selects automatically and permits inferred dimensions.
Validation includes inherited ANN settings and variants.
Multi-valued modes
Section titled “Multi-valued modes”multi is inherited when omitted and defaults to false (one value).
true selects the type’s default: set for INT, FLOAT, DOUBLE, DECIMAL, BOOLEAN, IP, DATE, and
STRING; array for TEXT, VECTOR, and GEO_POINT. Explicit "set" and "array"
select a mode. Array-only primary types reject "set"; ID is reserved for
the single-valued id field.
- Set: each document’s column values are sorted and deduplicated after coercion and normalization. Numbers and dates use ascending numeric order; strings use byte order after normalization and any long-term fitting, whether indexed or column-only.
- Array: columns preserve source order and repeats after coercion and normalization. Stored source, when enabled, keeps source order and repeats independently of the column mode. Numeric fields have no stored-source copy.
Bare multi-valued numeric, date, and STRING fields sort by their minimum
ascending and maximum descending in both modes. Per-document numeric reducers
such as min(f), max(f), count(f), sum(f), and avg(f) consume the stored
values; count(f) also counts STRING, ID and GEO_POINT values, so a set
counts a repeated value once and an array counts each repeat. Default facets
count each document once per bucket in either
mode, even when several values fall in that bucket. Indexed STRING postings
also contain each document once per term in both modes.
The declaration resolves against each representation’s own type. With true,
a TEXT primary uses an array and its STRING variant uses a set. With "set",
set-capable representations use sets and array-only variants use arrays.
Explicit "array" applies to every representation.
The authored schema view echoes true, false, "set", or "array",
preserving inheritance through type-changing templates. The resolved view
reports the effective mode: true for the type’s default, "array" for an
array on a set-capable type, and false for a single-valued representation.
Changing the effective mode starts a new introduction generation and affects
newly indexed data; reindex original values to convert existing documents.
Stored resources
Section titled “Stored resources”Stored values are grouped by stored_resource, and each group is compressed
in chunks of whole documents. The top-level stored_resources map declares
the groups; the default _stored_ always exists, and without an entry it uses
LZ4 with 16KB chunks of at most 128 documents. A field may name only _stored_
or a declared group, and a group name starts and ends with _ (for example
_stored_logs_), so it never collides with a field or template name.
{ "fields": {"message": {"type": "text", "stored_resource": "_stored_logs_"}}, "stored_resources": { "_stored_logs_": {"codec": "zstd"} }}| Key | Meaning |
|---|---|
codec |
lz4 (default): fast to write and read. zstd: much smaller for repetitive text such as logs, at more CPU per chunk |
zstd_level |
zstd only: 1 (fastest) to 22 (smallest); default 3 |
chunk_size_bytes |
target uncompressed bytes per chunk, 1024 to 4194304; default 16384, or 65536 with zstd |
max_docs_per_chunk |
most documents per chunk, 1 to 65536; default 128, or 1024 with zstd |
zstd_dictionary |
zstd only: enable one dictionary per resource, shared by its segments; trained by a merge |
zstd_dictionary_bytes |
dictionary capacity upper bound, 1024 to 262144; default 65536. Requires zstd_dictionary |
Larger chunks compress better, but returning one stored document decompresses
its whole chunk. Chunks are compressed as documents are indexed, so the codec
and level also set indexing cost; levels above the default compress much more
slowly for little further gain. Each segment records its codec and chunk
limits, so a change applies to segments written afterwards. A merge copies
compressed chunks from inputs that already match its output codec, chunk
limits and dictionary and recompresses the others; the level alone does not trigger
recompression. Inputs where more than 1% of chunks were sealed before filling
(each small flush ends in one) are also recompressed, so documents indexed a
few at a time still end up in full-size chunks. mode=set replaces the named
entries and keeps the others.
A dictionary holds text shared by documents of a resource, such as common values and log templates, so chunks can refer to it. That matters most for small chunks, which compress poorly on their own. For retrieval-heavy workloads (many single-document fetches), prefer small chunks plus a dictionary over large chunks:
"stored_resources": { "_stored_": {"codec": "zstd", "zstd_dictionary": true, "chunk_size_bytes": 4096, "max_docs_per_chunk": 32}}The first merge with at least 100 times the configured capacity (at most 8 MiB) of live stored data trains the resource’s dictionary. Earlier segments have none; later segments share the dictionary and each stores its bytes once. Capacity is an upper bound: training can produce a smaller dictionary. The benefit is greatest with small chunks; with large chunks, decoding can be slower.
There is no automatic retraining. Changing zstd_dictionary,
zstd_dictionary_bytes or the codec clears the current dictionary so a later
merge can train another. After a restart, the dictionary from the largest
live segment carrying one is used.
Text analysis
Section titled “Text analysis”A text analyzer applies an ordered list of char_filters to the whole value,
then one tokenizer, then an ordered list of token filters. Both lists are
optional.
Cleanup before tokenization
Section titled “Cleanup before tokenization”{ "fields": { "body": { "type": "text", "analyzer": { "char_filters": [ "html_strip", "nfkc_cf", {"name": "mapping", "params": {"mappings": ["- =>", "\u2014 =>"]}} ], "tokenizer": "unicode_word", "filters": ["fold", "kstem"] } } }}Character filters run in the written order, separately for each field value.
For example, HTML stripping followed by character removal can decode -
and then delete the resulting hyphen. Reversing those filters leaves that
hyphen in the tokenizer’s input. Removing - from foo-bar produces one
foobar token, instead of two words separated by punctuation.
| Character filter | Behavior |
|---|---|
nfkc_cf |
Unicode NFKC case folding, including default-ignorable removal, before segmentation. |
mapping |
Apply case-sensitive, leftmost-longest rules from params.mappings. Map to an empty replacement to delete text. |
html_strip |
Extract text, remove tags/comments/declarations and script/style contents, and decode named and numeric HTML character references once. |
All except mapping take no parameters. Lowercase, accent folding, stemming,
and possessive removal are token filters only. nfkc_cf is also available as a
char filter because compatibility normalization before segmentation can change
word boundaries.
mapping takes a list such as ["ph => f", "\u2014 => "]. The longest
source at each input position wins; rule order does not break ties, and duplicate
decoded sources are rejected. Replacements are not rescanned by the same filter.
Use multiple mapping filters to apply successive rewrites. An empty list is a
no-op; empty sources are invalid.
Each rule has exactly one unescaped =>; ASCII whitespace at either edge of
each side is trimmed. Use ordinary JSON escapes for characters, as in
"\u2014 =>" to delete an em dash. Mapping-level escapes are needed only to
preserve edge whitespace or include a literal =>: "x => \\u0020" maps
x to a space, and "\\u003d> => arrow" maps => to arrow. Double the
backslash in JSON for these mapping-level escapes. Mapping escapes accept
\\, \n, \r, \t, \b, \f, and \uXXXX, including surrogate pairs.
Invalid UTF-8, malformed escapes, unknown parameters, and non-string rules are
rejected. For folded matching, put nfkc_cf before mapping and write folded
sources in the rules.
html_strip removes inline tags without separating text: H<sub>2</sub>O
becomes H2O. Its inline list follows Lucene, including img, input,
select, and textarea; other tags, including br and unknown names, emit
newlines. Comments/declarations disappear; script/style regions become separators.
Quotes protect attribute values only immediately after = and optional
whitespace, so title=O'Reilly works. Unterminated ordinary tags preserve <;
unterminated comments and script/style regions consume the remainder. CDATA
delimiters disappear but contents stay literal, including < and &.
Named/numeric references decode once, including legacy semicolonless names;
unknown references stay literal and invalid numeric scalars become U+FFFD.
Decoded <b> stays text. It never executes scripts, fetches resources,
or expands document-defined entities. This extracts indexing text; it is neither
a browser rendering model nor an HTML sanitizer for safe display.
Cleanup runs for analyzed queries. Prefix/fuzzy terms and TEXT range bounds
use normalizing character filters before token normalization; html_strip is
excluded. Wildcard/regex literals retain per-codepoint normalization, while
character classes remain exact. Multi-character mapping sources do not span
adjacent pattern literals, so ph => f applies to a prefix term but not the
ph in ph*. Literal normalization currently rejects expansions beyond 32
codepoints per input codepoint. HTML markup and entities are not decoded inside
patterns. Already-analyzed phrase terms remain verbatim.
Stored source values retain the submitted text.
An analyzer specifying only char_filters keeps the inherited tokenizer and
token filters. Schema changes affect future indexing; reindex existing documents
to apply new cleanup rules to them.
Tokenizers and token filters
Section titled “Tokenizers and token filters”A tokenizer is followed by the configured token filters:
| Component | Meaning |
|---|---|
unicode_word |
Tokenizer: Unicode word segmentation (UAX#29). |
whitespace |
Tokenizer: split on whitespace. The default when an analyzer lists filters but no tokenizer. |
keyword |
Tokenizer: the whole value as one token. |
nfkc_cf |
Filter: Unicode NFKC_Casefold, including default-ignorable removal. |
lowercase |
Filter: Unicode lowercasing only. |
fold |
Filter: accent and diacritic folding. |
english_possessive |
Filter: remove a trailing 's. |
kstem |
Filter: KStem English stemming; optional possessive parameter. |
stop |
Filter: remove tokens in the required non-empty params.words list. |
Prefer nfkc_cf to lowercase on Unicode-segmented text: bare lowercasing
leaves equal-looking strings unequal (composed versus decomposed forms, final
sigma), while NFKC case folding makes them compare equal. It also removes
invisible default-ignorable characters such as soft hyphens, ZWJ/ZWNJ, and
variation selectors: co<U+00AD>operate matches cooperate. Tokens that become
empty are dropped without a position gap. Normalization follows Unicode 15.1.0
and stream-safe NFC; runs exceeding 30 non-starters in
NFKD receive U+034F (CGJ), so those adversarial inputs can differ from ICU.
stop takes authored words, for example
{"name":"stop","params":{"words":["THE","Cafe\u0301"]}}. Stop words are
analyzed like field text up to the stop filter, including character filters,
tokenization, and preceding token filters. Each must produce exactly one token;
analyzed duplicates collapse. A schema set rejects words producing zero or
several tokens. Loading a stored schema logs a warning and drops those words,
so schemas saved under an older analysis version still load. There is no
ignore_case parameter: preceding filters determine how words and tokens meet.
For example, stop after kstem analyzes ponies to pony. Later filters do not
affect the stop set.
Removed tokens leave position holes before the next emitted token, preserving
phrase distances. Trailing holes are preserved between indexed values, in
addition to the fixed 100-position gap, and between phrase.words entries.
Stop does not remove prefix/fuzzy/range terms or wildcard/regex literals during
normalization, and cannot appear in STRING normalizers.
kstem implements Bob Krovetz’s dictionary-based English stemming algorithm:
ponies becomes pony, while recognized dictionary words are preserved or
replaced by their dictionary mapping. It runs at both index and query time
for _t fields. In custom analyzers, put it after lowercase or nfkc_cf,
and after fold if used. Tokens containing anything outside lowercase ASCII
letters, or with lengths outside 3-49 letters, pass through unstemmed.
english_possessive removes one trailing apostrophe followed by s or S,
matching Lucene’s EnglishPossessiveFilter. It accepts ASCII apostrophes, right
single quotation marks (U+2019), and fullwidth apostrophes (U+FF07). Other
apostrophes and a bare trailing apostrophe are unchanged: Winter's becomes
Winter; don't and dogs' are left alone. The tokenizer may already discard
a trailing apostrophe before filtering.
Bare "kstem" includes equivalent possessive removal by default, stripping
the suffix before stemming within one filter stage. _t uses this default, so
title_t:winter matches Winter's Tale. To request stemming alone, disable
possessive removal explicitly:
{"name": "kstem", "params": {"possessive": false}}The standalone filter is useful without stemming. When placing it before
kstem, set possessive: false on KStem so removal happens only once.
Possessive removal applies even when the remaining word is non-ASCII or outside
KStem’s length range. Source spelling, offsets, and positions are preserved.
Prefix, wildcard, regex, and fuzzy query terms receive the field’s case and
accent folding without possessive removal or stemming. Neither
english_possessive nor kstem is available in STRING normalizers.
For Unicode text without English stemming, use _un, or configure
unicode_word with ["nfkc_cf", "fold"] to retain accent folding as well.
Named analyzers
Section titled “Named analyzers”An analyzer can be a registered name instead of a definition. A bare string is shorthand for the object form:
{"fields": {"body": {"type": "text", "analyzer": "english"}}}{"fields": {"body": {"type": "text", "analyzer": {"name": "english"}}}}A named analyzer takes no tokenizer or filters alongside the name.
char_filters next to the name replace the analyzer’s own character filters,
just as setting only char_filters does under an inherited analyzer. Built in:
| Name | Expands to |
|---|---|
standard |
unicode_word + ["nfkc_cf"] |
english |
unicode_word + ["nfkc_cf", "fold", "kstem"], the same chain as _t, with kstem’s default possessive=true |
Plugins can register more (a language bundle, for example). The schema stores
the name, and expands it through the registry each time the collection loads,
so a changed registration applies at the next load, to queries and new
documents alike. Reindex after such a change so existing segments match. The
resolved view reports the current expansion in analyzer and
the name in analyzer_preset. A schema naming an analyzer or component that this build
does not have is rejected on write; a collection already holding such a schema
fails to load with that message while the node keeps serving its other
collections.
Field variants
Section titled “Field variants”A document supplies one value per logical field, and the schema can index that
value several ways. The primary keeps the field’s name; each variant indexes
the same submitted value under <field>__<label>.
See Documents and values for the worked example:
searching author names by word and faceting on their whole values with the
built-in _name template. Its schema definition
is shown below, followed by the rules for inheritance and customization.
Variant names and constraints
Section titled “Variant names and constraints”Labels are ASCII: a letter first, then letters, digits, or single underscores.
self is reserved case-insensitively, and labels differing only by case
collide. Label lookup otherwise uses the authored case. __ is reserved in
authored field names, template names, labels, and final document keys after
field mapping. It is only used in requests to select a representation.
author_name__self selects the primary; it does not create another physical
field.
Physical names are limited to 127 bytes. With variants, a template’s maximum
instance name length is 127 - 2 - longest label length. The __self
selector is not a physical name and does not consume that budget.
multi, stored, and stored_resource belong to the logical field. A variant
may not set them or declare nested variants or defaults. There is one stored
source on the primary when storage is enabled. Neither id nor _version_
may have variants, including inherited ones, and a variant cannot have type id.
The primary and every variant must use types from the same group:
| Group | Allowed types |
|---|---|
| Scalar | text, string, int, float, double, decimal, boolean, ip, date. |
| Vector | vector; every branch must resolve to the same positive dims. |
| Geo | geo_point. |
| Range | long_range, double_range, date_range. |
Default bindings for variants
Section titled “Default bindings for variants”For a field with variants, default bindings choose which representation to
use when a request names the field without a variant selector, such as
author_name. Fields without variants use the field itself; there is nothing
to configure.
With variants, defaults.search selects the representation for text search,
and defaults.value selects it for operations such as sorting and faceting.
Explicit bindings, including inherited ones, take precedence. An omitted
binding is inferred from the resolved field types:
| Primary type | Search binding | Value binding |
|---|---|---|
text |
self |
Its sole string variant, or self if none exists. |
string |
Its sole text variant, or self if none exists. |
self |
| Other types | self |
self |
A text field with one string variant therefore searches words but sorts and
facets on whole values without a defaults block. A string field with one
text variant searches words through that variant and uses the primary for
whole-value operations.
If a text primary has multiple string variants, set defaults.value; if a
string primary has multiple text variants, set defaults.search. Otherwise,
the schema is rejected as ambiguous. Use "self" to keep an operation class
on the primary, or a variant label to select a particular representation.
The _name template explicitly sets defaults.value to s, preserving that
choice even if another string variant is added.
Inference uses types after inheritance. Each operation still checks the chosen representation’s index and column capabilities; it does not look for an alternative if those checks fail.
| Operation | Bare field name uses |
|---|---|
| Match, phrase, simple query, expression terms and phrases, prefix, fuzzy, wildcard, regex | defaults.search, including a match inside a filter. |
any_of / :=, ranges, field and range facets, sort, column expressions, metrics |
defaults.value, then the operation’s normal type and capability checks. |
Exists (f:*), kNN, geo |
The primary physical field. |
| Retrieval | The primary’s source store, else its own typed column; never the value default. |
An explicit selector bypasses both bindings: author_name__s selects that
variant and author_name__self selects the primary. Exists on a bare name
tests primary presence; an explicit selector tests that representation, which
can differ for documents indexed before it was added.
A binding names self or a label in the field’s effective variant map. See
Searching for query and projection examples,
and use ?explain=resolved to see which physical
field a request used.
Templates and inheritance
Section titled “Templates and inheritance”Templates can carry variants and defaults. The default schema includes
_name and its multi-valued form _names:
{ "templates": { "_name": { "type": "text", "analyzer": {"tokenizer": "unicode_word", "filters": ["nfkc_cf", "fold"]}, "variants": {"s": {"parent": "_s"}}, "defaults": {"value": "s"} }, "_names": {"parent": "_name", "multi": true} }}author_name uses _name; author_name__s selects its original whole-name
string variant. The root is resolved before the label, so the tail _s never
picks the default string template independently.
_name uses word search with case and accent folding, without English
stemming or possessive removal. Its string variant preserves the supplied
spelling for facets and sorting, so normalize names before indexing if you
need to.
A bare type: text would use the whitespace analyzer.
variants and defaults each inherit atomically: absent inherits, present
replaces the whole object, and {} clears it. There is no per-label merge.
For example, add fields to the
authors collection using the default _name
template:
{ "fields": { "inherited": { "parent": "_name" }, "replaced": { "parent": "_name", "variants": { "raw": "string" }, "defaults": {} }, "cleared": { "parent": "_name", "variants": {}, "defaults": {} } }}inherited keeps s and the value binding. replaced has only raw; clearing
the inherited defaults lets its value binding infer raw.
cleared clears both the variants and their inherited defaults, leaving only
the primary. Within a present defaults, an omitted binding follows the
inference rules above. An empty defaults: {} restores inference; use an
explicit "self" binding to pin the primary.
A variant’s own parent borrows physical settings only: type, index, column,
analyzer, normalizer, long_terms, and vector settings. It does not inherit
that parent’s shape, storage, variants, or defaults. Parents name authored
fields or templates, never derived selectors.
A new suffix can inherit _t’s word analysis and add a whole-value variant.
Templates can be supplied when the collection is created:
{ "name": "templates", "schema": { "templates": { "_title": { "parent": "_t", "variants": { "s": "string" } } } }}A field named book_title now indexes both book_title and book_title__s.
The _title template inherits _t’s case folding, accent folding, and English
stemming; its string variant keeps the whole value for sorting, faceting,
and exact lookup through the inferred value binding. See
field retrieval
for retrieving a particular variant.
STRING normalization and length
Section titled “STRING normalization and length”normalizer takes filters only: lowercase, nfkc_cf, and fold, using the
same component syntax as analyzer filters. Each input element stays one whole
value. Normalization applies at ingest and to query literals, facet selected
values, and range bounds, so a literal accepted at ingest finds its document
later. Absent normalizer inherits; [] clears it.
Long terms
Section titled “Long terms”IDs, indexed STRING values after normalization, and analyzed TEXT tokens share
a 255-byte term space. Terms of 255 bytes or less stay unchanged. The
long_terms policy says what happens to longer ones; the default, hash128,
keeps long values distinct with no limit on value length:
| Policy | Over-limit term becomes |
|---|---|
hash128 (default) |
The first 230 bytes, backed off to a UTF-8 boundary, followed immediately by 25 base36 characters hashing the whole term. |
truncate |
The first 255 bytes, backed off to a UTF-8 boundary. Values sharing that prefix merge, including IDs for overwrite and delete. |
reject |
Reject documents and requests containing over-limit terms. |
The hash128 suffix is unseeded XXH3_128 of the whole term, taken as one
128-bit number from its canonical 16 bytes (high 64 bits then low 64 bits,
both big-endian) and written in 0-9 and a-z, most significant digit first,
zero padded to 25 digits. A
300-byte value in a _s field therefore facets as its first 230 bytes plus a
25-character tail such as ...131pru8lxm5t1cohchv0tgjtg.
What the policy means at query time:
- Exact queries (
any_of,:=), range bounds, facetselectedvalues, overwrite, and delete apply the same transform as ingest, so full values round-trip. TEXT exact membership applies it to the single analyzed term; match and phrase apply it per analyzed token. - Different long values keep distinct IDs, exact matches, and facet buckets except for hash collisions.
- A prefix query longer than the kept prefix falls back to that prefix and
returns a superset. Wildcard and regex queries do the same when their common
leading literal prefix exceeds the limit; other patterns operate on the
stored term bytes, hash suffix included, so they can miss a long value or
match its hash characters. Fuzzy distance also compares transformed bytes,
so similarity between discarded tails is not measured. When a wildcard,
regex, or fuzzy query can reach a term of 252 bytes or more and a segment
holds one for that field, the response carries a
fitted_termswarning. - The stored source, when enabled, keeps the full bytes before normalization or hashing and is used to return the exact value if its column representation was truncated.
- Column-only strings (
index: "none") have no term-space limit, but cannot serve field facets or exact queries.
Absent long_terms inherits through the parent chain, otherwise defaults to
hash128; an explicit policy overrides inheritance. Variants may set the policy
or borrow it through their own parent; they do not inherit it from their
logical primary. The policy applies only to ID, TEXT, and indexed STRING
representations.
The operations
Section titled “The operations”| Request | Meaning |
|---|---|
GET /collections/{c}/_schema |
the authored schema (valid write body) |
GET /collections/{c}/_schema?view=resolved |
effective representations, bindings, and schema coverage (HTTP only) |
POST /collections/{c}/_schema |
mode=set (default): set the named definitions, keep the rest |
POST /collections/{c}/_schema?mode=replace_all |
replace the whole schema |
POST /collections/_create |
create a collection, optionally with a schema installed before it becomes visible |
The same operations are available over gRPC as luxir.Admin/GetSchema,
SetSchema, and CreateCollection; see the gRPC API.
Resolved view
Section titled “Resolved view”The default GET, also available as ?view=authored, stays sparse: explicit
parents, variants, and defaults are returned without inheritance expansion.
The resolved view expands everything, per physical representation. After
adding the fields in Templates and inheritance,
inspect the authors collection:
{ "generation": 1, "fields": { "inherited": { "bindings": { "search": "inherited", "value": "inherited__s" }, "representations": { "s": { "analyzer": null, "column": true, "coverage_complete": false, "index": "match", "introduced_generation": 1, "long_terms": "hash128", "multi": false, "name": "inherited__s", "normalizer": [], "oldest_generation": 0, "stored": false, "type": "string" }, "self": { "analyzer": { "filters": ["nfkc_cf", "fold"], "tokenizer": "unicode_word" }, "column": false, "coverage_complete": false, "index": "match", "introduced_generation": 1, "long_terms": "hash128", "multi": false, "name": "inherited", "normalizer": null, "oldest_generation": 0, "stored": true, "stored_resource": "_stored_", "type": "text" } } } }}(Abbreviated to the inherited field; the real response lists every declared
field.) The resolved view is diagnostic, not a schema write body. It contains:
| Property | Meaning |
|---|---|
generation |
Current schema generation; the initial persisted schema is generation 1. |
fields |
Declared concrete fields and their variants; suffix-template instances are not listed. |
fields.<f>.bindings |
search and value, each naming the effective physical field. |
fields.<f>.representations |
Definitions keyed by self or variant label. |
| Representation settings | name, type, multi, index, column, stored, analyzer (always the expanded definition; analyzer_preset adds the registered name when one was used), normalizer, effective long_terms for term-backed representations, plus stored_resource and vector settings when applicable. These describe the structures available to operations. |
ann (vector representations) |
Effective ANN settings in the authored vocabulary, including defaults and inherited values. Contains codec, nlist, and only the selected codec’s pq or rabitq object. Zero nlist, pq.m and rabitq.bits retain their automatic policies. ANN-only changes preserve introduction and coverage generations. |
introduced_generation |
Start of this representation’s current continuous availability. Redefining a representation or removing and re-adding its label starts a new introduction. |
oldest_generation |
Minimum schema generation across all committed live segments, even those lacking this field; null for an empty index. |
coverage_complete |
True for an empty index or when that minimum reaches the introduction generation. |
Coverage is derived from segment schema generations, not from counting documents that contain the field. Merges keep the minimum input generation and cannot establish backfill. Deleting some documents from an old segment does not refine its generation.
Changes to a live collection
Section titled “Changes to a live collection”Schema changes apply to new requests and do not rewrite existing documents.
Adding a field or variant leaves older documents without its values. Adding
a variant can also change an inferred binding immediately. For a staged
rollout, explicitly pin the affected binding to its current representation
(for example, "defaults": {"value": "self"}) when adding the variant.
Reindex from your source data, then select the new variant explicitly or set
defaults: {} to restore inference for both bindings. To keep one binding
pinned, retain it in the defaults object and omit only the other. Older
documents without a new author__s variant will not match its exact queries
or appear in its facet buckets.
Schema edits are not checked for compatibility with existing data. To change a field’s type or analysis, use a new field or variant label and reindex. Removing and reusing a name does not erase its old indexed values.
Limits
Section titled “Limits”- Schema edits do not rewrite existing documents; see Changes to a live collection.
- Analyzed text has no value column; use a string variant to sort text values or facet on whole values.
english_possessiveandkstemare analyzer filters only; STRING normalizers acceptlowercase,nfkc_cf, andfold.
Nested paths and qualified field names
Section titled “Nested paths and qualified field names”Declare nested paths and their fields by flat qualified names, for example
"items": "nested" and "items.color": "string" in fields.
Each intermediate component must be a nested path. Global suffix templates
also work under paths: lines_nested.color_s needs no explicit declaration
in the default schema. A nested path cannot declare scalar value settings,
variants, indexing, columns or storage. Objects may contain nested paths
recursively, up to 32 levels.
A nested path’s one setting is reverse_map (default true), described in
Declaring nested paths. It can be changed
on a live collection without reindexing.
Dots separate path components. Each component starts with an ASCII letter
and contains letters, digits or underscores. A physical name, including
a generated __variant suffix, must fit in 127 bytes.
Vector ANN overlays can be built on child fields using their qualified
physical names, for example vec.lines_nested.embedding_v.
Request field names are relative to their view: in the items view, color
resolves to items.color. See Nested documents for indexing,
queries, operations on child rows, and limits.