Elasticsearch Mappings

The ElasticMappingResolver provides intelligent field resolution based on Elasticsearch index mappings. It automatically handles analyzed vs non-analyzed fields, nested documents, and field types.

Overview

When you configure UseMappings(), the parser:

1. Loads field mappings from your Elasticsearch index
2. Resolves field names to their correct paths
3. Automatically uses keyword sub-fields for sorting and aggregations
4. Detects nested fields for proper query wrapping
5. Identifies field types for appropriate query generation

Configuration

From Elasticsearch Client

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index"));

From Type Mapping

var parser = new ElasticQueryParser(c => c
    .UseMappings<MyDocument>(client));

With Custom Mapping Builder

var parser = new ElasticQueryParser(c => c
    .UseMappings<MyDocument>(
        mappingBuilder: m => m
            .Properties(p => p
                .Text(n => n.Title, t => t
                    .Fields(f => f.Keyword("keyword")))
                .Keyword(n => n.Status)
                .Date(n => n.Created)
                .Nested(x => x.Comments, n => n.Properties(np => np))),
        client,
        "my-index"));

From Mapping Function

var parser = new ElasticQueryParser(c => c
    .UseMappings(
        getMapping: () => GetCachedMapping(),
        inferrer: client.Infer));

Field Resolution

Automatic Keyword Field Detection

For text fields with keyword sub-fields, the resolver automatically uses the keyword field for:

• Sorting
• Aggregations
• Exact match queries

// Mapping:
// "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }

var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index"));

// For queries - uses analyzed "title" field
var query = await parser.BuildQueryAsync("title:search terms");

// For aggregations - automatically uses "title.keyword"
var aggs = await parser.BuildAggregationsAsync("terms:title");

// For sort - automatically uses "title.keyword"
var sort = await parser.BuildSortAsync("title");

Field Type Detection

The resolver detects field types for appropriate query handling:

var resolver = parser.Configuration.MappingResolver;

// Check field types
bool isNested = resolver.IsNestedPropertyType("comments");
bool isGeo = resolver.IsGeoPropertyType("location");
bool isNumeric = resolver.IsNumericPropertyType("price");
bool isDate = resolver.IsDatePropertyType("created");
bool isBoolean = resolver.IsBooleanPropertyType("active");
bool isAnalyzed = resolver.IsPropertyAnalyzed("description");

ElasticMappingResolver API

Getting Field Information

var resolver = parser.Configuration.MappingResolver;

// Get full field mapping
var mapping = resolver.GetMapping("user.name");
if (mapping.Found)
{
    Console.WriteLine($"Full path: {mapping.FullPath}");
    Console.WriteLine($"Property type: {mapping.Property?.GetType().Name}");
}

// Get the Elasticsearch property
IProperty property = resolver.GetMappingProperty("status");

// Get resolved field name
string resolved = resolver.GetResolvedField("user");

// Get non-analyzed field for sorting
string sortField = resolver.GetSortFieldName("title");

// Get non-analyzed field for aggregations
string aggField = resolver.GetAggregationsFieldName("category");

// Get field type enum
FieldType fieldType = resolver.GetFieldType("price");

Field Type Enum

GetFieldType returns a FieldType enum with values such as:

• Text: text, match_only_text, search_as_you_type (legacy string also supported)
• Keyword: keyword, constant_keyword, wildcard
• Numeric: long, unsigned_long, integer, short, byte, double, float, half_float, scaled_float
• Date: date, date_nanos
• Range: integer_range, float_range, long_range, double_range, date_range, ip_range
• Geo: geo_point, geo_shape, point, shape
• Structured: nested, object, flattened, join
• Other: boolean, ip, binary, completion, murmur3, token_count, percolator, alias, rank_feature, rank_features, histogram, dense_vector, version

Unrecognized types return FieldType.None.

FieldType fieldType = resolver.GetFieldType("price");
if (fieldType == FieldType.Double || fieldType == FieldType.Float)
{
    // Handle numeric field
}

Nested Document Handling

For a detailed explanation of how visitors traverse nested query structures, field scoping rules, and the full AST breakdown, see Nested Queries and Visitor Traversal.

Automatic Nested Query Wrapping

When UseNested() is enabled, queries on nested fields are automatically wrapped. This includes individual field queries, grouped queries, negated groups, exists/missing queries, wildcard queries, aggregations, and sorting:

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .UseNested());

// Query on nested field
var query = await parser.BuildQueryAsync("comments.author:john");

// Automatically generates:
// {
//   "nested": {
//     "path": "comments",
//     "query": {
//       "term": { "comments.author": "john" }
//     }
//   }
// }

// Negated nested groups also produce correct structure
query = await parser.BuildQueryAsync("NOT comments:(comments.author:spammer)");

// Exists/missing on nested fields are wrapped automatically
query = await parser.BuildQueryAsync("_exists_:comments.author");

Nested Field Detection

var resolver = parser.Configuration.MappingResolver;

// Check if field is nested
bool isNested = resolver.IsNestedPropertyType("comments");

// Get the nested path for a field
// "comments.author" -> "comments"

Filtered Nested Queries

When multiple logical types share a nested array, use UseNestedFilter() to inject a discriminator filter:

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .UseNestedFilter((nestedPath, originalField, resolvedField, context) =>
    {
        if (nestedPath is "comments")
            return new TermQuery { Field = "comments.type", Value = "public" };

        return null;
    })
    .UseNested());

// Query: comments.author:john
// Produces:
// {
//   "nested": {
//     "path": "comments",
//     "query": {
//       "bool": {
//         "must": [
//           { "term": { "comments.author": "john" } },
//           { "term": { "comments.type": "public" } }
//         ]
//       }
//     }
//   }
// }

// Aggregation: max:comments.rating
// Produces: nested > filter(comments.type=public) > max(comments.rating)

// Sort: -comments.rating
// Produces: sort with nested(path=comments, filter=term(comments.type, public))

Mapping Extensions

Adding Keyword Sub-Fields

Use extension methods to add standard sub-fields to your mappings:

using Foundatio.Parsers.ElasticQueries.Extensions;

var createIndexResponse = await client.Indices.CreateAsync("my-index", c => c
    .Map<MyDocument>(m => m
        .Properties(p => p
            // Add .keyword sub-field
            .Text(n => n.Title, t => t.AddKeywordField())

            // Add .sort sub-field with lowercase normalizer
            .Text(n => n.Name, t => t.AddSortField())


            // Add both .keyword and .sort sub-fields
            .Text(n => n.Description, t => t.AddKeywordAndSortFields())
        )));

Sub-Field Names

using Foundatio.Parsers.ElasticQueries.Extensions;

// Default sub-field names
string keywordField = ElasticMappingExtensions.KeywordFieldName; // "keyword"
string sortField = ElasticMappingExtensions.SortFieldName;       // "sort"

Sort Normalizer

Add a lowercase normalizer for case-insensitive sorting:

var createIndexResponse = await client.Indices.CreateAsync("my-index", c => c
    .Settings(s => s.Analysis(a => a.AddSortNormalizer()))
    .Map<MyDocument>(m => m
        .Properties(p => p
            .Text(n => n.Name, t => t.AddSortField())
        )));

Refreshing Mappings

The resolver loads the server mapping on first use. After that, an unresolved field is the signal that the mapping may have changed, so it can trigger a rate-limited reload:

Trigger	Setting	Default	Behavior
A field could not be resolved	UnmappedFieldRefreshInterval	5 seconds	Minimum time between completed miss-driven reload attempts. Must be greater than zero.
A reload is already running	MappingRefreshWaitTimeout	30 seconds	How long a resolution waits to join an in-flight reload instead of issuing its own.

A field that cannot be resolved is the normal outcome for fields created at runtime — dynamic templates (including the idx.* custom field templates used by Foundatio.Repositories) only add a field to the index mapping after the first document that uses it is indexed. Because of that, an unresolved field reloads the server mapping on its own short interval and is never blocked by the mapping having been loaded at startup.

Concurrent reload attempts are coalesced into a single server mapping fetch per resolver. This protection is local to the resolver instance, so create one long-lived resolver per concrete index and process. Creating a resolver per request defeats both caching and request coalescing. With the five-second default, each resolver that is continuously receiving misses can make up to about 12 reload attempts per minute, excluding cold starts and explicit refreshes. There is no distributed cache or cross-process coordination, so this ceiling applies independently to each resolver in each process.

Before this miss-specific cooldown, successful and null mapping fetches were normally limited by the one-minute mapping refresh interval, while a callback exception could be retried by every sequential miss. The five-second default therefore improves dynamic-field discovery while placing the same finite ceiling on successful, null, and exceptional automatic reloads.

var resolver = parser.Configuration.MappingResolver;

// Reduce reload pressure for typo-heavy workloads.
resolver.UnmappedFieldRefreshInterval = TimeSpan.FromSeconds(30);

Waiting For An In-Flight Reload

Only one automatic mapping reload runs at a time. Other resolutions that need a fresh mapping wait for that reload rather than issuing their own. An explicit RefreshMapping() supersedes an obsolete in-flight load, so the next resolution can fetch the replacement mapping without waiting for the old callback to drain. That hard-invalidation case can temporarily overlap the obsolete physical callback. If the wait exceeds MappingRefreshWaitTimeout, the resolution gives up and treats the field as unmapped, and a warning is logged. A single resolution joins at most once; it does not repeat the same wait as both an initial load and a miss-driven reload.

The mapping callback must have its own finite timeout and must not call back into the same resolver. The built-in client factories use the Elasticsearch client's configured request timeout. Configure that timeout below MappingRefreshWaitTimeout if every joining resolution must observe the fetch result; otherwise a joining resolution can time out while the single fetch continues.

resolver.MappingRefreshWaitTimeout = TimeSpan.FromMinutes(2);

Residual Staleness

The five-second default is a cooldown, not a correctness deadline. Callback latency, failures, and a reload that began before the field was created can extend the stale window. If an automatic reload fails or returns no mapping, the resolver retains its last known mapping and throttles the next attempt. During that window an unresolved field is still treated as unmapped. Set AllowUnresolvedFields to false when failing validation is safer than generating a query from incomplete mapping information. This catches unresolved query fields, including default fields actually used by unfielded terms and included queries. Unused default fields and configured restricted fields are not treated as query references. Mapping validation runs after field resolution and reports original query names, including aliases, even when a visitor context is reused. Runtime fields defined in the context or discovered by its runtime-field resolver remain valid. Each operation reuses runtime lookup results, including misses and failures, so field resolution and validation do not invoke the callback twice for the same field. A later operation can retry the callback. The resolver's boolean type checks do not expose a separate "mapping unknown" state, so use explicit refresh for application-controlled changes that require immediate correctness.

Successful field resolutions do not trigger periodic reloads. New fields and sub-fields create misses and are discovered automatically, but changes to an already resolved alias or runtime-field definition require an explicit refresh.

Forcing a Full Refresh

RefreshMapping() bypasses the miss throttle and discards the current mapping snapshot. Call it after Elasticsearch acknowledges an application-controlled mapping change; the next resolution fetches or joins the new mapping.

resolver.RefreshMapping();

Do not call RefreshMapping() after every indexed document. Dynamically materialized fields already use the miss-driven reload path, while repeated full invalidation discards useful caches and can continually supersede in-flight fetches. Foundatio.Repositories should keep its long-lived per-index resolver and rely on that path for ordinary custom-field saves.

Cache Memory

Successful resolutions are cached by canonical field path for the lifetime of the mapping snapshot, which keeps the cache bounded by the mapping itself no matter how many distinct spellings callers ask for. Unknown field names are not cached in the long-lived snapshot, so caller-controlled misses cannot grow process state. A parser operation remembers its own positive and negative resolutions only until that operation finishes, preventing downstream visitors from repeating the same network-backed lookup. Direct query, sort, and aggregation helpers also release their temporary results on return or failure, so reusing a visitor context does not retain stale mappings across operations. Cache keys preserve exact field-name casing. Reloading the mapping publishes a new snapshot and atomically discards resolutions derived from the old one, unless the optional revision check below confirms that the mapping is unchanged. The operation's runtime-field result dictionary is allocated only when a runtime-field resolver is actually used.

Reusing Unchanged Mappings

If your mapping owner provides a cheap, reliable revision for the complete mapping, set ServerMappingRevisionResolver before sharing the resolver:

resolver.ServerMappingRevisionResolver = mapping =>
    mapping.Meta?.TryGetValue("resolver_revision", out var revision) == true
        ? revision?.ToString()
        : null;

Equal, nonempty revisions allow automatic reloads to reuse the merged property tree and successful field resolutions. The reload still fetches the mapping and observes the normal miss cooldown; this saves local rebuilding and allocations, not HTTP requests or deserialization. No mapping serialization is performed to compare revisions. RefreshMapping() always discards the cached state, even if the revision is unchanged.

The revision must identify the concrete index and change whenever any mapping detail changes, including fields added dynamically. Elasticsearch does not maintain this custom _meta value for you. Use it only when your mapping owner guarantees that contract; a deployment schema version or latest partition name alone is insufficient. Leave the callback unset if you do not have such a revision. Null or empty revisions also retain the default behavior of rebuilding after every successful reload.

The built-in client factories expect one concrete index, or a target whose indices have equivalent mappings. They do not merge heterogeneous mappings from rollover aliases or data streams.

Custom Mapping Resolver

Use an asynchronous loader when obtaining the mapping requires network I/O. The parser's asynchronous query, sort, and aggregation paths await this loader without blocking a request thread. Cancelling one resolver lookup cancels only that caller's wait; the shared fetch continues for other callers and is bounded by the resolver lifetime and the loader's transport timeout.

The application owns the resolver, including one created by UseMappingsWithAsyncLoader: dispose parser.Configuration.MappingResolver when the parser is no longer in use. Do not dispose it per request when it is shared. Configure refresh settings before sharing the resolver. Typed Field helpers preserve inferred canonical names while using the same single-load budget as string helpers.

Custom asynchronous loaders use the named CreateWithAsyncLoader and UseMappingsWithAsyncLoader entry points. Keeping them separate from the synchronous delegate overloads preserves source compatibility for existing callers.

var customResolver = ElasticMappingResolver.CreateWithAsyncLoader(
    getMappingAsync: cancellationToken => LoadMappingAsync(cancellationToken),
    inferrer: client.Infer,
    logger: logger);

Synchronous loader overloads remain available for compatibility. An asynchronous parser call configured with a synchronous loader must invoke that loader synchronously, and an explicit synchronous resolver call configured with only an asynchronous loader waits synchronously for it. That compatibility path dispatches the single shared load to the thread pool to avoid synchronization-context deadlocks, but the caller still blocks. Synchronous Parse preserves the caller thread for synchronous visitors; server request paths should use the asynchronous resolver and parser APIs. The built-in Elasticsearch client factories supply both forms: asynchronous parser paths call Indices.GetMappingAsync, while synchronous resolver calls retain Indices.GetMapping.

On the default task scheduler, the asynchronous loader starts inline and does not use Task.Run. When a custom synchronization context or task scheduler is active, only the loader boundary is dispatched to the thread pool so a synchronous caller joining the same shared fetch cannot block the loader's captured continuation. Library-owned awaits still use AnyContext().

The loaded mapping and its successful field memoization belong to an immutable resolver-local snapshot; an external cache client is neither required nor used. RefreshMapping() remains synchronous because it only invalidates that local snapshot and performs no I/O.

For an already synchronous or in-memory source, use the compatibility overload:

var customResolver = ElasticMappingResolver.Create(
    getMapping: () => {
        // Return cached or custom mapping
        return _cachedMapping;
    },
    inferrer: client.Infer,
    logger: logger);

var parser = new ElasticQueryParser(c => c
    .UseMappings(customResolver));

Field Mapping Structure

The FieldMapping class contains:

public class FieldMapping
{
    // Whether the field was found in mappings
    public bool Found { get; }

    // The full resolved path (e.g., "data.user.name")
    public string FullPath { get; }

    // The Elasticsearch IProperty for the field
    public IProperty Property { get; }
}

Best Practices

1. Use Consistent Sub-Field Naming

// Always use .keyword for exact matching
// Always use .sort for case-insensitive sorting
.Text(n => n.Title, t => t
    .Fields(f => f
        .Keyword("keyword", k => k.IgnoreAbove(256))
        .Keyword("sort", k => k.Normalizer("lowercase"))))

2. Cache Mapping Resolution

The resolver caches mappings automatically, but you can also:

// Create resolver once and reuse
var resolver = ElasticMappingResolver.Create(client, "my-index");

var parser1 = new ElasticQueryParser(c => c.UseMappings(resolver));
var parser2 = new ElasticQueryParser(c => c.UseMappings(resolver));

3. Handle Dynamic Mappings

For correctness-sensitive indices with dynamic mappings, fail validation if a field is still unresolved after the bounded reload attempt rather than generating a query from incomplete mapping information:

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .SetValidationOptions(new QueryValidationOptions {
        AllowUnresolvedFields = false
    }));

4. Log Mapping Issues

var parser = new ElasticQueryParser(c => c
    .SetLoggerFactory(loggerFactory)
    .UseMappings(client, "my-index"));

// Mapping resolution issues will be logged

Troubleshooting

Field Not Found

var resolver = parser.Configuration.MappingResolver;
var mapping = resolver.GetMapping("unknown_field");

if (!mapping.Found)
{
    // Field doesn't exist in mapping
    // Check: spelling, case sensitivity, nested path
}

Wrong Field Type Used

// Check what type the resolver sees
var fieldType = resolver.GetFieldType("my_field");
Console.WriteLine($"Field type: {fieldType}");

// Check if analyzed
bool isAnalyzed = resolver.IsPropertyAnalyzed("my_field");
Console.WriteLine($"Is analyzed: {isAnalyzed}");

Nested Queries Not Working

// Ensure UseNested() is configured
var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .UseNested());  // Required for nested support

// Verify field is detected as nested
bool isNested = resolver.IsNestedPropertyType("comments");

Next Steps

• Nested Queries and Visitor Traversal - How visitors handle nested document queries
• Elasticsearch Integration - Full parser guide
• Query Syntax - Query syntax reference
• Aggregation Syntax - Aggregation reference