--- url: /aggregations.md --- # Aggregation Syntax Examples: * Single metric: `min:field` * Multiple metric: `min:field max:field` * Nested bucketed: `terms:(field min:field max:field)` * Example result: ```json { "aggregations": { "terms_field1": { "buckets": [ { "key": "value1", "doc_count": 102, "min_field2": { "value": 2 }, "max_field2": { "value": 30 } }, { "key": "value2", "doc_count": 76, "min_field2": { "value": 0 }, "max_field2": { "value": 87 } } ] } } } ``` * Multiple levels nested: `terms:(field1 min:field2 max:field2 terms:(field2 min:field2 max:field2))` * Terms sorted by nested max: `terms:(field min:field +max:field)` # Metric Aggregations The aggregations in this family compute metrics based on values extracted in one way or another from the documents that are being aggregated. ## `min` A single-value metrics aggregation that keeps track and returns the minimum value among numeric values extracted from the aggregated documents. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `min:field` * Missing value: `min:field~0` ## `max` A single-value metrics aggregation that keeps track and returns the maximum value among the numeric values extracted from the aggregated documents. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `max:field` * Missing value: `max:field~0` ## `avg` A single-value metrics aggregation that computes the average of numeric values that are extracted from the aggregated documents. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `avg:field` * Missing value: `avg:field~1` ## `sum` A single-value metrics aggregation that sums up numeric values that are extracted from the aggregated documents. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `sum:field` * Missing value: `sum:field~1` ## `stats` A multi-value metrics aggregation that computes stats over numeric values extracted from the aggregated documents. The stats that are returned consist of: `min`, `max`, `sum`, `count` and `avg`. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `stats:field` * Missing value: `stats:field~0` ## `exstats` A multi-value metrics aggregation that computes stats over numeric values extracted from the aggregated documents. The `exstats` aggregations is an extended version of the `stats` aggregation, where additional metrics are added such as `sum_of_squares`, `variance`, `std_deviation` and `std_deviation_bounds`. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `exstats:field` * Missing value: `exstats:field~0` ## `cardinality` A single-value metrics aggregation that calculates an approximate count of distinct values. Modifiers: * `~` Sets the value to use when documents are missing a value. Examples: * Basic: `cardinality:field` * Missing value: `cardinality:field~0` ## `percentiles` A multi-value metrics aggregation that calculates one or more percentiles over numeric values extracted from the aggregated documents. Percentiles show the point at which a certain percentage of observed values occur. For example, the 95th percentile is the value which is greater than 95% of the observed values. Percentiles are often used to find outliers. In normal distributions, the 0.13th and 99.87th percentiles represents three standard deviations from the mean. Any data which falls outside three standard deviations is often considered an anomaly. When a range of percentiles are retrieved, they can be used to estimate the data distribution and determine if the data is skewed, bimodal, etc. Assume your data consists of website load times. The average and median load times are not overly useful to an administrator. The max may be interesting, but it can be easily skewed by a single slow response. By default, the percentile metric will generate a range of percentiles: `1, 5, 25, 50, 75, 95, 99` Modifiers: * `~` Sets a list of percentiles to override the defaults. Percentiles are `,` delimited. Examples: * Basic: `percentiles:field` * Custom percentile buckets: `percentiles:field~25,50,75` # Bucketed Aggregations Bucket aggregations don’t calculate metrics over fields like the metrics aggregations do, but instead, they create buckets of documents. Each bucket is associated with a criterion (depending on the aggregation type) which determines whether or not a document in the current context "falls" into it. In other words, the buckets effectively define document sets. In addition to the buckets themselves, the bucket aggregations also compute and return the number of documents that "fell into" each bucket. Bucket aggregations, as opposed to metrics aggregations, can hold sub-aggregations. These sub-aggregations will be aggregated for the buckets created by their "parent" bucket aggregation. ## `histogram` A multi-bucket values source based aggregation that can be applied on numeric values extracted from the documents. It dynamically builds fixed size (a.k.a. interval) buckets over the values. For example, if the documents have a field that holds a price (numeric), we can configure this aggregation to dynamically build buckets with interval 5 (in case of price it may represent $5). When the aggregation executes, the price field of every document will be evaluated and will be rounded down to its closest bucket - for example, if the price is 32 and the bucket size is 5 then the rounding will yield 30 and thus the document will "fall" into the bucket that is associated with the key 30. Modifiers: * `~` Sets the interval. Must be a positive decimal. Examples: * Basic: `histogram:field` * Interval of 5: `histogram:field~5` ## `date` A multi-bucket aggregation similar to the histogram except it can only be applied on date values. Modifiers: * `~` Sets the interval. Available expressions are: `year`, `quarter`, `month`, `week`, `day`, `hour`, `minute`, `second` * Time values can also be specified via abbreviations. `d` = days, `h` = hours, `m` = minutes, `s` = seconds, `ms` = milliseconds. Examples: `90m`, `1d` * `^` Sets the time zone. Time zones may either be specified as an ISO 8601 UTC offset (e.g. `+01:00` or `-08:00`) or as a timezone id, an identifier used in the TZ database like `America/Los_Angeles`. * `@missing` Value to use when documents are missing a value for the field. * `@offset` The offset parameter is used to change the start value of each bucket by the specified positive (+) or negative offset (-) duration, such as `1h` for an hour, or `1d` for a day. Examples: * Basic: `date:field` * 1 hour interval: `date:field~1h` * Year interval: `date:field~year` * 2h timezone: `date:field^2h` * 1 hour interval and -5h timezone: `date:field~year^-5h` * 1 hour offset: `date:(field @offset:1h)` ## `geogrid` A multi-bucket aggregation that works on geo fields and groups points into buckets that represent cells in a grid. The resulting grid can be sparse and only contains cells that have matching data. Each cell is labeled using a geohash which is of user-definable precision. High precision geohashes have a long string length and represent cells that cover only a small area. Low precision geohashes have a short string length and represent cells that each cover a large area. Geohashes used in this aggregation can have a choice of precision between 1 and 12. Modifiers: * `~` Sets the precision. Must be a number between 1 and 12. Examples: * Basic: `geogrid:field` * Precision of 5: `geogrid:field~5` ## `terms` A multi-bucket value source based aggregation where buckets are dynamically built - one per unique value. Modifiers: * `~` Sets the size to define how many term buckets should be returned out of the overall terms list. * `^` Sets the minimum number of matching documents that must exist for the term to be included. * `@include` Terms that match this pattern will be included in the result. * `@exclude` Terms that match this pattern will be excluded from the result. * `@missing` Value to use when documents are missing a value for the field. * `@min` Sets the minimum number of matching documents that must exist for the term to be included. Examples: * Basic: `terms:field` * Return top 5 terms: `terms:field~5` * Return terms excluding "value": `terms:(field @exclude:value)` * Sorted descending by nested max: `terms:(field -max:field)` ## `missing` A field data based single bucket aggregation, that creates a bucket of all documents in the current document set context that are missing a field value (effectively, missing a field or having the configured NULL value set). This aggregator will often be used in conjunction with other field data bucket aggregators (such as ranges) to return information for all the documents that could not be placed in any of the other buckets due to missing field data values. Examples: * Basic: `missing:field` # Other Aggregations ## `tophits` Returns a list of the top hits for the parent aggregation. Since a field name is not used with this aggregation, the specified field name (`tophits:_`) will be ignored. Modifiers: * `~` Sets the size to define how many term buckets should be returned out of the overall terms list. * `@include` Will include the specified field in the resulting top hits document. * `@exclude` Will exclude the specified field in the resulting top hits document. Examples: * Basic: `terms:(field tophits:_)` * Return terms excluding "value": `terms:(field tophits:(_ @exclude:value))` --- --- url: /guide/aggregation-syntax.md --- # Aggregation Syntax Foundatio.Parsers supports dynamic aggregation expressions that compile to Elasticsearch aggregations. This enables end users to build custom analytics, charts, and dashboards. ## Overview Aggregation expressions follow a simple syntax: ``` operation:field operation:(field modifiers subaggregations) ``` ### Quick Examples ```csharp using Foundatio.Parsers.ElasticQueries; var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // Single metric var aggs = await parser.BuildAggregationsAsync("min:price"); // Multiple metrics aggs = await parser.BuildAggregationsAsync("min:price max:price avg:price"); // Nested bucket with metrics aggs = await parser.BuildAggregationsAsync("terms:(category min:price max:price)"); ``` ## Metric Aggregations Metric aggregations compute values from document fields. ### min Returns the minimum value among numeric values. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` min:field min:field~0 ``` **Example:** ```csharp var aggs = await parser.BuildAggregationsAsync("min:price"); var aggs = await parser.BuildAggregationsAsync("min:price~0"); // Use 0 for missing ``` ### max Returns the maximum value among numeric values. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` max:field max:field~0 ``` ### avg Computes the average of numeric values. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` avg:field avg:field~0 ``` ### sum Sums numeric values. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` sum:field sum:field~0 ``` ### stats Multi-value aggregation returning `min`, `max`, `sum`, `count`, and `avg`. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` stats:field stats:field~0 ``` **Response structure:** ```json { "stats_price": { "count": 100, "min": 10.0, "max": 500.0, "avg": 125.5, "sum": 12550.0 } } ``` ### exstats Extended stats including `sum_of_squares`, `variance`, `std_deviation`, and `std_deviation_bounds`. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` exstats:field exstats:field~0 ``` ### cardinality Approximate count of distinct values. | Modifier | Description | |----------|-------------| | `~` | Value for missing documents | ``` cardinality:field cardinality:field~0 ``` ### percentiles Calculates percentiles over numeric values. | Modifier | Description | |----------|-------------| | `~` | Comma-separated percentile values (default: 1, 5, 25, 50, 75, 95, 99) | ``` percentiles:field percentiles:field~25,50,75,90,99 ``` **Example:** ```csharp // Default percentiles var aggs = await parser.BuildAggregationsAsync("percentiles:response_time"); // Custom percentiles aggs = await parser.BuildAggregationsAsync("percentiles:response_time~50,90,95,99"); ``` ## Bucket Aggregations Bucket aggregations group documents into buckets. They can contain sub-aggregations. ### terms Creates buckets for each unique value. | Modifier | Description | |----------|-------------| | `~` | Number of buckets to return (size) | | `^` | Minimum document count | | `@include` | Include pattern | | `@exclude` | Exclude pattern | | `@missing` | Value for missing documents | | `@min` | Minimum document count | **Sorting:** * `+field` - Sort ascending by nested aggregation * `-field` - Sort descending by nested aggregation ``` terms:field terms:field~10 terms:(field @exclude:value) terms:(field -max:amount) ``` **Examples:** ```csharp // Top 10 categories var aggs = await parser.BuildAggregationsAsync("terms:category~10"); // Exclude specific values aggs = await parser.BuildAggregationsAsync("terms:(status @exclude:deleted)"); // With nested metrics, sorted by max amount descending aggs = await parser.BuildAggregationsAsync("terms:(category -max:amount min:amount)"); // Include only matching patterns aggs = await parser.BuildAggregationsAsync("terms:(status @include:active* @include:pending)"); ``` **Response structure:** ```json { "terms_category": { "buckets": [ { "key": "electronics", "doc_count": 150, "max_amount": { "value": 999.99 }, "min_amount": { "value": 9.99 } }, { "key": "clothing", "doc_count": 120, "max_amount": { "value": 299.99 }, "min_amount": { "value": 19.99 } } ] } } ``` ### date Date histogram aggregation for time-series data. | Modifier | Description | |----------|-------------| | `~` | Interval: `year`, `quarter`, `month`, `week`, `day`, `hour`, `minute`, `second`, or duration (`1h`, `30m`, `1d`) | | `^` | Timezone (e.g., `+01:00`, `America/Los_Angeles`) | | `@missing` | Value for missing documents | | `@offset` | Bucket offset (e.g., `+6h`, `-1d`) | ``` date:field date:field~month date:field~1h date:field~day^America/New_York date:(field~week @offset:1d) ``` **Examples:** ```csharp // Daily buckets var aggs = await parser.BuildAggregationsAsync("date:created~day"); // Hourly buckets with timezone aggs = await parser.BuildAggregationsAsync("date:created~1h^America/New_York"); // Monthly with nested metrics aggs = await parser.BuildAggregationsAsync("date:(created~month min:amount max:amount sum:amount)"); // With offset (start buckets at 6am instead of midnight) aggs = await parser.BuildAggregationsAsync("date:(created~day @offset:6h)"); ``` ### histogram Numeric histogram with fixed intervals. | Modifier | Description | |----------|-------------| | `~` | Interval (positive decimal) | ``` histogram:field histogram:field~10 histogram:field~0.5 ``` **Examples:** ```csharp // Price ranges in $10 increments var aggs = await parser.BuildAggregationsAsync("histogram:price~10"); // With nested metrics aggs = await parser.BuildAggregationsAsync("histogram:(price~50 avg:quantity)"); ``` ### geogrid Geohash grid aggregation for geographic data. | Modifier | Description | |----------|-------------| | `~` | Precision (1-12, higher = smaller cells) | ``` geogrid:field geogrid:field~5 ``` **Examples:** ```csharp // Default precision var aggs = await parser.BuildAggregationsAsync("geogrid:location"); // Higher precision (smaller cells) aggs = await parser.BuildAggregationsAsync("geogrid:location~7"); ``` ### missing Creates a bucket for documents missing a field value. ``` missing:field ``` **Example:** ```csharp // Count documents without a category var aggs = await parser.BuildAggregationsAsync("missing:category"); ``` ## Other Aggregations ### tophits Returns top documents for each bucket. Use `_` as the field name. | Modifier | Description | |----------|-------------| | `~` | Number of hits to return | | `@include` | Fields to include in results | | `@exclude` | Fields to exclude from results | ``` tophits:_ tophits:_~5 tophits:(_ @include:title @include:date) tophits:(_ @exclude:large_field) ``` **Examples:** ```csharp // Top 3 documents per category var aggs = await parser.BuildAggregationsAsync("terms:(category tophits:_~3)"); // With field filtering aggs = await parser.BuildAggregationsAsync("terms:(category tophits:(_ ~5 @include:title @include:price))"); ``` ## Nested Aggregations Bucket aggregations can contain sub-aggregations: ```csharp // Terms with metrics var aggs = await parser.BuildAggregationsAsync("terms:(category min:price max:price avg:price)"); // Multiple levels of nesting aggs = await parser.BuildAggregationsAsync( "terms:(category terms:(subcategory min:price max:price))"); // Date histogram with terms breakdown aggs = await parser.BuildAggregationsAsync( "date:(created~month terms:(status count:_))"); ``` **Response structure for nested aggregations:** ```json { "terms_category": { "buckets": [ { "key": "electronics", "doc_count": 150, "min_price": { "value": 9.99 }, "max_price": { "value": 999.99 }, "avg_price": { "value": 249.99 } } ] } } ``` ## Sorting Bucket Aggregations Sort buckets by nested metric aggregations: | Prefix | Direction | |--------|-----------| | `+` | Ascending | | `-` | Descending | ```csharp // Sort by max price descending var aggs = await parser.BuildAggregationsAsync("terms:(category -max:price)"); // Sort by min price ascending aggs = await parser.BuildAggregationsAsync("terms:(category +min:price)"); // Sort by count (default is descending by doc_count) aggs = await parser.BuildAggregationsAsync("terms:(category~10)"); ``` ## Field Aliases Aggregations support [field aliases](./field-aliases): ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseFieldMap(new Dictionary { { "user", "data.user.identity" }, { "amount", "transaction.amount" } })); // Uses aliased fields var aggs = await parser.BuildAggregationsAsync("terms:(user sum:amount)"); ``` ## Complete Example ```csharp using Foundatio.Parsers.ElasticQueries; using Nest; var client = new ElasticClient(); var parser = new ElasticQueryParser(c => c .SetLoggerFactory(loggerFactory) .UseMappings(client, "orders")); // Build complex aggregation var aggs = await parser.BuildAggregationsAsync(@" date:(created~month terms:(category~5 sum:amount avg:amount cardinality:customer_id ) sum:amount ) "); // Execute search with aggregations var response = await client.SearchAsync(s => s .Index("orders") .Size(0) // Only aggregations, no hits .Query(q => q.Range(r => r.Field(f => f.Created).GreaterThan("now-1y"))) .Aggregations(aggs)); // Process results foreach (var monthBucket in response.Aggregations.DateHistogram("date_created").Buckets) { Console.WriteLine($"Month: {monthBucket.KeyAsString}"); foreach (var categoryBucket in monthBucket.Terms("terms_category").Buckets) { var sum = categoryBucket.Sum("sum_amount").Value; var avg = categoryBucket.Average("avg_amount").Value; var uniqueCustomers = categoryBucket.Cardinality("cardinality_customer_id").Value; Console.WriteLine($" {categoryBucket.Key}: ${sum:N2} total, ${avg:N2} avg, {uniqueCustomers} customers"); } } ``` ## Validation Validate aggregation expressions: ```csharp var parser = new ElasticQueryParser(c => c .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "category", "price", "created" }, AllowedOperations = { "terms", "date", "min", "max", "avg" } })); var result = await parser.ValidateAggregationsAsync("terms:(category min:price)"); if (!result.IsValid) { Console.WriteLine($"Invalid: {result.Message}"); } ``` ## Next Steps * [Query Syntax](./query-syntax) - Query expression reference * [Elasticsearch Integration](./elastic-query-parser) - Full ElasticQueryParser guide * [Validation](./validation) - Restrict allowed operations --- --- url: /guide/custom-visitors.md --- # Custom Visitors Create custom visitors to implement specialized query transformations, validations, or data extraction. ## Creating a Basic Visitor ### Non-Mutating Visitor Extend `QueryNodeVisitorBase` for read-only traversal: ```csharp using Foundatio.Parsers.LuceneQueries.Nodes; using Foundatio.Parsers.LuceneQueries.Visitors; public class FieldCollectorVisitor : QueryNodeVisitorBase { public HashSet Fields { get; } = new(); public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { if (!string.IsNullOrEmpty(node.Field)) Fields.Add(node.Field); return Task.CompletedTask; } public override Task VisitAsync(TermRangeNode node, IQueryVisitorContext context) { if (!string.IsNullOrEmpty(node.Field)) Fields.Add(node.Field); return Task.CompletedTask; } public override Task VisitAsync(ExistsNode node, IQueryVisitorContext context) { if (!string.IsNullOrEmpty(node.Field)) Fields.Add(node.Field); return Task.CompletedTask; } } ``` Usage: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("status:active AND created:[2024-01-01 TO 2024-12-31]"); var visitor = new FieldCollectorVisitor(); await visitor.AcceptAsync(result, new QueryVisitorContext()); foreach (var field in visitor.Fields) { Console.WriteLine($"Field: {field}"); } // Output: status, created ``` ### Mutating Visitor Extend `MutatingQueryNodeVisitorBase` to modify nodes: ```csharp public class FieldPrefixVisitor : MutatingQueryNodeVisitorBase { private readonly string _prefix; public FieldPrefixVisitor(string prefix) { _prefix = prefix; } public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { if (!string.IsNullOrEmpty(node.Field) && !node.Field.StartsWith(_prefix)) { node.Field = _prefix + node.Field; } return Task.CompletedTask; } public override Task VisitAsync(TermRangeNode node, IQueryVisitorContext context) { if (!string.IsNullOrEmpty(node.Field) && !node.Field.StartsWith(_prefix)) { node.Field = _prefix + node.Field; } return Task.CompletedTask; } } ``` Usage: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("status:active"); var visitor = new FieldPrefixVisitor("data."); await visitor.AcceptAsync(result, new QueryVisitorContext()); string query = await GenerateQueryVisitor.RunAsync(result); // Output: "data.status:active" ``` ## Chainable Visitors For use with `ElasticQueryParser` or `ChainedQueryVisitor`, implement `IChainableQueryVisitor`: ```csharp public class CustomFilterVisitor : ChainableQueryVisitor { public override async Task VisitAsync(GroupNode node, IQueryVisitorContext context) { // Process group nodes if (node.Field == "@custom") { // Custom processing logic await ProcessCustomFilter(node, context); } // Continue traversal to child nodes await base.VisitAsync(node, context); } private Task ProcessCustomFilter(GroupNode node, IQueryVisitorContext context) { // Implementation return Task.CompletedTask; } } ``` ### Adding to ElasticQueryParser ```csharp var parser = new ElasticQueryParser(c => c .AddVisitor(new CustomFilterVisitor(), priority: 100)); var query = await parser.BuildQueryAsync("@custom:(filter)"); ``` ## Visitor with Result Return a value from traversal: ```csharp public class QueryComplexityVisitor : QueryNodeVisitorWithResultBase { private int _complexity = 0; public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { _complexity += 1; return Task.CompletedTask; } public override Task VisitAsync(TermRangeNode node, IQueryVisitorContext context) { _complexity += 2; // Ranges are more complex return Task.CompletedTask; } public override Task VisitAsync(GroupNode node, IQueryVisitorContext context) { _complexity += 1; return base.VisitAsync(node, context); } public override Task AcceptAsync(IQueryNode node, IQueryVisitorContext context) { _complexity = 0; return base.AcceptAsync(node, context); } protected override int GetResult() { return _complexity; } } ``` Usage: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("(a:1 AND b:2) OR c:[1 TO 10]"); var visitor = new QueryComplexityVisitor(); int complexity = await visitor.AcceptAsync(result, new QueryVisitorContext()); Console.WriteLine($"Complexity: {complexity}"); ``` ## Real-World Example: Custom Filter Resolution This example shows a visitor that resolves custom filter syntax to Elasticsearch queries: ```csharp using Foundatio.Parsers.ElasticQueries.Extensions; using Foundatio.Parsers.LuceneQueries.Nodes; using Foundatio.Parsers.LuceneQueries.Visitors; using Nest; /// /// Resolves @custom:(filter) syntax to actual queries. /// Example: @custom:(premium) -> terms query for premium user IDs /// public class CustomFilterVisitor : ChainableQueryVisitor { private readonly IUserService _userService; public CustomFilterVisitor(IUserService userService) { _userService = userService; } public override async Task VisitAsync(GroupNode node, IQueryVisitorContext context) { if (node.Field == "@custom" && node.Left != null) { string filterName = GetFilterName(node); var query = await ResolveFilter(filterName); if (query != null) { // Set the resolved query on the parent node node.Parent?.SetQuery(query); } // Clear the node's children (they've been processed) node.Left = null; node.Right = null; } await base.VisitAsync(node, context); } private string GetFilterName(GroupNode node) { if (node.Left is TermNode term) return term.Term; return GenerateQueryVisitor.Run(node.Left); } private async Task ResolveFilter(string filterName) { switch (filterName?.ToLowerInvariant()) { case "premium": var premiumIds = await _userService.GetPremiumUserIdsAsync(); return new TermsQuery { Field = "user_id", Terms = premiumIds }; case "active": return new TermQuery { Field = "status", Value = "active" }; default: return null; } } } ``` Usage: ```csharp var parser = new ElasticQueryParser(c => c .AddVisitor(new CustomFilterVisitor(userService), priority: 50)); // Resolves @custom:(premium) to a terms query var query = await parser.BuildQueryAsync("@custom:(premium) AND category:electronics"); ``` ## Example: Date Range Expansion Expand relative date expressions: ```csharp public class DateRangeExpansionVisitor : ChainableMutatingQueryVisitor { public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { if (IsDateField(node.Field) && IsRelativeDate(node.Term)) { var (start, end) = ExpandRelativeDate(node.Term); // Replace term node with range node var rangeNode = new TermRangeNode { Field = node.Field, Min = start, Max = end, MinInclusive = true, MaxInclusive = true }; node.ReplaceSelf(rangeNode); } return Task.CompletedTask; } private bool IsDateField(string field) { return field?.EndsWith("_date") == true || field?.EndsWith("_at") == true || field == "created" || field == "updated"; } private bool IsRelativeDate(string term) { return term == "today" || term == "yesterday" || term == "this_week" || term == "last_week"; } private (string start, string end) ExpandRelativeDate(string term) { return term switch { "today" => ("now/d", "now"), "yesterday" => ("now-1d/d", "now-1d/d"), "this_week" => ("now/w", "now"), "last_week" => ("now-1w/w", "now-1w/w"), _ => (term, term) }; } } ``` ## Example: Query Logging Visitor Log all queries for analytics: ```csharp public class QueryLoggingVisitor : QueryNodeVisitorBase { private readonly ILogger _logger; private readonly List _terms = new(); private readonly List _fields = new(); public QueryLoggingVisitor(ILogger logger) { _logger = logger; } public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { _fields.Add(node.Field ?? "_default"); _terms.Add(node.Term); return Task.CompletedTask; } public override async Task AcceptAsync(IQueryNode node, IQueryVisitorContext context) { _terms.Clear(); _fields.Clear(); var result = await base.AcceptAsync(node, context); _logger.LogInformation( "Query executed. Fields: {Fields}, Terms: {Terms}", string.Join(", ", _fields.Distinct()), string.Join(", ", _terms)); return result; } } ``` ## Visitor Priority When using chained visitors, priority determines execution order (lower runs first): ```csharp var parser = new ElasticQueryParser(c => c // Field resolution first .AddVisitor(new FieldResolverQueryVisitor(resolver), priority: 10) // Then include expansion .AddVisitor(new IncludeVisitor(), priority: 20) // Then custom processing .AddVisitor(new CustomFilterVisitor(), priority: 50) // Validation last .AddVisitor(new ValidationVisitor(), priority: 100)); ``` ## Best Practices ### 1. Call Base Implementation Always call the base implementation to continue traversal: ```csharp public override async Task VisitAsync(GroupNode node, IQueryVisitorContext context) { // Your logic here // Continue to child nodes await base.VisitAsync(node, context); } ``` ### 2. Handle Null Fields ```csharp public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { if (string.IsNullOrEmpty(node.Field)) return Task.CompletedTask; // Process node return Task.CompletedTask; } ``` ### 3. Use Context for State ```csharp public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { // Get state from context var userId = context.GetValue("UserId"); // Store results in context var fields = context.GetCollection("ReferencedFields"); fields.Add(node.Field); return Task.CompletedTask; } ``` ### 4. Make Visitors Stateless When Possible ```csharp // Prefer stateless visitors public class StatelessVisitor : ChainableQueryVisitor { public override Task VisitAsync(TermNode node, IQueryVisitorContext context) { // Use context for state, not instance fields var count = context.GetValue("TermCount"); context.SetValue("TermCount", count + 1); return Task.CompletedTask; } } ``` ## Next Steps * [Visitors](./visitors) - Built-in visitors reference * [Elasticsearch Integration](./elastic-query-parser) - Elasticsearch-specific visitors * [Validation](./validation) - Query validation --- --- url: /guide/elastic-query-parser.md --- # Elasticsearch Integration The `ElasticQueryParser` extends the base Lucene parser to build NEST query objects for Elasticsearch. It provides a powerful replacement for Elasticsearch's `query_string` query with additional features. ## Installation ```bash dotnet add package Foundatio.Parsers.ElasticQueries ``` ## Basic Usage ```csharp using Foundatio.Parsers.ElasticQueries; using Nest; var client = new ElasticClient(); var parser = new ElasticQueryParser(c => c .SetLoggerFactory(loggerFactory) .UseMappings(client, "my-index")); // Build a query QueryContainer query = await parser.BuildQueryAsync("status:active AND created:>2024-01-01"); // Use in search var response = await client.SearchAsync(s => s .Index("my-index") .Query(_ => query)); ``` ## Configuration ### Basic Configuration ```csharp var parser = new ElasticQueryParser(c => c // Logging .SetLoggerFactory(loggerFactory) // Default fields for unqualified terms .SetDefaultFields(new[] { "title", "description", "content" }) // Field mappings from Elasticsearch .UseMappings(client, "my-index") // Field aliases .UseFieldMap(new Dictionary { { "user", "data.user.identity" }, { "created", "metadata.createdAt" } }) // Query includes .UseIncludes(new Dictionary { { "active", "status:active AND deleted:false" } }) // Validation .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "status", "name", "created" } }) // Geo query support .UseGeo(location => ResolveGeoLocation(location)) // Nested document support .UseNested()); ``` ### Configuration Methods | Method | Description | |--------|-------------| | `SetLoggerFactory(factory)` | Set logging factory | | `SetDefaultFields(fields)` | Default fields for unqualified terms | | `UseMappings(client, index)` | Load mappings from Elasticsearch | | `UseFieldMap(map)` | Static field alias map | | `UseFieldResolver(resolver)` | Dynamic field resolver | | `UseIncludes(includes)` | Query include definitions | | `UseGeo(resolver)` | Geo location resolver | | `UseNested()` | Enable nested document support | | `SetValidationOptions(options)` | Validation configuration | | `UseRuntimeFieldResolver(resolver)` | Runtime field support | ## Building Queries ### Simple Queries ```csharp var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // Term query var query = await parser.BuildQueryAsync("status:active"); // Range query query = await parser.BuildQueryAsync("price:[100 TO 500]"); // Boolean query query = await parser.BuildQueryAsync("status:active AND category:electronics"); // Phrase query query = await parser.BuildQueryAsync("title:\"quick brown fox\""); ``` ### With Scoring By default, queries are wrapped in a `bool` filter (no scoring). Enable scoring for relevance: ```csharp var context = new ElasticQueryVisitorContext { UseScoring = true }; var query = await parser.BuildQueryAsync("title:search terms", context); ``` Or use the search mode helper: ```csharp var context = new ElasticQueryVisitorContext().UseSearchMode(); var query = await parser.BuildQueryAsync("search terms", context); ``` ### From Parsed AST ```csharp // Parse first var ast = await parser.ParseAsync("status:active"); // Build query from AST var query = await parser.BuildQueryAsync(ast, context); ``` ## Building Aggregations ```csharp var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // Build aggregations AggregationContainer aggs = await parser.BuildAggregationsAsync( "terms:(category min:price max:price avg:price)"); // Use in search var response = await client.SearchAsync(s => s .Index("my-index") .Size(0) .Aggregations(aggs)); ``` ### Complex Aggregations ```csharp // Date histogram with nested terms var aggs = await parser.BuildAggregationsAsync( "date:(created~month terms:(status sum:amount))"); // Multiple aggregations aggs = await parser.BuildAggregationsAsync( "min:price max:price avg:price terms:category~10"); ``` ## Building Sort ```csharp var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // Build sort (- for descending, + for ascending) IEnumerable sort = await parser.BuildSortAsync("-created +name"); // Use in search var response = await client.SearchAsync(s => s .Index("my-index") .Sort(sort)); ``` ## Validation ### Validate Before Building ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "status", "name", "created" }, AllowLeadingWildcards = false })); // Validate query var result = await parser.ValidateQueryAsync("status:active"); if (!result.IsValid) { Console.WriteLine($"Invalid: {result.Message}"); return; } // Validate aggregations result = await parser.ValidateAggregationsAsync("terms:category"); // Validate sort result = await parser.ValidateSortAsync("-created"); ``` ### Automatic Validation `BuildQueryAsync` automatically validates and throws on failure: ```csharp try { var query = await parser.BuildQueryAsync("invalid:field"); } catch (QueryValidationException ex) { Console.WriteLine($"Validation failed: {ex.Message}"); } ``` ## Visitor Management ### Adding Custom Visitors ```csharp var parser = new ElasticQueryParser(c => c // Add to all visitor chains (query, aggregation, sort) .AddVisitor(new MyCustomVisitor(), priority: 100) // Add only to query visitor chain .AddQueryVisitor(new QueryOnlyVisitor(), priority: 50) // Add only to aggregation visitor chain .AddAggregationVisitor(new AggOnlyVisitor(), priority: 50) // Add only to sort visitor chain .AddSortVisitor(new SortOnlyVisitor(), priority: 50)); ``` ### Visitor Chain Management ```csharp var parser = new ElasticQueryParser(c => c // Remove a visitor .RemoveVisitor() // Replace a visitor .ReplaceVisitor(new MyFieldResolver(), newPriority: 15) // Add before/after specific visitor .AddVisitorBefore(new PreValidationVisitor()) .AddVisitorAfter(new PostResolverVisitor())); ``` ## Geo Queries ### Configuration ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseGeo(async location => { // Resolve location string to coordinates if (location.Length == 5 && int.TryParse(location, out _)) { // Zip code lookup var coords = await geocoder.ResolveZipCode(location); return $"{coords.Lat},{coords.Lon}"; } return location; })); ``` ### Proximity Queries ```csharp // Within 75 miles of zip code var query = await parser.BuildQueryAsync("location:75044~75mi"); // Within 10 kilometers of coordinates query = await parser.BuildQueryAsync("location:51.5,-0.1~10km"); ``` ### Bounding Box Queries ```csharp // Bounding box var query = await parser.BuildQueryAsync("location:[51.5,-0.2 TO 51.4,-0.1]"); ``` ## Nested Documents Enable nested document support: ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseNested()); // Queries on nested fields are automatically wrapped var query = await parser.BuildQueryAsync("comments.author:john"); ``` ## Runtime Fields ### Opt-In Runtime Fields ```csharp var parser = new ElasticQueryParser(c => c .UseOptInRuntimeFieldResolver(async field => { // Return runtime field definition if field should be runtime if (field == "full_name") { return new ElasticRuntimeField { Name = "full_name", FieldType = ElasticRuntimeFieldType.Keyword, Script = "emit(doc['first_name'].value + ' ' + doc['last_name'].value)" }; } return null; })); // Enable runtime fields for specific query var context = new ElasticQueryVisitorContext(); context.EnableRuntimeFieldResolver(true); var query = await parser.BuildQueryAsync("full_name:John*", context); // Access runtime fields to include in search var runtimeFields = context.RuntimeFields; ``` ### Always-On Runtime Fields ```csharp var parser = new ElasticQueryParser(c => c .UseRuntimeFieldResolver(async field => { // Always check for runtime fields return await GetRuntimeFieldDefinition(field); })); ``` ## Complete Example ```csharp using Foundatio.Parsers.ElasticQueries; using Foundatio.Parsers.ElasticQueries.Visitors; using Nest; public class SearchService { private readonly IElasticClient _client; private readonly ElasticQueryParser _parser; public SearchService(IElasticClient client, ILoggerFactory loggerFactory) { _client = client; _parser = new ElasticQueryParser(c => c .SetLoggerFactory(loggerFactory) .UseMappings(client, "products") .SetDefaultFields(new[] { "name", "description" }) .UseFieldMap(new Dictionary { { "category", "category.keyword" }, { "brand", "brand.keyword" } }) .UseIncludes(new Dictionary { { "available", "status:active AND inventory:>0" }, { "sale", "discount:>0" } }) .UseGeo(ResolveLocation) .UseNested() .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "name", "description", "category", "brand", "price", "status", "inventory", "location" }, AllowLeadingWildcards = false, AllowedMaxNodeDepth = 10 })); } public async Task SearchAsync( string query, string aggregations = null, string sort = null, int page = 1, int pageSize = 20) { // Validate inputs var validation = await _parser.ValidateQueryAsync(query); if (!validation.IsValid) throw new ArgumentException(validation.Message); // Build query with scoring for relevance var context = new ElasticQueryVisitorContext().UseSearchMode(); var esQuery = await _parser.BuildQueryAsync(query, context); // Build search request var searchDescriptor = new SearchDescriptor() .Index("products") .From((page - 1) * pageSize) .Size(pageSize) .Query(_ => esQuery); // Add aggregations if provided if (!string.IsNullOrEmpty(aggregations)) { var aggs = await _parser.BuildAggregationsAsync(aggregations); searchDescriptor.Aggregations(aggs); } // Add sort if provided if (!string.IsNullOrEmpty(sort)) { var sortFields = await _parser.BuildSortAsync(sort); searchDescriptor.Sort(sortFields); } var response = await _client.SearchAsync(searchDescriptor); return new SearchResult { Total = response.Total, Items = response.Documents.ToList(), Aggregations = response.Aggregations }; } private string ResolveLocation(string location) { // Implement location resolution return location; } } ``` ## Next Steps * [Elasticsearch Mappings](./elastic-mappings) - Mapping resolver details * [Aggregation Syntax](./aggregation-syntax) - Aggregation expression reference * [Query Syntax](./query-syntax) - Query syntax reference * [Custom Visitors](./custom-visitors) - Create custom visitors --- --- url: /guide/elastic-mappings.md --- # Elasticsearch Mappings The `ElasticMappingResolver` provides intelligent field resolution based on Elasticsearch index mappings. It automatically handles analyzed vs non-analyzed fields, nested documents, and field types. ## Overview When you configure `UseMappings()`, the parser: 1. Loads field mappings from your Elasticsearch index 2. Resolves field names to their correct paths 3. Automatically uses keyword sub-fields for sorting and aggregations 4. Detects nested fields for proper query wrapping 5. Identifies field types for appropriate query generation ## Configuration ### From Elasticsearch Client ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index")); ``` ### From Type Mapping ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client)); ``` ### With Custom Mapping Builder ```csharp var parser = new ElasticQueryParser(c => c .UseMappings( mappingBuilder: m => m .Properties(p => p .Text(t => t.Name(n => n.Title) .Fields(f => f.Keyword(k => k.Name("keyword")))) .Keyword(k => k.Name(n => n.Status)) .Date(d => d.Name(n => n.Created)) .Nested(n => n.Name(x => x.Comments))), client, "my-index")); ``` ### From Mapping Function ```csharp var parser = new ElasticQueryParser(c => c .UseMappings( getMapping: () => GetCachedMapping(), inferrer: client.Infer)); ``` ## Field Resolution ### Automatic Keyword Field Detection For text fields with keyword sub-fields, the resolver automatically uses the keyword field for: * Sorting * Aggregations * Exact match queries ```csharp // Mapping: // "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } } var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // For queries - uses analyzed "title" field var query = await parser.BuildQueryAsync("title:search terms"); // For aggregations - automatically uses "title.keyword" var aggs = await parser.BuildAggregationsAsync("terms:title"); // For sort - automatically uses "title.keyword" var sort = await parser.BuildSortAsync("title"); ``` ### Field Type Detection The resolver detects field types for appropriate query handling: ```csharp var resolver = parser.Configuration.MappingResolver; // Check field types bool isNested = resolver.IsNestedPropertyType("comments"); bool isGeo = resolver.IsGeoPropertyType("location"); bool isNumeric = resolver.IsNumericPropertyType("price"); bool isDate = resolver.IsDatePropertyType("created"); bool isBoolean = resolver.IsBooleanPropertyType("active"); bool isAnalyzed = resolver.IsPropertyAnalyzed("description"); ``` ## ElasticMappingResolver API ### Getting Field Information ```csharp var resolver = parser.Configuration.MappingResolver; // Get full field mapping var mapping = resolver.GetMapping("user.name"); if (mapping.Found) { Console.WriteLine($"Full path: {mapping.FullPath}"); Console.WriteLine($"Property type: {mapping.Property?.GetType().Name}"); } // Get the NEST property IProperty property = resolver.GetMappingProperty("status"); // Get resolved field name string resolved = resolver.GetResolvedField("user"); // Get non-analyzed field for sorting string sortField = resolver.GetSortFieldName("title"); // Get non-analyzed field for aggregations string aggField = resolver.GetAggregationsFieldName("category"); // Get field type enum FieldType fieldType = resolver.GetFieldType("price"); ``` ### Field Type Enum ```csharp public enum FieldType { Unknown, Text, Keyword, Date, Boolean, Long, Integer, Short, Byte, Double, Float, HalfFloat, ScaledFloat, GeoPoint, GeoShape, Nested, Object, // ... other types } ``` ## Nested Document Handling ### Automatic Nested Query Wrapping When `UseNested()` is enabled, queries on nested fields are automatically wrapped: ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseNested()); // Query on nested field var query = await parser.BuildQueryAsync("comments.author:john"); // Automatically generates: // { // "nested": { // "path": "comments", // "query": { // "term": { "comments.author": "john" } // } // } // } ``` ### Nested Field Detection ```csharp var resolver = parser.Configuration.MappingResolver; // Check if field is nested bool isNested = resolver.IsNestedPropertyType("comments"); // Get the nested path for a field // "comments.author" -> "comments" ``` ## Mapping Extensions ### Adding Keyword Sub-Fields Use extension methods to add standard sub-fields to your mappings: ```csharp using Foundatio.Parsers.ElasticQueries.Extensions; var createIndexResponse = await client.Indices.CreateAsync("my-index", c => c .Map(m => m .Properties(p => p // Add .keyword sub-field .Text(t => t.Name(n => n.Title).AddKeywordField()) // Add .sort sub-field with lowercase normalizer .Text(t => t.Name(n => n.Name).AddSortField()) // Add both .keyword and .sort sub-fields .Text(t => t.Name(n => n.Description).AddKeywordAndSortFields()) ))); ``` ### Sub-Field Names ```csharp using Foundatio.Parsers.ElasticQueries.Extensions; // Default sub-field names string keywordField = ElasticMappingExtensions.KeywordFieldName; // "keyword" string sortField = ElasticMappingExtensions.SortFieldName; // "sort" ``` ### Sort Normalizer Add a lowercase normalizer for case-insensitive sorting: ```csharp var createIndexResponse = await client.Indices.CreateAsync("my-index", c => c .Settings(s => s.AddSortNormalizer()) .Map(m => m .Properties(p => p .Text(t => t.Name(n => n.Name).AddSortField()) ))); ``` ## Refreshing Mappings Mappings are automatically refreshed from Elasticsearch at most once per minute. In most production scenarios, this automatic refresh is sufficient. For unit tests where you're creating or modifying indices and need immediate visibility of changes, you can force a refresh: ```csharp var resolver = parser.Configuration.MappingResolver; // Force refresh from Elasticsearch (primarily for unit tests) resolver.RefreshMapping(); ``` ## Custom Mapping Resolver Create a custom resolver for special cases: ```csharp var customResolver = ElasticMappingResolver.Create( getMapping: () => { // Return cached or custom mapping return _cachedMapping; }, inferrer: client.Infer, logger: logger); var parser = new ElasticQueryParser(c => c .UseMappings(customResolver)); ``` ## Field Mapping Structure The `FieldMapping` class contains: ```csharp public class FieldMapping { // Whether the field was found in mappings public bool Found { get; } // The full resolved path (e.g., "data.user.name") public string FullPath { get; } // The NEST IProperty for the field public IProperty Property { get; } } ``` ## Best Practices ### 1. Use Consistent Sub-Field Naming ```csharp // Always use .keyword for exact matching // Always use .sort for case-insensitive sorting .Text(t => t.Name(n => n.Title) .Fields(f => f .Keyword(k => k.Name("keyword").IgnoreAbove(256)) .Keyword(k => k.Name("sort").Normalizer("lowercase")))) ``` ### 2. Cache Mapping Resolution The resolver caches mappings automatically, but you can also: ```csharp // Create resolver once and reuse var resolver = ElasticMappingResolver.Create(client, "my-index"); var parser1 = new ElasticQueryParser(c => c.UseMappings(resolver)); var parser2 = new ElasticQueryParser(c => c.UseMappings(resolver)); ``` ### 3. Handle Dynamic Mappings For indices with dynamic mappings: ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .SetValidationOptions(new QueryValidationOptions { // Allow fields not in current mapping AllowUnresolvedFields = true })); ``` ### 4. Log Mapping Issues ```csharp var parser = new ElasticQueryParser(c => c .SetLoggerFactory(loggerFactory) .UseMappings(client, "my-index")); // Mapping resolution issues will be logged ``` ## Troubleshooting ### Field Not Found ```csharp var resolver = parser.Configuration.MappingResolver; var mapping = resolver.GetMapping("unknown_field"); if (!mapping.Found) { // Field doesn't exist in mapping // Check: spelling, case sensitivity, nested path } ``` ### Wrong Field Type Used ```csharp // Check what type the resolver sees var fieldType = resolver.GetFieldType("my_field"); Console.WriteLine($"Field type: {fieldType}"); // Check if analyzed bool isAnalyzed = resolver.IsPropertyAnalyzed("my_field"); Console.WriteLine($"Is analyzed: {isAnalyzed}"); ``` ### Nested Queries Not Working ```csharp // Ensure UseNested() is configured var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseNested()); // Required for nested support // Verify field is detected as nested bool isNested = resolver.IsNestedPropertyType("comments"); ``` ## Next Steps * [Elasticsearch Integration](./elastic-query-parser) - Full parser guide * [Query Syntax](./query-syntax) - Query syntax reference * [Aggregation Syntax](./aggregation-syntax) - Aggregation reference --- --- url: /guide/field-aliases.md --- # Field Aliases Field aliases allow you to map user-friendly field names to actual field paths in your data. This is useful for: * Hiding complex nested field paths from users * Providing backward compatibility when field names change * Creating domain-specific query languages * Abstracting internal data structures ## Static Field Maps The simplest approach is a static dictionary mapping aliases to actual field names: ```csharp using Foundatio.Parsers.LuceneQueries; using Foundatio.Parsers.LuceneQueries.Visitors; var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("user:john"); // Define field mappings var fieldMap = new FieldMap { { "user", "data.user.identity" }, { "created", "metadata.createdAt" }, { "status", "workflow.currentStatus" } }; // Apply field resolution var resolved = await FieldResolverQueryVisitor.RunAsync(result, fieldMap); // Result: data.user.identity:john Console.WriteLine(resolved.ToString()); ``` ### With ElasticQueryParser ```csharp using Foundatio.Parsers.ElasticQueries; var parser = new ElasticQueryParser(c => c .UseFieldMap(new Dictionary { { "user", "data.user.identity" }, { "created", "metadata.createdAt" }, { "status", "workflow.currentStatus" } })); // Query uses aliases var query = await parser.BuildQueryAsync("user:john AND status:active"); // Internally resolves to: data.user.identity:john AND workflow.currentStatus:active ``` ## Hierarchical Field Resolution For nested field structures, use hierarchical resolution. This resolves parent paths and preserves child segments: ```csharp using Foundatio.Parsers.LuceneQueries.Visitors; var fieldMap = new Dictionary { { "user", "data.profile.user" }, { "user.address", "data.profile.user.location" } }; // Convert to hierarchical resolver var resolver = fieldMap.ToHierarchicalFieldResolver(); // Resolution examples: // "user" -> "data.profile.user" // "user.name" -> "data.profile.user.name" (parent resolved, child preserved) // "user.address" -> "data.profile.user.location" (exact match) // "user.address.city" -> "data.profile.user.location.city" (parent resolved) ``` ### How Hierarchical Resolution Works 1. Check for exact match in the map 2. If not found, split the field by `.` and check for parent matches 3. Replace the matched parent with its mapping, preserve remaining segments ```csharp var map = new Dictionary { { "original", "replacement" }, { "original.nested", "otherreplacement" } }; var resolver = map.ToHierarchicalFieldResolver(); // Examples: await resolver("notmapped", null); // "notmapped" (no change) await resolver("original", null); // "replacement" await resolver("original.hey", null); // "replacement.hey" await resolver("original.nested", null); // "otherreplacement" await resolver("original.nested.hey", null); // "otherreplacement.hey" ``` ## Dynamic Field Resolvers For complex resolution logic, use a custom resolver function: ```csharp using Foundatio.Parsers.LuceneQueries.Visitors; var parser = new ElasticQueryParser(c => c .UseFieldResolver(async (field, context) => { // Custom resolution logic if (field.StartsWith("custom.")) { return field.Replace("custom.", "data.custom_fields."); } // Check a database or external service var mapping = await GetFieldMappingFromDatabase(field); if (mapping != null) return mapping; // Return null to keep original field name return null; })); ``` ### Resolver Signature ```csharp public delegate Task QueryFieldResolver(string field, IQueryVisitorContext context); ``` The resolver receives: * `field` - The field name to resolve * `context` - The visitor context with additional data Return: * The resolved field name, or * `null` to keep the original field name ## Combining Static and Dynamic Resolution You can combine static maps with dynamic resolution: ```csharp var staticMap = new Dictionary { { "user", "data.user.identity" }, { "status", "workflow.status" } }; var parser = new ElasticQueryParser(c => c .UseFieldResolver(async (field, context) => { // First check static map if (staticMap.TryGetValue(field, out var mapped)) return mapped; // Then apply dynamic logic if (field.StartsWith("meta.")) return field.Replace("meta.", "metadata."); // Check hierarchical resolution for nested fields var hierarchical = staticMap.ToHierarchicalFieldResolver(); return await hierarchical(field, context); })); ``` ## Field Resolution in Groups Field aliases work with grouped queries: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("(user.name:john OR user.email:john@example.com) status:active"); var fieldMap = new FieldMap { { "user.name", "profile.fullName" }, { "user.email", "profile.emailAddress" }, { "status", "account.status" } }; var resolved = await FieldResolverQueryVisitor.RunAsync(result, fieldMap); // Result: (profile.fullName:john OR profile.emailAddress:john@example.com) account.status:active ``` ## Field Resolution with Aggregations Field aliases apply to aggregations as well: ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseFieldMap(new Dictionary { { "category", "product.category.keyword" }, { "price", "product.pricing.amount" } })); // Aggregation uses aliases var aggs = await parser.BuildAggregationsAsync("terms:(category min:price max:price)"); // Resolves to: terms:(product.category.keyword min:product.pricing.amount max:product.pricing.amount) ``` ## Preserving Original Field Names The original field name is preserved in node metadata for reference: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("user:john"); var fieldMap = new FieldMap { { "user", "data.user.identity" } }; var context = new QueryVisitorContext(); await FieldResolverQueryVisitor.RunAsync(result, fieldMap, context); // Access original field name var termNode = result.Left as TermNode; string original = termNode.GetOriginalField(); // "user" string resolved = termNode.Field; // "data.user.identity" ``` ## Validation with Field Aliases When using validation, specify allowed fields using the alias names (not resolved names): ```csharp var parser = new ElasticQueryParser(c => c .UseFieldMap(new Dictionary { { "user", "data.user.identity" }, { "status", "workflow.status" } }) .SetValidationOptions(new QueryValidationOptions { // Use alias names in allowed fields AllowedFields = { "user", "status", "created" } })); // Valid - uses allowed alias var result = await parser.ValidateQueryAsync("user:john"); // result.IsValid == true // Invalid - uses resolved name directly result = await parser.ValidateQueryAsync("data.user.identity:john"); // result.IsValid == false (unless also in AllowedFields) ``` ## Best Practices ### 1. Use Descriptive Alias Names ```csharp // Good - clear, domain-specific names var fieldMap = new Dictionary { { "author", "document.metadata.author.name" }, { "published", "document.metadata.publishedDate" } }; // Avoid - cryptic abbreviations var fieldMap = new Dictionary { { "a", "document.metadata.author.name" }, { "pd", "document.metadata.publishedDate" } }; ``` ### 2. Document Your Aliases Maintain documentation of available aliases for API consumers: ```csharp /// /// Available field aliases for the search API: /// - user: Maps to data.user.identity /// - status: Maps to workflow.currentStatus /// - created: Maps to metadata.createdAt /// ``` ### 3. Handle Unknown Fields Gracefully ```csharp var parser = new ElasticQueryParser(c => c .UseFieldResolver(async (field, context) => { if (knownAliases.TryGetValue(field, out var mapped)) return mapped; // Option 1: Return null to keep original (permissive) return null; // Option 2: Use validation to reject unknown fields (strict) // Configure AllowedFields in validation options }) .SetValidationOptions(new QueryValidationOptions { AllowUnresolvedFields = false // Reject unknown fields })); ``` ### 4. Consider Case Sensitivity ```csharp var parser = new ElasticQueryParser(c => c .UseFieldResolver(async (field, context) => { // Case-insensitive lookup var key = fieldMap.Keys.FirstOrDefault(k => k.Equals(field, StringComparison.OrdinalIgnoreCase)); return key != null ? fieldMap[key] : null; })); ``` ## Next Steps * [Query Includes](./query-includes) - Reusable query macros * [Validation](./validation) - Validate and restrict queries * [Elasticsearch Integration](./elastic-query-parser) - Full parser configuration --- --- url: /README.md --- # Foundatio.Parsers Documentation This folder contains the VitePress documentation site for Foundatio.Parsers. ## Development ```bash # Install dependencies npm install # Start development server npm run dev # Build for production npm run build # Preview production build npm run preview ``` ## Structure * `.vitepress/config.ts` - VitePress configuration * `index.md` - Homepage * `guide/` - Documentation pages * `public/` - Static assets ## Deployment Documentation is automatically deployed to GitHub Pages when changes are pushed to the `main` branch. --- --- url: /guide/getting-started.md --- # Getting Started This guide walks you through installing Foundatio.Parsers and using it to parse your first query. ## Installation Install the package for your use case: ::: code-group ```bash [Lucene Parser Only] dotnet add package Foundatio.Parsers.LuceneQueries ``` ```bash [Elasticsearch Integration] dotnet add package Foundatio.Parsers.ElasticQueries ``` ```bash [SQL/EF Core Integration] dotnet add package Foundatio.Parsers.SqlQueries ``` ::: ## Basic Usage ### Parsing a Query The `LuceneQueryParser` parses query strings into an Abstract Syntax Tree (AST): ```csharp using Foundatio.Parsers.LuceneQueries; using Foundatio.Parsers.LuceneQueries.Visitors; var parser = new LuceneQueryParser(); // Parse a query string var result = parser.Parse("field:value AND other:[1 TO 10]"); // Inspect the AST structure Console.WriteLine(DebugQueryVisitor.Run(result)); ``` Output: ``` Group: Left - Group: Operator: And Left - Term: Field: field Term: value Right - Term: Field: other TermMin: 1 TermMax: 10 MinInclusive: True MaxInclusive: True ``` ### Regenerating the Query Use `GenerateQueryVisitor` to convert the AST back to a query string: ```csharp var parser = new LuceneQueryParser(); var result = parser.Parse("field:[1 TO 2]"); string generatedQuery = GenerateQueryVisitor.Run(result); Console.WriteLine(generatedQuery); // Output: field:[1 TO 2] ``` ### Async Parsing For async workflows, use `ParseAsync`: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("status:active"); string query = await GenerateQueryVisitor.RunAsync(result); ``` ## Elasticsearch Integration The `ElasticQueryParser` extends the base parser to build NEST query objects: ```csharp using Foundatio.Parsers.ElasticQueries; using Nest; var client = new ElasticClient(); var parser = new ElasticQueryParser(c => c .SetLoggerFactory(loggerFactory) .UseMappings(client, "my-index")); // Build a query QueryContainer query = await parser.BuildQueryAsync("status:active AND created:>2024-01-01"); // Use in a search var response = await client.SearchAsync(s => s .Index("my-index") .Query(_ => query)); ``` ### Building Aggregations ```csharp var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // Build aggregations from expression AggregationContainer aggs = await parser.BuildAggregationsAsync( "terms:(status min:amount max:amount avg:amount)"); var response = await client.SearchAsync(s => s .Index("my-index") .Size(0) .Aggregations(aggs)); ``` ### Building Sort ```csharp var parser = new ElasticQueryParser(c => c.UseMappings(client, "my-index")); // Build sort from expression (- for descending, + for ascending) var sort = await parser.BuildSortAsync("-created +name"); var response = await client.SearchAsync(s => s .Index("my-index") .Sort(sort)); ``` ## SQL/Entity Framework Core Integration The `SqlQueryParser` generates Dynamic LINQ expressions for EF Core: ```csharp using Foundatio.Parsers.SqlQueries; using Microsoft.EntityFrameworkCore; var parser = new SqlQueryParser(c => c .SetDefaultFields(["Name", "Description"])); // Get context from your DbContext await using var db = new MyDbContext(); var context = parser.GetContext(db.Products.EntityType); // Convert query to Dynamic LINQ string dynamicLinq = await parser.ToDynamicLinqAsync( "status:active AND price:>100", context); // Use with EF Core var results = await db.Products .Where(parser.ParsingConfig, dynamicLinq) .ToListAsync(); ``` ## Field Aliases Map user-friendly field names to actual field paths: ```csharp var parser = new ElasticQueryParser(c => c .UseFieldMap(new Dictionary { { "user", "data.user.identity" }, { "created", "metadata.createdAt" } })); // Query uses aliases var query = await parser.BuildQueryAsync("user:john AND created:>2024-01-01"); // Internally resolves to: data.user.identity:john AND metadata.createdAt:>2024-01-01 ``` ## Query Validation Validate queries before execution: ```csharp var parser = new ElasticQueryParser(c => c .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "status", "name", "created" }, AllowLeadingWildcards = false, AllowedMaxNodeDepth = 5 })); var result = await parser.ValidateQueryAsync("status:active"); if (!result.IsValid) { Console.WriteLine($"Invalid query: {result.Message}"); foreach (var error in result.ValidationErrors) { Console.WriteLine($" - {error.Message} at position {error.Index}"); } } ``` ## Query Includes Define reusable query macros: ```csharp var includes = new Dictionary { { "active", "status:active AND deleted:false" }, { "recent", "created:>now-7d" } }; var parser = new ElasticQueryParser(c => c .UseIncludes(includes)); // Expands @include:active to (status:active AND deleted:false) var query = await parser.BuildQueryAsync("@include:active AND @include:recent"); ``` ## Next Steps * [Query Syntax](./query-syntax) - Complete query syntax reference * [Aggregation Syntax](./aggregation-syntax) - Aggregation expression reference * [Field Aliases](./field-aliases) - Advanced field mapping * [Validation](./validation) - Query validation options * [Visitors](./visitors) - Understanding the visitor pattern ## LLM-Friendly Documentation For AI assistants and Large Language Models, we provide optimized documentation formats: * [LLMs Index](/llms.txt) - Quick reference with links to all sections * [Complete Documentation](/llms-full.txt) - All docs in one LLM-friendly file These files follow the [llmstxt.org](https://llmstxt.org/) standard. --- --- url: /guide/query-includes.md --- # Query Includes Query includes (also called macros) allow you to define reusable query fragments that expand inline. This is useful for: * Storing commonly used filters * Creating user-defined saved searches * Building complex queries from simple building blocks * Providing shortcuts for frequently used conditions ## Basic Usage ### Syntax Include a stored query using the `@include:` prefix: ``` @include:name ``` ### Simple Example ```csharp using Foundatio.Parsers.LuceneQueries; using Foundatio.Parsers.LuceneQueries.Visitors; var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("@include:active"); // Define includes var includes = new Dictionary { { "active", "status:active AND deleted:false" } }; // Expand includes var expanded = await IncludeVisitor.RunAsync(result, includes); // Result: (status:active AND deleted:false) Console.WriteLine(expanded.ToString()); ``` ## With ElasticQueryParser ```csharp using Foundatio.Parsers.ElasticQueries; var includes = new Dictionary { { "active", "status:active AND deleted:false" }, { "recent", "created:[now-7d TO now]" }, { "highvalue", "amount:>=1000" } }; var parser = new ElasticQueryParser(c => c .UseIncludes(includes)); // Single include var query = await parser.BuildQueryAsync("@include:active"); // Multiple includes query = await parser.BuildQueryAsync("@include:active AND @include:recent"); // Combine with other conditions query = await parser.BuildQueryAsync("@include:active AND category:electronics"); ``` ## Dynamic Include Resolution For includes stored in a database or external service: ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes(async (name) => { // Load from database var savedSearch = await db.SavedSearches .FirstOrDefaultAsync(s => s.Name == name); return savedSearch?.Query; })); // Resolves include from database var query = await parser.BuildQueryAsync("@include:my-saved-search"); ``` ### Async Include Resolver ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes(async (name) => { // Call external API var response = await httpClient.GetAsync($"/api/includes/{name}"); if (response.IsSuccessStatusCode) { return await response.Content.ReadAsStringAsync(); } return null; // Include not found })); ``` ## Nested Includes Includes can reference other includes: ```csharp var includes = new Dictionary { { "active", "status:active AND deleted:false" }, { "recent", "created:[now-7d TO now]" }, { "active-recent", "@include:active AND @include:recent" } }; var parser = new ElasticQueryParser(c => c.UseIncludes(includes)); // Expands to: ((status:active AND deleted:false) AND (created:[now-7d TO now])) var query = await parser.BuildQueryAsync("@include:active-recent"); ``` ## Recursive Include Detection The parser automatically detects and prevents infinite recursion: ```csharp var includes = new Dictionary { { "a", "@include:b" }, { "b", "@include:a" } // Circular reference! }; var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("@include:a"); var context = new QueryVisitorContext(); var expanded = await IncludeVisitor.RunAsync(result, includes, context); // Check for errors var validation = context.GetValidationResult(); if (!validation.IsValid) { Console.WriteLine(validation.Message); // Output: "Recursive include detected: a" } ``` ## Skipping Includes You can conditionally skip include expansion: ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes( includeResolver: name => includes.GetValueOrDefault(name), shouldSkipInclude: (name, context) => { // Skip includes starting with "admin_" for non-admin users if (name.StartsWith("admin_") && !context.GetValue("IsAdmin")) return true; return false; })); // Set context var context = new ElasticQueryVisitorContext(); context.SetValue("IsAdmin", false); // admin_reports include will be skipped var query = await parser.BuildQueryAsync("@include:admin_reports", context); ``` ## Tracking Referenced Includes The validation result tracks which includes were referenced: ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes(includes) .SetValidationOptions(new QueryValidationOptions())); var context = new ElasticQueryVisitorContext(); var query = await parser.BuildQueryAsync("@include:active AND @include:recent", context); var validation = context.GetValidationResult(); // Get referenced includes foreach (var include in validation.ReferencedIncludes) { Console.WriteLine($"Used include: {include}"); } // Get unresolved includes (not found) foreach (var include in validation.UnresolvedIncludes) { Console.WriteLine($"Missing include: {include}"); } ``` ## Validation Options Control include behavior with validation options: ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes(includes) .SetValidationOptions(new QueryValidationOptions { // Fail if an include cannot be resolved AllowUnresolvedIncludes = false })); // Throws QueryValidationException if include not found try { var query = await parser.BuildQueryAsync("@include:nonexistent"); } catch (QueryValidationException ex) { Console.WriteLine(ex.Message); // Output: "Include 'nonexistent' could not be resolved" } ``` ## Combining with Field Aliases Includes work with field aliases - the alias resolution happens after include expansion: ```csharp var includes = new Dictionary { { "active", "status:active" } // Uses alias "status" }; var fieldMap = new Dictionary { { "status", "workflow.currentStatus" } }; var parser = new ElasticQueryParser(c => c .UseIncludes(includes) .UseFieldMap(fieldMap)); // 1. Expands @include:active to (status:active) // 2. Resolves status to workflow.currentStatus var query = await parser.BuildQueryAsync("@include:active"); // Final: workflow.currentStatus:active ``` ## Use Cases ### Saved Searches Allow users to save and reuse searches: ```csharp public class SavedSearchService { private readonly ElasticQueryParser _parser; private readonly IRepository _repository; public SavedSearchService(IRepository repository) { _repository = repository; _parser = new ElasticQueryParser(c => c .UseIncludes(ResolveSavedSearch)); } private async Task ResolveSavedSearch(string name) { var search = await _repository.GetByNameAsync(name); return search?.Query; } public async Task BuildQuery(string userQuery) { // User can reference saved searches: @include:my-filter AND category:books return await _parser.BuildQueryAsync(userQuery); } } ``` ### Role-Based Filters Automatically apply filters based on user role: ```csharp var roleFilters = new Dictionary { { "user-filter", "organization_id:{userId}" }, { "admin-filter", "*" }, // No filter for admins { "manager-filter", "department_id:{departmentId}" } }; var parser = new ElasticQueryParser(c => c .UseIncludes(async name => { var template = roleFilters.GetValueOrDefault(name); if (template == null) return null; // Replace placeholders with actual values return template .Replace("{userId}", currentUser.Id) .Replace("{departmentId}", currentUser.DepartmentId); })); ``` ### Query Templates Create parameterized query templates: ```csharp var templates = new Dictionary { { "date-range", "created:[{start} TO {end}]" }, { "price-range", "price:[{min} TO {max}]" } }; // Note: This requires custom expansion logic var parser = new ElasticQueryParser(c => c .UseIncludes(name => { // Parse template name and parameters // e.g., "date-range:2024-01-01:2024-12-31" var parts = name.Split(':'); if (parts.Length < 2) return templates.GetValueOrDefault(parts[0]); var template = templates.GetValueOrDefault(parts[0]); if (template == null) return null; // Simple parameter substitution for (int i = 1; i < parts.Length; i++) { template = template.Replace($"{{{i-1}}}", parts[i]); } return template; })); ``` ## Best Practices ### 1. Use Descriptive Names ```csharp // Good var includes = new Dictionary { { "active-users", "status:active AND type:user" }, { "last-30-days", "created:[now-30d TO now]" } }; // Avoid var includes = new Dictionary { { "q1", "status:active AND type:user" }, { "d", "created:[now-30d TO now]" } }; ``` ### 2. Document Available Includes ```csharp /// /// Available query includes: /// - active: Active, non-deleted records /// - recent: Created in the last 7 days /// - high-priority: Priority >= 8 /// ``` ### 3. Validate Include Content ```csharp public async Task SaveInclude(string name, string query) { // Validate the query before saving var parser = new LuceneQueryParser(); try { var result = await parser.ParseAsync(query); // Additional validation... return true; } catch (FormatException) { return false; } } ``` ### 4. Handle Missing Includes Gracefully ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes(name => { var query = includes.GetValueOrDefault(name); if (query == null) { logger.LogWarning("Include not found: {Name}", name); } return query; })); ``` ## Next Steps * [Validation](./validation) - Validate queries and includes * [Field Aliases](./field-aliases) - Map field names * [Custom Visitors](./custom-visitors) - Create custom query transformations --- --- url: /guide/query-syntax.md --- # Query Syntax The query syntax is based on [Lucene query syntax](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html) and is compatible with [Elasticsearch query\_string](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html). ## Basic Queries ### Term Queries Match documents where a field contains a specific value: | Syntax | Description | Example | |--------|-------------|---------| | `field:value` | Exact match | `status:active` | | `field:"quoted value"` | Exact phrase match | `name:"John Smith"` | | `value` | Search default fields | `error` | ```csharp var parser = new LuceneQueryParser(); // Simple term var result = parser.Parse("status:active"); // Quoted phrase result = parser.Parse("name:\"John Smith\""); // Default field search (when configured) result = parser.Parse("error"); ``` ### Existence Queries Check if a field has any value or is missing: | Syntax | Description | |--------|-------------| | `_exists_:field` | Field has any value | | `_missing_:field` | Field has no value (null or missing) | ```csharp // Find documents with a title var result = parser.Parse("_exists_:title"); // Find documents without a description result = parser.Parse("_missing_:description"); ``` ### Wildcard Queries Use wildcards for partial matching: | Wildcard | Description | Example | |----------|-------------|---------| | `*` | Matches zero or more characters | `name:john*` | | `?` | Matches exactly one character | `name:jo?n` | ```csharp // Prefix match var result = parser.Parse("name:john*"); // Single character wildcard result = parser.Parse("code:A?123"); ``` ::: warning Leading Wildcards Leading wildcards (`*value`) can be expensive. Use [validation options](./validation) to disable them if needed. ::: ### Regex Queries Use regular expressions enclosed in forward slashes: ``` field:/regex/ ``` Example: ```csharp // Match email patterns var result = parser.Parse("email:/.*@example\\.com/"); ``` ## Range Queries Range queries filter numeric or date fields within bounds. ### Bracket Syntax | Syntax | Description | |--------|-------------| | `[min TO max]` | Inclusive on both ends | | `{min TO max}` | Exclusive on both ends | | `[min TO max}` | Inclusive min, exclusive max | | `{min TO max]` | Exclusive min, inclusive max | Examples: ```csharp // Inclusive range: 1 <= value <= 5 var result = parser.Parse("field:[1 TO 5]"); // Exclusive range: 1 < value < 5 result = parser.Parse("field:{1 TO 5}"); // Mixed: 1 <= value < 5 result = parser.Parse("field:[1 TO 5}"); ``` ### Shorthand Syntax Use `..` as shorthand for inclusive ranges: ```csharp // Equivalent to field:[1 TO 5] var result = parser.Parse("field:1..5"); ``` ### Unbounded Ranges Use `*` for unbounded sides: ```csharp // All values before 2024 var result = parser.Parse("date:{* TO 2024-01-01}"); // All values 10 and above result = parser.Parse("count:[10 TO *]"); ``` ### Comparison Operators Use comparison operators for single-sided ranges: | Operator | Description | Equivalent | |----------|-------------|------------| | `>` | Greater than | `{value TO *}` | | `>=` | Greater than or equal | `[value TO *]` | | `<` | Less than | `{* TO value}` | | `<=` | Less than or equal | `{* TO value]` | ```csharp // Greater than 10 var result = parser.Parse("age:>10"); // Greater than or equal to 10 result = parser.Parse("age:>=10"); // Less than 100 result = parser.Parse("price:<100"); // Less than or equal to 100 result = parser.Parse("price:<=100"); ``` ## Boolean Operators Combine queries with boolean logic: | Operator | Description | Alternative | |----------|-------------|-------------| | `AND` | Both conditions must match | `&&` | | `OR` | Either condition must match | `\|\|` | | `NOT` | Negate the following condition | `!` | ### Examples ```csharp // AND - both must match var result = parser.Parse("status:active AND type:user"); // OR - either must match result = parser.Parse("status:active OR status:pending"); // NOT - exclude matches result = parser.Parse("status:active AND NOT deleted:true"); // Complex boolean result = parser.Parse("((status:active AND type:user) OR type:admin) AND NOT deleted:true"); ``` ### Prefix Operators Use prefix operators for required/excluded terms: | Prefix | Description | |--------|-------------| | `+` | Term must be present (required) | | `-` | Term must not be present (excluded) | ```csharp // Required term var result = parser.Parse("+status:active"); // Excluded term result = parser.Parse("-deleted:true"); // Combined result = parser.Parse("+status:active -deleted:true type:user"); ``` ## Grouping Use parentheses to group clauses and control precedence: ```csharp // Group OR conditions var result = parser.Parse("(status:active OR status:pending) AND type:user"); // Nested groups result = parser.Parse("((a:1 OR b:2) AND c:3) OR d:4"); ``` ### Field Grouping Apply a field to multiple values: ```csharp // Field applies to all terms in group var result = parser.Parse("status:(active OR pending OR review)"); ``` ## Date Math Date fields support date math expressions for relative dates. ### Anchor Date Expressions start with an anchor: * `now` - Current date/time * `2024-01-01||` - Specific date followed by `||` ### Math Operations | Operation | Description | |-----------|-------------| | `+1d` | Add 1 day | | `-1d` | Subtract 1 day | | `/d` | Round down to day | ### Supported Units | Unit | Description | |------|-------------| | `y` | Years | | `M` | Months | | `w` | Weeks | | `d` | Days | | `h` or `H` | Hours | | `m` | Minutes | | `s` | Seconds | ### Examples Assuming current time is `2024-06-15 12:00:00`: | Expression | Result | |------------|--------| | `now` | 2024-06-15 12:00:00 | | `now+1h` | 2024-06-15 13:00:00 | | `now-1d` | 2024-06-14 12:00:00 | | `now-7d` | 2024-06-08 12:00:00 | | `now/d` | 2024-06-15 00:00:00 | | `now-1M/M` | 2024-05-01 00:00:00 | ```csharp // Last 7 days var result = parser.Parse("created:[now-7d TO now]"); // Last month result = parser.Parse("created:[now-1M/M TO now/M]"); // Future dates result = parser.Parse("expires:[now TO now+30d]"); ``` ## Geo Proximity Queries Filter documents by geographic distance from a point. ### Syntax ``` geofield:location~distance ``` Where: * `location` can be a geohash, coordinates, or resolvable location (zip code, city) * `distance` is a number followed by a unit (`mi`, `km`, `m`) ### Examples ```csharp // Within 75 miles of a geohash var result = parser.Parse("location:abc123~75mi"); // Within 75 miles of a zip code (requires geo resolver) result = parser.Parse("location:75044~75mi"); // Within 10 kilometers result = parser.Parse("location:51.5,-0.1~10km"); ``` ### Configuration Geo queries require a location resolver: ```csharp var parser = new ElasticQueryParser(c => c .UseGeo(location => { // Resolve location string to coordinates if (location == "75044") return "32.9,-96.8"; return location; })); ``` ## Geo Range Queries Filter documents within a geographic bounding box. ### Syntax ``` geofield:[topLeft TO bottomRight] ``` ### Examples ```csharp // Bounding box with geohashes var result = parser.Parse("location:[u4pruydqqv TO u4pruydr2n]"); // Bounding box with coordinates result = parser.Parse("location:[51.5,-0.2 TO 51.4,-0.1]"); ``` ## Nested Document Queries When using Elasticsearch, queries on nested document fields work automatically with the `ElasticQueryParser`: ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseNested()); // Query nested field - automatically wrapped in nested query var query = await parser.BuildQueryAsync("comments.author:john"); ``` ::: info Elasticsearch Limitation Standard Elasticsearch `query_string` does not support nested documents. Foundatio.Parsers automatically detects nested fields and wraps queries appropriately. ::: ## Boosting Boost the relevance of specific terms: ```csharp // Boost a term var result = parser.Parse("title:important^2"); // Boost a phrase result = parser.Parse("title:\"very important\"^3"); ``` ## Fuzzy Queries Use `~` for fuzzy matching (edit distance): ```csharp // Fuzzy match with default edit distance var result = parser.Parse("name:john~"); // Fuzzy match with specific edit distance result = parser.Parse("name:john~2"); ``` ## Escaping Special Characters Escape special characters with backslash: ``` + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ / ``` ```csharp // Escape colon in value var result = parser.Parse("url:https\\://example.com"); // Escape parentheses result = parser.Parse("name:John\\ \\(Jr\\)"); ``` ## Complete Example ```csharp using Foundatio.Parsers.LuceneQueries; using Foundatio.Parsers.LuceneQueries.Visitors; var parser = new LuceneQueryParser(); // Complex query string query = @" (status:active OR status:pending) AND created:[now-30d TO now] AND NOT deleted:true AND (name:john* OR email:*@example.com) "; var result = parser.Parse(query); // Debug the AST Console.WriteLine(DebugQueryVisitor.Run(result)); // Regenerate (normalized) string normalized = GenerateQueryVisitor.Run(result); ``` ## Next Steps * [Aggregation Syntax](./aggregation-syntax) - Dynamic aggregation expressions * [Field Aliases](./field-aliases) - Map field names * [Validation](./validation) - Validate and restrict queries --- --- url: /query.md --- # Query Syntax The query syntax is based on Lucene syntax. ## Basic * `field:value` exact match * `field:"Eric Smith"` exact match with quoted string value * `_exists_:field` matches if there is any value for the field * `_missing_:field` matches if there is not a value for the field ## Ranges Ranges can be specified for date or numeric fields. Inclusive ranges are specified with square brackets `[min TO max]` and exclusive ranges with curly brackets `{min TO max}`. * `datefield:[2012-01-01 TO 2012-12-31]` matches all days in 2012 for `datefield` * `numberfield:[1 TO 5]` matches any number between 1 and 5 on `numberfield` * `numberfield:1..5` shorthand for above query * `numberfield:1..5` shorthand for above query * `datefield:{* TO 2012-01-01}` matches dates before 2012 * `numberfield:[10 TO *]` matches values 10 and above * `numberfield:[1 TO 5}` matches numbers from 1 up to but not including 5 Ranges with one side unbounded can use the following syntax: * `age:>10` matches age greater than 10 * `age:>=10` matches age greater or equal to 10 * `age:<10` matches age less than 10 * `age:<=10` matches age less than or equal to 10 ## Boolean operators `AND` and `OR` are used to combine filter criteria. `NOT` can be used to negate filter criteria `((field:quick AND otherfield:fox) OR (field:brown AND otherfield:fox) OR otherfield:fox) AND NOT thirdfield:news` ## Grouping Multiple terms or clauses can be grouped together with parentheses, to form sub-queries: `(field:quick OR field:brown) AND otherfield:fox` ## Date Math Parameters which accept a formatted date value understand date math. The expression starts with an anchor date, which can either be `now`, or a date string ending with ||. This anchor date can optionally be followed by one or more maths expressions: `+1h` Add one hour `-1d` Subtract one day Supported units are: * `y` Years * `M` Months * `w` Weeks * `d` Days * `h` Hours * `H` Hours * `m` Minutes * `s` Seconds Assuming now is `2001-01-01 12:00:00`, some examples are: * `now+1h` now in milliseconds plus one hour. Resolves to: 2001-01-01 13:00:00 * `now-1h` now in milliseconds minus one hour. Resolves to: 2001-01-01 11:00:00 ## Geo Proximity Queries Geo proximity queries allow filtering data by documents that have a geo field value located within a given proximity of a geo coordinate. Locations can be resolved using a provided geo coding function that can translate something like "Dallas, TX" into a geo coordinate. Examples: * Within 75 miles of abc geohash: geofield:abc~75mi * Within 75 miles of 75044: geofield:75044~75mi ## Geo Range Queries Geo range queries allow filtering data by documents that have a geo field value located within a bounding box. It uses the same syntax as other Lucene range queries, but the range is the top left and bottom right geo coordinates. Examples: * Within coordinates rectangle: geofield:\[geohash1..geohash2] ## Nested Document Queries Elasticsearch does not support querying nested documents using the query\_string query, but when using this library queries on those fields should work automatically. --- --- url: /guide/sql-query-parser.md --- # SQL Integration The `SqlQueryParser` extends the base Lucene parser to generate Dynamic LINQ expressions for Entity Framework Core. This enables powerful query capabilities for SQL databases. ## Installation ```bash dotnet add package Foundatio.Parsers.SqlQueries ``` ## Basic Usage ```csharp using Foundatio.Parsers.SqlQueries; using Microsoft.EntityFrameworkCore; var parser = new SqlQueryParser(c => c .SetDefaultFields(new[] { "Name", "Description" })); // Get context from your DbContext await using var db = new MyDbContext(); var context = parser.GetContext(db.Products.EntityType); // Convert query to Dynamic LINQ string dynamicLinq = await parser.ToDynamicLinqAsync( "status:active AND price:>100", context); // Use with EF Core var results = await db.Products .Where(parser.ParsingConfig, dynamicLinq) .ToListAsync(); ``` ## Configuration ### Basic Configuration ```csharp var parser = new SqlQueryParser(c => c // Logging .SetLoggerFactory(loggerFactory) // Default fields for unqualified search terms .SetDefaultFields(new[] { "Name", "Description", "Tags" }) // Search operator for default fields .SetSearchOperator(SqlSearchOperator.Contains) // Full-text search fields .SetFullTextFields(new[] { "Name", "Description" }) // Field aliases .UseFieldMap(new Dictionary { { "user", "CreatedBy.Name" } }) // Query includes .UseIncludes(new Dictionary { { "active", "Status == \"Active\"" } }) // Validation .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "Name", "Status", "Price", "Created" } }) // Navigation depth limit .SetFieldDepth(3)); ``` ### Configuration Methods | Method | Description | |--------|-------------| | `SetLoggerFactory(factory)` | Set logging factory | | `SetDefaultFields(fields)` | Default fields for unqualified terms | | `SetSearchOperator(operator)` | Default search operator (Equals, Contains, StartsWith) | | `SetFullTextFields(fields)` | Fields using SQL full-text search | | `SetSearchTokenizer(tokenizer)` | Custom search term tokenization | | `SetDateTimeParser(parser)` | Custom DateTime parsing | | `SetDateOnlyParser(parser)` | Custom DateOnly parsing | | `SetFieldDepth(depth)` | Maximum navigation property depth | | `UseFieldMap(map)` | Static field alias map | | `UseFieldResolver(resolver)` | Dynamic field resolver | | `UseIncludes(includes)` | Query include definitions | | `SetValidationOptions(options)` | Validation configuration | ## Query Context The `SqlQueryVisitorContext` provides entity metadata for query generation: ```csharp // Get context from entity type var context = parser.GetContext(db.Products.EntityType); // Context contains: // - Fields: List of EntityFieldInfo with type information // - ValidationOptions: Automatically populated from entity properties // - SearchTokenizer, DateTimeParser, etc. from configuration ``` ### EntityFieldInfo Each field has metadata: ```csharp public class EntityFieldInfo { public string Name { get; } // Property name public string FullName { get; } // Full path (e.g., "Category.Name") public bool IsNumber { get; } // Numeric type public bool IsDate { get; } // DateTime type public bool IsDateOnly { get; } // DateOnly type public bool IsBoolean { get; } // Boolean type public bool IsCollection { get; } // Collection navigation public bool IsNavigation { get; } // Navigation property public EntityFieldInfo Parent { get; } // Parent for nested fields } ``` ## Query Translation ### Term Queries ```csharp // Exact match "status:active" // Generates: Status == "active" // Quoted phrase "name:\"John Smith\"" // Generates: Name == "John Smith" // Wildcard (suffix) "name:john*" // Generates: Name.StartsWith("john") // Wildcard (contains) "name:*john*" // Generates: Name.Contains("john") ``` ### Range Queries ```csharp // Inclusive range "price:[100 TO 500]" // Generates: (Price >= 100 AND Price <= 500) // Exclusive range "price:{100 TO 500}" // Generates: (Price > 100 AND Price < 500) // Comparison operators "price:>100" // Generates: Price > 100 "price:>=100" // Generates: Price >= 100 ``` ### Boolean Operators ```csharp // AND "status:active AND price:>100" // Generates: Status == "active" AND Price > 100 // OR "status:active OR status:pending" // Generates: Status == "active" OR Status == "pending" // NOT "NOT status:deleted" // Generates: NOT (Status == "deleted") ``` ### Existence Queries ```csharp // Field exists (not null) "_exists_:description" // Generates: Description != null // Field missing (null) "_missing_:description" // Generates: Description == null ``` ## Default Field Search When no field is specified, the query searches default fields: ```csharp var parser = new SqlQueryParser(c => c .SetDefaultFields(new[] { "Name", "Description" }) .SetSearchOperator(SqlSearchOperator.Contains)); var context = parser.GetContext(db.Products.EntityType); // Search term without field string sql = await parser.ToDynamicLinqAsync("laptop", context); // Generates: Name.Contains("laptop") OR Description.Contains("laptop") ``` ### Search Operators | Operator | Description | Generated SQL | |----------|-------------|---------------| | `Equals` | Exact match | `Field == "value"` | | `Contains` | Contains substring | `Field.Contains("value")` | | `StartsWith` | Starts with | `Field.StartsWith("value")` | ## Full-Text Search For SQL Server full-text search: ```csharp var parser = new SqlQueryParser(c => c .SetDefaultFields(new[] { "Name", "Description" }) .SetFullTextFields(new[] { "Name", "Description" })); var context = parser.GetContext(db.Products.EntityType); string sql = await parser.ToDynamicLinqAsync("laptop", context); // Generates: FTS.Contains(Name, "\"laptop*\"") OR FTS.Contains(Description, "\"laptop*\"") ``` ### FTS Helper Class The `FTS` class wraps `EF.Functions.Contains`: ```csharp // In your queries db.Products.Where(p => FTS.Contains(p.Name, "search term")); // Equivalent to db.Products.Where(p => EF.Functions.Contains(p.Name, "search term")); ``` ## Navigation Properties The parser automatically handles EF Core navigation properties: ### Non-Collection Navigation ```csharp // Query on related entity "category.name:electronics" // Generates: Category.Name == "electronics" ``` ### Collection Navigation ```csharp // Query on collection items "tags.name:sale" // Generates: Tags.Any(t => t.Name == "sale") ``` ### Depth Limiting ```csharp var parser = new SqlQueryParser(c => c .SetFieldDepth(2)); // Limit navigation depth // "category.parent.grandparent.name" would be limited ``` ## Date Handling ### Date Math ```csharp // Relative dates "created:>now-7d" // Generates: Created > DateTime.Parse("2024-01-08") // 7 days ago // Date ranges "created:[now-30d TO now]" // Generates: (Created >= ... AND Created <= ...) ``` ### Custom Date Parsing ```csharp var parser = new SqlQueryParser(c => c .SetDateTimeParser(value => { if (value.StartsWith("now")) return ParseDateMath(value); return DateTime.Parse(value); }) .SetDateOnlyParser(value => { if (value.StartsWith("now")) return DateOnly.FromDateTime(ParseDateMath(value)); return DateOnly.Parse(value); })); ``` ## Validation ### Automatic Field Validation The parser automatically populates allowed fields from the entity type: ```csharp var context = parser.GetContext(db.Products.EntityType); // context.ValidationOptions.AllowedFields contains all entity properties ``` ### Custom Validation ```csharp var parser = new SqlQueryParser(c => c .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "Name", "Status", "Price" }, RestrictedFields = { "InternalNotes" }, AllowLeadingWildcards = false })); var result = await parser.ValidateAsync(query, context); if (!result.IsValid) { Console.WriteLine($"Invalid: {result.Message}"); } ``` ## Entity Type Filtering Control which properties and navigations are exposed: ```csharp var parser = new SqlQueryParser(c => c // Filter properties .UseEntityTypePropertyFilter(property => { // Exclude internal properties return !property.Name.StartsWith("Internal"); }) // Filter navigation properties .UseEntityTypeNavigationFilter(navigation => { // Exclude audit navigations return navigation.Name != "AuditLogs"; }) // Filter skip navigations (many-to-many) .UseEntityTypeSkipNavigationFilter(skipNav => { return true; // Include all })); ``` ## Complete Example ```csharp using Foundatio.Parsers.SqlQueries; using Microsoft.EntityFrameworkCore; public class ProductSearchService { private readonly MyDbContext _db; private readonly SqlQueryParser _parser; public ProductSearchService(MyDbContext db, ILoggerFactory loggerFactory) { _db = db; _parser = new SqlQueryParser(c => c .SetLoggerFactory(loggerFactory) .SetDefaultFields(new[] { "Name", "Description" }) .SetFullTextFields(new[] { "Name", "Description" }) .UseFieldMap(new Dictionary { { "category", "Category.Name" }, { "brand", "Brand.Name" } }) .UseIncludes(new Dictionary { { "available", "Status == \"Active\" AND Inventory > 0" }, { "sale", "DiscountPercent > 0" } }) .SetFieldDepth(2) .SetValidationOptions(new QueryValidationOptions { AllowLeadingWildcards = false, AllowedMaxNodeDepth = 5 })); } public async Task> SearchAsync(string query) { var context = _parser.GetContext(_db.Products.EntityType); // Validate var validation = await _parser.ValidateAsync(query, context); if (!validation.IsValid) throw new ArgumentException(validation.Message); // Convert to Dynamic LINQ string dynamicLinq = await _parser.ToDynamicLinqAsync(query, context); // Execute query return await _db.Products .Include(p => p.Category) .Include(p => p.Brand) .Where(_parser.ParsingConfig, dynamicLinq) .ToListAsync(); } public async Task> SearchPagedAsync( string query, string sort = null, int page = 1, int pageSize = 20) { var context = _parser.GetContext(_db.Products.EntityType); string dynamicLinq = await _parser.ToDynamicLinqAsync(query, context); var baseQuery = _db.Products .Include(p => p.Category) .Where(_parser.ParsingConfig, dynamicLinq); // Get total count int total = await baseQuery.CountAsync(); // Apply sorting if (!string.IsNullOrEmpty(sort)) { // Parse sort: "-created +name" -> "Created desc, Name asc" var sortParts = sort.Split(' ', StringSplitOptions.RemoveEmptyEntries) .Select(s => s.StartsWith("-") ? $"{s.Substring(1)} desc" : $"{s.TrimStart('+')} asc"); baseQuery = baseQuery.OrderBy(_parser.ParsingConfig, string.Join(", ", sortParts)); } // Apply paging var items = await baseQuery .Skip((page - 1) * pageSize) .Take(pageSize) .ToListAsync(); return new PagedResult { Items = items, Total = total, Page = page, PageSize = pageSize }; } } ``` ## Troubleshooting ### Dynamic LINQ Errors ```csharp try { var results = await db.Products .Where(parser.ParsingConfig, dynamicLinq) .ToListAsync(); } catch (ParseException ex) { // Dynamic LINQ parsing error Console.WriteLine($"Parse error: {ex.Message}"); } ``` ### Field Not Found ```csharp var context = parser.GetContext(db.Products.EntityType); // Check available fields foreach (var field in context.Fields) { Console.WriteLine($"{field.FullName}: {field.GetType().Name}"); } ``` ### Navigation Depth Issues ```csharp // If queries on deep navigations fail, increase depth var parser = new SqlQueryParser(c => c .SetFieldDepth(4)); // Allow deeper navigation ``` ## Next Steps * [Query Syntax](./query-syntax) - Query syntax reference * [Validation](./validation) - Query validation * [Field Aliases](./field-aliases) - Field mapping --- --- url: /guide/troubleshooting.md --- # Troubleshooting This guide covers common issues and their solutions when using Foundatio.Parsers. ## Parse Errors ### Unexpected Character **Error:** `Unexpected character at position X` **Cause:** The query contains invalid syntax. **Solution:** ```csharp try { var result = await parser.ParseAsync(query); } catch (FormatException ex) { // Get position information var cursor = ex.Data["cursor"] as Cursor; if (cursor != null) { Console.WriteLine($"Error at line {cursor.Line}, column {cursor.Column}"); } Console.WriteLine($"Message: {ex.Message}"); } ``` **Common causes:** * Unbalanced parentheses: `(status:active` * Invalid range syntax: `field:[1 TO]` * Unescaped special characters: `url:https://example.com` **Fix:** Escape special characters: ```csharp // Escape colons in values "url:https\\://example.com" // Or use quotes "url:\"https://example.com\"" ``` ### Empty Query **Error:** Query returns no results or empty AST. **Solution:** Check for empty or whitespace-only queries: ```csharp if (string.IsNullOrWhiteSpace(query)) { // Handle empty query return new MatchAllQuery(); } var result = await parser.ParseAsync(query); ``` ## Validation Errors ### Field Not Allowed **Error:** `Field 'X' is not allowed` **Cause:** Field is not in the `AllowedFields` list. **Solution:** ```csharp var options = new QueryValidationOptions(); options.AllowedFields.Add("status"); options.AllowedFields.Add("name"); options.AllowedFields.Add("created"); // Add all fields users should be able to query ``` ### Field Restricted **Error:** `Field 'X' is restricted` **Cause:** Field is in the `RestrictedFields` list. **Solution:** Remove from restricted list or use a different field: ```csharp var options = new QueryValidationOptions(); options.RestrictedFields.Remove("fieldname"); ``` ### Leading Wildcard Not Allowed **Error:** `Leading wildcards are not allowed` **Cause:** Query contains `*value` and `AllowLeadingWildcards` is false. **Solution:** ```csharp // Option 1: Allow leading wildcards var options = new QueryValidationOptions { AllowLeadingWildcards = true }; // Option 2: Rewrite query to use trailing wildcard // Instead of: *smith // Use: smith* (if appropriate for your use case) ``` ### Query Too Deep **Error:** `Query exceeds maximum depth of X` **Cause:** Query nesting exceeds `AllowedMaxNodeDepth`. **Solution:** ```csharp var options = new QueryValidationOptions { AllowedMaxNodeDepth = 15 // Increase limit }; ``` ### Unresolved Field **Error:** `Field 'X' could not be resolved` **Cause:** Field doesn't exist in Elasticsearch mapping and `AllowUnresolvedFields` is false. **Solution:** ```csharp // Option 1: Allow unresolved fields var options = new QueryValidationOptions { AllowUnresolvedFields = true }; // Option 2: Add field alias var parser = new ElasticQueryParser(c => c .UseFieldMap(new Dictionary { { "user_field", "actual.field.path" } })); // Option 3: Refresh mappings (if recently added field, wait for auto-refresh or force it) parser.Configuration.MappingResolver.RefreshMapping(); ``` ## Elasticsearch Issues ### Query Returns No Results **Possible causes:** 1. **Analyzed vs keyword field:** ```csharp // Check if field is analyzed var resolver = parser.Configuration.MappingResolver; bool isAnalyzed = resolver.IsPropertyAnalyzed("title"); // For exact matches on analyzed fields, use .keyword "title.keyword:\"Exact Title\"" ``` 2. **Case sensitivity:** ```csharp // Elasticsearch is case-sensitive by default // Use lowercase or configure analyzer "status:Active" // May not match "active" ``` 3. **Filter vs query context:** ```csharp // By default, queries are wrapped in filter (no scoring) // For full-text search, enable scoring var context = new ElasticQueryVisitorContext { UseScoring = true }; var query = await parser.BuildQueryAsync("search terms", context); ``` ### Nested Query Not Working **Cause:** Nested support not enabled or field not detected as nested. **Solution:** ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseNested()); // Enable nested support // Verify field is nested var resolver = parser.Configuration.MappingResolver; bool isNested = resolver.IsNestedPropertyType("comments"); ``` ### Geo Query Fails **Cause:** Geo resolver not configured or returns invalid coordinates. **Solution:** ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .UseGeo(location => { // Ensure valid coordinates are returned Console.WriteLine($"Resolving: {location}"); var coords = ResolveLocation(location); Console.WriteLine($"Resolved to: {coords}"); return coords; })); ``` ### Aggregation Field Error **Error:** `Fielddata is disabled on text fields by default` **Cause:** Trying to aggregate on an analyzed text field. **Solution:** ```csharp // The parser should automatically use .keyword sub-field // If not, check your mapping has keyword sub-field: // "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } } // Or use field alias var parser = new ElasticQueryParser(c => c .UseFieldMap(new Dictionary { { "title", "title.keyword" } })); ``` ## SQL/EF Core Issues ### Dynamic LINQ Parse Error **Error:** `No property or field 'X' exists in type 'Y'` **Cause:** Field name doesn't match entity property. **Solution:** ```csharp // Check available fields var context = parser.GetContext(db.Products.EntityType); foreach (var field in context.Fields) { Console.WriteLine($"{field.FullName} ({field.GetType().Name})"); } // Use correct casing (EF Core is case-sensitive) "Status:active" // Not "status:active" ``` ### Navigation Property Error **Error:** Query on navigation property fails. **Solution:** ```csharp // Increase navigation depth var parser = new SqlQueryParser(c => c .SetFieldDepth(3)); // Check if navigation is included var context = parser.GetContext(db.Products.EntityType); var navFields = context.Fields.Where(f => f.IsNavigation); ``` ### Full-Text Search Not Working **Cause:** Full-text catalog not configured or fields not indexed. **Solution:** ```csharp // Ensure full-text catalog exists // CREATE FULLTEXT CATALOG ftCatalog AS DEFAULT; // Ensure full-text index exists on table // CREATE FULLTEXT INDEX ON Products(Name, Description) KEY INDEX PK_Products; // Configure parser var parser = new SqlQueryParser(c => c .SetFullTextFields(new[] { "Name", "Description" })); ``` ## Include Issues ### Include Not Expanding **Cause:** Include name not found in resolver. **Solution:** ```csharp var parser = new ElasticQueryParser(c => c .UseIncludes(async name => { var include = await GetInclude(name); if (include == null) { Console.WriteLine($"Include not found: {name}"); } return include; })); ``` ### Recursive Include Detected **Error:** `Recursive include detected: X` **Cause:** Include references itself directly or indirectly. **Solution:** ```csharp // Check your include definitions for cycles var includes = new Dictionary { { "a", "@include:b" }, { "b", "@include:a" } // Cycle! }; // Fix by removing the cycle var includes = new Dictionary { { "a", "@include:b" }, { "b", "status:active" } // No cycle }; ``` ## Performance Issues ### Slow Query Parsing **Cause:** Complex queries or many visitors. **Solution:** ```csharp // Reuse parser instance private readonly ElasticQueryParser _parser; public MyService() { _parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index")); } // Don't create new parser for each request ``` ### Slow Mapping Resolution **Cause:** Mapping fetched from Elasticsearch on every query. **Solution:** ```csharp // Mappings are cached by default (auto-refresh at most once per minute) // Manual refresh is typically only needed in unit tests: parser.Configuration.MappingResolver.RefreshMapping(); // For production, create resolver once and share var resolver = ElasticMappingResolver.Create(client, "my-index"); var parser1 = new ElasticQueryParser(c => c.UseMappings(resolver)); var parser2 = new ElasticQueryParser(c => c.UseMappings(resolver)); ``` ## Debugging ### Enable Logging ```csharp var parser = new ElasticQueryParser(c => c .SetLoggerFactory(loggerFactory)); // Or for LuceneQueryParser with tracing var parser = new LuceneQueryParser { Tracer = new LoggingTracer(logger) }; ``` ### Inspect AST ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync(query); // Print AST structure string debug = DebugQueryVisitor.Run(result); Console.WriteLine(debug); ``` ### Inspect Generated Query ```csharp // Regenerate query string from AST string regenerated = GenerateQueryVisitor.Run(result); Console.WriteLine($"Parsed as: {regenerated}"); ``` ### Inspect Elasticsearch Request ```csharp var response = await client.SearchAsync(s => s .Index("my-index") .Query(_ => query)); // Get the actual request sent string request = response.GetRequest(); Console.WriteLine(request); // Check for errors if (!response.IsValid) { Console.WriteLine(response.GetErrorMessage()); } ``` ## Getting Help If you're still having issues: 1. **Check the tests** - The test files contain many examples: * `tests/Foundatio.Parsers.LuceneQueries.Tests/` * `tests/Foundatio.Parsers.ElasticQueries.Tests/` * `tests/Foundatio.Parsers.SqlQueries.Tests/` 2. **Enable detailed logging** - Use `LogLevel.Trace` for maximum detail 3. **Open an issue** - [GitHub Issues](https://github.com/FoundatioFx/Foundatio.Parsers/issues) 4. **Join Discord** - [Foundatio Discord](https://discord.gg/6HxgFCx) --- --- url: /guide/validation.md --- # Validation Foundatio.Parsers provides comprehensive query validation to ensure queries are safe, well-formed, and within allowed boundaries. This is essential when accepting queries from untrusted sources like user input or APIs. ## Basic Validation ### Syntax Validation Validate query syntax without executing: ```csharp using Foundatio.Parsers.LuceneQueries; var result = await QueryValidator.ValidateQueryAsync("status:active AND created:>2024-01-01"); if (result.IsValid) { Console.WriteLine("Query is valid"); } else { Console.WriteLine($"Invalid: {result.Message}"); foreach (var error in result.ValidationErrors) { Console.WriteLine($" Position {error.Index}: {error.Message}"); } } ``` ### Throw on Invalid Use `ValidateQueryAndThrowAsync` to throw an exception for invalid queries: ```csharp try { await QueryValidator.ValidateQueryAndThrowAsync("invalid::"); } catch (QueryValidationException ex) { Console.WriteLine($"Validation failed: {ex.Message}"); Console.WriteLine($"Errors: {ex.Result.ValidationErrors.Count}"); } ``` ## Validation Options Configure validation behavior with `QueryValidationOptions`: ```csharp var options = new QueryValidationOptions { AllowedFields = { "status", "name", "created" }, RestrictedFields = { "password", "secret" }, AllowLeadingWildcards = false, AllowUnresolvedFields = false, AllowUnresolvedIncludes = false, AllowedMaxNodeDepth = 10, AllowedOperations = { "terms", "date", "min", "max" }, RestrictedOperations = { "tophits" } }; var result = await QueryValidator.ValidateQueryAsync(query, options); ``` ### Option Reference | Option | Type | Default | Description | |--------|------|---------|-------------| | `AllowedFields` | `HashSet` | Empty (all allowed) | Whitelist of allowed field names | | `RestrictedFields` | `HashSet` | Empty | Blacklist of restricted field names | | `AllowLeadingWildcards` | `bool` | `true` | Allow `*value` wildcards | | `AllowUnresolvedFields` | `bool` | `true` | Allow fields not in mapping | | `AllowUnresolvedIncludes` | `bool` | `false` | Allow includes that don't resolve | | `AllowedMaxNodeDepth` | `int` | `0` (unlimited) | Maximum query nesting depth | | `AllowedOperations` | `HashSet` | Empty (all allowed) | Whitelist of aggregation operations | | `RestrictedOperations` | `HashSet` | Empty | Blacklist of aggregation operations | | `ShouldThrow` | `bool` | `false` | Throw exception on validation failure | ## Field Restrictions ### Allowed Fields (Whitelist) Only allow specific fields: ```csharp var options = new QueryValidationOptions(); options.AllowedFields.Add("status"); options.AllowedFields.Add("name"); options.AllowedFields.Add("created"); // Valid var result = await QueryValidator.ValidateQueryAsync("status:active", options); // result.IsValid == true // Invalid - field not in whitelist result = await QueryValidator.ValidateQueryAsync("password:secret", options); // result.IsValid == false ``` ### Restricted Fields (Blacklist) Block specific fields: ```csharp var options = new QueryValidationOptions(); options.RestrictedFields.Add("password"); options.RestrictedFields.Add("apiKey"); options.RestrictedFields.Add("secret"); // Valid var result = await QueryValidator.ValidateQueryAsync("status:active", options); // result.IsValid == true // Invalid - field is restricted result = await QueryValidator.ValidateQueryAsync("password:test", options); // result.IsValid == false ``` ### Field Aliases and Validation When using field aliases, validation uses the alias names (not resolved names): ```csharp var parser = new ElasticQueryParser(c => c .UseFieldMap(new Dictionary { { "user", "data.user.identity" } }) .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "user", "status" } // Use alias names })); // Valid - uses allowed alias var query = await parser.BuildQueryAsync("user:john"); // Invalid - uses resolved name directly // (unless "data.user.identity" is also in AllowedFields) await Assert.ThrowsAsync(() => parser.BuildQueryAsync("data.user.identity:john")); ``` ## Wildcard Restrictions ### Disable Leading Wildcards Leading wildcards (`*value`) can be expensive. Disable them: ```csharp var options = new QueryValidationOptions { AllowLeadingWildcards = false }; // Valid - trailing wildcard var result = await QueryValidator.ValidateQueryAsync("name:john*", options); // result.IsValid == true // Invalid - leading wildcard result = await QueryValidator.ValidateQueryAsync("name:*smith", options); // result.IsValid == false // result.Message contains "wildcard" ``` ## Depth Restrictions Limit query nesting to prevent complex queries: ```csharp var options = new QueryValidationOptions { AllowedMaxNodeDepth = 5 }; // Valid - shallow nesting var result = await QueryValidator.ValidateQueryAsync("(a:1 AND b:2)", options); // result.IsValid == true // Invalid - too deep result = await QueryValidator.ValidateQueryAsync( "((((((a:1 AND b:2))))))", options); // result.IsValid == false ``` ## Unresolved Field Handling Control behavior when fields don't exist in mappings: ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .SetValidationOptions(new QueryValidationOptions { AllowUnresolvedFields = false })); // If "nonexistent" is not in the index mapping var result = await parser.ValidateQueryAsync("nonexistent:value"); // result.IsValid == false // result.UnresolvedFields contains "nonexistent" ``` ## Aggregation Validation ### Allowed Operations Whitelist specific aggregation operations: ```csharp var options = new QueryValidationOptions(); options.AllowedOperations.Add("terms"); options.AllowedOperations.Add("date"); options.AllowedOperations.Add("min"); options.AllowedOperations.Add("max"); var result = await QueryValidator.ValidateAggregationsAsync( "terms:(status min:amount)", options); // result.IsValid == true result = await QueryValidator.ValidateAggregationsAsync( "tophits:_", options); // result.IsValid == false - tophits not in allowed list ``` ### Restricted Operations Blacklist specific operations: ```csharp var options = new QueryValidationOptions(); options.RestrictedOperations.Add("tophits"); // Expensive operation var result = await QueryValidator.ValidateAggregationsAsync( "terms:(status tophits:_)", options); // result.IsValid == false ``` ## Validation Result The `QueryValidationResult` provides detailed information: ```csharp var result = await QueryValidator.ValidateQueryAsync(query, options); // Basic status bool isValid = result.IsValid; string message = result.Message; // Detailed errors foreach (var error in result.ValidationErrors) { Console.WriteLine($"Error at position {error.Index}: {error.Message}"); } // Referenced fields foreach (var field in result.ReferencedFields) { Console.WriteLine($"Field used: {field}"); } // Unresolved fields (not found in mapping) foreach (var field in result.UnresolvedFields) { Console.WriteLine($"Unknown field: {field}"); } // Referenced includes foreach (var include in result.ReferencedIncludes) { Console.WriteLine($"Include used: {include}"); } // Unresolved includes foreach (var include in result.UnresolvedIncludes) { Console.WriteLine($"Missing include: {include}"); } // Query depth int depth = result.MaxNodeDepth; // Operations used (for aggregations) foreach (var op in result.Operations) { Console.WriteLine($"Operation: {op.Key}, Count: {op.Value}"); } ``` ## With ElasticQueryParser ### Validate Before Building ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "status", "name", "created" }, AllowLeadingWildcards = false, AllowedMaxNodeDepth = 10 })); // Validate query var validation = await parser.ValidateQueryAsync("status:active"); if (!validation.IsValid) { return BadRequest(validation.Message); } // Build query (also validates and throws if invalid) var query = await parser.BuildQueryAsync("status:active"); ``` ### Validate Aggregations ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .SetValidationOptions(new QueryValidationOptions { AllowedOperations = { "terms", "date", "min", "max", "avg" } })); var validation = await parser.ValidateAggregationsAsync( "terms:(status min:amount)"); if (!validation.IsValid) { return BadRequest(validation.Message); } ``` ### Validate Sort ```csharp var parser = new ElasticQueryParser(c => c .UseMappings(client, "my-index") .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "created", "name", "status" } })); var validation = await parser.ValidateSortAsync("-created +name"); if (!validation.IsValid) { return BadRequest(validation.Message); } ``` ## Context-Based Validation Access validation results from the context: ```csharp var parser = new ElasticQueryParser(c => c .SetValidationOptions(options)); var context = new ElasticQueryVisitorContext(); var query = await parser.BuildQueryAsync("status:active", context); // Get validation result from context var validation = context.GetValidationResult(); // Check validity if (!context.IsValid()) { var errors = context.GetValidationErrors(); var message = context.GetValidationMessage(); } // Throw if invalid context.ThrowIfInvalid(); ``` ## Custom Validation Add custom validation errors: ```csharp var context = new ElasticQueryVisitorContext(); // Add custom validation error context.AddValidationError("Custom error message", position: 5); // Check result var validation = context.GetValidationResult(); // validation.IsValid == false ``` ## Best Practices ### 1. Always Validate User Input ```csharp public async Task Search([FromQuery] string query) { var validation = await parser.ValidateQueryAsync(query); if (!validation.IsValid) { return BadRequest(new { error = "Invalid query", details = validation.Message, errors = validation.ValidationErrors }); } // Safe to execute var results = await ExecuteSearch(query); return Ok(results); } ``` ### 2. Use Whitelists Over Blacklists ```csharp // Preferred - explicit whitelist var options = new QueryValidationOptions { AllowedFields = { "status", "name", "created", "category" } }; // Less secure - blacklist can miss fields var options = new QueryValidationOptions { RestrictedFields = { "password" } // What about "apiKey"? }; ``` ### 3. Limit Query Complexity ```csharp var options = new QueryValidationOptions { AllowedMaxNodeDepth = 10, AllowLeadingWildcards = false }; ``` ### 4. Log Validation Failures ```csharp var validation = await parser.ValidateQueryAsync(query); if (!validation.IsValid) { logger.LogWarning( "Invalid query from user {UserId}: {Query}. Errors: {Errors}", userId, query, validation.Message); return BadRequest(validation.Message); } ``` ### 5. Provide Helpful Error Messages ```csharp var validation = await parser.ValidateQueryAsync(query); if (!validation.IsValid) { var response = new { error = "Query validation failed", message = validation.Message, allowedFields = options.AllowedFields, errors = validation.ValidationErrors.Select(e => new { position = e.Index, message = e.Message }) }; return BadRequest(response); } ``` ## Next Steps * [Field Aliases](./field-aliases) - Map field names * [Query Includes](./query-includes) - Reusable query macros * [Visitors](./visitors) - Custom query transformations --- --- url: /guide/visitors.md --- # Visitors Foundatio.Parsers uses the visitor pattern for AST traversal and transformation. Visitors allow you to inspect, modify, or extract information from parsed queries. ## Visitor Pattern Overview When a query is parsed, it creates an Abstract Syntax Tree (AST) of nodes. Visitors traverse this tree, performing operations on each node type. ```mermaid graph TD Query[Query String] --> Parser[LuceneQueryParser] Parser --> AST[AST Tree] AST --> V1[Visitor 1] V1 --> V2[Visitor 2] V2 --> V3[Visitor 3] V3 --> Output[Result] ``` ## Built-in Visitors ### GenerateQueryVisitor Converts an AST back to a query string: ```csharp using Foundatio.Parsers.LuceneQueries; using Foundatio.Parsers.LuceneQueries.Visitors; var parser = new LuceneQueryParser(); var result = parser.Parse("status:active AND created:>2024-01-01"); // Generate query string from AST string query = GenerateQueryVisitor.Run(result); // Output: "status:active AND created:>2024-01-01" // Async version query = await GenerateQueryVisitor.RunAsync(result); ``` #### With Default Operator ```csharp var context = new QueryVisitorContext { DefaultOperator = GroupOperator.And }; var result = parser.Parse("value1 value2"); string query = await GenerateQueryVisitor.RunAsync(result, context); // Output: "value1 AND value2" ``` ### DebugQueryVisitor Outputs the AST structure for debugging: ```csharp var parser = new LuceneQueryParser(); var result = parser.Parse("(status:active OR status:pending) AND type:user"); string debug = DebugQueryVisitor.Run(result); Console.WriteLine(debug); ``` Output: ``` Group: Operator: And Left - Group: HasParens: True Operator: Or Left - Term: Field: status Term: active Right - Term: Field: status Term: pending Right - Term: Field: type Term: user ``` ### FieldResolverQueryVisitor Resolves field aliases: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("user:john"); var fieldMap = new FieldMap { { "user", "data.user.identity" } }; var resolved = await FieldResolverQueryVisitor.RunAsync(result, fieldMap); // Result: data.user.identity:john ``` ### IncludeVisitor Expands query includes/macros: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("@include:active"); var includes = new Dictionary { { "active", "status:active AND deleted:false" } }; var expanded = await IncludeVisitor.RunAsync(result, includes); // Result: (status:active AND deleted:false) ``` ### ValidationVisitor Validates queries against rules: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("status:active"); var context = new QueryVisitorContext(); context.SetValidationOptions(new QueryValidationOptions { AllowedFields = { "status", "name" } }); await ValidationVisitor.RunAsync(result, context); var validation = context.GetValidationResult(); Console.WriteLine($"Valid: {validation.IsValid}"); ``` ### InvertQueryVisitor Inverts query negation: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("status:active"); var invertVisitor = new InvertQueryVisitor(); var inverted = await invertVisitor.AcceptAsync(result, new QueryVisitorContext()); string query = inverted.ToString(); // Output: "(NOT status:active)" ``` #### With Non-Inverted Fields ```csharp // Some fields should not be inverted var invertVisitor = new InvertQueryVisitor( nonInvertedFields: new[] { "organization_id", "tenant_id" }); var result = await parser.ParseAsync("status:active organization_id:123"); var inverted = await invertVisitor.AcceptAsync(result, context); // Output: "(NOT status:active) organization_id:123" ``` ### RemoveFieldsQueryVisitor Removes specific fields from queries: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("status:active AND secret:value AND name:john"); var removeVisitor = new RemoveFieldsQueryVisitor(new[] { "secret" }); var cleaned = await removeVisitor.AcceptAsync(result, new QueryVisitorContext()); string query = cleaned.ToString(); // Output: "status:active AND name:john" ``` ### CleanupQueryVisitor Simplifies and cleans up the AST: ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("((status:active))"); var cleaned = await CleanupQueryVisitor.RunAsync(result); // Removes unnecessary nesting ``` ### TermToFieldVisitor Converts standalone terms to field queries (used for sort expressions): ```csharp var parser = new LuceneQueryParser(); var result = await parser.ParseAsync("-created +name"); var converted = await TermToFieldVisitor.RunAsync(result); // Converts terms to field nodes for sort processing ``` ## Chained Visitors Multiple visitors can be chained together with priority ordering: ```csharp using Foundatio.Parsers.LuceneQueries.Visitors; var chainedVisitor = new ChainedQueryVisitor(); // Add visitors with priority (lower runs first) chainedVisitor.AddVisitor(new FieldResolverQueryVisitor(fieldResolver), priority: 10); chainedVisitor.AddVisitor(new IncludeVisitor(), priority: 20); chainedVisitor.AddVisitor(new ValidationVisitor(), priority: 30); // Run all visitors var result = await chainedVisitor.AcceptAsync(ast, context); ``` ### Managing Chained Visitors ```csharp var chain = new ChainedQueryVisitor(); // Add visitor chain.AddVisitor(new MyVisitor(), priority: 100); // Remove visitor by type chain.RemoveVisitor(); // Replace visitor chain.ReplaceVisitor(new NewVisitor(), newPriority: 50); // Add before/after specific visitor chain.AddVisitorBefore(new MyVisitor()); chain.AddVisitorAfter(new MyVisitor()); ``` ## Visitor Context Visitors receive a context object for sharing data: ```csharp var context = new QueryVisitorContext(); // Set values context.SetValue("UserId", "123"); context.SetValue("IsAdmin", true); // Get values string userId = context.GetValue("UserId"); bool isAdmin = context.GetValue("IsAdmin"); // Default operator context.DefaultOperator = GroupOperator.And; // Default fields for unqualified terms context.DefaultFields = new[] { "title", "description" }; // Query type context.QueryType = QueryTypes.Query; // or Aggregation, Sort ``` ### Context Extensions ```csharp // Field resolver context.SetFieldResolver(async (field, ctx) => { return fieldMap.GetValueOrDefault(field); }); var resolver = context.GetFieldResolver(); // Include resolver context.SetIncludeResolver(async name => includes.GetValueOrDefault(name)); var includeResolver = context.GetIncludeResolver(); // Validation context.SetValidationOptions(options); var validation = context.GetValidationResult(); context.AddValidationError("Error message", position: 5); context.ThrowIfInvalid(); ``` ## AST Node Types Visitors can handle different node types: | Node Type | Description | Key Properties | |-----------|-------------|----------------| | `GroupNode` | Boolean group | `Left`, `Right`, `Operator`, `HasParens` | | `TermNode` | Single term | `Field`, `Term`, `IsQuotedTerm`, `Boost`, `Proximity` | | `TermRangeNode` | Range query | `Field`, `Min`, `Max`, `MinInclusive`, `MaxInclusive` | | `ExistsNode` | Existence check | `Field` | | `MissingNode` | Missing check | `Field` | ### Node Properties ```csharp // Common to all field nodes string field = node.Field; string unescapedField = node.UnescapedField; bool isNegated = node.IsNegated; string prefix = node.Prefix; // +, -, or null // TermNode specific string term = termNode.Term; bool isQuoted = termNode.IsQuotedTerm; bool isRegex = termNode.IsRegexTerm; string boost = termNode.Boost; string proximity = termNode.Proximity; // TermRangeNode specific string min = rangeNode.Min; string max = rangeNode.Max; bool minInclusive = rangeNode.MinInclusive; bool maxInclusive = rangeNode.MaxInclusive; // GroupNode specific IQueryNode left = groupNode.Left; IQueryNode right = groupNode.Right; GroupOperator op = groupNode.Operator; // And, Or, Default bool hasParens = groupNode.HasParens; ``` ### Node Data Dictionary Nodes have a data dictionary for storing metadata: ```csharp // Store custom data node.Data["CustomKey"] = "CustomValue"; // Retrieve data var value = node.Data.GetValueOrDefault("CustomKey"); // Extension methods for common data node.SetOriginalField("alias"); string original = node.GetOriginalField(); node.SetTimeZone("America/New_York"); string tz = node.GetTimeZone(); ``` ## Visitor Base Classes | Base Class | Description | |------------|-------------| | `QueryNodeVisitorBase` | Non-mutating visitor | | `MutatingQueryNodeVisitorBase` | Can modify nodes | | `QueryNodeVisitorWithResultBase` | Returns a result | | `ChainableQueryVisitor` | Can be chained | | `ChainableMutatingQueryVisitor` | Chainable and mutating | ## Next Steps * [Custom Visitors](./custom-visitors) - Create your own visitors * [Elasticsearch Integration](./elastic-query-parser) - Elasticsearch-specific visitors * [Field Aliases](./field-aliases) - Field resolution --- --- url: /guide/what-is-foundatio-parsers.md --- # What is Foundatio.Parsers? Foundatio.Parsers is a production-grade query parsing library for .NET that provides extensible Lucene-style query syntax parsing with specialized support for Elasticsearch and SQL/Entity Framework Core. ## Key Features * **Lucene Query Syntax** - Parse standardized query syntax compatible with Lucene and Elasticsearch * **Elasticsearch Integration** - Enhanced `query_string` replacement with dynamic queries, aggregations, and sorting * **SQL/EF Core Integration** - Generate Dynamic LINQ expressions for Entity Framework Core * **Visitor Pattern** - Extensible AST traversal for custom query transformations * **Field Aliases** - Static and dynamic field name mapping * **Query Includes** - Macro expansion for reusable query fragments * **Validation** - Syntax validation, field restrictions, and operation limits ## Architecture Foundatio.Parsers uses a layered architecture built on the visitor pattern: ```mermaid graph TD subgraph input [Input] QueryString[Query String] end subgraph parsing [Parsing Layer] LuceneParser[LuceneQueryParser] ElasticParser[ElasticQueryParser] SqlParser[SqlQueryParser] end subgraph ast [AST Nodes] GroupNode[GroupNode] TermNode[TermNode] TermRangeNode[TermRangeNode] ExistsNode[ExistsNode] end subgraph visitors [Visitor Pipeline] FieldResolver[FieldResolverVisitor] IncludeVisitor[IncludeVisitor] ValidationVisitor[ValidationVisitor] CustomVisitor[Custom Visitors] end subgraph output [Output] ElasticQuery[NEST QueryContainer] SqlQuery[Dynamic LINQ String] GeneratedQuery[Query String] end QueryString --> LuceneParser LuceneParser --> ElasticParser LuceneParser --> SqlParser LuceneParser --> GroupNode GroupNode --> TermNode GroupNode --> TermRangeNode GroupNode --> ExistsNode GroupNode --> FieldResolver FieldResolver --> IncludeVisitor IncludeVisitor --> ValidationVisitor ValidationVisitor --> CustomVisitor ElasticParser --> ElasticQuery SqlParser --> SqlQuery CustomVisitor --> GeneratedQuery ``` ### Parsing Layer The core `LuceneQueryParser` parses query strings into an Abstract Syntax Tree (AST). The `ElasticQueryParser` and `SqlQueryParser` extend this base parser with specialized visitor chains for their respective outputs. ### AST Nodes Queries are represented as a tree of typed nodes: | Node Type | Description | Example | |-----------|-------------|---------| | `GroupNode` | Groups child nodes with boolean operators | `(a OR b) AND c` | | `TermNode` | Single term or phrase query | `field:value`, `field:"quoted"` | | `TermRangeNode` | Range query | `field:[1 TO 10]`, `field:>5` | | `ExistsNode` | Field existence check | `_exists_:field` | | `MissingNode` | Field absence check | `_missing_:field` | ### Visitor Pipeline Visitors traverse the AST to transform, validate, or extract information. Multiple visitors can be chained together with priority ordering. ## Use Cases ### Dynamic Search APIs Expose powerful search capabilities to end users: ```csharp // User-provided query string userQuery = "status:active AND created:>2024-01-01"; var parser = new ElasticQueryParser(c => c .UseMappings(client, index) .SetValidationOptions(new QueryValidationOptions { AllowedFields = { "status", "created", "name" } })); var query = await parser.BuildQueryAsync(userQuery); ``` ### Custom Dashboards and Views Let users build custom aggregations: ```csharp // User-defined aggregation string userAgg = "terms:(status date:created~month min:amount max:amount)"; var aggs = await parser.BuildAggregationsAsync(userAgg); ``` ### Query Translation Translate queries between different backends: ```csharp // Parse once, output to multiple formats var luceneParser = new LuceneQueryParser(); var ast = await luceneParser.ParseAsync("field:value AND other:[1 TO 10]"); // Generate Elasticsearch query var elasticParser = new ElasticQueryParser(); var esQuery = await elasticParser.BuildQueryAsync(ast); // Generate SQL var sqlParser = new SqlQueryParser(); var sqlQuery = await sqlParser.ToDynamicLinqAsync(ast, context); ``` ## Packages | Package | Description | |---------|-------------| | `Foundatio.Parsers.LuceneQueries` | Core Lucene query parser and visitors | | `Foundatio.Parsers.ElasticQueries` | Elasticsearch/NEST integration | | `Foundatio.Parsers.SqlQueries` | SQL/Entity Framework Core integration | ## Next Steps * [Getting Started](./getting-started) - Installation and basic usage * [Query Syntax](./query-syntax) - Complete query syntax reference * [Elasticsearch Integration](./elastic-query-parser) - Building Elasticsearch queries * [SQL Integration](./sql-query-parser) - Entity Framework Core integration