Validation

Foundatio.Parsers provides comprehensive query validation to ensure queries are safe, well-formed, and within allowed boundaries. This is essential when accepting queries from untrusted sources like user input or APIs.

Basic Validation

Syntax Validation

Validate query syntax without executing:

using Foundatio.Parsers.LuceneQueries;

var result = await QueryValidator.ValidateQueryAsync("status:active AND created:>2024-01-01");

if (result.IsValid)
{
    Console.WriteLine("Query is valid");
}
else
{
    Console.WriteLine($"Invalid: {result.Message}");
    foreach (var error in result.ValidationErrors)
    {
        Console.WriteLine($"  Position {error.Index}: {error.Message}");
    }
}

Throw on Invalid

Use ValidateQueryAndThrowAsync to throw an exception for invalid queries:

try
{
    await QueryValidator.ValidateQueryAndThrowAsync("invalid::");
}
catch (QueryValidationException ex)
{
    Console.WriteLine($"Validation failed: {ex.Message}");
    Console.WriteLine($"Errors: {ex.Result.ValidationErrors.Count}");
}

Validation Options

Configure validation behavior with QueryValidationOptions:

var options = new QueryValidationOptions
{
    AllowedFields = { "status", "name", "created" },
    RestrictedFields = { "password", "secret" },
    AllowLeadingWildcards = false,
    AllowUnresolvedFields = false,
    AllowUnresolvedIncludes = false,
    AllowedMaxNodeDepth = 10,
    AllowedOperations = { "terms", "date", "min", "max" },
    RestrictedOperations = { "tophits" }
};

var result = await QueryValidator.ValidateQueryAsync(query, options);

Option Reference

Option	Type	Default	Description
AllowedFields	HashSet<string>	Empty (all allowed)	Whitelist of allowed field names
RestrictedFields	HashSet<string>	Empty	Blacklist of restricted field names
AllowLeadingWildcards	bool	true	Allow *value wildcards
AllowUnresolvedFields	bool	true	Allow fields not in mapping
AllowUnresolvedIncludes	bool	false	Allow includes that don't resolve
AllowedMaxNodeDepth	int	0 (unlimited)	Maximum query nesting depth
AllowedOperations	HashSet<string>	Empty (all allowed)	Whitelist of aggregation operations
RestrictedOperations	HashSet<string>	Empty	Blacklist of aggregation operations
ShouldThrow	bool	false	Throw exception on validation failure

Field Restrictions

Allowed Fields (Whitelist)

Only allow specific fields:

var options = new QueryValidationOptions();
options.AllowedFields.Add("status");
options.AllowedFields.Add("name");
options.AllowedFields.Add("created");

// Valid
var result = await QueryValidator.ValidateQueryAsync("status:active", options);
// result.IsValid == true

// Invalid - field not in whitelist
result = await QueryValidator.ValidateQueryAsync("password:secret", options);
// result.IsValid == false

Restricted Fields (Blacklist)

Block specific fields:

var options = new QueryValidationOptions();
options.RestrictedFields.Add("password");
options.RestrictedFields.Add("apiKey");
options.RestrictedFields.Add("secret");

// Valid
var result = await QueryValidator.ValidateQueryAsync("status:active", options);
// result.IsValid == true

// Invalid - field is restricted
result = await QueryValidator.ValidateQueryAsync("password:test", options);
// result.IsValid == false

Field Aliases and Validation

When using field aliases, validation uses the alias names (not resolved names):

var parser = new ElasticQueryParser(c => c
    .UseFieldMap(new Dictionary<string, string> {
        { "user", "data.user.identity" }
    })
    .SetValidationOptions(new QueryValidationOptions {
        AllowedFields = { "user", "status" }  // Use alias names
    }));

// Valid - uses allowed alias
var query = await parser.BuildQueryAsync("user:john");

// Invalid - uses resolved name directly
// (unless "data.user.identity" is also in AllowedFields)
await Assert.ThrowsAsync<QueryValidationException>(() =>
    parser.BuildQueryAsync("data.user.identity:john"));

Wildcard Restrictions

Disable Leading Wildcards

Leading wildcards (*value) can be expensive. Disable them:

var options = new QueryValidationOptions
{
    AllowLeadingWildcards = false
};

// Valid - trailing wildcard
var result = await QueryValidator.ValidateQueryAsync("name:john*", options);
// result.IsValid == true

// Invalid - leading wildcard
result = await QueryValidator.ValidateQueryAsync("name:*smith", options);
// result.IsValid == false
// result.Message contains "wildcard"

Depth Restrictions

Limit query nesting to prevent complex queries:

var options = new QueryValidationOptions
{
    AllowedMaxNodeDepth = 5
};

// Valid - shallow nesting
var result = await QueryValidator.ValidateQueryAsync("(a:1 AND b:2)", options);
// result.IsValid == true

// Invalid - too deep
result = await QueryValidator.ValidateQueryAsync(
    "((((((a:1 AND b:2))))))", options);
// result.IsValid == false

Unresolved Field Handling

Control behavior when fields don't exist in mappings:

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .SetValidationOptions(new QueryValidationOptions {
        AllowUnresolvedFields = false
    }));

// If "nonexistent" is not in the index mapping
var result = await parser.ValidateQueryAsync("nonexistent:value");
// result.IsValid == false
// result.UnresolvedFields contains "nonexistent"

Aggregation Validation

Allowed Operations

Whitelist specific aggregation operations:

var options = new QueryValidationOptions();
options.AllowedOperations.Add("terms");
options.AllowedOperations.Add("date");
options.AllowedOperations.Add("min");
options.AllowedOperations.Add("max");

var result = await QueryValidator.ValidateAggregationsAsync(
    "terms:(status min:amount)", options);
// result.IsValid == true

result = await QueryValidator.ValidateAggregationsAsync(
    "tophits:_", options);
// result.IsValid == false - tophits not in allowed list

Restricted Operations

Blacklist specific operations:

var options = new QueryValidationOptions();
options.RestrictedOperations.Add("tophits");  // Expensive operation

var result = await QueryValidator.ValidateAggregationsAsync(
    "terms:(status tophits:_)", options);
// result.IsValid == false

Validation Result

The QueryValidationResult provides detailed information:

var result = await QueryValidator.ValidateQueryAsync(query, options);

// Basic status
bool isValid = result.IsValid;
string message = result.Message;

// Detailed errors
foreach (var error in result.ValidationErrors)
{
    Console.WriteLine($"Error at position {error.Index}: {error.Message}");
}

// Referenced fields
foreach (var field in result.ReferencedFields)
{
    Console.WriteLine($"Field used: {field}");
}

// Unresolved fields (not found in mapping)
foreach (var field in result.UnresolvedFields)
{
    Console.WriteLine($"Unknown field: {field}");
}

// Referenced includes
foreach (var include in result.ReferencedIncludes)
{
    Console.WriteLine($"Include used: {include}");
}

// Unresolved includes
foreach (var include in result.UnresolvedIncludes)
{
    Console.WriteLine($"Missing include: {include}");
}

// Query depth
int depth = result.MaxNodeDepth;

// Operations used (for aggregations)
foreach (var op in result.Operations)
{
    Console.WriteLine($"Operation: {op.Key}, Count: {op.Value}");
}

With ElasticQueryParser

Validate Before Building

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .SetValidationOptions(new QueryValidationOptions {
        AllowedFields = { "status", "name", "created" },
        AllowLeadingWildcards = false,
        AllowedMaxNodeDepth = 10
    }));

// Validate query
var validation = await parser.ValidateQueryAsync("status:active");
if (!validation.IsValid)
{
    return BadRequest(validation.Message);
}

// Build query (also validates and throws if invalid)
var query = await parser.BuildQueryAsync("status:active");

Validate Aggregations

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .SetValidationOptions(new QueryValidationOptions {
        AllowedOperations = { "terms", "date", "min", "max", "avg" }
    }));

var validation = await parser.ValidateAggregationsAsync(
    "terms:(status min:amount)");

if (!validation.IsValid)
{
    return BadRequest(validation.Message);
}

Validate Sort

var parser = new ElasticQueryParser(c => c
    .UseMappings(client, "my-index")
    .SetValidationOptions(new QueryValidationOptions {
        AllowedFields = { "created", "name", "status" }
    }));

var validation = await parser.ValidateSortAsync("-created +name");

if (!validation.IsValid)
{
    return BadRequest(validation.Message);
}

Ordering Operators

Sort and aggregation ordering accept only the + (ascending) and - (descending) prefixes. The boolean negation operators NOT and ! are query operators and are reported as validation errors in sort and aggregation expressions:

var validation = await parser.ValidateSortAsync("!created");
// validation.IsValid == false
// validation.Message == "Boolean operator (!) is not supported in sort expressions for field (created): use + for ascending or - for descending order."

validation = await parser.ValidateAggregationsAsync("terms:(category NOT max:price)");
// validation.IsValid == false
// validation.Message == "Boolean operator (NOT) is not supported in aggregation expressions for field (price): use + for ascending or - for descending order."

validation = await parser.ValidateAggregationsAsync("terms:(!category)");
// validation.IsValid == false
// validation.Message == "Boolean operator (!) is not supported in aggregation expressions for field (category): use + for ascending or - for descending order."

This applies to the entire expression, including aggregation primary fields, nested metrics, and operators combined with a direction prefix, such as terms:(NOT +category). When an include resolver is configured for an ordering context, expansion cannot hide a negated include such as !@include:ordering or NOT @include:ordering.

Operators must precede the field name. Inputs such as max:!price or !max:+price fail parsing before ordering validation; see operator placement. Escaped, quoted, and regex payload text containing ! or NOT is not a Boolean operator.

With the default ShouldThrow = false, the string-based ValidateSortAsync and ValidateAggregationsAsync methods return these errors in QueryValidationResult. The corresponding string-based BuildSortAsync and BuildAggregationsAsync methods throw QueryValidationException, whose Result contains the validation errors. The static QueryValidator methods enforce the same ordering rule, including their AndThrowAsync variants.

AST overloads and custom pipelines

These guarantees apply to the string-based APIs with the built-in validation pipeline. BuildSortAsync(IQueryNode, ...) and BuildAggregationsAsync(IQueryNode, ...) are lower-level APIs and do not rerun validation. They expect an appropriately processed AST. Use the string-based APIs for untrusted expressions. Custom pipelines must retain the ordering checks before destructive transformations and reject an invalid context before using the generated output.

Migrating existing ordering expressions

This is a behavioral breaking change: it rejects syntax that older versions accepted even though public method signatures are unchanged. Replace !created or NOT created with -created when descending sort order is intended. For contradictory input such as NOT +created, choose +created or -created explicitly rather than relying on prefix precedence.

Aggregation negation previously could be silently ignored. Remove it from primary fields (terms:(!category) becomes terms:(category)), and choose an explicit + or - on the sub-aggregation when ordering is intended (terms:(category -max:price), for example). Do not blindly replace an ignored operator with descending order, since that changes the result ordering.

This ordering restriction does not apply to NOT, !, or required/prohibited clauses in search queries, including SQL queries. For changes to required clauses and prefixed includes, see the query upgrade guidance.

For custom visitor pipelines, preserve operator state until it can be validated. The built-in AssignOperationTypeVisitor checks a primary field term before removing it, and IncludeVisitor checks an include node before replacing it. These narrow checks do not move field, operation, or nesting restrictions ahead of normalization and resolution.

Context-Based Validation

Access validation results from the context:

var parser = new ElasticQueryParser(c => c
    .SetValidationOptions(options));

var context = new ElasticQueryVisitorContext();
var query = await parser.BuildQueryAsync("status:active", context);

// Get validation result from context
var validation = context.GetValidationResult();

// Check validity
if (!context.IsValid())
{
    var errors = context.GetValidationErrors();
    var message = context.GetValidationMessage();
}

// Throw if invalid
context.ThrowIfInvalid();

Custom Validation

Add custom validation errors:

var context = new ElasticQueryVisitorContext();

// Add custom validation error
context.AddValidationError("Custom error message", index: 5);

// Check result
var validation = context.GetValidationResult();
// validation.IsValid == false

Best Practices

1. Always Validate User Input

public async Task<IActionResult> Search([FromQuery] string query)
{
    var validation = await parser.ValidateQueryAsync(query);
    if (!validation.IsValid)
    {
        return BadRequest(new {
            error = "Invalid query",
            details = validation.Message,
            errors = validation.ValidationErrors
        });
    }
    
    // Safe to execute
    var results = await ExecuteSearch(query);
    return Ok(results);
}

2. Use Whitelists Over Blacklists

// Preferred - explicit whitelist
var options = new QueryValidationOptions {
    AllowedFields = { "status", "name", "created", "category" }
};

// Less secure - blacklist can miss fields
var options = new QueryValidationOptions {
    RestrictedFields = { "password" }  // What about "apiKey"?
};

3. Limit Query Complexity

var options = new QueryValidationOptions {
    AllowedMaxNodeDepth = 10,
    AllowLeadingWildcards = false
};

4. Log Validation Failures

var validation = await parser.ValidateQueryAsync(query);
if (!validation.IsValid)
{
    logger.LogWarning(
        "Invalid query from user {UserId}: {Query}. Errors: {Errors}",
        userId, query, validation.Message);
    
    return BadRequest(validation.Message);
}

5. Provide Helpful Error Messages

var validation = await parser.ValidateQueryAsync(query);
if (!validation.IsValid)
{
    var response = new {
        error = "Query validation failed",
        message = validation.Message,
        allowedFields = options.AllowedFields,
        errors = validation.ValidationErrors.Select(e => new {
            position = e.Index,
            message = e.Message
        })
    };
    
    return BadRequest(response);
}

Next Steps

• Field Aliases - Map field names
• Query Includes - Reusable query macros
• Visitors - Custom query transformations