Introduction
New in Gravwell 5.3, the “eval” module now supports accelerated filtering! This new feature can greatly improve performance when using eval to perform complex filtering on extracted values. If you’re already using eval in your queries, and you have acceleration enabled, there’s nothing you need to do except enjoy an improvement in your query performance!
This article will cover the basics of acceleration, and how you can now use “eval” to accelerate on complex filter expressions.
Acceleration Basics
Gravwell’s acceleration system makes searching through large volumes of data fast. When enabled, it works by creating a “dictionary” of words, IP addresses, numbers, and other fields, that it finds in textual data as it’s ingested. Some types of binary data, such as PCAP, netflow, and IPFIX, are also supported.
Acceleration is used automatically when you add a filter to an extraction module, such as in this query:
tag=gravwell syslog Appname==webserver | ...
In the above example, we ask the “syslog” module to extract “Appname”, and only from entries where “Appname” is equal to “webserver”. Since this query will drop all entries that don’t contain the word “webserver”, Gravwell can simply go to the acceleration “dictionary” and lookup all the places in your ingested data where that word exists, and retrieve only those records from disk. By doing this, Gravwell avoids the costly task of retrieving all data from disk, only to discard most of it in the query.
You can combine filters in queries as well. Consider the following query:
tag=gravwell syslog Appname==webserver Hostname==g1 | ...
In this example, the acceleration system looks for entries that contain both the words “webserver” and “g1”.
One key filtering operation this approach cannot do is multifiltering. This means filters that contain expressions in the form of “this or that”. Consider a query where you want to extract all syslog entries where the “Appname” is equal to “webserver” or “indexer”. The only way to construct this query is with eval:
tag=gravwell syslog Appname
| eval ( Appname == “webserver” || Appname == “indexer”)
| ...
This approach works, but until Gravwell 5.3, it didn’t enable any kind of acceleration, potentially making this query slower than it needs to be.
Acceleration with Eval
Beginning in Gravwell 5.3, the query above will now engage the acceleration system, looking for either “webserver” or “indexer”. Let’s look at a few examples.
Example: Simple filtering
tag=gravwell syslog Appname | eval Appname == “webserver”
In this example, we do a simple filter on the “Appname” field that was extracted from syslog. This is functionally equivalent to:
tag=gravwell syslog Appname == “webserver”
Both of these queries will yield the same result, and both will now engage the acceleration system.
Example: Complex filtering
Extraction modules can filter on just a single value of a given extraction (eg Appname == “webserver”), but cannot express more complex expressions. Eval supports accelerating on more than one filter for a given extraction. For example:
tag=gravwell syslog Appname
| eval ( Appname == “webserver” || Appname == “indexer”)
| ...
In this example, eval will use acceleration to look for either “webserver” or “indexer”.
Example: Complex filtering with fields that cannot be accelerated
Eval will automatically determine which parts of an expression can be accelerated, and engage acceleration on just those terms. For example:
tag=gravwell syslog Appname ProcID
| eval ( ProcID > 5 && Appname == “indexer” )
| ...
In this example, the “ProcID” portion of the expression cannot be accelerated, since it’s not an equality operation. But since it’s being ANDed with an equality operation, we can still accelerate on the term “indexer”. Eval does this automatically.
Consider a different form of this example:
tag=gravwell syslog Appname ProcID
| eval ( ProcID > 5 || Appname == “indexer” )
| ...
In this example, the two operations in the expression are being ORed. This expression cannot be accelerated, because even if “indexer” isn’t present, there’s still a chance that the entry will be passed, based on the value of ProcID. Again, eval determined this automatically and will not engage acceleration here.
Lastly, consider a much more complex example:
tag=gravwell kv A B C D E
| eval ( (A == “foo” || B == “bar” || C == “baz”) && (D == “gravwell” || E == “shoes”) )
| ...
In this example, eval actually rewrites the expression to be in a form that can be accelerated. By using a little boolean algebra, eval produces the following acceleration checks, any one of which will allow the entry to be pulled from disk
“foo” and “gravwell”
“foo” and “shoes”
“bar” and “gravwell”
“bar” and “shoes”
“baz” and “gravwell”
“baz” and “shoes”
Again, eval does all of this automatically. You just need to write your eval expressions as you normally would, and eval will take care of the rest.
Conclusion
Accelerated filtering with eval is a powerful new feature in Gravwell 5.3 that enables you to search data with acceleration in more complex ways than before. It also may improve the performance of existing queries that use eval, without needing to modify your queries.
