Gravwell 5.4.0 is full of new features, and today we’ll be starting a series of blog posts showcasing some of the major updates. First is an overview of the updated eval search module.
Wait, a “new” eval module?
Well sort of… The eval module has been undergoing replacement for several months now, and if you’ve used eval in Gravwell 5.2 or 5.3, you’ve probably used the “new” one. However, Gravwell 5.4 represents the completion of the core features in the new eval, so we’ll be going over most of it here.
Full documentation on eval can be found in the Gravwell documentation.
What about old eval?
“Old” eval still exists and will automatically be invoked if you provide an eval program that doesn’t compile under the new eval syntax but does in the old syntax. In the case that your program that was running under the old eval does compile in the new one, the new module will be used.
Performance improvements
We saw in an earlier blog post how eval can accelerate your searches. In addition to this, eval can now run on indexers (in most cases), meaning that if you have a clustered Gravwell deployment, using eval does not cause your search to collapse. This allows your clustered deployment to maximize search parallelism.
New integer types
Eval already supported most of the Gravwell enumerated value types such as strings, ints, timestamps, IPs, etc. Beginning in Gravwell 5.4, eval also now supports all eight EV integer types:
int (equivalent to int64)
uint8
int8
uint16
int16
uint32
int32
uint64
int64
These types are used in the same way as a regular int, and you can cast to these types by using the type keyword (eg “foo = uint8(bar);”).
The “int” type still works, and is an alias to “int64”.
Persistent variables
Eval programs execute once for every entry in the search pipeline. Normally, eval doesn’t save state from one execution to the next. Starting in Gravwell 5.4, eval now supports “persistent variables”, which are simply regular variables that persist across all executions of eval in a given search. Let’s take a look at an example.
tag=data
eval
var count = 0;
if ( count == 10 ) {
return false;
}
count++;
In this example, we use eval to implement a limit function. The actual “limit” module is much faster, so keep this as an example only. The first statement in the eval program is “var count”, which declares a persistent variable named count, and initializes it to 0. This initialization happens when the search begins, and isn’t executed on every entry.
From here we simply increment the count variable until we reach a count of 10, and then start dropping every entry after. The “count” variable isn’t produced as an EV that other modules can use, and is accessible only within eval.
Note: Using persistent variables will cause eval to “collapse” the search pipeline. If you’re running on a clustered Gravwell deployment, this can impact performance.
Loops
Both old and new eval (until now) have disallowed any loop operations for the sake of performance. Eval now supports C-style for loops, including “break” and “continue” keywords. For example, to replace the string “foo” with an equal number of “*” characters (for masking a password perhaps), we can use eval with a for loop:
tag=data
eval
for ( i = 0; i < len(foo); i++ ) {
temp = temp + “*”;
}
foo = temp;
Keep in mind that eval won’t let any for loops execute more than one million iterations. After this eval will exit with an error!
Return and dropping entries
This one is a simple, but powerful, addition to eval: return statements. Now you can generate more complex eval programs that can drop or pass entries with a return statement. For example, if you wanted to drop all entries that have an EV named “foo”, and create an EV named “bar” for all others, you could do something like:
tag=data
eval
if ( has(foo) ) {
return false;
} else {
bar = “I didn’t get dropped!”;
}
New builtins
Eval has a lot of new built in functions. Many of these are simple string manipulation convenience functions, but eval also adds support for random numbers, encoding (more on this next), and several advanced math functions.
In addition to these, eval now provides a few new key convenience functions:
has()
Returns true if the EV specified exists.
delete()
Deletes the named EV from the entry.
in()
in() takes an EV or expression as the first argument, and then takes one or more expressions as input. If any of the expressions, treated as strings, are equal to the first argument, in() returns true. This is useful for constructing large “or” statements. For example:
in(foo, “fritz”, “john”, “kris”)
Will return true if the strings “fritz”, “john”, or “kris” are equal to the EV named foo.
Example: Building custom JSON
Before we end, let’s take a look at a more complex eval program that demonstrates some of the new features. This example generates a JSON encoded output containing the counts of several common application names in the gravwell tag. We’ll start with the whole example and then break it down piece by piece.
tag=gravwell
syslog -s Appname
| sort by time asc
| eval
var blob;
if ( blob == "") {
// we don't have any data yet, so create our json blob from scratch
blob = json_object(Appname, 1);
} else {
count = int(json_get(blob, Appname));
if ( count == 0 ) {
// a new key was found, add it to the blob
blob = json_set(blob, Appname, 1);
} else {
blob = json_set(blob, Appname, count + 1);
}
}
output = json_pretty(blob);
| last
| table output
Let’s start with the non-eval parts, because that’s a lot simpler to grok. We extract the Appname field from syslog records in the gravwell tag, sort the data by time, and we ask for just the last entry (eval will execute over all entries, but we only care to see the last one).
Now onto eval:
First we declare a persistent variable named “blob”. We’ll use this to store a JSON encoded blob of data. If the blob is empty, it must mean this is the first time eval has executed, so we’ll simply create a new blob with the builtin function “json_object”. Eval has several JSON encoding helpers, and supports all of the native JSON types.
If the blob already exists, we’ll use it to retrieve the count for that given Appname. If the count doesn’t exist, we’ll create it as it’s the first time we’ve seen that particular Appname. If it does exist, we’ll increment it and store it back to the blob.
Finally, because persistent variables aren’t accessible outside of eval, we’ll pretty-print the JSON, and assign it to an EV named “output”. Let’s see this in action:
Fantastic!
The JSON encoding functionality is especially useful when combined with Gravwell Alerts, as it can be used for integrations with external tools.
Conclusion
Eval provides several new features that can be used to perform complex accelerated filtering, data manipulation, and more. To find out more, contact us at info@gravwell.io
