With Gravwell 3.2.4 we've introduced a new search module: kv, short for 'key-value'. This module is designed to help you extract key-value data from text entries without having to hand-craft regular expressions. It also interfaces with the fulltext indexer automatically, so you can analyze your indexed data more quickly.
Lots of data sources like to dump ugly-looking entries which are packed with information but not as convenient to parse as, say, JSON or XML. Consider this (generated) syslog entry:
<5>1 2019-09-27T09:33:01.362929-06:00 unicorn chatter 14747 - [sofiamartinez461@test.net source-address="1.192.254.26" source-port=47734 destination-address="15.176.86.127" destination-port=626 useragent="Mozilla/5.0 (Windows NT 6.2; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0"]
Before the kv module, you'd have to build a regex that gets uglier and harder to understand the more fields you extract:
tag=syslog regex "source-port=(?P<sourceport>\S+).+destination-port=(?P<destport>\S+)"
Now, we can pull out the same fields with a much simpler kv invocation:
tag=syslog kv "source-port" "destination-port"
(Note: "source-port" and "destination-port" are in quotes because they contain hyphens, which would break the names into separate tokens if not quoted. See our documentation for a detailed description of tokenization rules)
We can also apply filtering at extraction time:
tag=syslog kv "source-port"==47734
Although the kv module defaults to looking for keys and values separated by an equals sign ("foo = bar"), the module is flexible. Suppose management just got back from a webinar and mandated that your logs must now look like this:
2019-09-30 13:13:23 User EQUALS john.floren, Domain EQUALS gravwell.io, Hostname EQUALS floren-laptop
We need to specify that keys and values are separated by the word "EQUALS" and that the entries may be separated by spaces and commas. We do this with the `-sep` and `-d` flags, respectively:
tag=logs kv -sep EQUALS -d " ," User==john.floren Domain Hostname | table
We can also handle cases where the values are wrapped in quotes and may contain delimiters by adding the `-q` flag:
2019-09-30 13:13:23 User EQUALS john.floren, Domain EQUALS gravwell.io, Hostname EQUALS floren-laptop, Application EQUALS "Microsoft Surveillance Reporting"
tag=logs kv -q -sep EQUALS -d " ," User==john.floren Domain Hostname Application | table
Our new kv module can make short work of a variety of data formats, and it's only one of the ways Gravwell can help you make sense of your data--no matter how ugly! You can click the buttons below to schedule a demo with Gravwell...