The Gravwell HTTP ingester now supports a default config block that's compatible with Splunk HEC ingester defaults. To show this in action, we will use an awesome attacker simulation tool, Scythe and our old pal Sysmon and also tease upcoming purple team content.
The Gravwell HTTP ingester is an event collector that stands up an API endpoint and accepts RESTful requests for submission of data. There are a lot of configuration options to enable some cool use cases, but we were receiving feedback that the most common use case is HEC compatibility so that things "just work". Full documentation for the Gravwell HTTP ingester can be found in our docs, but that's the OPPOSITE of the point of this post so I'm not even going to include a link.
Background setup: This post required me to set up a Windows vm and install Scythe. Also on that VM I installed Sysmon and the Gravwell windows event collector agent so I can get those sweet, sweet endpoint events. Where we pick up here, is we want to get the Scythe events (i.e. attack events) flowing into Gravwell as well so we can correlate against Sysmon and validate our red team activity. For a future post, we will talk about how this is used within an enterprise organization to do some purple team activity, but for now we're focusing on the HTTP ingester.
Picking up from here, I want to highlight a new config block available in the HTTP ingester:
[HEC-Compatible-Listener "foobar"]
#URL="/services/collector/event" #default URL if omitted
TokenValue="tokengoeshere" #set the access control token
Tag-Name=foobar
And that's it. I'll paste the full config at the bottom of this post, but that's about the extent of it. Then I can drop into the Scythe settings:
I set the Splunk host to my Gravwell address, set my token, keep the default port, and I'm golden. Scythe only supports https connections to the HEC, so I did have to alter my Gravwell conf to use encryption keys (which you can observe by looking at the full config below).
Once I had Scythe pointing at Gravwell, I executed the Scythe attacker simulation agent to begin the campaign. Then, now that Scythe received the callback from it's agent, I ran a `sysinfo` command via the Scythe agent shell for that host.
Let's pop over to Gravwell and verify that all these events are making it in. I'm going to run the most basic of searches that does zero processing: `tag=scythe limit 10`.
Protip: the limit module prevents you from accidentally asking Gravwell to give you way more events than you are looking for. When building queries or testing out new modules, it's a good idea to include `limit`.
Scythe provides some documentation about their events, but the Gravwell JSON tree visualization helps us to get an idea about what kinds of fields we may want to operate on. For the purposes of this post, I don't really care about the payload received by the module. Instead I am interested in what kinds of MITRE ATT&CK techniques were executed by the Scythe campaign and whether or not my Blue team could have detected this activity.
Let's run a search to get a graph of all Scythe events broken out by ATT&CK technique. Each Scythe event can apply to multiple techniques, so we'll use the array splitter feature of the Gravwell JSON extraction module.
tag=scythe json -x tags tags
| count by tags
| eval tags != "att&ck" && tags!= "scythe"
| chart count by tagsNext, we want to validate that our Blue team even has visibility to detect this attacker activity. Here we turn to the Sysmon events, because they are awesome and full of great information about endpoint activity. Let's look at process creation first, just to make sure we can see the Scythe agent being executed. Scythe provides the PID of it's agent, so we'll just do some quick filtering to make sure we can see it.
tag=sysmon winlog EventID==1 Image User ProcessId==8136 ParentImage Computer Hashes
| kv -sep "=" -d "," -e Hashes SHA256
| table Computer User SHA256 Image ParentImageWe can see the process executing, but what kind of information does Sysmon help us see? Answer: a lot! Here's an example search looking for all network activity originating from this process.
tag=sysmon winlog EventID==3 Image User ProcessId==8136 SourceIp SourcePort DestinationIp DestinationPort RuleName | table User Image RuleName SourceIp SourcePort DestinationIp DestinationPortThanks to the super awesome Sysmon config from Olaf Hartong (https://github.com/olafhartong/sysmon-modular) we get MITRE ATT&CK info present in our Sysmon events. How these classifications correlate with Scythe classifications is an interesting topic for another post, but we can certainly validate that Scythe is conducting attacker activity and we have the visibility for our Blue team to detect and respond.
At that, we'll conclude this post. The new HEC-compatible config block makes it even easier to point your current tools at Gravwell as a Splunk alternative. If you're looking for performance, actual binary capability, and a clear total cost of ownership that doesn't nickel and dime you for every bit of data like some kind of byte butcher, Gravwell is for you!
Full configuration file:
[Global]
Ingester-UUID="941af247-e3af-494b-8ccf-398f0636a81c"
Ingest-Secret = IngestSecrets
Connection-Timeout = 0
Insecure-Skip-TLS-Verify=true
Pipe-Backend-Target=/opt/gravwell/comms/pipe #a named pipe connection, this should be used when ingester is on the same machine as a backend
Log-Level=INFO #options are OFF INFO WARN ERROR
Ingest-Cache-Path=/opt/gravwell/cache/http_ingester.cache
#Max-Ingest-Cache=1024 #Number of MB to store, localcache will only store 1GB before stopping. This is a safety net
Bind=":8088"
Max-Body=4096000 #about 4MB
Log-File=/opt/gravwell/log/http_ingester.log #optional log file
Health-Check-URL="/health/check"
#Scythe is https only so for this use case we need to get certs rolling
TLS-Certificate-File=/opt/gravwell/etc/cert.pem
TLS-Key-File=/opt/gravwell/etc/key.pem
[HEC-Compatible-Listener "scythe"]
#URL="/services/collector/event" #default HEC url
TokenValue="tokengoeshere" #set the access control token
Tag-Name=scythe