Making things awk-ward: Gravwell and AWK

Today we’ll be going over a recently added module: awk.

The awk module is a fully functional implementation of the awk programming language, and conforms to the POSIX awk specification.

For those of you who aren’t familiar with awk, it is used for filtering, pattern matching, and textual data processing. awk is similar in scope to the UNIX tools sed and grep, and is usually packaged with those tools as yet another way to filter and transform text on the command line. 

awk is both the name of the command line tool in UNIX-like operating systems, and the programming language it implements. The POSIX awk specification describes the language, but since awk is nearly 50 years old, you can probably find examples of most concepts just by searching for it.

Using awk in Gravwell is identical to using it on the command line, with one difference – The Gravwell awk module operates on individual entries and does not carry state from one entry to the next in the way that the command line tool operates on individual lines. Let’s look at an example:

tag=input awk -o output "{ print $3 }" | table DATA output

In this example, we simply use the awk module to print the third element of the input data. By default, awk splits on whitespace (which of course you can change by using the BEGIN {FS=...} awk syntax). 

You’ll notice that we also have the flag “-o output”. This allows us to set the enumerated value awk should write its output into. By default, awk will simply rewrite the data field (or the enumerated value used if the -e flag is specified). 

Awk can implement arbitrarily complex programs. For example:

tag=input awk -o output "
BEGIN {FS=','}
{
   sum = 0
   i = 0
   while (i < 3) {
      sum += $i
      i++
   }
   average = sum / 3
   print 'Average:',average
}"
| table DATA output

This example assumes that the input is a three column CSV of numbers, and it calculates the average of the columns, setting the output to the “output” EV:

The Takeaway

Gravwell's awk module is a powerful way to quickly filter and transform data when you need functionality outside of the scope of other Gravwell modules. Since it conforms to the awk specification, you can even just copy/paste your awk programs into Gravwell and get searching immediately!