When we started Gravwell years ago, we knew it was going to be a significant undertaking requiring some serious tooling under the hood. Building a custom data lake and analytics platform from scratch that can scale to hundreds of TB/day ain't easy. We chose Go for a lot of reasons and that choice has paid dividends in terms of what we've been able to accomplish in so short a time.
This post is about our tooling, and some of the lessons we have learned along the way in managing a large Go codebase. A few weeks ago Gravwell made the switch to Go modules on both our open source github repositories and our internal repo. Let's talk about about our planned workflow going forward and a few caveats we've run into.
The Existing Setup
Gravwell has two separate groups of repositories:
- Open-source repos on Github
- Private mono-repo hosted on an internal server
The private repository used dep to manage dependencies. We specifically excluded the github.com/gravwell repositories from our dep configs; instead, we would update those repositories manually using `go get` or `git pull`. This made it easier to hack on the libraries as needed.
The open-source repos did not use dep or any other form of dependency management tool. Whenever we released a new version of Gravwell, we would tag all of our open-source repositories with the same version name (e.g. v3.2.2); the intent was to make it easy to pull up the appropriate version of the library if a customer had a problem with their installation.
Versioning
Like many Go things, Go's modules feature is highly opinionated. For one thing, it follows semver strictly. Under semver, going from v1.x to v2.x means you have made a breaking change to the API.
Go has decided that modules-enabled packages must explicitly include the version in the import path for versions 2 and higher, e.g. the go.mod file should declare "module github.com/example/foo/v2" and you should import "github.com/example/foo/v2" in your code.
Because we'd been tagging our open source repos along with Gravwell releases, we were already into major version 3. Therefore, the ingest library's go.mod file says "module github.com/gravwell/ingest/v3", and we went through ALL our code to re-write imports from "github.com/gravwell/ingest" to "github.com/gravwell/ingest/v3". With a little bit of sed magic, this was actually reasonably painless.
Some of our libraries depend on each other; the ingest library depends on the timegrinder library, for example. This doesn't mesh well with our old practice of tagging all repositories at the time of a new Gravwell release--we would have to make sure to tag and push timegrinder before we tag the ingest library, etc. Our old practice also meant we couldn't cut a new patch version of a library if we add functionality. For that reason, we moved to a new model:
- Gravwell libraries (ingest, timegrinder, netflow) will use normal semantic versioning from version 3.2.3 onward. This means they will no longer be tagged when we release a new version of Gravwell, but only when the library itself changes.
- Gravwell open-source tool repositories such as the ingesters and generators repositories will continue to be tagged along with the mainline Gravwell releases.
We had originally tagged our libraries along with the Gravwell releases so we would know that a customer running Gravwell 3.2.2 was using the ingest library version 3.2.2 as well. Because we are now tracking the open-source libraries in our go.mod files, the library version is preserved in our revision history, leaving us free to decouple library versions.
Hacking on Libraries
The purpose of Go modules is to lock builds to a specific version of a dependency. However, we will occasionally need to modify our own libraries and test those modifications. Suppose we want to add a new time format to the timegrinder library. We can check out the timegrinder repository locally and make modifications, but if we then go into the ingesters repo and try to build one of those ingesters to test the new time format, it will be built with the timegrinder version specified in go.mod, not the version we just modified.
To get around this, we will be using the replace directive in go.mod. Although you can modify the file directly, `go mod edit` is easiest. In this example, we could run the following within the ingesters repository:
go mod edit -replace github.com/gravwell/timegrinder/v3=/home/john/work/timegrinder
Now, `go build` in the ingesters repository will use our local version of timegrinder.
Of course, once we've completed the modifications and pushed the changes upstream, we can't commit a go.mod file which includes a reference to a local directory! First, we drop the replace directive:
go mod edit -dropreplace github.com/gravwell/timegrinder/v3
Then, from the ingesters repo, we can run `go get github.com/gravwell/timegrinder/v3@master` to force it to use the tip of the master branch. Once we tag the next release on the timegrinder repo, we will be able to say `go get -u` in the ingesters repository to once again point at a release version of timegrinder rather than a specific commit.
An Interesting Go Proxy Caveat
By default, Go will actually fetch modules from a proxy/mirror, as detailed in this blog post. It also verifies module checksums against values maintained by the mirror.
We ran into a problem with this. I had set up Go modules in the Gravwell ingesters repository while using the default setting: GOPROXY="https://proxy.golang.org,direct". Kris, on the other hand, is a more paranoid kind of guy and had immediately set GOPROXY="direct", meaning Go fetches all repositories directly from Github/Bitbucket/etc.
One of our dependencies is github.com/klauspost/compress. Go modules defaulted to the latest version, v1.8.4. Everything built fine for me. However, when Kris tried to build the module which depends on that library, he saw an error:
verifying github.com/klauspost/compress@v1.8.4: checksum mismatch
downloaded: h1:kecSN5NBXafCCyyzBGF3lsKqPbU1HdtG7usSTs3rE3I=
go.sum: h1:Udk++ps4wOTuOpzZ3wTZxXP/6wEBELAJv3+DY+tlFqw=
SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.
Spooky! The checksum of the library he was downloading from Github did not match the checksum I had received earlier. Was someone MITM'ing Kris's connection to github? Was Google being evil and replacing something on the proxy?
Nope! The Go team appears to take a very strict view of releases: once you've tagged a release, it's final. Us mere mortals, however, sometimes mess up and want to re-tag. I've done it, and Klaus did too (I confirmed via email). This means that Go's proxy server had fetched the first commit he tagged as v1.8.4, but when Kris went directly to Github he got a different commit for v1.8.4, leading to the checksum error.
The solution? Well, for the time being, we've just had Kris set his GOPROXY variable to match mine. When Klaus cuts a v1.8.5 and we update our go.mod file, this problem should go away. For our own libraries, we have to remember that once a tag is pushed to Github, it should be considered immutable for the purposes of Go modules.
Conclusion
In the weeks since we made the switch, we've rapidly become accustomed to the new Go modules workflow. The biggest advantage is that we can be sure everyone on the team is building from the exact same dependencies all the time without having to commit a vendor directory. There have been a few bumps in the road, but the wins seem to outnumber the inconveniences.
If you're moving your own project over to Go modules and want some additional information--or if we've made a mistake about modules in this post--please feel free to email us at info@gravwell.io. If you came to this post looking for help with Go modules and now you're wondering what the heck Gravwell is, you can check out our free Trial by clicking the button below:
