Other articles in this series:

Go modules considered harmful

Go modules are a fundamentally misguided and harmful change in the design of the Go ecosystem. I decline to adopt them or to use software which requires use of them.

Origin of the Cancer

The origin of the misguided Modules proposal appears to be this series of blog posts by rsc.

“We need to add package versioning to Go.”

I reject this premise.

“Versioning will let us enable reproducible builds,”

False premise. Reproducible builds are already wholly feasible without “modules”.

“, so that if I tell you to try the latest version of my program, I know you're going to get not just the latest version of my code but the exact same versions of all the packages my code depends on, so that you and I will build completely equivalent binaries.”

This insinuates that it is the legitimate demarcation of a software developer to dictate the precise point versions of the dependencies upon which their software relies. I both reject this premise and explicitly oppose it. It is not the legitimate demarcation of an upstream software developer to so dictate dependency versions and they will not be allowed to do so. This is a decision to be made by distributions and distributions alone.

“Even when there are newer versions of my dependencies, the go command shouldn't start using them until asked.”

I expressly reject this behaviour as expressly harmful and something I do not want. Any Go software I maintain should use the latest dependencies independent of my involvement or approval. Moreover, no Go repository I maintain will ever contain any file identifying specific point versions of its dependencies.

“Although we must add versioning,”

Why? Unsupported assertion.

“This proposal [...] eliminates vendoring,”

This is literally the diametric opposite of the truth. Modules seek to entrench vendoring as the officially “correct” way to develop Go software because they seek to have Go software developers prescribe specific versions of their software's dependencies. It's completely irrelevent whether that is done by actually checking in the files of dependencies into their own repository (bundling) or referencing specific versions of their dependencies via cryptographic hash. Sane distributions reject bundling absolutely for good reason and should reject Go modules for the same reasons.

“deprecates GOPATH in favor of a project-based workflow,”

This is a disaster and in no way desirable. $GOPATH is one of the best code management approaches I've seen in any programming language. It's such a good approach, in fact, that I have a work-in-progress package manager designed to maintain Node.js and other TypeScript/JavaScript code inside of $GOPATH, with JavaScript packages given Go-like import strings; I also intend to adopt it for C/C++ projects. You will take $GOPATH from my cold, dead hands.

“and provides for a smooth migration from dep and its predecessors.”

Nobody needed to be using dep in the first place. This is essentially developers operating bad practices in Go development but considering their practices normative trying to make their practices an official part of Go (and thus officially normative).

“It was clear in the very first discussions of goinstall that we needed to do something about versioning. Unfortunately, it was not clear, at least to us on the Go team, exactly what to do. When go get needs a package, it always fetches the latest copy, delegating the download and update operations to version control systems like Git or Mercurial. This ignorance of package versioning has led to at least two significant shortcomings.”

This problem was already solved by gopkg.in, which allows package import paths to designate a major version (read: not a precise version). New point releases of a major version are automatically picked up and adopted, as they should be. There is a valid argument to be made that a specific service run by a specific person (gopkg.in) should not be instrumental in the use of versioning in Go, but the Modules proposal lacks any desirability compared to gopkg.in. Functionality such as that of gopkg.in, based around major versions only, could easily have been added to go get.

As noted:

“In March 2014, Gustavo Niemeyer created gopkg.in, advertising “stable APIs for the Go language.” The domain is a version-aware GitHub redirector, allowing import paths like gopkg.in/yaml.v1 and gopkg.in/yaml.v2 to refer to different commits (perhaps on different branches) of a single Git repository. Following semantic versioning, authors are expected to introduce a new major version when making a breaking change, so that later versions of a v1 import path can be expected to be drop-in replacements for earlier ones, while a v2 import path may be a completely different API.”

Regrettably, the author concludes that the complete opposite functionality is preferable:

“Instead of concluding from Hyrum's law that semantic versioning is impossible, I conclude that builds should be careful to use exactly the same versions of each dependency that the author did, unless forced to do otherwise. That is, builds should default to being as reproducible as possible.”

While reproducible builds are a desirable thing in themselves, the above quote makes the erroneous assumption that it should be the package developer deciding what dependency versions their software should be combined with. Rather, it is the legitimate demarcation of the distribution to decide this.

“The second significant shortcoming of go get is that, without a concept of versioning, it cannot ensure or even express the idea of a reproducible build. There is no way to be sure that your users are compiling the same versions of your code's dependencies that you did.”

There should be tools which can facilitate reproducible builds in Go. There is no reason why that tool should be go get, and certainly no reason why those tools should be based on precise dependency versions nominated by dependency consumers.

“In November 2013, the Go 1.2 FAQ also added this basic advice:
If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) [...]

This is known as bundling and is a prohibited practice in all sane distributions for good reason. It's well known to me that Google's internal monorepo is based around vendoring every dependency into their monorepo. It's also Google's practice to check in binaries of dependencies into their monorepo. Google's internal software development practices are fundamentally misguided and nobody should be seeking to emulate them.

“That spring, Jason Buberel surveyed the Go package management landscape to understand what could be done to unify these multiple efforts and avoid duplication and wasted work. His survey made it clear to us on the Go team that the go command needed direct support for vendoring without import rewriting. At the same time, Daniel Theophanes started a specification for a file format to describe the exact provenance and version of code in a vendor directory. In June 2015, we accepted Keith's proposal as the Go 1.5 vendor experiment, optional in Go 1.5 and enabled by default in Go 1.6. We encouraged all vendoring tool authors to work with Daniel to adopt a single metadata file format.”

The adoption of vendoring support in Go 1.5 was fundamentally misguided and the adoption of vendoring in any way is a fundamentally misguided practice. That the Go ecosystem has been evolved in a way that advocates, presumes and now in fact expects the use of vendoring is deeply disappointing.

“Incorporating the concept of vendoring into the Go toolchain allowed program analysis tools like go vet to better understand projects using vendoring,”

There shouldn't be any projects using vendoring. Or modules, which are effectively the same thing. Which this post now seems to be admitting, contrary to its previous assertion that it “eliminates vendoring”.

“Nearly all pain in package management systems is caused by trying to tame incompatibility. For example, most systems allow package B to declare that it requires package D 6 or later, and then allow package C to declare that it requires D 2, 3, or 4, but not 5 or later. If you are writing package A, and you want to use both B and C, then you are out of luck: there is no one single version of D that can be chosen to build both B and C into A. There is nothing you can do about it: these systems say that what B and C did was acceptable—they effectively encourage it—so you are just stuck.”

This is a good thing. I do not want multiple versions of a given package built into any binary of mine. Firstly, it leads to senseless binary bloat. Secondly, it's not even a sound practice: if package A depends on package X and package B depends on package X, and I get a pointer to a structure defined in package X from package A, and expect in my own code to be able to pass it to a function accepting such a pointer in package B, this is completely broken if A and B are referring to different versions of X. It is entirely correct that there must be only one version in any given $GOPATH.

“Creating v2.0.0, which in semantic versioning denotes a major break, therefore creates a new package with a new import path, as required by import compatibility. Because each major version has a different import path, a given Go executable might contain one of each major version. This is expected and desirable.”

Having multiple major versions of a package in a binary is not desirable and I decline to permit it.

“It keeps programs building and allows parts of a very large program to update from v1 to v2 independently.”

No thanks.

I think gopkg.in is a decent idea because it allows package authors to release new breaking major versions and allow me to upgrade to those versions synchronously as I make the necessary changes to my own code. However, it's incredibly unlikely I'd ever allow assorted dependencies of my projects to depend on two different major versions of the same package, as this is just senseless code bloat. The correct response to this is to fix the package which depends on the old version.

Exceptionally if you absolutely must, gopkg.in does already permit you to incorporate different major versions of the same package, as they are basically given different names. Moreover, it does so without having packages nominate precise point versions of their dependencies. Nonetheless, this in no way should be a default or recommended practice.

“Nearly all package managers today, including dep and cargo, use the newest allowed version of packages involved in the build. I believe this is the wrong default, for two important reasons.”

Here we have the start of the disaster. The entirety of the design of Go modules flows from this premise, which I reject.

“First, the meaning of “newest allowed version” can change due to external events, namely new versions being published.”

That is literally what I want.

“Maybe tonight someone will introduce a new version of some dependency, and then tomorrow the same sequence of commands you ran today would produce a different result.”

If you consider this undesirable, distributions have already solved this problem, not even just for Go: they reference all software they build and package by commit (or tarball) hash. The important distinction is that it's the distribution deciding what versions of packages to combine.

Essentially, a distinction should be made between software repositories and “distribution repositories” (e.g. Nixpkgs). A distribution repository is used to facilitate reproducible builds. Any given HEAD of a distribution repository describes how to combine multiple software repositories of specific, cryptographically identified versions and produce a reproducible result.

There are two important principles to be observed here:

Go modules fail at this because they demand that package authors commit files cryptographically nominating precise point versions of their dependencies to their software repositories. They conflate software repositories and distribution repositories as one and the same. Not only that, they do it in a way likely to lead to the unexamined proliferation of multiple versions of the same packages inside a single tree or binary. Ironically, Go modules actually undermines the ability of people to maintain control over what versions of packages are used in distribution repositories.

“This proposal takes a different approach, which I call minimal version selection. It defaults to using the oldest allowed version of every package involved in the build. This decision does not change from today to tomorrow, because no older version will be published.”

No thanks. (Also, does this mean I can foil this scheme by numbering my software backwards, starting at v100.9999.9999 and counting down? Perhaps I should do that.)

“Minimal version selection delivers reproducible builds by default, without a lock file.”

go.sum is literally a lockfile.

“Goinstall and old go get invoke version control tools like git and hg directly to download code, leading to many problems, among them fragmentation: users without bzr cannot download code stored in Bazaar repositories, for example. In contrast, modules are always zip archives served over HTTP. Before, go get had special cases to choose the version control commands for popular code hosting sites. Now, vgo has special cases to use those hosting sites' APIs to fetch archives. The uniform representation of modules as zip archives makes possible a trivial protocol for and implementation of a module-downloading proxy. Companies or individuals can run proxies for any number of reasons, including security and wanting to be able to work from cached copies in case the originals are removed.”

This is another bad move, as it represents an attempt to centralise module distribution. One of the best innovations of go get was its use of distributed package distribution. More on this later, however.

“The most significant change, though, is the end of GOPATH as a required place to work on Go code. Because the go.mod file includes the full module path and also defines the version of every dependency in use, a directory with a go.mod file marks the root of a directory tree that serves as a self-contained work space, separate from any other such directories.”

As a software developer, it is not my place to prescribe specific point versions of software my software depends on and I decline to do so. Moreover, $GOPATH is the best package organisation methodology I've ever seen and I'm not interested in moving away from it.

“I'm excited for Go to take the long-overdue step of adding versions to its working vocabulary. Some of the most common problems that developers run into when using Go are the lack of reproducible builds,”

People have been making reproducible builds with Go for years.

“the inability of GOPATH to comprehend multiple versions of a package”

I consider this a feature.

“In some later release, we'll remove support for the old, unversioned go get.”

I will literally fork go(1) over this if I have to.

The Cancer Continues

The blogpost series continues, in which the author then amazingly admits that allowing multiple versions of a package is fundamentally unsound:

“One common objection to the semantic import versioning approach is that package authors today expect that there is only ever one copy of their package in a given build. Allowing multiple packages at different major versions may cause problems due to unintended duplications of singletons. An example would be registering an HTTP handler. If my/thing registers an HTTP handler for /debug/my/thing, then having two copies of the package will result in duplicate registrations, which causes a panic at registration time. Another problem would be if there were two HTTP stacks in the program. Clearly only one HTTP stack can listen on port 80; we wouldn't want half the program registering handlers that will not be used. Go developers are already running into problems like this due to vendoring inside vendored packages.”

However, the author then adds:

“Moving to vgo and semantic import versioning clarifies and simplifies the current situation though.”

This is not the “current situation”; I have never used vendoring and never would. This is only the “current situation” in projects already engaging in the fundamentally bad practice of vendoring. This confirms my suspicions that — much like, say, Docker — Go modules are motivated by a desire to accommodate the nauseating and backwards software development practices commonly found inside large enterprises.

“One of the key reasons to allow both v1 and v2 of a package to coexist in a large program is to make it possible to upgrade the clients of that package one at a time and still have a buildable result.”

Again, this can (if absolutely necessary) already be done with gopkg.in, in a way that doesn't rely on package repositories prescribing specific versions of their own dependencies. gopkg.in is a centralised service and if logic had been added to go(1) to implement gopkg.in-like functionality, I probably would have welcomed it. These are all arguments for Go modules... on the grounds that they solve problems which were already solved by gopkg.in. This is a remarkably underwhelming argument.

The Cancer Continues II

The series continues further. The absurdity of one theme of these blogposts now reaches a deafening crescendo. Namely, there is an insistence throughout that semantic versioning should be used; that major versions should indicate compatibility; that if you make API-breaking changes to a package, you should also change its name, gopkg.in style... while at the same time simultaneously insisting the opposite.

“The second algorithm is the behavior of go get -u: download and use the latest version of everything. This mode fails by using versions that are too new: if you run go get -u to download A, it will correctly update to B 1.2, but it will also update to C 1.3 and E 1.3, which aren't what A asks for, may not have been tested, and may not work.”

In this example, if updating to C 1.3 and E 1.3 causes breakage, the author of these packages has violated the very principle the author of this post is claiming should be upheld, that of semantic versioning. The author appears to simultaneously argue in favour of and against semantic versioning.

At best, perhaps author is arguing that semantic versioning should be adopted but that upstream developers cannot be trusted to abide by it anyway. But this is irrelevant; it's against my policy to accommodate other people's incompetence, as this just aids them in getting away with it. It also raises the question of what the point is of adopting semantic versioning if library consumers aren't going to make constructive use of its conventions.

“Minimal version selection is very simple. It achieves simplicity by eliminating all flexibility about what the answer must be: the build list is exactly the versions specified in the requirements.”

This is a rather self-serving argument, since the rest of this post is literally describing algorithms to modify these build lists to get behaviour which would be automatic in other systems.

“If you are familiar with the way most other systems approach version selection, or if you remember my Version SAT post from a year ago, probably the most striking feature of Minimal version selection is that it does not solve general Boolean satisfiability, or SAT.”

Neither does a conventional system based on semantic versioning, which always uses the latest version within a given major version. Which is basically demonstrated by the fact that this very blogpost describes an algorithm for doing just that, updating modules to the latest versions. SAT only comes into it if you have some module which refuses to work with a later version inside the same major version, but I don't tolerate such packages. Any such package would be fixed, forked, or fired by me. Shoddy code shall not be permitted to intercede in the upgrading of packages. Thus the entire set of “problems” Go modules is trying to “solve” is of no interest to me whatsoever.

The Cancer Continues III

The series continues further.

“We want to encourage more developers to tag releases of their packages, instead of expecting that users will just pick a commit hash that looks good to them. Tagging explicit releases makes clear what is expected to be useful to others and what is still under development.”

This is fine. I do this already.

“We want to allow multiple modules to be developed in a single source code repository but versioned independently.”

Not only is this a bad idea, note that these two goals seem mutually exclusive. They want different modules in the same repository to be able to have different versions, but they also want versions to be tagged by the repository. The author's proposed solution to this is to allow repositories to contain subdirectories like “v2” as an alternative to a tag. I have no idea why you would want to do this as it completely defeats the point of a version control system. At least it's optional.

“We want to move away from invoking version control tools such as bzr, fossil, git, hg, and svn to download source code. These fragment the ecosystem: packages developed using Bazaar or Fossil, for example, are effectively unavailable to users who cannot or choose not to install these tools. The version control tools have also been a source of exciting security problems. It would be good to move them outside the security perimeter.”

I don't particularly care how code is distributed per se, but I only provide my code as Git repositories. While people are free to derive other forms, such as tarballs, from these if they please (for example, Github does this automatically), I don't regard these derived items as definitive.

Since the new modules proposal can only consume ZIP files, rather than VCS repositories, anyone wanting to host their own modules server is therefore required to provide ZIP files as a normative distribution of their source code. I decline to do so.

Moreover, this raises a very obvious question. At various times in these posts, the author has indicated that it should be possible to reference commits by commit hash. But there is no way of confirming cryptographically that a tarball is an accurate embodiment of a Git hash, short of retrieving the Git repository anyway. So these objectives seem antithetical to reproducible builds, a claimed objective, because they necessarily involve trusting a central party (see below) to be an oracle to provide a mapping for Git commit hashes to ZIP files. This completely defeats the point of reproducible builds, part of which is based around not trusting specific central parties to be trustworthy distributors of artefacts derived from source code repositories. The author does not appear to understand what “reproducible builds” means.

“We want to make it easy for individuals and companies to put caching proxies in front of go get downloads, whether for availability (use a local copy to ensure the download works tomorrow) or security (vet packages before they can be used inside a company). We want to make it possible, at some future point, to introduce a shared proxy for use by the Go community, similar in spirit to those used by Rust, Node, and other languages.”

Stop, stop, stop. This is explicitly an agenda to get rid of Go's distributed package retrieval system and replace it with something centrally controlled and administered by golang.org — in fact, such a service has now been set up. This is wholly unacceptable centralisation, and a very depressing move to “what every other programming language with a package manager does”.

“At the same time, the design must work well without assuming such a proxy or registry.”

It's still reasonable to assume that this design is going to encourage people to use this registry by default, will create large amounts of dependence on it, and will therefore lead automatically to large amounts of centralisation in Go package distribution which weren't previously present.

“We want to eliminate vendor directories. They were introduced for reproducibility and availability, but we now have better mechanisms. Reproducibility is handled by proper versioning, and availability is handled by caching proxies.”

Modules seek to eliminate vendor directories in name only — by entrenching vendoring itself directly into the packaging system as a canonical practice. See above.

“To start, it suffices to create a repository and tag a commit, using a semver-formatted tag like v0.1.0. The leading v is required, and having three numbers is also required. Although vgo itself accepts shorthands like v0.1 on the command line, the canonical form v0.1.0 must be used in repository tags, to avoid ambiguity.”

Note that this appears to be trying to force Go developers to adopt a specific version numbering scheme, beyond merely “first component indicates breaking changes”. For example, it assumes a library uses three-part version numbers rather than, say, two-part version numbers.

“Authors also need to be able to deprecate a version, to indicate that it should not be used anymore. This is not yet implemented in the vgo prototype, but one way it could work would be to define that on code hosting sites, the existence of a tag v1.0.0+deprecated (ideally pointing at the same commit as v1.0.0) would indicate that the commit is deprecated. It is of course important not to remove the tag entirely, because that will break builds.”

At this point, I simply do not believe that the author isn't aware that this is an insane proposal. This is the kludgiest idea in this entire proposal given so far. Using the existence of a tag to add one bit of metadata to another tag is in no way sane or acceptable as a proposal.

“With the combination of the lightweight repository conventions, which mostly match what developers are already doing, and the support for known code hosting sites, we expect that most open source activity will be unaffected by the move to modules, other than simply adding a go.mod to each repository.”

No go.mod file will ever be committed to any repository I control, as to do so would require me to express precise point versions of that repository's dependencies, which I decline to do.

The article ends with a section entitled “The End of Vendoring”, which is amazing because it would be more accurately titled “The Beginning of Vendoring for Everyone, Whether You Like It Or Not”.


See also: Position statement on Go modules

Other articles in this series: