Interface evolution

Overview

Software components of all kinds coevolve with each other, leading to version hell when you try to select versions of them that work together. The components have an interface between them that is never fully explicit in practice. Since that interface is not fully stated or checked, and since these interfaces drift over time, you need some kind of outside system to reason about which versions work well together. The problem occurs widely, including with Maven Central, Debian-based Linux distributions, and the Node Package Manager.

The approach I studied is based on the idea of a universe of things that are visible to a participant at a given time. While a participant inside a system will talk about “the universe”, they really mean “my universe”. You can have a very simple dependency system if you have some form of testing and control for the overall universe that components are shared in.

I first noticed this approach being used with Debian, where the majority of dependencies between packages were very simple. The reason it works is that they have some control over the overall set of packages that are available. If you upload a package that breaks other ones, you are quickly informed and are expected to roll back your change or to fix it. Later in life, I saw this used at Google for their C++, Java, Go, and other software components. At Google, teams share components with each other via one large mono-repository, and like with Debian, teams generally share and use one latest version of each component with each other.

Supporting theory

I have researched some of the theory that can be used to work within a package universe.

The following papers explore type checking for interface evolution. I started with Java’s deprecation annotation, and I added a “forthcoming” keyword for the opposite of deprecation, meaning that an interface is going to gain a new method, in the future. I could have really used “forthcoming” at Google, sometimes, for certain kinds of changes, and I think it’s worth considering in standard programming languages.

I thought this work was going to be a warm-up toy exercise, in preparation for looking for more difficult cases to study, but I already found a gap in Java’s deprecation checker. Java allows a deprecated method to override a non-deprecated method, and because of this, it cannot always detect when a deprecated method is being called. I updated the Scala compiler to forbid this case, because I think you want to get a warning from the compiler whenever a deprecated method is still in use. With the way Java works, you can have a warning-free compile but still not be ready to remove something that is deprecated.

I also wrote a position paper on how package universes work, in general.

Practical usage in industry

I have worked with multiple companies to dig out of an internal version hell and find a way for teams to share their work together more conveniently.

The most common strategy I found to work well is to transition to trunk-based development in a mono-repo. The reason it works well is that workers don’t have to know how it works. They just know that they can’t push if they break any tests, including the tests of their coworkers.

Packaging tools

I developed experimental tools for Squeak and for Scala to experiment with how package universes can work.