Keeping versions simple

Today I learned that, while there is a rich and subtle syntax for Python dependency specifications, it’s probably simpler and wiser to ignore it and just use explicit version bounds like ">=0.5.1,<1.0.0".

In particular, I learned that caret requirements like "^0.5.1" are very counterintuitive, since their behavior is qualitatively different if the major version is zero.

A surprising error: semantic versioning versus the caret

What led me to noticing this was this bug report from Johno about vertexauth:

I’ve updated claudette, had to downgrade anthropic for vertexauth (pip install -U anthropic[vertex] fails because “vertexauth 0.1.1 requires anthropic[vertex]<0.38.0,>=0.37.1, but you have anthropic 0.40.0 which is incompatible.”)

Reading this, it is clear that vertexauth required an anthropic version higher than 0.37.1 but lower than 0.38. But why would the author of vertexauth want to require that? I am that author, and I didn’t!

In my mind, the main purpose of any library (this one especially) was for it to be easy to use. And to that end a library should not require a highly specific dependency. Why? Because that will be awkward for the user, and because fundamentally a library should not require its user to fix or change things to use it. The way to do that, is to accommodate as wide a range of dependency versions as possible.

Recall that, according to semantic versioning, the major version is the first of the three digits, and only an increase in the major version is intended to indicate a backward-incompatible change. So if vertexauth works with 0.37.something, then it should also work fine with 0.38.something, and so on up to but not necessarily including 1.0.0. So I had not aimed to define a dependency more restrictive than this.

In fact, what I had written was the following:

anthropic = {version = "^0.37.1", extras = ["vertex"]}

What does ^ mean? Here’s the rub. The poetry tool’s documentation describes dependency specifiers with these examples:

These do suggest that ^0.37.1 would allow any version equal to or higher than 0.37.1, all the way up to a major version bump, of 1.0.0. But it’s not so! Reading further, it turns out the exact definition of caret requirements is as follows (emphasis mine):

Caret requirements allow SemVer compatible updates to a specified version. An update is allowed if the new version number does not modify the left-most non-zero digit in the major, minor, patch grouping. For instance, if we previously ran poetry add requests@^2.13.0 and wanted to update the library and ran poetry update requests, poetry would update us to version 2.14.0 if it was available, but would not update us to 3.0.0. If instead we had specified the version string as ^0.1.13, poetry would update to 0.1.14 but not 0.2.0. 0.0.x is not considered compatible with any other version.

In other words, when used with a version number with leading zeroes, the caret requirement behaves as if the first non-zero version marks incompatibility even if it is minor or patch version. You can also see it in how these examples are different from the ones above.

Reasons and takeaways

I got to wondering, why does caret behaves so oddly?

I’m not sure but I suspect the reason lies in this paragraph of the semver spec:

Major version zero (0.y.z) is for initial development. Anything MAY change at any time. The public API SHOULD NOT be considered stable.

That is, if the major version is 0, then in theory the entire version number actually implies no stability or compatibility guarantees at all. The API could break with any version change at all. So based on this close reading of the special meaning of zero-major versions, it makes sense that to have a caret specifier which behaves differently for them.

However, out here in reality, I think hardly anyone treats zero-major versions this way at all. anthropic 0.38 will work fine wherever 0.37 works. In short, it seems like caret requirement specifiers work oddly because they are closely following the letter of the SemVer spec, which no one follows in practice.

In the end, my takeaway here is that caret-based requirements are too clever for me, and maybe too clever for themselves, as are perhaps tilde-based requirements and wildcard-based requirements. But two inequality bounds are a familiar syntax, since I see inequalities every day, and they are expressive of every likely intent.

This reminds me of one way to handle the complex rules governing when you may omit a semicolon at the end of a javascript statement: ignore them. Instead, always put a semicolon there always, since it is always legal.

In general, if there is a simple and reliable way to do something, and also a subtle and finnicky way, I’d prefer to skip the subtlety and the potential errors it brings. I’ll save my capacity for subtlety for where it’s needed, rather than waste it on trivia!