Monthly Archives: October 2021

Immutable Kubernetes Selectors

it’s easy to be frustrated with immutable selectors on deployments and statefulSets, i know i was for a while, but after reading a bit more about it tonight, i feel it was a good call.

btw, jobs and immutability are a different ball of wax that i’m not addressing here.

the opener on this issue explains a large part of it, especially this bit: “users are not expected to change the selectors” – i.e. the controllers all the way down from deploymentController|statefulSetController->replicaSetController->scheduler->kubelet were designed to rely on these selectors not changing, but not as an oversight; as an opinionated decision that is a major enabler. – a better way to think of these labels might be something closer to a unique identifier, and by making them immutable they’re avoiding the need to track the changing of unique identifiers in a state machine.

– not that all labels act like UIDs, only the ones you use as selectors in parent objects.

the emphasis on parent objects is key here. the serviceController does not need the same constraint because services are essentially orthogonal or sisters to the things to which their selectors point. but if the selector of a deployment changes, the deploymentController has no way to know what that selector was previously, and therefore has no way to know which, if any, existing replicaSets are related to that deployment. it proceeds to find no replicaSets matching the selector, the existing ones are now orphaned, and it creates new ones.

apparently a lot of people were shooting themselves in the foot, and experiencing orphan “storms” before the community decided to stop defaulting spec.selector to spec.template.labels (which was defaulted from metadata.labels and is seemingly the only reason they ever thought it was OK to make the selectors mutable on deployments (they were never mutable on statefulSets.))

it’s easy to think of Kubernetes labels as just visual cues, and sometimes they are just that, but labels are also a first class citizen around which the core Kubernetes code and algorithms are designed. they give you the magical combination of A) a powerful arbitrary key value labeling and searching, and B) the dependability of UIDs, or reference-ability across the database. the trade off is, immutable labelSelectors.

i.e. you gotta delete and recreate that sh*t if you wanna change the labelSectors.

P.S. I haven’t blogged for a while mostly because I try too hard to get it all perfect. So I’m trying a new, less time consuming approach, as seen above.