Should combo handle complex upgrades?

[joelanford]

If a combo Template contains an object that has complex upgrade semantics (e.g. CRDs, workloads, maybe PVs/PVCs), how does a combo Combination handle those upgrades gracefully?

Should we build these APIs to specifically preclude upgrades from happening (e.g. by forcing the user to delete the Combination and create a replacement to trigger a delete/recreate flow instead of an in-place update)?

To do that, would we need to make both the Template and Combination immutable?

[joelanford]

Another option that @exdx mentioned would be to leave cluster apply functionality out of the Combination controller and delegate to another controller to actually deploy the rendered combination to the cluster.

[joelanford]

make both the Template and Combination immutable

Even that might not be enough. I could create a Template and Combination referencing that template, and then delete the Template and replace it with a new one of the same name. It seems like we'd have to avoid re-rendering the Combination from its Template after the first rendering succeeds.

[tylerslaton]

make both the Template and Combination immutable

Even that might not be enough. I could create a Template and Combination referencing that template, and then delete the Template and replace it with a new one of the same name. It seems like we'd have to avoid re-rendering the Combination from its Template after the first rendering succeeds.

This is something that Nick brought up pretty early on. We could create a hash of the original template and if doesn't match what the Combination is expecting we don't apply it. Thoughts?

[joelanford]

I think that would solve the technical problem, but IMO that would be a pretty wonky user experience.

Thinking through a scenario here:

1. Create a Template named foo-tmpl (with UID 1)
2. Create a Combination named foo-combo (with UID 1)
3. Combo controller renders manifest for foo-combo from foo-tmpl and applies it to the cluster
4. Delete foo-tmpl (open question: what happens to foo-combo's on-cluster resources here?)
5. Create a new foo-tmpl (with UID 2 now) (open question: what happens to foo-combo's on-cluster resources here?)

One way to answer this that avoids upgrades happening is that in step 4, the Combination controller treats a "template not found" error (because it was just deleted) as an empty template and then deletes everything it was managing. Then in step 5 when a new foo-tmpl shows up, nothing is on the cluster so step 5 is the same as step 2 (i.e. just create everything).

So TL;DR: One solution is:

1. Template objects are immutable
2. Combination objects are immutable
3. When the combination controller gets a "Not Found" error looking up the template, we delete any existing objects owned by the Combination.