Ingester: support zone-aware replication

[timonegk]

What this PR does:
Add support for zone-aware replication to ingesters. Similar to #632, but with less duplication.
I have documented the migration process and tested it in a cluster with six ingesters.

Which issue(s) this PR fixes:
Fixes #203.

Checklist

• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX], [DEPENDENCY]

[kd7lxl]

Why would you modify the chart when the chart already supports another technique for zone awareness that doesn't require further modification? Is there something I'm missing?

[nschad]

another technique for zone awareness

That requires the admission controller, otherwise you won't get the env-vars. If that is something you are ok with running, then yes you are correct. Otherwise there is currently no true "stand-a-lone" way of deploying zone-aware ingesters

right?

[kd7lxl]

another technique for zone awareness

That requires the admission controller, otherwise you won't get the env-vars. If that is something you are ok with running, then yes you are correct. Otherwise there is currently no true "stand-a-lone" way of deploying zone-aware ingesters

right?

Yes, I would run the admission controller. There is no need for the admission controller to be deployed in the same chart as cortex (nor would I couple them), so this is possible today. A guide doc may be the only contribution needed.

A strong benefit of the admission controller is that is does not require prior knowledge of the available zones. In contrast, the configuration method requires first collecting information about the target cluster and what zones are available, then configuring cortex for those zones. This is significant increased deployment complexity.

[nschad]

another technique for zone awareness

That requires the admission controller, otherwise you won't get the env-vars. If that is something you are ok with running, then yes you are correct. Otherwise there is currently no true "stand-a-lone" way of deploying zone-aware ingesters
right?

Yes, I would run the admission controller. There is no need for the admission controller to be deployed in the same chart as cortex (nor would I couple them), so this is possible today. A guide doc may be the only contribution needed.

A strong benefit of the admission controller is that is does not require prior knowledge of the available zones. In contrast, the configuration method requires first collecting information about the target cluster and what zones are available, then configuring cortex for those zones. This is significant increased deployment complexity.

on the other hand, this PR allows you have to one Deployment/StatefulSet per zone which can be beneficial if you want to do per-zone rollouts. For example facialited by the grafana rollout operator

I think there is a case to be made for both. Also the admission controller (even though the code is not complicated) is not maintained.

[timonegk]

@kd7lxl we are mostly benefitting from this change because of faster rollouts. With zone-aware ingesters, you can roll out one zone at a time, so a rollout out is O(zones) instead of O(ingesters). For us, that brings rollout times down from several hours to about 15 minutes.
If I understand your proposal correctly, this is not something we could achieve with the admission controller.

[kd7lxl]

@kd7lxl we are mostly benefitting from this change because of faster rollouts. With zone-aware ingesters, you can roll out one zone at a time, so a rollout out is O(zones) instead of O(ingesters). For us, that brings rollout times down from several hours to about 15 minutes. If I understand your proposal correctly, this is not something we could achieve with the admission controller.

That is a valuable benefit!

[nschad]

what's that? zone-default ?

[timonegk]

It's part of the migration process. When the write path is in migration (migration.writePath=true), the default zone should be excluded so that we only write to the new zones.
But the zone name should be default, not zone-default. I fixed it, but that means I have to test again.

[timonegk]

With the wrong zone name the test worked because there were still 2/3 replicas in the other ingester zones. I will test again tomorrow with the fixed value.