Kubernetes Forum Sydney 2019

If you’ve deployed your own Kubernetes cluster, care and feeding of an etcd cluster is a necessary evil. At the end of the day, etcd is the place where your cluster’s buck stops, and despite the marketing hype, operating an etcd cluster at scale is not a set and forget experience.

This talk tells a story of how my team grew from etcd novices to delivering a well-monitored, reasonably resilient, etcd system that could be upgraded in less than half an hour per cluster, online, with no downtime.

After this talk, you will have:
- a better understanding of etcd's sharp edges, and what you can do to avoid catching yourself on them
- some insights on key etcd metrics to keep an eye on and why
- what we tried with upgrades, what worked, and what didn't, and how you can avoid stepping in our potholes
- some war stories, like when we accidentally made 80k namespaces in Kube and filled our etcd.