Upgrading Clusters and Nodes

Transcript:

Another important operational consideration with Kubernetes is ensuring that your cluster remains up to date, both the control plane as well as your worker nodes. Kubernetes, the project, releases updates every three or four months, generally about three times a year. And after a certain period of time, they stop maintaining those older versions. That means those older versions won't get security updates. And as you build out on top of Kubernetes, you're inevitably going to find a feature that was released recently that you want to leverage. This module, I'm going to take a quick look at one possible procedure for updating your cluster and nodes safely. This is a procedure that I've used a number of times. In certain cases, your managed cluster may not provide the level of control that is required to do this, and you may have to upgrade in place. But if possible, I suggest using an approach like this. The first step of any upgrade is to check and see if you are utilizing any of the APIs that may be removed in the version you're upgrading to. So as the Kubernetes APIs evolve, they add new versions, old versions may get taken away. And so you want to make sure that you do not have any of those older resources deployed into your cluster. And if you do, you should upgrade them to the newer versions. They have an upgrade path where they wait a few versions before they remove any deprecated resources to allow you to move to the new version, get the new resource type, get the new resource version, upgrade your resources to unblock the next upgrade. There's an open source tool that can help to do that called kubent, or kubenotrouble. Here's the GitHub repo for it. I'm gonna run it against my clusters just to verify that I'm not using any. This will scan your cluster, check what API version you're using, and check all of the Kubernetes resources that are deployed to make sure that none of those are being removed. And it will warn you if they are. Once you're sure that you're not using any of those deprecated APIs, you want to upgrade your control plane. Now your control plane can be ahead of your worker nodes in terms of version, but it used to be by one minor version. I believe they've relaxed that to now be two minor versions. So you'll want to upgrade your control plane version first. And then once your control plane is upgraded, you want to upgrade your nodes. There's two paths to doing this. You could either upgrade the nodes in place. Usually what this means is that the cloud provider, such as NGKE, has a concept of a node pool with multiple nodes in it. It will add a new node with the newer version, cordon and drain the older node, which will move traffic onto the new one, and cycle through the nodes in your node pool one at a time or multiple at a time, depending on the configuration settings you've applied. The other option, which I think is safer, is to deploy an entirely new set of nodes using the newer version. Once those have come up and are healthy, you can then shift the workloads that are running in your cluster onto those new nodes. And once that is done, you can delete the old nodes. This blue-green approach is safer because if something goes wrong in that upgrade process, it's very easy to shift the workloads back because you already have a set of healthy nodes, and you're much less likely to encounter downtime due to having an upgrade issue and not having available capacity at the ready. I'm calling out here that some people do away with this problem entirely by moving to an entirely new cluster. So rather than upgrading your control plane, some people back up the state of an entire cluster and shift it over. If you have a bunch of stateful workloads running your cluster, this is much more difficult to do with no downtime. But I'm just calling out there that some people feel that that is safer. So not only just shifting onto a new set of worker nodes, but also shifting onto an entirely new cluster is a different approach. Also, it's important to call out here that the blue-green approach that I'm talking about of creating new nodes and destroying the old ones is much easier to achieve in a cloud environment than in an on-premises environment. Because of that, you're more likely to upgrade in place with a long-lived physical server that you're managing on-premises rather than needing to have a whole new set of hardware to move your workloads onto. And so with that, let's go ahead and upgrade our GKE cluster using this process. We'll start by navigating to module 13. The first step is gonna be to run kubectl against our cluster. Let's make sure we're pointed at our GKE cluster. It will use your default kubectl context. So I'm running just the kubectl command that should have been installed by dev box. Here it's using Rego, which is a language that you can use to specify types of policies. It's checking against all of the different Kubernetes APIs that were deprecated in each of these versions or that are planned to be deprecated in the future. We got no warnings back, so that means we are good to go and there are no deprecated APIs being used. So we can proceed to the next step. If we look at our Kubernetes cluster in the console and click into it, we can see I am on version 129.6. Then we want to list the available versions that we can upgrade to. GKE maintains three release channels. There's a stable, which upgrades the most slowly, regular, a bit faster, and rapid has even more versions available. As you can see, only the rapid channel can you use the 1.30 versions. So let's make sure that our cluster is indeed using that rapid version. So we'll update the release channel accordingly. That runs the gcloud container clusters command and passes it the rapid channel. And let's pick one of these versions that we want to upgrade to. Let's grab the default version for the rapid channel. That seems fine. And let's specify it in our task here. Paste that in. And so this version variable is going to get substituted in when we call our container clusters upgrade command. And specifically, I'm telling it to upgrade the control plane. So let's do that. So here it issued that command, passing it the version. It's warning me that this is going to block other operations on the cluster. That's fine. Now this is just upgrading the control plane. It shouldn't impact the worker nodes at all. So I wouldn't expect any of my workloads to move and they should continue running just fine while this upgrade is happening. If I reload the page here on Google Cloud, you'll see that my cluster is now in an upgrading state. This will take a few minutes behind the scenes. So after about five minutes, that control plane upgrade completed. You can see here this warning. You can see here this note that auto upgrade is enabled. That node upgrade would happen in the background. And GKE has been around enough and has tested enough of these upgrades that auto upgrade in place for a node pool is generally pretty darn safe. But if you want to take one extra degree of safety, you can do what I'm about to show you and provision a new node pool on that new version and shift the workloads onto it. To do that, we're going to run the node pools create command. It's going to be the same machine type as before, two nodes within it and passing in our cluster and zone. Just as a reminder, previously we had just the default pool. Now we have the default pool and the updated node pool. You can see the versions are different because the new node pool is going to use the same version that the control plane is using. If we do K get nodes within the cluster, we can see the four nodes here, three of them are ready and one of them is not. Now all four nodes are ready. At this point, we want to shift the workloads from the old node pool to the new one. I have this command here. You can do what's called coordinating a node and that will apply a taint such that new workloads will not be scheduled onto it. Then you can issue the drain command, which will terminate all the workloads on an existing node. There's some additional options here that can help if things are getting stuck around, if there's empty directory volumes that would prevent a workload from being deleted or getting stuck for another reason, you can add the force flag. The first thing that I'm going to do though is to just cordon the nodes. So if I just want to cordon the two old nodes, let's do K get nodes. Let's do K get nodes. And then we'll issue a K cordon on that node and that node. You can see that the status has changed from ready to ready scheduling disabled. Because this is now cordon, new workloads will not be scheduled onto it. If I do a K get pods, if I do a K get pods dash O wide, you can see that these are currently all scheduled onto the default node pool. But if I do a K rollout restart on a deployment, as you can see, they were now scheduled onto the new node pool. Great. Any workloads that you're worried about having downtime for, you would roll out restart and as those new pods are scheduled, because we've cordoned the old nodes, they'll get scheduled onto that new updated pool. Now just to force everything else, once you've moved the workloads that you're worried about downtime for, you can issue this drain command and specifically you can pass it the force flag, which is kind of like taking a sledgehammer to it and just saying terminate all the pods regardless of their current state. So let's go ahead and do that just to force everything else over. It's evicting a bunch of pods. And so eventually, once you've restarted all the workloads that you're worried about downtime for, the only things running on your old node pool are gonna be daemon sets. And that's okay, because we already have one copy of that running on each of our new nodes as well. So we can delete that node pool safely. So we can issue a gcloud container node pools delete. We'll say yes. And then we'll do a K get pods dash O wide. And then we'll do a K get pods dash O wide. We'll say yes. And at this point, Google is gonna go off and delete this node pool from our cluster. Now, a few minutes later, the original default node pool is gone. We've got our updated node pool here. If we look at the nodes in the cluster, we see only the two nodes. Awesome. And so, like I said, doing an in-place upgrade can work and it will cycle one node at a time. However, having that additional capacity ready to shift the workloads back to if something were to go wrong is a nice safeguard to have.