Upgrade, update, or migration? Why getting it wrong wrecks your timeline

Underneath that, you have minor versions. Um typically, what you have is you have a major version that you install and you lock into and then every 6 months a new minor version comes out and that's a kind of a gate where more significant changes happen that allows teams to kind of prepare for and have a known cadence shift in major changes. These are not nearly as significant as major version shifts, but they're still like a kernel bump, for example, which could cause compatibility or breakage with the the kernel the cabbie stability. Uh so you may have to like resign or redo your K mods for example or some software is going to have a bigger shift and allows you kind of prepare for it rather than it just happening one day in the middle of the week.

And that gets us to the last part which is the updates. Once you've installed a version of Linux like Rocky Linux at 9.6 or in this case you can see 8.10. Every time you run a DNF update which is their package manager command to pull down the latest packages available upstream, those individual packages will update. So going back up in the bottom, if you're just running a DNF update, that's just updating whatever packages are ready and available uh on an individual package basis. Then you look at the minor version which is a larger bump, a larger shift. Um they're making bigger changes during these minor version bumps, but they're still relatively, you know, light.

There's a there's a upgrade path that's been tested extensively, but this is more what something someone would call an upgrade, especially for an enterprise. These bumps can be more serious depending on how much customization you have. And then when we're really talking about upgrades more broadly, we're talking about upgrades from a major version. Um I'm moving from 8 to 9, 9 to 10. These are significantly bigger shifts and for most enterprises, something is going to need to be updated on your application stack to ensure that this is a smooth transition. >> Got it. So based on that, walk us through what happens outside of a version, but when we're you're going to be moving let's say from maybe one Linux distribution to another.

Why are some of those uh moves also maybe straightforward and then why are a lot of other parts of that uh a lot of work? >> Absolutely. I I in a typical server environment, like a fleet environment, you're going to have many different Linux distributions because there are many different use cases in your fleet. You've got databases, and you've got servers, and you've got, I know, runners, and you've got orchestrators, and you've got you've got long-term batches. So, you're going to have different needs for even in one rack, you might have different operating systems running. And you have to maintain all of them. Um when we look at them, they're in roughly families and schools of thought.

Enterprise Linux is one school of thought, and that kind of comes from the Fedora family. Ubuntu was one school of thought, and that's downstream of Debian. And these are generally for enterprise the two major schools of thought. Um and they're both start from the Linux kernel, but then they fork, and they start they even can have a lot of the same user space packages that they used to define them. But, they have different uh repository structures. They've got different metadata structures. They've got different ways to install. They have different package managers. And the further you go down the stream, the more divergence and iteration there is.

The first thing to look at from a migration standpoint is um within the scope of, say, the Fedora family. Uh Rocky Linux and RC Pro is binary compatible with other Enterprise Linux uh distributions. That'd be Oracle Linux, AlmaLinux, CentOS Stream, and Red Hat Enterprise Linux. They're binary compatible. And what this effectively means is that if you have your application stack running on one of these Enterprise Linux, if you move to the same version of another Enterprise Linux distribution, things are generally going to work out. They're it's the same philosophy, mostly times the same versions, same packages. There will be subtle differences in the metadata or the wrapper or certain packages, but by and large, they're the same binary in terms of you're going to it's going to be compatible meaning that application stack is just going to keep running.

It's going to be the same compatibility with you that you see on one versus the other. That doesn't mean you can just take packages from all of these different Enterprise Linux distributions and throw them into a pot and call it a day. It's they're designed to work together and that's where that metadata and that glue that makes these distributions a distribution and not just a bunch of packages in a bucket unique. So, if you take Oracle Linux packages and a random AlmaLinux package and a Rocky Linux package and an OLCI package and suddenly they're going to start conflicting cuz they're they're expecting the same they're they're have the same packages, but they're expecting them to be maybe in different places or communicate slightly differently or have different release packages or update slightly differently or from different places.

And that's where when you migrate even within the Enterprise Linux family, you generally have to do a full migration. Now, when you're talking about a migration to say Ubuntu from a Rocky Linux or back vice versa, that is a far different discussion. Effectively, it's not compatible. You're not just going to be able to swap out the foundation. You have to do a clean installation. >> So, for those binary compatible moves, where you're going from let's say Red Hat to RLC, uh I know that we have CIQ is specifically built out some migration resources. I think we call it migrate to RLC. What what does that actually do?

And and when should you be reaching to use that? >> What it effectively does is it takes a think of it as a house and you've built a house on top of this plot of land. Um if you're moving from Red Hat to RHEL C Pro, you're effectively swapping out the foundation. Because the the foundation is all of those binary compatible packages. Um and from the intent of your your you know, your application stack, your custom proprietary packages, your code, it broadly shouldn't notice a difference. You have to do a complete transition though. Uh you don't want to be mixing and matching them. Technically it works, but you never know which particular package is going to play nicely with the other packages.

So, it's safest just to do a complete transition. You get into things like signing keys and checksums and other things that you don't want to deal with. Um the the migration like this, it's never 100% guaranteed because you may have some customization that you've done or some particular packages that are expecting one of the other. Sometimes uh packages or software even explicitly looks for an identifier to say, "I only work on Red Hat Linux." And any other Linux, I'm not going to play nicely with I'm going to throw an error because they're hard-coded to one of those like metadata pieces. But, again, outside of those, those are relatively light things to solve.

Um that's where as long as you're moving within the same version, you're going to have, you know, most of the time a tool like migrate to RHEL C is going to handle all of that by just lifting and shifting everything over. So, your base base moves underneath you, but it's all compatible. Most of the time you're not going to notice things or there's some minor polish. When you start making moves to uh distributions with the larger deltas or to a different version, that's where things start getting messy. Because your customization um may be hardwired or tuned to particular package versions or particular versions that don't uh transition well.

And so, the larger that delta is, the more risk that that migration doesn't play nicely and you're going to want to just do a clean install. >> Yeah. So, with especially with those higher risk things, I know that when we talked in the past, you always are kind of a very good proponent of uh maybe doing a clean start. Uh can you tell me why and in the realm of defining terms like we're doing that, would you still consider that a {quote} migration? If you were going to be doing some of those hard clean starts on a different distribution. >> I would absolutely still consider those a migration.

It's it's the same amount of work whether you're starting I mean you're still having to make a plan, you're still having to test it, you're still having to go through all the same preparation and motions to make the shift. But there are different considerations. When you're moving your application stack over to a fresh install, the there's more tuning you have to put into place. But you are you're not it's it's much more straightforward bringing the tuning on a fresh install up to match what you wanted. If you're using a tool like a sender, you can develop a run book, establish your you know, your deltas and your tuning, spin up a VM, identify the difference, update the run book.

I mean it's a very safe approach and you're not there's nothing kind of left over. There's no cropped or dead or you know, rust that's been accumulated over time from various updates in that install. It's fresh. If you are in a situation where you have a pet where you really can't transition it over clean for whatever reason, then you are just naturally going to have to account for all of the weird little things that have accumulated over time, all the configurations, all the random packages you've installed, any drift there's been from the default distribution, and when you make that migration, no one tool is going to know your setup perfectly and what the correct target transition is.

It's going to try to transition you to whatever the updated version is. If you are on RLC Rocky Linux 8 and you're transitioning to Rocky Linux 9, well, it's going to update you to the Glibc version that is latest on Rocky Linux 9. But, if you have some tuning or customization or packages that are dependent on older versions of Glibc, you're going to have to reconcile that as part of the migration and ensure that not just that Glibc version is getting updated, but the applications that depend on that Glibc version are also adapted to that newer version of Glibc, and you're going to have to

kind of hold those two things true at once, or you're going to have to make sure that when you make that transition, every single one of these little nuances gets handled and cleaned up before you can spin it back, you know, spin up the back. So, you have to do not just one little thing at once, one little change at a time. You're doing a thousand changes all at once, and then you've got to make sure that that runbook uh completes everything all at once, and then do that across hundreds of nodes, perhaps. So, there is that migration is just there's more complexity and there's more room for error and there's more risk and there's a lot more preparation that you have to do and testing you have to do before you make that big of a migration.

So, it's always easier to do it on clean install. Um it's not that it's impossible to do a migration, um you know, an existing pet and upgrade it to a newer version, but you, you know, the kind of rule of thumb is it's if 50/50 whether it's going to be an easy transition and you don't even notice anything, or it takes weeks of tuning and polish to make sure that when you do that upgrade in production, it just lands. >> It sounds like there's uh there's always going to be some factor that you need to kind of consider. And on that note, you were kind of talking towards it a little bit, but theoretically, let's say I'm about to kick off a Linux project.

We're going to be moving, you know, half of our fleet over to uh a a new distribution. What's What would you say should should be like the things that I need to do before I even put a date on the calendar to say this is when we're going to get this done. >> The any type of migration is not going to happen overnight. Um or rather, it's one of those things, it's an overnight success years in the making. Um there's it's preparation, preparation, preparation. Um the the mean the obvious ones are setting up a a dev environment with a snapshot production, and then just running the first test.

Try migrate it, try updating it, see what breaks. What you want to avoid is having to spend weeks iterating through error after error after error um sequentially. You want to find as much of the uh challenges up front so you can get a full list of all the work you have to do to make that transition. Uh a tool like um migrate to RHC is already going to use run that against your dev environment, see what happens. It'll help you identify most of the blockages and the changes that you have to account for in your application stack. And you kind of get a an upfront look and work on these things in parallel.

But uh the the other part is just generally taking inventory of what you have changed, your delta from a default environment, which things you've done differently, uh and then things that are maybe not necessarily thought of. Uh cryptography is a big like often a big issue. When enterprises have their stack and they depend on certain cryptographic algorithms, and they migrate to a newer or you know, upgrade to a newer version, um sometimes those that cryptography is old, it's no longer supported. Or if you're migrating from not from FIPS, where you have FIPS mode on, but it's not actually compliant, to RHEL C Pro, where when FIPS mode's on, it's actually tested by a lab and certain algorithms are turned off because they're not allowed, you may be surprised.

And often we have customers who are like, "Well, I'm not using that algorithm, it's fine." And then they go deploy, and they go, "Oh, I didn't know this random package over here in my stack was using that algorithm. Now I've got to go update that software stack." And that's where a lot of the time goes. It's not necessarily the migration to the new Linux. Linux is pretty stable. It's your application stack depending on old things or old packages that are no longer available, or they've changed in this new version, and updating your application stack and your software and your custom, you know, your custom code is where more of the time goes into bringing into alignment with that new version.

>> Uh So, uh you talked about this earlier and on that note, too, kind of just tying this in, uh you said you were talking about like the pet servers, and I I I can't remember any company that I've worked at who didn't have a just a cadre of pet servers kind of hanging out there, um which from my perspective means they're each kind of like bespoke, they're each kind of like this unique snowflake on how they how they run them. When you're planning out to maybe convert half or maybe all of those pet servers over to kind of a new distribution, >> Mhm. >> uh does that mean I have to build like dev versions of each one of them and test them out?

Or what does that what does that look like? >> should. You certainly should. Um I would never recommend yellowing uh a pet server migration or update or upgrade. The This is one where you're going to run into challenges. CIQ support our our support engineers are very experienced in these types of things. They can provide and typically provide a lot of advice and support for our customers when they're making these transitions because they've seen it before. And uh there are even opportunities if you need, you know, some additional hands-on keyboard help. CIQ can support customers with a hands-on keyboard transitions when the transition is just too complicated.

Um you're not, you know, you're not doing this all the time. Your company isn't making these migrations or these upgrades all the time. Uh so you may want a partner like CIQ to come in and help you because we've seen it before and we know what, you know, what gremlins are out there and where the pitfalls are and we can save you a lot of uh time and pain in making that transition. Uh that doesn't mean you can't do it yourself. Um and we can't, you know, we can't adapt your own proprietary code for you. But we can definitely help you identify where the root cause is, whether it's in the code, whether it's in a package, or um hopefully save you a lot of pain from kind of on a going on a goose chase to uh figure out where this migration is going to break down.

But the number one thing is when you hit that run book to make that migration, you want to have, you know, the highest possible confidence that you've already s- looked around corners and found what is going to break and it's very well tested in a staging uh version. >> Got it. And then conversely, uh if I've got a large amount of nodes that are fairly similar, uh does that process look different? Or do I just need to test it once on a single dev box? Or is there some problems that you see at scale that need to be addressed when you're doing that? >> That is a great data cuz if you have a you look at a tool like a Cinder Pro, the whole caveat of a Cinder Pro is that you anybody can get one machine up and running.

What happens when you have hundreds or thousands? Um drift is inevitable. And a tool like a Cinder Pro ensures that when you're pushing upgrades across thousands of nodes, it identifies when a drift happens that causes one or two or nodes to go drift away from your standard, what you've set it, what this should be happening, and identify when failures happen on given nodes that shouldn't happen. And helps you identify those and focus your attention on remediating those outliers, um whether it's automation that you have or whether you have to SSH in yourself. But that's just evidential of there's no perfect scenario. There's going to be nuances about one of your in nodes um for who knows what reason.

It just maybe there's some uh some timing issues, maybe there's some latency, and the a step in the deployment gets skipped and suddenly it gets bungled. Um so it's inevitable that that happens. Uh but you just can't be you have to prepare for that to happen, and tools like a Cinder help you focus your effort on those hot spots that need attention rather than kind of hit and script and stare at the terminal and hoping everything goes well. Um and then when something does catch on fire, you've got to go dig through logs to figure out, "Okay, which of these 2,000 nodes is the one that didn't like the upgrade?" >> Got it.

Um more complicated than I had kind of anticipated going into, but I mean this is why we have you as is you like to kind of think of like all the permutations that you need to do. Brady, if people uh watching this need to take away a single thing from today, what would you uh advise them as that single takeaway from us? >> Um I'd say the first thing um is kind of just understanding the difference between an upgrade, an update, and a migration. And a migration being a shift of distribution. What is the distribution I'm running on this node? Is it Rocky Linux and I'm moving, you know, to RHEL 8 Pro?

That's a migration. No matter how small or how easy, it's still a transition and there's going to be nuances. Uh the next is updates of individual packages versus upgrades. And if you want to throw up the slide real quick, uh maybe an easy way to frame this up is uh I like to talk about it in terms of cars cuz everybody knows about cars. Um when you're talking about a major version, um a migration would be like switching models. Like I'm moving from uh a Toyota Tacoma and I'm switching over to a Bronco. These are uh migrations to a different model or different distribution. And then within that model, like the Bronco, there's major versions and there's minor versions.

Major versions are like generations. Every generation, it's still a Bronco, but there's some pretty significant feature updates. It might have changes to the engine or the interior or the design, the look. Um but even within those generational shifts, there's minor versions. Every year, there's refinement. They'd make uh remove like bugs or issues in the engine or clears up any uh you know, known problems. Uh you never want to, you know, most people don't want to buy first year of a new generation cuz there's kinks to work out. Um unless you just really love it. But if you wait a couple years into the later years of that new vehicle generation, you're going to have a lot of those kinks worked out and you're going to feel a lot more comfortable that that's going to be a nice stable car.

And that's the trade-off between do I want the the thing that's most stable uh on like an an um older or later minor version of a major version, like 8.10, very stable, but it's not getting all those new features that that new Bronco generation has. I want all the fancy Bluetooth and the the new hotness and the cool new look. So, I'm going to jump ahead to version 10, but there are going to be kinks that the manufacturer hasn't yet worked out in exchange for all of those new features. Uh and so, that's I think that conceptual model is really easy to kind of compare when I'm doing an upgrade, what am I really making that trade-off on?

>> Thanks, Brady. Um like this was super helpful on this. For everyone who's watching, whether you're the one who's writing the plan on what to do or you're the one building and running the scripts, uh before your next Linux plot project gets a date, how you define what you're doing sets that scope, as Brady was saying. It sets the budget, probably, and expectations of how things are going to go. To get it right up front, and then the rest of the plan uh uh is important, uh and then everything else has something to stand on. If you'd like some help sizing your project or you own uh that you own or you're tackling your next Linux move, we'd love to schedule a discovery call with you and kick around some options that you may have.

Uh you can do this by visiting us at our website at ciq.com. We would love to to hear from you. As always, uh thank you for watching, and we'll see you on the next one.