In a bustling IT department, nestled in a company that relied heavily on its servers and databases, a significant challenge loomed on the horizon. The clock was ticking down to the end-of-life support for RHEL (Red Hat Enterprise Linux) 7.9. The company, like many others, depended on this operating system to run various critical applications, MySQL databases, and environments such as pre-production, production, and user acceptance testing (UAT). With over 250 servers at stake, the risk of non-compliance was a formidable adversary.
Enter the Cloud Service Provider (CSP), the hero of our story. They were tasked with the monumental project of upgrading all the servers to RHEL 8.8 within a strict 35-day deadline. Let’s embark on this journey of challenges, strategies, and solutions as the CSP navigates the treacherous waters of the RHEL upgrade.
Gathering the Troops
Before charging into battle, the CSP had to understand the lay of the land. They began by collecting comprehensive data about the existing landscape. They needed to know which versions of RHEL the application and database servers were running and how these servers were distributed across different environments. This information was crucial for planning the upgrade strategy.
The servers were spread across pre-production, UAT, development, and production environments. The CSP meticulously segregated the servers based on their roles and environments. Armed with this detailed map, they met with the customer to present their plan.
The Grand Strategy
The CSP’s strategy was methodical and well-structured. They proposed to tackle the upgrade in phases to minimize disruption
- Phase One: Upgrade UAT and pre-production application servers to RHEL 8.8.
- Phase Two: Upgrade production application servers.
- Phase Three: Upgrade UAT and pre-production database servers.
- Phase Four: Upgrade production database servers.
Application servers were to be upgraded in place, while database servers would undergo a parallel build upgrade. This meant building new servers with RHEL 8.8, replicating the data, and then redirecting traffic to these new servers.
Facing the Dragons: Risks and Mitigation
Every quest has its perils, and this upgrade project was no exception. The CSP identified several risks and devised mitigation strategies to handle them.
- Compatibility Issues: There was a risk that applications might not be compatible with RHEL 8.8. To mitigate this, the CSP planned to thoroughly test the upgraded UAT application servers.
- In-place Upgrade Challenges for Databases: In-place upgrades for database servers were deemed risky. Hence, a parallel build approach was chosen.
- Post-Upgrade Compatibility Problems: If any issues arose post-upgrade, a rollback strategy was ready, utilizing existing snapshots to restore services.
- Stakeholder Communication: The CSP identified key stakeholders, teams, and points of contact, ensuring clear communication and timely approvals throughout the project.
The Battle Begins: Execution Stage
After weeks of planning and strategizing, the CSP was ready to execute the upgrade. The servers were categorized into critical and non-critical, with non-critical servers being upgraded during the day and critical ones during off-hours.
First, the non-critical application servers were targeted. The team sent out emails to all relevant stakeholders, informing them of the scheduled downtime. Once the services on these servers were stopped, a snapshot backup was taken. The operating system engineers then upgraded the RHEL version from 7.9 to 8.8. Post-upgrade, the application team verified that all services were running smoothly before giving the green light.
This process was repeated for the critical application servers, but during nighttime to avoid disrupting business operations. If any application failed to function correctly after the upgrade, the team immediately rolled back to RHEL 7.9 using the snapshot.
This meticulous approach ensured that all 150+ application servers were successfully upgraded without a hitch.
The Final Frontier: Database Servers
With the application servers upgraded, the focus shifted to the most critical component: the database servers. These servers were essential for the smooth functioning of the applications, handling data retrieval, storage, and other vital activities.
The CSP decided against an in-place upgrade for the database servers. Instead, they opted for a parallel build approach, constructing new servers with the exact configuration of the existing ones, installing RHEL 8.8, and replicating the data.
The database servers were categorized based on their criticality and upgrade timing. Non-critical pre-production database servers were the first to be upgraded. Six database servers were successfully upgraded and handed over to the customer’s application team for testing.
A Sudden Twist: The Unexpected Problem
Just when it seemed everything was under control, the upgraded database servers began to crash repeatedly. The team monitored the situation for eight hours, hoping it would stabilize. When the problem persisted, they raised a high-priority ticket with the original equipment manufacturer (OEM).
The OEM analyzed the logs and recommended upgrading the underlying database OS, MySQL, to version 8.0.37. This posed a new challenge since the Digital Asset Management (DAM) tool was not compatible with MySQL 8.0.37.
Crafting the Solution: Customer Collaboration
The CSP presented the customer with two options
- Upgrade RHEL to 8.8, MySQL to 8.0.37, and deal with DAM incompatibility.
- Upgrade RHEL to 8.8, MySQL to 8.0.34 (compatible with DAM), and thoroughly test in the lower environment before full deployment.
The customer chose the second option. The CSP upgraded six pre-production servers and conducted rigorous testing over four days. The results were positive, leading to the decision to proceed with upgrading all database servers to RHEL 8.8, MySQL 8.0.34, and DAM version 10.
The Victory March
With the solution in hand, the CSP executed the upgrades smoothly, ensuring all database servers were brought up to the required specifications without any issues. The journey was arduous, but the CSP’s careful planning, thorough testing, and effective communication led to a successful upgrade project.
Lessons Learned
This grand tale of the RHEL upgrade project is a testament to the power of meticulous planning, teamwork, and adaptability. The challenges faced and the solutions crafted offer valuable lessons for any organization undertaking similar projects. The CSP’s journey from initial assessment to final execution serves as a guiding light for ensuring smooth and effective upgrades in the ever-evolving landscape of technology.
