We are just finishing up a late night change for the ELDAP servers. The memory in the server is being increased from 2GB to 3GB at Sun's recommendation. It went relatively smooth except for the command line config tool for the DPS. Apparently it did not behave exactly like in the dev environment, and Lee had to use the GUI. Otherwise, it was uneventful.
Wow. Now that's a subject line. Well, I'm starting to bring them online. I've got the bastion host moved over for each site, and I doing the final tweaks like IP addresses, firewall settings, and such to make sure that all of the necessary connectivity is there.
The new Nagios server is sending email notifications! The email flood has been released.
I've been spending most of today working through the various outstanding issues that we have for the Nagios server migration. Most of them are related to missing service check types in the new configuration. Where it's not possible to use the default check, I'm porting over the custom check over to the new server and debugging it. I think I'm going to get rid of the MySQL port check since it generates connection errors on the server side, and causes the client to become blocked. I've also temporarily disabled notifications while all of this is going on.
The new Nagios service is intermittently running as new hosts and services are added to the configuration. At this time, sendmail is shutdown to prevent the alerts from going out. It's basically trial and error to make sure that the checks in the current environment are carried over. It's tedious work without a build document.
I've been working lately on rebuilding our Nagios installation on a RHE5 server. The new server is going to take advantage of Fruity to aid with the configuration. While not entirely different, there are some which caused me to slow my pace a bit. The first was the introduction of yum rather than up2date to manage RPMs. After a little bit of tinkering to find the yum commands that where equivalent to the up2date syntax that I was used to, I was back on my merry way. The hurdle that I'm working on is the SELinux environment. The SELinux policies are much more stringent than in RHEL4. I've never previously run into a situation where SELinux prevented root from deleting or modifying files. I could go the "off-fix-on" route, but I figured that I'd better take the time to get this mastered since RHEL5 systems are going to be a reality in our environment. The source files for the policies are not installed, but now tools exist to create modular policies, which is a nice addition. In addition, I'm taking the time to layer on all of the miscellaneous security tools that I built into our production RHEL4 web environment.
Web-srv2 is officially blank, and I've submitted the request to have it removed from the data center. That's one more Ultra2 that has been taken care of.
I booted web-srv2 from the install CD to wipe the drives, and one of the drives does not power up. Luckily this happened after the service was transferred to the new servers. Otherwise, I'd be rebuilding this server from spare parts.
I just submitted the request to remove web-srv1 from the data center. Web-srv1 was the work horse that handled the majority of the requests for the shared web environment. The Sun 220R handled roughly 2 million requests per day for the main Vanderbilt website. The Apache configuration file was approximately 100 pages when printed. Here's a picture for those that are curious.
Today we say goodbye our old friend Helios. For those who do know know Helios, this was the machine where web developers in the Vanderbilt community uploaded their content. When I first started in ITS, Helios ran on a Sun SparcStation 10, a piece of hardware that was introduced in 1992. It ran that way until it had a catastrophic hardware failure and would not boot. I rebuilt Helios on a slightly new SparcStation 20 (introduced by Sun in 1994). It also got a serious upgrade that day- a 150 MHz processor and 256 MB memory. Helios was an integral part of the web environment until June 3rd when the cutover to the new Linux servers occurred. Today I wiped the 1 GB and 4 GB drives, and submitted the request to remove the hardware from the Hill data center.
Helios will make a brief stop in storage before continuing its path to wherever surplus equipment goes. It won't be alone for long though. Web-srv1 and Web-srv2 will soon be joining Helios in storage.