Sun Identity Management (IDM): The team has decided to create another classification of servers to add some additional flexibilty into the development progress. The new classification will be called preproduction, and it will combine some of the elements of the existing test and production classifications. I also removed the Directory Proxy Server (DPS) configurations from the CSM. This caused a small problem with production transaction routing, and a temporary out-of-service serverfarm was created so that the server was still accessible.
Test F5 LTM Implementation: The two LTMs that we purchased are now ready to take up their roles as test hardware. I had to modify our server naming policy to accommodate the new device types and roles. I also renamed all of our initial testing configurations to meet the new standards.
Bastion Host Patching: I patched the two AppHosting bastion hosts up to current RHEL4 standards. During the patching process, remote access was available by using the alternate host.
mysql-srv1 Decommissioning: The Sun Solaris MySQL server has been officially decommissioned. It is now out of service and waiting for the disks to be wiped.
thecard.vanderbilt.edu: I created a virtual on the SSL servers for Dining Services and change the DNS to point to the new site. This new configuration allows Dining Services to offer additional services on their main site beyond those currently provided by the vendor.
change.its.vanderbilt.edu: I created a new virtual host on the ITS web servers for the new change management website. The new virtual host involves an additional vserver and serverfarm on the CSM as well as some modifications to the existing mod_security rules.
ldap.vunetid.vanderbilt.edu: I updated the SSL certificate use by the two Solaris LDAP servers.
My Magic queue is entirely empty. Somebody submit something quick…
Sun Identity Management: I have been involved in the various architectural and development processes for this project. We are re-architecting to work around the DPS which has caused availability issues with the LDAP services.
F5 LTM Training: I attended the F5 LTM Essentials and F5 LTM Advanced training courses in Atlanta. This training was needed for our proposed load balancing solution for the Sun IDM project.
Web Application Scanning: In a coordinated effort with the Network Security tea, several hand selected websites on the main Vanderbilt web server were used as evaluation content for two web application scanners. These sites were selected because of suspicious log entries for requests to these sites. The scans involved a discovery phase, last minute database backup, an evaluation phase, and a reporting phase. In addition to evaluating the scanning productions, we were also able to pass the findings on to our customers so that they could fix their applications.
Proofpoint Upgrade: I was an unexpected participant in the Proofpoint upgrades. The upgrade script did not perform as expected, and exited before installing all of the upgraded packages. This left the servers in a state where the kernel did not match the modules. I got the servers online to a point where the vendor could resolve the issues for us.
Just ran across a new feature in Evolution- the attachment reminder. I'm guessing that the word quote is a common sign of an email attachment.
Identity Management: The new LDAP servers used in the IDM project currently suffer from some stability problems due to issues with the Directory Proxy Server (DPS). The DPS is being used to ensure that the correct SSL certificate is presented to the client. While the vendor is working to correct the problem, the IDEV and AppHosting teams are working on an alternate solution. I built two new development LDAP servers to be used by the IDEV team. These new servers more closely resemble the architecture and configuration of the production servers. We are currently testing the use of stunnel as a DPS replacement. While stunnel does not offer the same feature set as the DPS, it does provide the base SSL functionality and is considerably more light weight. We are also evaluating F5 LTM appliances in the test environment to see how well they integrate. We currently have a pair of evaulation appliances that we will use to front-end the new test LDAP servers that I built.
IMAP-to-Exchange Migration Tool: I have scripted the necessary tasks to prepare IMAP accounts for import into Exchange. The process essentially makes a duplicate of the user accounts so that the original is unmodified. I still need to create documentation to detail the process for the rest of the team.
Search Engine Replacement Project: I have attended several meetings and a couple of product demonstration. I am passing this project to Troy Osborn.
ITS Website Redesign: My involvement in this project has been relatively minimal this month while the vendor continues development efforts on the site. My activities this month included adjusting ModSecurity rules, re-importing a database for the vendor after a failed upgrade,and identifying application errors in the logs.
Operational Issues: The majority of the operational issues for this month have been relatively routine, and most of the web requests were related to creating redirects and such. I did create a new virtual host for a website that was being publicized the next day in a national magazine.
We are just finishing up a late night change for the ELDAP servers. The memory in the server is being increased from 2GB to 3GB at Sun's recommendation. It went relatively smooth except for the command line config tool for the DPS. Apparently it did not behave exactly like in the dev environment, and Lee had to use the GUI. Otherwise, it was uneventful.
Wow. Now that's a subject line. Well, I'm starting to bring them online. I've got the bastion host moved over for each site, and I doing the final tweaks like IP addresses, firewall settings, and such to make sure that all of the necessary connectivity is there.
Helix Server Hardware Migration: I moved the Helix virtual machines from the Intel resource pool over to the AMD resource pool. These two particular VMs had better performance on the AMD hardware due to a system call issue. I was also involved in changing the content NFS mounts over to a new network.
Web MySQL Patching: The BEVS01 server was patched to current RHEL4 standards. I also implemented a script to reset the client error counts at periodic intervals to alleviate issues with incorrect code and occassional vulnerability scanners.
Sun Identity Management: I am providing support for various aspects of the IDM projects, and this has consumed the majority of time for this month. I performed the CSM configuration for IDM gateway servers. This configuration is slightly different from our typical least connections load balancing method in that all connections are routed to the primary server until it becomes unavailable, and then connections are routed to the secondary. I also built the eight virtual machines for the LDAP component of the project. These were a complete rebuild from the current ELDAP01 and CLDAP01 since the file systems needed to be resized to avoid Nagios warning errrors when local database backups are being run. The VMs were resized from 10GB to 18GB. It took approximately two days to get everyone to agree on an acceptable file system layout. I also spent some time ensuring that each application on the LDAP servers could be started and stopped as a non-privileged user. As the LDAP servers were being configured by the IDEV team, I installed and configured the ESX servers at the VUH and Sungard data centers. I now have the VMs moved over to their respective servers, and I am in the process of verifying that all necessary connectivity is in place. I am still in the process of updating the server commissioning documents for these VMs.
Documentation: I spent a couple of nights documenting some procedures so that work can be shared among the team. The design specification for automating the IMAP to Exchange migrations; however, it needs to be tested and verified. I am also in the process of documenting a break fix procedure our Diamond IP management application.
CSM Support: I have also performed some intermittent CSM support for other App Hosting team members.
There are a few benefits to coming to work on Memorial Day. For starters, there is essentially no traffic on the way to work and I get my pick of the parking spots. The office is also really quiet, which means that I'll get to work undistracted for a few hours. I'm trying to finish up some of the IDM project tasks that I had started on last Friday, but I did not get to complete because of the campus power outage.
Sun Identity Management: I am making arrangements to get the hardware installed in the VUH and Sungard data centers.
Nagios Enhancement: The new Nagios server has been very stable, and we are continually adding new devices and services to be monitored.
Web Security Incidents: I spent nearly three whole days investigating, cleaning, and fixing various web applications on the primary web servers. Due to the open nature of this post, I won't divulge specifics of what occurred, but needless to say, there was an opportunity for some site owners to secure their websites. The situation was corrected with minimal impact to users and no impact to the core sites.
Cohosted Web Server Patching: The cohosted web servers were patched to current RHEL4 standards.
Streaming Servers Move: HelixA and HelixB have been moved to a new network for performance reasons. This was also used an opportunity to re-evaluate the local firewall rules.
Blog Services: There are now 55 organizational and 105 personal blogs on the blogs servers. I have started the testing to begin the upgrade of the blogs to WordPress 2.5.
PBX Pager Phase II: The Iolan TCP serial port was removed from Jprod1 and Jprod2. It will be moved to the Jtest1 server for development use.
Team Dynamics: Now that Troy has officially started for the team, I've started transitioning some of my operational work to his near-empty list. Troy is now the primary administrator for the Jprod servers, the webmail servers, the Sitemason servers, and the development Bluestem servers. We are also getting him acclimated to being on the other side of the operational tasks.
