Exchange Support: The vast majority of my time this month has been spent work with the ECS team to support the new Exchange environment. The consolidation process is very complex, and it requires coordination from many people within the Vanderbilt community. The Exchange 2007 environment has been my re-introduction into the world of Microsoft Windows. I have been involved recently in monitoring server performance.

Enterprise Linux Reference Platform: We are continuing to meet and define the base RHEL standard.

Exchange 2007 Support: I have been tasked with several items supporting the Exchange 2007 environment.

Sun Identity Management [SIDM]: The new SIDM LDAP services are live, available, and in-use by the general Vanderbilt community. The team is in the process ensuring that all clients are properly migrated to the new service.  Only a few clients are still connecting to the Solaris-based service, and we are working with the owners to move them over.

Enterprise Linux Reference Platform [ELRP]: The team has produced a server build with minimal packages to use as the base system.  We are in the process of determining the proper configuration for the installed packaging.

PHP5/MySQL5 Web Server Migrations: The project to upgrade all of the shared web services to RHEL5 is essentially complete. All of the content has been migrated, and the sites are running smoothly. This migration gives the web community access to PHP5 and MySQL5.  It also brings along a much more robust ModSecurity ruleset.  So far the vast majority of the help desk tickets have been related to applications with the new ModSecurity rules.

[WIN] ITS Website Redesign: The cutover to the new ITS website is scheduled for this weekend. The change will involve exporting the test database, cleaning up all of the links, and importing the data on the production server. I’ll also need to copy the necessary files over to the production server.  The last step will involve moving the IP addresses from the old servers over to the new servers. The CSM will pass the incoming connections to the new servers.

PHP5/MySQL5 Upgrades: The cutover deadline for the cut over of the first round of server has been extended until mid-June. This round of migration is progressing slowly. There are 58 folders left on the WWW4 servers 29 virtual hosts left on the vanity domain servers. This remaining content is under review by the site owners and awaiting migration approval.

ITS Website Redesign: The final cutovef date has been scheduled.

Sun Identity Management: All of the LDAP virtual machines have been built and are accessible via our bastion hosts. The networking issues with the F5 LTMs have been resolved, and this appears to have revealed a possible issue with the ESX servers behind them. I have opened a ticket with the VUH helpdesk, and I am working with their staff to resolve the issue.

Security Investigations: I spent several days working with the Network Security team to identify and mitigate a security issue.

Sun Identity Management: All of the ELDAP and CLDAP VMs are now online.  I am working through verifying connectivity to each VM and service point.  I have also been alerted to some latency issues with the LTM, and I am in the process to debugging this.  I have reconfigured the secondary LTM in a slightly different configuration for comparison testing.  The best working configuration will a synced over the the other peer.

ITS Website Redesign: This project is on track. All of the content is being reviewed for accuracy and relevance.  Some minor presentation issues are also being addressed.

PHP5/MySQL5 Upgrade: All of the customers for this migration have been identified. The specifications for each servers are also being collected so that each service pair is appropriately sized.

Exchange 2007: I am working with the primary project resources to ensure that the new Exchange 2007 services are available via the LTMs.  This configuration is slightly different from our preferred architecture.  The new Exchange servers are not directly behind the LTM, and the connections are being proxied over to the servers. This implies some additional configuration to get the connection to go through.  The only deteriment to this configuration is that the source client IP address is lost due to the source NATing that takes place.

Shibboleth Rebuild: I am working with the primary project resources to ensure that the new Shibboleth servers are available via the LTMs.  I have recommended that the recently built test Shibboleth server be moved behind the test LTMs to ensure that testing procedures are valid.

Unix Team Cross Training: The Unix team has expanded our weekly one hour meeting from our typical status updates and info distribution. We are now attempting to work in cross training, technology demos, and team project support. During our last meeting, we covered the basic UltraSeek admin tasks, OSSEC capabilities and agent install, DiamondIP zone creation, and patch administration. This is an effort to eliminate operational bottlenecks that develop when the primary admin is the defacto expert for a particular technology.

Sun Identity Management: Both of the ESX servers are fully operational in the VUH data center, and all of the virtual machines have been configured for the new network location. Both of the F5 LTMs are online; however, I only have limited network connectivity to the VIPs that I have created. I’m working with the VUH staff to diagnose the connectivity issues.

ITS Website Redesign: All of the virtual hosts for the miscellaneous ITS websites (except the main ITS website) have been moved to other servers in preparation for the final jump to the Drupal-based site.  The last major step is the content reviews.

Web Service Resource Allocations: I increased the disk space allocation for the WWW and WWW4 web server pairs. Content growth has been fairly steady on both services, and the Nagios service checks were starting to alert at warning level. The MySQL server VM also received more memory and disk space.

MySQL InnoDB Reconfiguration: The existing InnoDB tables on the shared MySQL server were stored in a single file, which made it difficult to determine which customers were consuming an inordinate amount of resources. It took about two hours to re-import all of the databases using InnoDB tables. Some of them had to be imported twice due to corruption issues from an unknown source.

Cohosted Server Patching: The Greeklife, Honor Council, and VICC websites were upgraded to current RHEL standards. The vmware-tools were also updated.

Exchange 2007 Deployment: I have done miscellaneous tasks in support of this project such as configuring the service point VIPs on the F5 LTM and debugging connectivity issues. The CAS VIPs also has a pair of minor iRules to forward connections.

Sun Identity Management: I am continuing my work to bring the VMs online  in the VUH data center, but I’m being diverted by operational events and tasks. Both ESX servers are fully online and accessible after the latest network revision. We are in the process of implementing slamd to load test the new service.  The test MIS Business Objects servers are now connecting to the new service.

ITS Website Redesign: All of the ITS-manage virtual hosts have been migrated from the old servers to the new servers, except for the main ITS website and the change management website. The Drupal-based ITS website is still undergoing content revisions. The change website will be migrated in the near future. Cutover to the Drupal-based site is pending completion of the content revision.

Nagios Data Merge: The data from the old Nagios server has been merged into the archives on the new Nagios server. This process involved translating server and service check names between the two servers. All of the monitoring data is now on the new server.

Sitemason Enhancements: We have put a lot of effort into identifying performance issues with the Sitemason service and implementing potential fixes. The database server has been reconfigured to handle an increased connection volume. The too many clients errors were somewhat sporatic and do not always coincide with the MyVU traffic. We have built and are currently testing a clustered configuration.

ITS Server Migration: Two new RHEL5 virtual servers have been deployed, and the first of the ITS websites has been moved over. swdist.vanderbilt.edu has been transitioned over. More websites will be transitioned over in the near future.

Tertiary Web Presence: A business continuity website has been established for the main Vanderbilt website by utilizing services from Rackspace. This was was given to the team (aloing with business continuity DNS) as a special work effort to complete the implementation.

Backup Client Upgrades: The Unix team has completed upgrading the Legato client on our servers. This enables for active management of the backup process for the storage team.

Sun Identity Management: We are using a new web application to track our project tasks. This has been a great help, since I now have defined goals to meet. The only drawback is that there is a bug in the application which prevents me from updating the status of my task, and I don’t really like this since it displays my stuff as Not Started, so I send frequent email updates. I have moved the two CLDAP virtual machines that are slated for production behind the LTMs, and tested basic network connectivity. I’m in the process of doing the same for the ELDAP virtual machines.

NetTracker Data Archive: The NetTracker data was archived to the rolling 13 month window.

Backup Client Upgrades: I have begun upgrading the Legato client on my systems.

ITS Website Redesign: I have gone through two rounds of vulnerability remediation with the Network Security team. Many of the issues are false positives that are induced because of our security configuration. There were a couple of legitimate issues such a plain text authentication. This was something that our vendor was supposed to fix; however, I created some RewriteRules to remediate.

Website Maintenance: I had to add more storage to the WWW4 server. This was a really easy change to add another virtual disk to the VMs and grow the file system. I have received multiple helpdesk tickets to disable the ModSecurity rules for several websites. Rather than disable the rules, which would have left them vulnerable to the thousands of daily hack attempts, I tested and modified several rules to relax the restrictions under certain conditions. Unfortunately, the typical web advice on most forums is to disable ModSecurity for things like WordPress and Drupal when it becomes a problem.