CSM
This month in the CSM space we tried to identify changes that could be done without interruption that would be good candidates for pre-approved standard changes. We have produced a list of 6 items to be discussed during the next change management meeting.
We also have been having issues backing up servers that are in the CSM server vlan. The reason being hostname issues mostly. This is because the servers have a hostname, but they also have to respond to the DNS name clients are using to reference the load balanced configuration. To fix this issue we had to add aliases in the backup environment for all ip addresses on the server, as well as the hostname and DNS name.
We have been working to load balance some new services such as Sharepoint and the mail delivery environment. These are currently being moved and tested behind the CSM for load balancing and fail over
Backup Server
We began the work on deploying the new backup server. First we had to load an OS that is in line with the organizational direction of OS deployment which is RedHat Enterprise Linux. The backup administrators have requested a different network configuration than we have implemented in ITS before. It was requested to provide a 3GB up/down connection as well as the capabilities for the 3gb up/down to be available simultaneously on different networks. I worked with RedHat to get the most optimal configuration to support this requirement. We are using a network protocol called IEEE 802.3ad (LACP, or dynamic link aggregation) which gives us the capabilities of bonding Ethernet ports together to gain a bigger pipe. The pipe is limited by the size of the pipe to the switch it is connected to. To achieve the goal of multiple network presence, we configured the server to use VLAN tagging. With these two configurations together, it provides the desired throughput to the backup administrators.
To retrofit the current environment to simulate this new networking configuration, as well as to avoid an upcoming outage for a rack, and not take an outage in the backup environment, we had to move the backup storage node from the high density. We also took the liberty to set the backup storage node networking configuration to VLAN tag with IEEE 802.3ad to provide the same capabilities to the storage node that the new backup server will have.
Virtual Infrastructure
We have begun to analyze our virtual infrastructure, to evaluate different ways to perform capacity management. We have identified a method to provide real-time statistics on the capacities in the virtual infrastructure by performing some database queries to the Virtual Center Management database. The technicalities of how it is going to work still needs to be done.
Since the move to the administration network for managing these servers, we had lost the capabilities to backup the virtual center management server because of the fact that 1) the administration network is non-routeable 2) the backup server does not currently have network presence on the administration network and 3) windows machines have to have a static route manually created for machines on multiple networks. We added the static route to the Virtual Center management station, and backups are back functional.
In an effort of continuous improvement. We deployed HP management agents on all the HP hardware in our virtual environment. This allows us to see real-time system health, and be pre-emptive in finding resolutions for these types of problems without outages.
We have started to identify ways and processes to streamline request to the virtual administrators. As a part of this effort we created a distribution list and a Sharepoint site dedicated for all things Virtual internal to ITS.
Infrastructure Development had a need for a virtual machine to be the backup AD cert server. So we provisioned the requested resources from our virtual environment to fulfill this request.
Streaming Media Services
We have been working on replacing the current streaming media environment, for life cycle reasons. However, we took this opportunity to make very big enhancements to the environment. We deployed a redundant server architecture for managing load, as well as fail over capabilities. The server OS changed from windows to Linux, which will provide at least a %300 performance increase. We migrated the archive media structure from locally attached storage, to network attached storage, which make it substantially easier to share this data to different streaming media servers, that we have deployed and plan to deploy in the future. We also deployed them on virtual machines, which will help tremendously in disaster recovery, and scalability.
We also made changes to the flash server. The flash server took advantage of the network attached storage by mounting it as a samba / cifs share. However, we noticed there are problems with this form of mounting a network attached storage system on a Linux server. So we made modifications to the share to allow NFS access for the Linux servers, and changed the mount type from samba to NFS on the flash server.
Operations
We were tasked with patching our servers before the students got back to campus. I scheduled and updated our RedHat enterprise Linux servers.
It was requested from the public affairs office to synchronized the theme of the new www.vanderbilt.edu web page and the webmail webpages. Made the corresponding changes and pushed them into production.
Amber lights on servers
We have had three amber lights on servers, all ESX servers. Two of them appear to be false positives as when we look at the hardware problems from the management station, there is no hardware problems, and the amber light turned green. The last one is am IBM server, that we have talked with IBM support about, and they recommend upgrading the BMC (Baseboard Management Controller). This is scheduled for this Sunday during the maintenance window.