Kenon Ewing

VTL
This month the storage team has been working with EMC on deploying a disk back end evaulation Virtual Tape Library solution. We worked with EMC for three days to get the initial installation complete. EMC is on site Tuesday and Wednesday next week to finish the configuration. Once this is complete we will begin a one month evaluation to ensure that this product will perform as needed in out environment. This will entail reconfiguring the backup server running networker to use the new VTL that we will present from the EMC evaluation unit. We are looking at creating four Virtual Tape Libraries (one for each Legato Networker storage node) with 12 virtual tape drives. This should substantially increase our throughput for backup clients, which in turn should give us the capability to do more work in less time.

Backup Activity Reporting
I have completed phase I of a custom application that will report statistics about what files we are backing up. This visibility will provide the capability to begin taking steps to managing the data that we backup, which in turn can effect the amount of data we have stored for Vanderbilt University. For instance, there are some servers we have already seen that have extremely large files like database backups and keep rotatations of these backups for weeks on end. If we could identify a smaller number of these backups (say one week as opposed to four) we could decrease the time needed to complete our daily backups, as well as ensure we have capacity for upcoming projects, as opposed to having to purchase more storage.

Library Replacement
ITS has begun looking into replacing our current backup eL700 robot library. We are currently comparing the different vendor products to see which one will suite ITS needs best. This is a very important decision as the library is how we clone all of our full savesets for offsite disaster recovery copies of production data. ITS is investigating the different achievable throughputs to assist us in making the correct decision, including growth considerations.

Backup Environment
ITS has purchased two new servers to license them as additional Legato storage nodes. Theses additional storage nodes and licenses will enable us to push more backup clients simultanesly as we will increase our server parallelism value from 64 simultaneous backup streams of savesets to 128 streams. This allows ITS to consider such implementations as allowing more clients to start at the same time, which should decrease our complete datacenter backup time to completion.

Graduation
In order to ensure the success of the graduation streaming media initiative, ITS identified network congestion issues associated with live streaming. Once we identified these, we took the necessary steps to corrent these congestion locations. The first step was creating a new isolated network in front of our main firewall, but inside the perimeter firewall. We then moved the primary network connection to the new network. We also created a new isolated network to facilitate the streaming mount points for archived media, and added this network to the Celerra. The access list for IP address that are allowed to connect to the Celerra streaming media share was modified to grant the new IP address on the new isolated network to see the archived media files.

IMSP Tools
In order to facilitate the retirement of the IMSP server that stores user options and address books for Mulberry users, ITS was asked to provide a web enabled tool that users to authenticate to and export their Mulberry address book that is stored on the IMSP server. ITS has to first establish a file transfer mechanism (SSH), and installed the software for the transfer mechanism. ITS then developed a tool that transfers an authenticated users configuration directory, which includes the address book, and parses for valid email addresses. Once the parse is complete, it then either does direct MySQL insert statements and loads a users VUwebmail address book, or it generates two files, one that a user can import directly into Microsoft Outlook. The decision on what output is generated is based on user selection after authentication.

EMC RM
ITS upgraded the flare code on our Clariion SP’s. However, even after being assured by EMC support that this would not break our RM/SE application for our exchange disaster recovery scenario, it actually did break this application. EMC support informed us that we needed to updat our navisphere agent and our navisphere command line interface application that runs on the exchange server and works in tandum with RM/SE. However, upon attempting this installation it actually crashed our exchange back end server. We then called EMC back, and they noticed we needed to apply to non-standard patches. These patches got the application to install, however we still could not create a disaster recovery copy of our exchange environment. Opened another call with EMC, and they informed us that the flare code version we upgraded to on our Clariion is not supported with EMC’s RM/SE, only RM. So we adjusted our priorites to upgrading to RM. EMC sent two individuals and one RM specialists to assist with our upgrade. Upon completion we got a successful copy, but the next day it was broken again. It has consistenly breaking for a week straight with the same error. EMC has been on site for five days straight, and we will be here again today investigating root causes, and attempting to provide fixes.

SUN L700
The SUN Storagetek L700 has been having a whirlwind of phyiscal device failures. First the gripper on the robot failed, and SUN came out and replaced it. Ths L700 ran without flaw for about 4 hours before it was offline again. Opened another case with SUN, and they came out and replaced the MPC board for the unit. Once the new MPC board was in place, we had to replicate the configuration for the L700 on the new MPC board. Once we had all configurations in place, the L700 returned to normal operations. Later in the month a tape drive failed, and SUN replaced that as well.
It seems as if the stability our L700 is minimal at best. This seems to be caused by normal usage over the lifetime of this device. It may be time to look into life cycle replacement for this unit.

IMSP Retirement
ITS has agreed to develop tools to assist in the retirement of the IMSP service. This tool is an on demand address book export. It provides the capability for a user to authenticate and choose whether to export their email contacts out of IMSP into either the webmail database or a TAB Delimited file ready for import to exchange. This tool is ~ 70% complete. It currently has the ability to authenticate a user and get the indivudal email contacts. There are two major task left to complete this effort. 1) Automated file copy - the LINUX team is getting me an SSH server built on the IMSP server to assist in an automated file copy process to get the users flat text file address book from IMSP to the web server running the tool. 2) further code modifications to the tool that allow the export of distribution lists as well as individual contacts.

Identity Management
The IDM project has scheduled for go live in March. In preparation for the go live, ITS has currently deployed two C-LDAP servers to provide directory services for Medical Center applications and an E-LDAP server to provide directory services for the general University community. ITS also has deployed IDM gateway servers for active directory provisioning events. ITS also increased the security of authentication services by ensuring encrypted network communication channels.

Storage Administration
ITS personnel changed, as I move from the system administration team to the storage administration team. This change requires knowledge transfer, which encapsulated time during this month.

Virtual Stabilization
Two new VLANS were trunked into the virtual infrastucture to support some new PBX applications that will run on virtual machines.
System Resource Reports - ITS spent time metering and gathering statistics about our virtual infrastucture depicting the host physical configuration and utilization as well as all virtual machine location and utilization.

Vacation
Personally, I was about one week this month for my first vacation. Upon return, I was fully revitalized and ready to hit the ground running.

IDM
ITS is currently looking at HA solutions for the Identity Management project. Utilizing our current CSM load balancer and possibly purchase Big IP F5 load balancers to handle the medical center, we will have a three server redundant architecture for epassword authentication. We will have a 6 server redundant architecture the HA medical center application needs. There will also be another two server load balanced in an active / passive configuration for gateway servers that translate the IDM instructions to AD transactions. This allows for real-time propagation of events inside the Identity Management service to Active Directory.
Backup
ITS updated tuning parameters to allow NFS to take advantage of the previous TCP and UDP tunings implemented about a month ago.
Virtual Environment
Snapshots – ITS has scripted an automated snapshot management solution for virtual machines. We are currently evaluating what virtual machines ITS will choose to perform weekly snapshots of specific virtual machines.
Firmware – We have had two small outages associated with outaded firmware on IBM x86 servers. The outdated firmware can cause ESX 3.0.2 to have connection issues with internal disk drives, thus when this has happened, All virtual machines continue to run, but all management to the ESX host in question is lost until reboot. Furthermore, since all the virtual machines are still powered on, we cannot move these virtual machines to other hardware because there is a lock file that disallows multiple right acess to the same virtual machines (and right so). So to fix this situation, ITS has to power off the ESX server in question to allow the release of the lock file on all the running virtual machines, which then allows the virtual machines to move hosts.
VCP – I have been studying to get my Virtual Center Professional (VCP) certification, and I am scheduled to take the test on Wed Feb 26. (wish me luck)

IDM
ITS has been working diligently on the identity management application. This month we put together the build documentation that should allow rapid deployment of these servers to provide the num identity management service. The Applications Hosting team built two pre-dployment VM’s from the build documentation that the Development team is now using to configure master to master replication. Upon completion of this task we should be able to build the production VM’s for this project.

Hardware Monitoring
We have been trying to get a handle on hardware events and their proper detection, notification, and resulotion in our environment. Since one of the most important parts of our infrastructure is the virtual environment, we started with those physical servers. We have 6 HP servers and 10 IBM servers. The HP servers have now been registered to our HP SIM monitoring server, and IBM are registered in our IBM Director, which will give us visibility to pre-empt into our hardware problems.

Sitemason
ITS released a new sitemason service this month. ITS worked with our vendor Sitemason to ensure a server that would handle the load that our current server sees. ITS deployed and tuned a VM for the front end web service and physical back end database service.

Training
ITS sent myself an one of my team members to RedHat Certified Architect Training with a focus on performance monitoring and system tuning. This entailed an in depth four day course followed by an examination for certification on the last day. I will have results within a few weeks.

Helix
ITS got the opportunity to upgrade the version of helix streaming server, which fixes a bug in the software that hinders the capability to perform two consecutive real live streaming events. ITS successfully deployed the upgrade to the helix development environment, and tested before we prepared to do the actual upgrade. This testing phase ensured a quick non-intrusive upgrade to the service capabilities.
There is a monitoring extension that is provided with the Helix streaming media application, which integrates with SNMP. However, since production we have been having problems getting the extension to function. The way it works is they provide a proxy SNMP service that connects the system SNMP configuration with the helix SNMP extensions. However, there was a port conflict with another application. ITS decided to change the other applications port number which allows Helix proxy SNMP service to function as expected.

Listserv
ITS has been moving forward with the list.vanderbilt.edu server replacement. This month we had to get the application into a testing capacity. In order to do so, we had to submit a security scan to get a baseline of the system configuration. After a few iterations of scanning and mitigating specific security concerns, we passed the security scan. Upon passing, we could then put the server on a publicly addressable address space, which puts the server into a testing capacity.

IDM
ITS identified a small need for two additional VM’s to support the Identity Management initiative. These VM’s would serve as a proxy service from IDM to different Identity applications (ie LDAP, AD, Novell.. etc). These servers essentially keep conversion tables that convert the IDM commands into the specific command for the environment being provisioned.
ITS also began discussing the monitoring model for the Identity management solution. This will consist of monitoring many service points as well as the servers that compose the cluster for the service point.
ITS also started conversations and putting together action plans for testing the load balancing capabilities of the F5 for this project. This involved ITS and MIS working together to try to get a sub configuration that ITS has access to and can manage, but MIS owns.

Amcom
ITS has been deploying the development environment for the OSIS servers. We have been doing a little coordination with the Vendor Amcom, to provide them assistance in getting all of their applications installed and configured.

CSM Test
ITS is upgrading firmware code revisions on the CSM load balancing module in our Cisco chassis. In order to ensure a non-interruptive upgrade, we had to test the firmware in our test lab. This consisted of getting two apache web servers serving similar content and coming up with various ways of breaking connections from CSM to the individual servers, and then verifying the service availability.

CSM Case
ITS is seeing issues with servers behind the CSM using a NAT configuration so that clients see the public VIP address rather than the private REAL IP address. For example, if there are two servers in a CSM serverfarm that is using NAT for the VIP address, and they want to communicate to one another over the VIP address, the connection does not make it to the other server. ITS opened a trouble ticket with Cisco about this issue and we are working towards resolution.

RHN DB
ITS finally had a successfully test using the latest version of RHN 5.0 and a tiered application database fashion. This provides new features and functionality, and will also bring stability to the RHN service ITS wants to provide.

Helix Upgrade

                ITS is preparing to upgrade the streaming media software to the latest release version, which will fix a problem that causes us to restart the service on the streaming servers.

IMSP Count

                In an effort to retire the IMSP service, ITS has been gathering statistics on how many users are attaching by day in a one month period.

VUNETID

                Have been working the the infrastructure development to replace the VUNETID system. Currently, ITS is in a phase where the new web server used to start on demand transactions, needs access to various tools that the current production environment has access to. Currently ITS is setting up a method for the new web server to check the quota for a given mailbox in the IMAP system

LISTSERV

                The new listserv application to replace majordomo has begun development and staging. This month ITS successfully got the application installed and configured, and now are awaiting the status of a baseline security scan. Once the scan comes back, we will put this on a public network, and begin testing.

LEGATO ESX

                ITS successfully architected and testing a backup / restore model that can used in the Virtual Infrastucture to backup VMDK files (essentially a ghost image). ITS is now looking at ways to automate this process.

IDM

                ITS has agreed to a proposed architecture for the Identity Mangement Solution, and are now looking for buy-in on the architecture from the customer. ITS is now aggressively working to get a functional test environment

AMCOM

                ITS has setup amcom access to development server using the applications hosting build network, and ITS applications hosting authentication standards.

Move VM’s

                This month an upgrade to the SAN infrastructure caused a quick need to move about 7 VM’s from Stevenson Center to Hill Center. ITS performed this task quickly and efficiently.

New ESX Servers

                ITS expanded the virtual resources available to two clusters this month. One went into the Intel Production Cluster, and the other went into the Co-Lo cluster.

Virtual Infrastructure
                This month in the virtual infrastructure, we enabled HA and DRS across the entire virtual environment. HA (High Availability) is a VMware technology that automatically powers on VM on available resources if the host they were running on faults. DRS (Dynamic Resource Scheduling) is a VMware technology that constantly evaluates a ESX Cluster. If monitors the hosts and VM’s in the cluster, and runs calculations to ensure that all VM’s are getting as many resources that the cluster can make available.