Kenon Ewing

SMTP

ITS has been looking into replacing the hardware supporting our mail delivery system for the University. Currently their are 5 inbound servers, two outbound servers, and two auto-responder servers running on physical hardware. ITS will consolidate these physical boxes into virtual machines as a result of this replacement effort. ITS successfully completed a POC and is now looking into a small testing initiative to continue this effort.

Exchange Consolidation

ITS is consolidating the VUMC and the University exchange environments in an effort to reduce overhead eliminate unnecessary redundancies. The effort will involve deploying a new exchange 2007 environment with an email archive offering, and systematically migrate users from University and VUMC to it. This month ITS successfully provided an architecture for storage solution as well as the email archive platform from concept to vendor, and has begun initiating purchasing processes for both.

BlueARC Upgrade

This month ITS expanded the capacity of the HPC NAS environment by 35TB of RAW capacity. This enables ITS to take on additional customers in need of HPC NAS space for research endeavors. However, in order to facilitate the new request, ITS has had to perform a number of maintenance task involving moving file systems around to free contiguous space on storage frames. ITS has nearly completed all maintenance tasks needed and will be complete in the coming week or two.

Virtual Desktop

ITS has been evaluating the plausibly and functionality of a Virtual Desktop Offering to the community to assist in lowering TCO of desktop computing resourced. This month ITS successfully created a POC environment using OS and application streaming to alleviate unnecessary computing and storage resources from being oversubscribed, but underutilized.

BlueArc upgrade
ITS is planning an upgrade of the bluearc high perfomance NAS environemtn. ITS will be expanding the capabilities from 6oTB to 750TB usable. ITS is also architecting a backup soluetion for the new 750TB environment. This will encompass 4 bluearc contollers and 2 i2k with 18 drives each. This provides enough throughput to perform full backups once a month and incrementals once a week.

SMTP Replacement
ITS has been working to replace the SMTP infrastucture from physical Sun servers to virtual Linux servers. ITS has completed the development work required for this effort, and is now looking into the deployment process. This will allow time to re-architect our email solution without running into server issues due to hardware problems.

Dedicated Backup Networks – This month, ITS has spent efforts adding network addresses to servers for increasing backup performance. ITS has implemented a tiered network configuration where co-located servers can backup over one dedicated network, and ITS servers can be backed up over another. This maintains a high security model, by not exposing ITS servers to co-servers without a firewall with the exception of backup purposes. We have seen significant reliability and performance increases from this effort.

SMTP Life Cycle Replacement – ITS has been researching replacing the SMTP environment as they are due for life cycle replacement. ITS is going to replace the physical solaris systems with virtual redhat systems. Substantial progress was made this month in this space. We successfully get messages to send and recieve from the new SMTP servers. We are trying to finalize the configurations for auto-responder. We anticipate being able to test the configurations some time next week.

Restores – ITS has been responsible for a substantial amount of data restores this month. We were able to provide the backups as for back as we allow data to be retained within SLA time frames..

Bluearc
ITS performed life cycle replacement upgrade from a TITAN head 1 to the newer generation of TITAN head. We have had to perform a number of benchmark test to evaluate bottle necks for the configuration. We initially installed a 10Gb capable TITAN head. However we noticed significant performance issues in this configuration due to the the slower connection speed of clients being 1Gb and 100Mb. This configuration caused multiple packets to get dropped and retransmitted. This increased the utilization of the network to the point were it was detrimental to service usage. Once identified, we implemented a new configuration involving 3 2Gb uplinks to the Bluearc on the 3 separate networks (ACCRE, Campus, MC). This configuration produced much better performance, and we have settled on this configuration currently. ITS does plan on re-evaluating a 10Gb infrastructure for the BlueArc.

VDI
ITS has been doing diligent research on a feasible and scalable virtual infrastructure to support desktop environments for users. ITS is currently researching the capabilities of VMware View. VMware View is capable of a seamless environment complete with OS and application streaming. This two aspects are particularly necessary because they are cost cutting measures by reducing the amount of disk space needed as well as reliability and scalability measures. Isolating the OS and the application layer makes faster patch/deployment and users can share the same disk space that these pieces occupy.

SMTP
ITS is life cycling the SMTP infrastructure for mail delivery. There has been a significant amount of work done to support this effort, however, ITS has had to focus on higher priority assignments recently. We are re-focusing efforts to complete this endeavor. This is a three part process. The first part is the delivery of mail and has been completed. The second is the receipt of mail and is currently making progress. The third is the transaction procedures to maintain the SMTP configuration through automated tools.

Backup
ITS has been consistently exercising continuous improvement in this service area. Currently we have tested a dedicated backup network configuration, and the results improved performance significantly. ITS has dediced to implement this full scale throughout our backup environments. ITS internal servers have been using this mechanism for a few months, and now we are putting in a isolated backup network to support co-located servers as well. ITS is confident that servers backing up over this network will experience much stabler and more reliable connection to the service.

Celerra Checkpoints

At the request of a user, ITS was asked to research a solution for user enabled restores from the EMC Celerra that performs our NAS operations. ITS identified Celerra Checkpoint functionality to serve this purpose. Celerra Checkpoints tracks the block changes for a share and stores them in a additional storage pool allocated solely for this purpose. There is a module for Windows (installed by defualt on Windows Vista) that allows user to right click and restore any file that they have permissions to see. The inital user that made this request no longer has a need, but ITS has identified a solution in case a similar need should arise in the future.

Metric Expansion

The ITS storage team spent extensive amount of time enhancing our the data we collect for storage metric reporting. We were asked to track not only the raw storage available, but to also report on usable storage. In order to provide this data we have to look at the raw storage without overhead, which means elimating the data that corresponds to hot spares. The storage has automated the gathering of this information.

Boinc

Vanderbilt has a team configured for the Boinc project for climate predictions. ITS has vowed to take on a large role of this team, thus we have asked ITS employees to install Boinc, and compute data for the climate prediction project.

VCMS Script

Due to the complexitiy of secure access to our virtual infrastucture, ITS has removed most of the manual configuration left up to users to access the virtual infrastucture. ITS has provided a Visual Basic sript that automatically configures putty with tunnel access to our Windows host used for managing Virtual Machines via our SSH bastion hosts.

Virtual Desktop

ITS has been attempting to get a feasible solution for managing desktops. ITS has deemed a feasible solution to use one vitual machine image delivered to multiple users (one-to-many). This signficantly decreases the need for storage for all virtual machines used in the virtual desktop environment, since there is only one virtual machine image that is shared between multiple desktops. ITS has been evaluating Citrix Provisioning Server to serve this need. This month ITS identified a issue with our current pilot plan with Citrix provisioning server that has to do with the fact that when a disk is removed from a virtual machine in VMware ESX 3.5u3 it also removes the controller for this disk. Thus when the virtual machine image is deployed to the virtual desktop and the OS begins to boot, it looks for the disk controller, and does not find it, thus it blue screens and does not boot. ITS is evaluating a workaround by leaving a 1MB disk associated with each VM which should leave the disk controller present, thus we should not blue screen on boot.

Backup Test Colo

Two of our large backup customers asked ITS to research a more viable solution for backing up clients with large amounts of data. One point of contention we wanted to remove from a potential bottleneck were the two firewall traversals that occur for network communication from these two large clients to the backup server. ITS decided to use a isolated network connected to the two clients and the backup server to see if it would decrease the amount of time needed to perform a full backup of these two clients. The results are extremely promising. ITS is researching the best way to implement this architecture into the backup environment.

VCMS Storage Consolidation

In an effort to use storage more efficiently, the storage and virtual teams have been working closely to consolidate storage presented to the ESX environment into fewer, but larger datastores accessible by all machines in a given datacenter. CUrrently, we have two storage presentations to different clusters inside of VMware. One cluster is over utilized and the other is under utilized. So rather than present new space to the over utilized environment, ITS has decided to merge these two storage presentations. This way any server could access the storage, and remove the need for different storage presentations.

VTL
THe storage team has been evaluating the EMC EDL Virtual Tape Library solution for a little over a month. It has seemed to fill the void ITS has been expericencing in the backup environment for the past few years. The EDL will give us growth capacity for one year, which allows us to begin to take on new backup clients. The EDL also speeds up read operations, so we can do multiple cloning sessions much faster than previous technologies would allow. We also increased the throughput with the EDL because we went from 12 target devices to 96 target devices which allows ITS to do more work in the same amount of time.

Backup Reporting
ITS is developing tools to allow us to get better information on the data we are backing up. For example, there are numerous backup clients with databases. The databases running cannot be backup up live without networker modules. Thus clinets without the networker module do not need to have thier hot databases backed up as theses will be corrupt upon restore. This application provides this granular visibaility, as well as additional data such as largest backup client, largest files, and the prevalent extension types.

Virtual Desktop
ITS has made big strides of improvement in our Virtual Desktop project. We have successfully configured pools to allow the automatic creation of windows XP VM’s and joining them to the domain. THe team is now looking into the phase 2 and phase 3 implementations of this project which will starts to reveals the cost savings capabilities of a service such as this. We are looking into a “one-to-many” configuration which means that we are only using one actual virtual machine stored on disk, that no user accesses (this is our golden image we use to create the VM’s for virtual desktop). THe next steps to achieve this goal is to create 8K snapshots of the golden image, therefore every copy of the golden image only uses 8K of additional disk space for overhead. Since these are snapshots and will get destroyed with some frequency, we do not encourage users to store data on the virtual machine. Instead we are looking into “roaming profiles” at an OU level, that would allow user settings to move with the user, but limited to the current organizational unit in active directory. Once we get this configuration we will look into phase 3 which is on demand application streaming. This allows us to install / manage software independently of the virtual machine itself. When users need an application on the application streaming server, it will be delivered to the VM on demand.

VTL
We have been evaluating the EMC EDL Virtual Tape Library Solution for two weeks now. It has been performing well, and has increased our ability to clone savesets as reads from disk are significantly faster than reads from tape. We did have some configurations difficulties as our Legato Networker client and media pool configurations would not work for restores as clients believed that they could restore from any storage node rather than only being able to restore from the storage node that backed up that client. SO essentially configured our clients to ensure that they would backup to a specific storage node and that storage node would be the only location to try to restore from.

We did have a significant fail over test due to a GBIC that went bad on our Cisco 6509 SAN switch. This caused an engine to fail within the VTL. The failover was not handled well at all, and after EMC investigated realized that our EDL configuration was incorrect for fail over. We have since corrected the mis-configuration and are set to test fail over again.

Training
I had a training session with EMC for Clariion management. We learned about provisioning raid group and luns in the Navisphere management GUI. I am now complete with 80% of the mandatory training needed for my new position within ITS. I am quickly getting ramped up in this role and now I am very confident in my capabilities within this role. I have Navisphere Analyzer performance class, NAS operations with the Celerra class, and finally SNIA to finish off the required training.

VDT
I have been assisting with the Virtual Desktop project which has made significant progress in the last two weeks. We have successfully configured our VDI implementation to auto provision VM’s so the administrators will not have to perform multiple manual tasks to entitle a new user for a Virtual Desktop. The process has been isolated down to adding users to an group in Active Directory. We also have successfully tested a “direct connect” configuration where clients use the Microsoft’s Remote Desktop Protocol to connect directly to the VM as opposed to using a SSL tunnel over HTTPS. However, SSL tunneling looks to be a supoerior solution from a security and performance stand point. I also did some research and got information about a one-to-many implementation which could be very beneficial to the University.

Dashboard Scripting
We have been working on automating our monthly dashboard report to minimize the amount of time spent gathering data for reports. This will allow the data to be programatically gathered, and we can spend more time on the presentation of the data. It also makes it easier for us to expand our reporting capabilities in the future.

VTL
This month the storage team has been working with EMC on deploying a disk back end evaulation Virtual Tape Library solution. We worked with EMC for three days to get the initial installation complete. EMC is on site Tuesday and Wednesday next week to finish the configuration. Once this is complete we will begin a one month evaluation to ensure that this product will perform as needed in out environment. This will entail reconfiguring the backup server running networker to use the new VTL that we will present from the EMC evaluation unit. We are looking at creating four Virtual Tape Libraries (one for each Legato Networker storage node) with 12 virtual tape drives. This should substantially increase our throughput for backup clients, which in turn should give us the capability to do more work in less time.

Backup Activity Reporting
I have completed phase I of a custom application that will report statistics about what files we are backing up. This visibility will provide the capability to begin taking steps to managing the data that we backup, which in turn can effect the amount of data we have stored for Vanderbilt University. For instance, there are some servers we have already seen that have extremely large files like database backups and keep rotatations of these backups for weeks on end. If we could identify a smaller number of these backups (say one week as opposed to four) we could decrease the time needed to complete our daily backups, as well as ensure we have capacity for upcoming projects, as opposed to having to purchase more storage.

Library Replacement
ITS has begun looking into replacing our current backup eL700 robot library. We are currently comparing the different vendor products to see which one will suite ITS needs best. This is a very important decision as the library is how we clone all of our full savesets for offsite disaster recovery copies of production data. ITS is investigating the different achievable throughputs to assist us in making the correct decision, including growth considerations.

Backup Environment
ITS has purchased two new servers to license them as additional Legato storage nodes. Theses additional storage nodes and licenses will enable us to push more backup clients simultanesly as we will increase our server parallelism value from 64 simultaneous backup streams of savesets to 128 streams. This allows ITS to consider such implementations as allowing more clients to start at the same time, which should decrease our complete datacenter backup time to completion.

Graduation
In order to ensure the success of the graduation streaming media initiative, ITS identified network congestion issues associated with live streaming. Once we identified these, we took the necessary steps to corrent these congestion locations. The first step was creating a new isolated network in front of our main firewall, but inside the perimeter firewall. We then moved the primary network connection to the new network. We also created a new isolated network to facilitate the streaming mount points for archived media, and added this network to the Celerra. The access list for IP address that are allowed to connect to the Celerra streaming media share was modified to grant the new IP address on the new isolated network to see the archived media files.

IMSP Tools
In order to facilitate the retirement of the IMSP server that stores user options and address books for Mulberry users, ITS was asked to provide a web enabled tool that users to authenticate to and export their Mulberry address book that is stored on the IMSP server. ITS has to first establish a file transfer mechanism (SSH), and installed the software for the transfer mechanism. ITS then developed a tool that transfers an authenticated users configuration directory, which includes the address book, and parses for valid email addresses. Once the parse is complete, it then either does direct MySQL insert statements and loads a users VUwebmail address book, or it generates two files, one that a user can import directly into Microsoft Outlook. The decision on what output is generated is based on user selection after authentication.

EMC RM
ITS upgraded the flare code on our Clariion SP’s. However, even after being assured by EMC support that this would not break our RM/SE application for our exchange disaster recovery scenario, it actually did break this application. EMC support informed us that we needed to updat our navisphere agent and our navisphere command line interface application that runs on the exchange server and works in tandum with RM/SE. However, upon attempting this installation it actually crashed our exchange back end server. We then called EMC back, and they noticed we needed to apply to non-standard patches. These patches got the application to install, however we still could not create a disaster recovery copy of our exchange environment. Opened another call with EMC, and they informed us that the flare code version we upgraded to on our Clariion is not supported with EMC’s RM/SE, only RM. So we adjusted our priorites to upgrading to RM. EMC sent two individuals and one RM specialists to assist with our upgrade. Upon completion we got a successful copy, but the next day it was broken again. It has consistenly breaking for a week straight with the same error. EMC has been on site for five days straight, and we will be here again today investigating root causes, and attempting to provide fixes.

SUN L700
The SUN Storagetek L700 has been having a whirlwind of phyiscal device failures. First the gripper on the robot failed, and SUN came out and replaced it. Ths L700 ran without flaw for about 4 hours before it was offline again. Opened another case with SUN, and they came out and replaced the MPC board for the unit. Once the new MPC board was in place, we had to replicate the configuration for the L700 on the new MPC board. Once we had all configurations in place, the L700 returned to normal operations. Later in the month a tape drive failed, and SUN replaced that as well.
It seems as if the stability our L700 is minimal at best. This seems to be caused by normal usage over the lifetime of this device. It may be time to look into life cycle replacement for this unit.

IMSP Retirement
ITS has agreed to develop tools to assist in the retirement of the IMSP service. This tool is an on demand address book export. It provides the capability for a user to authenticate and choose whether to export their email contacts out of IMSP into either the webmail database or a TAB Delimited file ready for import to exchange. This tool is ~ 70% complete. It currently has the ability to authenticate a user and get the indivudal email contacts. There are two major task left to complete this effort. 1) Automated file copy – the LINUX team is getting me an SSH server built on the IMSP server to assist in an automated file copy process to get the users flat text file address book from IMSP to the web server running the tool. 2) further code modifications to the tool that allow the export of distribution lists as well as individual contacts.