The DR storage array a Clariion CX4-240 was configured and brought online this month.  The storage groups have all been configured and the dedicated SAN switches have been configured as well.  All that is left is for the Exchange host to be brought online and connected to the dedicated SAN switches.

An additional expansion of 6 DAE of storage to the Array dedicated to Exchange is scheduled for a week from this coming Sunday.

E-mail Archive:

We have spent a lot of time around the SOW provided by Symantec and have ironed out the process required for our implementation.

We have installed one archive server and we have a couple of admins using it for their mail.

Projects:

VUspace replacement projects are being ramped up for replacement of the groups and user services.  The groups service will be migrated to NAS storage.

We deployed a CX4-480 storage array for the purposes of supporting Exchange 2007 and e-mail archive.  We have presented storage to Exchange backend servers and MSSQL servers.  The ESX environment will also be getting some much needed additional capacity from this expansion.

Work continues to support the new Exchange 2007 and e-mail archive environment, additionally we have launched some initial project work into looking at VUspace lifecycle replacement.

We added 66 TB of storage to our EMC EDL.  We had to upgrade the code on the array and the EDL engines in order to successfully install the new capacity.

We ordered and received two new storage arrays.  One is currently in the Hill Center and the other will be placed in an offsite location for disaster recovery purposes.  We configured and assigned the first batch of storage just this past week to the new Exchange 2007 database servers from the new CX4 at the Hill Center.

We have ordered new SAN blades which will be installed into our existing Cisco 9509’s to provide high bandwidth ports to the new Storage Arrays and performance intensive Exchange servers.  Additionally, a smaller SAN switch was purchased to go in the offsite location for DR.

We are still working on a migration path in our e-mail archive environment from the old EAS environment to the new environment.

We have been working on planning to resolve some fiber length issues in our SAN environment since the arrays and SAN switches have been moved to better facilitate data center cooling.  Along with these changes we’ll be coordinating installation of the new SAN blades sometime by early August.

The BlueArc storage we purchased a couple of months ago is now almost completely used.  We have migrated some shares around to provide more space to the users on those shares.  I have also restructured old storage pools to make better use of spindles underneath those file systems.

The backup environment has been relatively quiet.  We have included growth room in the Exchange projects to handle the upcoming load we are expecting from the deployment of those services.  Additionally, we are starting to see a need for additional storage in our NDMP pools to support NAS backup.

We are currently evaluating 4 different vendor archive solutions and we are in the process of putting together our final technical recommendation to ITS management.

Operationally our backup environment continues to run smoothly and we are even completing all of our cloning within 2-3 days of processing.  Since we have implemented the new backup infrastructure we have growen from ~157 TB of data backed up per month to 247 TB backed up monthly.  We are growing at a rate of approximately 85% per year.

I have architected a backup solution for the possible expanded BlueArc environment.  The environment will include two dedicated tape libraries, a SAN switch, two storage nodes, 36 LTO-4 drives, and 6800 LTO-4 media.  This solution will be able to produce monthly full backups and weekly incrementals of all of the data residing on the new service.  Additionally, we will be producing monthly clones of all full data.

E-mail archive is in the research phase.  We are looking at various vendors and their solutions to our archiving requirements.

One of our Co-lo customers of the backup service lost 2 databases approximately 6 minutes after the backup had completed.  They were able to restore their database back with only a 6 minute loss of data.

I worked extensively on the backup environment this month in an attempt to reduce the number of backup failures in our environment.  We have two basic types of failures in our environment currently.  We have servers that are losing network connection to the environment and we have Windows clients that are failing system state saves via the VSS facility in windows.  The network situation is being resolved by moving affected servers to the dedicated backup networks.  We have a dedicated network for CO-LO servers and a dedicated network for ITS servers.  The VSS problem appears to be mainly a lack of MS patches or hotfixes that apply specifically to the VSS writers.  Though some clients are failing the same way for different reasons (memory, service configuration etc).

This month we implemented the new BlueArc 3210 (10GB network) into our environment.  Due to network problems between 100 mb/s, 1000 mb/s and the 10 gb/s of the BlueArc we ended up backing out this change and swapping in the Titan 3200 (6 1 gigE links).  This appears to have resolved the issue we were seeing with a high number of retransmits on the network causing poor performance for the entire community.

Additionally, Kenon and I have been tasked with replacing the smtp environment.  We have only recently had time to devote to this project and all of the work up to this point has been done by Troy.  It looks like we may be unable to implement the new smtp environment on Linux due to some Perl module problems and may have to take a look at Windows 2008 as our platform.

Additionally, we are almost finsihed addressing the virtual drive to physical disk space problem.  Originally we had our virtual tapes configured for a capacity of 800 GB which is the native capacity for LTO-4.  Compressed this represents approximately 1.2 TB of data per virtual cartridge.  The problem was that our tapes were not expiring as often as we had hoped because of the sheer number of savesets on each volume.  I have reconfigured all of the pools but one to use 200 GB cartidges and this should correct the problem.

One more change we have made to address the amount of data on tape is to reduce our retention time on disk/onsite to one week.  Originally we retained 14 days, but this resulted in retaining 3 weeks due to the fact that a “full” backup will be kept if any incrementals dependant on that backup are not yet expired.  Lowering the retention time on standard backups (1 full, 6 incrementals) to 1 week still keeps 2 weeks worth of data and corrects the problem of keeping the third full.  All clients that backup at a “full” level every day are still set to be kept for two weeks as they are not affected by the full/incremental retention issue.

We tested backup of CO-LO customers Owen and Law over a private backend network in order to avoid a couple of firewalls.  During this test we realized a 400% improvement in performance.  This brings up the a real need for a dedicated backup network to improve performance across the environment.
We have done extensive work this month to present accurate and detailed statistics on the state of our storage environment so that the information can be used to determine how the storage environment is being used and to more accurately forecast future needs.

I wrote a script this month to analyze VUspace and determine how many users are actively using the service and how much of the data on the service is stale.