Kevin McDonalds Blog

• New Tape Library goes live!
  

  ITS life-cycled the core component of the backup environment in July, 2008. The old L700 tape library has been replaced with a Quantum i2000 Scalar tape library which boasts a dozen LTO4 drives, and eleven-hundred tape slots. This new library is a foundational component for future business recoverability for many departments at Vanderbilt.

• DNS Security Patch Applied
  

  On July 8th, 2008, the Technical Cyber Security Alert TA08-190B announcing vulnerability within multiple DNS implementations to cache poisoning, including BIND was released. This is the protocol used here at Vanderbilt for Domain Name Resolution. Many technology vendors have released patches for their products to close this major vulnerability. ITS applied the patch to the primary internet-facing DNS servers on July 19^th, 2008.

• Data Corruption Avoided
  

  Earlier this summer, EMC contacted ITS Storage Administration about a bug in a particular version of a hard-disk drive deployed at ITS. This bug caused data corruption when specific conditions occurred. ITS quickly scheduled a Flare code patch that would identify corruption before it occurred and then subsequently replaced the disk drive that was susceptible to the error condition.

• Mail Gateways Upgraded
  

  The ITS Electronic Message Administration team has upgraded the Proofpoint Email Gateways to version 5.03. This upgrade provided a more granular reputation model, advanced administration GUI and reporting capability.

• Web Application Scanning
  

  In a coordinated effort with the Network Security tea, several hand selected websites on the main Vanderbilt web server were used as evaluation content for two web application scanners. These sites were selected because of suspicious log entries for requests to these sites. The scans involved a discovery phase, last minute database backup, an evaluation phase, and a reporting phase. In addition to evaluating the scanning productions, we were also able to pass the findings on to our customers so that they could fix their applications.

• All Server Consoles moved to a protected VLAN
  

  This month ITS System Administration completed the year long project to move Server Consoles for HP and IBM servers to a protected and private administration network. This configuration is much more secure than the previous.

• Storage Upgrades in preparation of 1GB VUExchange Mailbox offering
  

  This month ITS Storage Administration completed the expansion of the EMC Clariion and Cisco 9509 SAN Directors to support the 1GB VUExchange default mailbox size.

• Vunetid server migrated to a Virtual Machine
  

  ITS Staff migrated the vunetid server from an out-dated server and up-supported operating system to RedHat Enterprise Linux running on a virtual machine. This has removed dozens of security holes and properly aligned this service with modern infrastructure.
  

• Operational and Administrative updates:
  
  • 166 Magic Tickets were processed in Applications Hosting.
  • 34 VUmail to VUExchange Mailbox Migrations
  • Two employees attended F5 Load Balancer (Local Traffic Management) training

• Virtual snapshots -

  ITS has released the Virtual Machine snapshot service. The "Snapshot Service" provides a snapshot of a real-time live image of a guest O/S system disk (or Hard Disk Drive) for a Virtual Machine. The frequency of the snapshot is recommended to be weekly at a cost of $150 yearly. The snapshot service is only offered to Virtual Machines. ITS System Administrators have created virtual server snapshots for multiple customers, including several for the Law School. These snapshots have proven beneficial in projects such as Sharepoint, where restoring from a snapshot has saved the project hours of rebuilding effort in re-establishing the test environment. The primary usage of snapshots is reverting to a previous state of the host operating system. This is useful for recovery from a bad change or a intrusion.

• Nettracker Database Upgrade

  ITS Database Administrators upgraded the Nettracker web server oracle client and Nettracker Oracle database versions applying several patches that will increase the security of the data kept on this system.

• Streaming Media Infrastructure - Enhanced

  ITS Staff have moved the Media Streaming Infrastructure components outside of the ITS Firewalls. This move should provide improved network bandwidth to the Media Streaming Infrastructure which better prepares Vanderbilt University for the upcoming commencement ceremonies. These servers are now protected by host based firewalls instead of the previous setup.

• Cohosted Web Server Patching

  The cohosted web servers were patched to current RedHat Enterprise Linux 4 standards. These are instrumental in enhancing the security and performance of these servers.

• Identifying web applications with security vulnerabilities

  ITS hosts web sites for many different departments with various levels of expertise in web site development and maintenance. As a result of an incident investigation, our administrators identified 987 sites that were open to vulnerabilities, and notified their owners (45). We also upgraded one user application that had a vulnerability and shut down 2 sites that could not be fixed and made the environment for all web site owners vulnerable.

• Operational and Administrative updates:
  • 151 Magic Tickets were processed in Applications Hosting.
  • 11 IMAP to Exchange Mailbox Migrations
  • 1 staff obtained Microsoft Certified Technology Specialist (MCTS)
  • 1 staff upgraded Microsoft Certified Systems Engineer to 2003.

• New Nagios 2.10 in production!
  

  The AppHosting team migrated the existing hosts and service configurations from the old Nagios to the new version, and it went live on Friday March 28^th, 2008. The quantity and quality of the checks is a great enhancement to ITS and Vanderbilt. The server is performing very well considering that it is an extremely trim server compared to the current server. Below is some data from the migration.
  

  Old            New
  

  Hosts Monitored        319            329
  

  Services Monitored        657            884
  

  Service Groups        16            35

• New DNS Infrastructure in Production!
  

  Application Hosting rolled out the new DNS architecture. The new system is running a fully integrated DNS/DHCP/IP Management solution and went in with zero downtime. There were only six errors reports in over 29,000 records throughout 430 separate domains that were converted. This includes an upgrade from BIND8 to BIND9, a redesign of how the DNS topography was layed out, and introduction of a new management layer – using Diamond IP.

  Currently, ITS is serving up over 1.5 million DNS queries per hour without any hiccups. The transition was transparent to the vast Vanderbilt community, and the entire Internet.
  

• New Mailgate Feature Deployed!
  

  Application Hosting staff deployed Proofpoint Dynamic Reputation (PDR) this month. PDR uses a combination of local, predictive behavioral data and globally-observed reputation-analyzed by powerful machine learning algorithms-to block incoming connections from malicious IP addresses. This process will help decrease the amount of SPAM that makes it through the mail gateways.

• Chancellor Announcement
  

  Application Hosting staff participated in the new Chancellor announcement with responsibilities such as removing the redirect to the interim chancellor site and ensuring that the new content was web accessible.

• MS Project 2007 Server deployed
  

  As part of the Sharepoint rollout, MS Project 2007 is now online and appears to be working as described. The only functionality not deployed was some configuration related settings which will be addressed during the “enhancement phase” of the SharePoint project.
  

• Sitemason Disk Space Addition
  

  Additional disk space was added to the virtual machine providing the web service for the Sitemason frontend. This was a relatively easy task involving the creation of another virtual disk and addition to the web content volume.

• Server Decommissioning
  

  The following physical servers were powered off and removed from the Hill Data Center to be prepped for disposal. The services of most of these machines have been turned into “Virtual Machines.”

1. meru2
2. samsara
3. sitemsn-fe
4. sitemsn-be
5. nde-syslog
6. apps3

• Hera Decommissioning
  

  The Projector and Port Block applications have been migrated over to the main ITS website after the power supply failure. The Xserve is now merely providing a redirect page to the current locations. All existing links to these two applications have been found and corrected. The change to remove the Xserve hardware has been scheduled for Apr 7^th, 2008.

• MC Intranet
  

  Application Hosting Staff added another aliased virtual interface to VICC virtual machine to handle the new MC intranet website., and created another Apache virtual host as a placeholder for the content.
  

• Operational and Administrative updates:
  
  • 150 Magic Tickets were processed in Applications Hosting.
  • 14 IMAP to Exchange Mailbox Migrations
  • 1 additional staff obtained the VCP Certification (VMWare Certified Professional)
  • 1additional staff attended the “Legato Networker” training
  • 1 Staff attended SNIA Storage Networking Foundations course
  • 1 Staff obtained the “2003 MCSE+Messaging” certification
  • 1 Staff obtained the MCSA for 2003 certification
  • Annual Staff Evaluations were finalized and delivered.

I am now BALOO Certified! What is BALOO? It is the Boy Scouts of America "Basic Adult Leader Outdoor Orientation." This means that I can lead a pack in a campout.

Basic Adult Leader Outdoor Orientation (BALOO) is a one-day training event that introduces leaders and parents to the skills needed to plan and conduct pack outdoor activities, particularly pack camping.

Participants who complete this training course will

* Understand the focus of the Cub Scout level of the BSA outdoor program.

* Gain the skills needed to plan and carry out a successful Cub Scout-level overnight activity.

* Learn more about the resources available from the BSA for carrying out this activity.

This training is required for any adult who is in charge of planning a pack campout.

• Unified Communications Pilot
  

  ITS Application Hosting is performing a pilot run of the Microsoft Exchange 2007 server environment. Getting email functional has been fairly simple, but integration with the Vanderbilt environment has been challenging.

• EMS SCALA Report
  

  In support of the Division of Student Life’s data mining needs, ITS System Administration staff implemented a SSIS package to run a TSQL query and export the results to an XML file.

• Virtual Machine Snapshots
  

  The ITS System Administration staff has enabled Virtual Machine snapshots as a feature of the VM Hosting service. This allows virtual co-located customers to take “moment-in-time” snapshots of their running virtual machines. This can be useful when recovering from a failure such as a bad software install or a system administrative error.

• Web Server O/S Patching
  

  The ITS System Administration staff has updated the security patches of the web infrastructure, allowing for a more secure web experience.
  

• Hydrastor Eval

  The ITS Storage Administration staff continued the evaluation of the NEC Hydrastor network de-duplication device in February.  This evaluation is needed to enhance the backup architecture.

• Speed Test Server Deployment
  

  Application Hosting worked with the ND&E Staff and has implemented http://speedtest.vanderbilt.edu/ which will test the upload and download connection speeds of a network client.

• Operational and Administrative updates:
  
  • 170 Magic Tickets were processed in Applications Hosting.
  • 13 IMAP to Exchange Mailbox Migrations
  • 2 Staff obtained the VMWare Certified Professional Certification
  • 6 Staff attended “Implementing & maintaining SQL Server 2005 database” course
  • 2 Staff attended the “Redhat System Monitoring and Performance Tuning” course
  • 1 Staff attended the “Legato Networker” training

• SMTP Mailgates are now 100% Load Balanced!    

  Early in January, the System Administration staff finished the production load balancing of the SMTP mailgates. This change is a huge step in providing seamless availability to our email customers, as well as providing the flexibility to scale to future needs without changing any public facing mail destinations. The Mail load is now evenly distributed amongst the multiple mailgate servers, which has resulted in a smoother operation and less disruptions from bursts of activities.

• Sitemason Migration

  The Sitemason services have been migrated to new servers.  The frontend web server is running in a Virtual Machine, and the backend database server is running on physical hardware. The site has been running pretty well, and everything looks to be performing as expected.  The frontend server was built with twice a much memory (4GB) as the original, and the system is using up to 95% of it at times.

• Office of Investments Application Hosting - PerTrac

  ITS is hosting an application called “PerTrac” for the Office of Investments. This month, ITS staff assisted the Pertrac technical consultants with the implementation and production rollout of their product. The following components were installed and configured: SharePoint, SQL Reporting Services, and major troubleshooting issues for them.

• Backup Enhancements – new methodology for Exchange backups.

  This month, ITS System Administrator staff enhanced the methid in which Microsoft Exchange backup processes were performed. They implemented Legato’s Exchange Backup product which dropped the backup window for the exchange servers from over seventeen hours to just over four hours. Additionally, this lowered online disk consumption in the MS Exchange environment that resulted in freeing up 1TB of Tier 3 diskspace to the storage environment.

• Operational and Administrative updates:
  • 179 Magic Tickets were processed in Applications Hosting.
  • 23 IMAP to Exchange Mailbox Migrations
  • Staff attended “Sharepoint end-user” training

Two staff obtained the Redhat Certified System Engineer (RHCE) certification.

“Nothing in the world is more dangerous than a sincere ignorance and conscientious stupidity.” - MLK

• Operational:
  • 152 Magic Tickets for Applications Hosting
  • Performed periodic database maintenance for Corporate Time
  • Applied all Sun recommended patches and security patches to approximately ½ of the Sun environment. The other ½ is scheduled for August.
• Security Enhancements:
  • Service Console Migration – to a private non-routed LAN.
• Messaging:
  • Changed various domain’s MX records in order to load balance across mail exchangers.  Has had immediate impact on load distribution across mail exchangers. 
  • Successfully tested VUMailguard Digest Authenticaiton via LDAPs with AD and our LDAP servers.
  • Created script to detect message loops on list server.  Has already detected two instances which were resolved immediately.
• Storage / Backup:
  • Implemented second SAN Director (Cisco 9509)
  • Finished evaluation of the EMC EDL. We are returning it.
  • Started evaluation of Legato Backup to Disk solution.
  • Started an evaluation of various de-duplication technologies including microsofts DPM and Data Domains hardware device.
• ESM / Metrics
  • Minor progress of monitoring improvements.
• Projects:
  • Web Architecture Migration:
    • Complete! We are now in the process of load balancing the entire web service.
  • Teamspeak Server
    
    • In production
  • Helix Streaming Media Server
    
    • Underway
  • VUNETID Batch, Web, and Database separation
    
    • Underway
  • Boinc Distributed Computing
    • Fully deployed
  • Magic 9.0 Upgrade:
    
    • Scheduled for August 3^rd.
  • Flash Streaming Media Server
    
    • Deployed
  • Load Balancing the Mailgate SMTP environment
    
    • In Analysis phase
  • SharePoint Implementation
    • Beta environment is 95% ready. Will be released in August.
  • CSM Implementation:
    • The following web services have been load balanced:
      • fevs01.its.vanderbilt.edu (its)
      • fevs02.its.vanderbilt.edu (vanity domains)
      • fevs03.its.vanderbilt.edu (blogs)
      • fevs04.its.vanderbilt.edu (studentorgs)
      • fesv06.its.vanderbilt.edu (www4)
  • OSIS
    
    • In production!
  • DNS/DHCP:
    • DiamondIP research continues
  • SCOM:
    • Project on hold pending a Nagios upgrade
  • VUSpace3
    
    • Kicked off project, evaluating various alternatives for replacement.
• Virtual Infrastructure
  • Added Blair School of Music
  • Added Vanderbilt Institute Research Group (VIRG)
  • Created a reliable ESX Test Environment
  • HA and DRS fully operational in the test environment.
  • Automated ESX HBA pathing / failover.
• Administrative
  • 5 staff attended ITIL foundation training (all are now certified)

6 staff attended Sharepoint System Administration training