Kevin McDonalds Blog

“Nothing in the world is more dangerous than a sincere ignorance and conscientious stupidity.” – MLK

• Operational:
  • 152 Magic Tickets for Applications Hosting
  • Performed periodic database maintenance for Corporate Time
  • Applied all Sun recommended patches and security patches to approximately ½ of the Sun environment. The other ½ is scheduled for August.
• Security Enhancements:
  • Service Console Migration – to a private non-routed LAN.
• Messaging:
  • Changed various domain’s MX records in order to load balance across mail exchangers.  Has had immediate impact on load distribution across mail exchangers. 
  • Successfully tested VUMailguard Digest Authenticaiton via LDAPs with AD and our LDAP servers.
  • Created script to detect message loops on list server.  Has already detected two instances which were resolved immediately.
• Storage / Backup:
  • Implemented second SAN Director (Cisco 9509)
  • Finished evaluation of the EMC EDL. We are returning it.
  • Started evaluation of Legato Backup to Disk solution.
  • Started an evaluation of various de-duplication technologies including microsofts DPM and Data Domains hardware device.
• ESM / Metrics
  • Minor progress of monitoring improvements.
• Projects:
  • Web Architecture Migration:
    • Complete! We are now in the process of load balancing the entire web service.
  • Teamspeak Server
    
    • In production
  • Helix Streaming Media Server
    
    • Underway
  • VUNETID Batch, Web, and Database separation
    
    • Underway
  • Boinc Distributed Computing
    • Fully deployed
  • Magic 9.0 Upgrade:
    
    • Scheduled for August 3^rd.
  • Flash Streaming Media Server
    
    • Deployed
  • Load Balancing the Mailgate SMTP environment
    
    • In Analysis phase
  • SharePoint Implementation
    • Beta environment is 95% ready. Will be released in August.
  • CSM Implementation:
    • The following web services have been load balanced:
      • fevs01.its.vanderbilt.edu (its)
      • fevs02.its.vanderbilt.edu (vanity domains)
      • fevs03.its.vanderbilt.edu (blogs)
      • fevs04.its.vanderbilt.edu (studentorgs)
      • fesv06.its.vanderbilt.edu (www4)
  • OSIS
    
    • In production!
  • DNS/DHCP:
    • DiamondIP research continues
  • SCOM:
    • Project on hold pending a Nagios upgrade
  • VUSpace3
    
    • Kicked off project, evaluating various alternatives for replacement.
• Virtual Infrastructure
  • Added Blair School of Music
  • Added Vanderbilt Institute Research Group (VIRG)
  • Created a reliable ESX Test Environment
  • HA and DRS fully operational in the test environment.
  • Automated ESX HBA pathing / failover.
• Administrative
  • 5 staff attended ITIL foundation training (all are now certified)

6 staff attended Sharepoint System Administration training

• Operational:
  • Started reviewing Weekly Incident Reports in the Tuesday Apphosting meeting.
    
  • Automated reports to generate top 20 mailboxes on IMAP and Exchange servers.
  • Successfully installed and configured SQL Reporting Services 2005 on Apphost1a
  • Implemented
    “Out Of office delivery” to the internet for the Exchange environment.
  • 121 Magic Tickets closed for Applications Hosting
  • Cleaned 20+ Sun Servers for disposal. Sold a couple on Ebay through Snappy Auctions.
  • Entered 206 engineering students for the annual PAVE program
  • Upgraded LDAP to 5.2SP2
• Email Oneness
  • Documented Restore process (Exchange, IMAP, and Groupmail)
  • Developed initial VU Email SLA
• Storage / Backup:
  • Completed the Legato 7.3 upgrade.
  • Completed the implementation of a second Legato Storage Node and worked through various issues.
  • Configured second Hill Center Cisco 9509 SAN Director. Implementation scheduled for June.
  • Owen has partially migrated to using the ITS backup Legato infrastructure.
  • Started the evaluation of a EMC Disk Library.
  • An interesting piece of information regarding backups.  As of 2/11 of this year we were backing up approximately 18.5 TB.  As of 5/29/07 we are averaging 21.25 TB.  So we are adding approximately 1 TB per month to our backup infrastructure!
  • Sold the Sun HDS 9970
  • Started the VUSpace replacement project (In Define phase.)
• DNS:
  • Separated the NetID Environment from VUMC
  • Continue feasibility of Infoblox appliance option.

     

• ESM / Metrics
  • Performed a GAP analysis on the state of system monitoring for ITS.
• Projects:
  • Web Architecture Migration:
    • This project is in the final stage.  The replacement ITS web services pages have been created, and are being populated with information and feedback from the web migrations. The new services signup form is also available. Nearly all of the student organization websites have been migrated to the studentorgs server.
    • Migrated 309 Websites and deleted 93 in May.
    • The MOO service has been migrated from Helios to the new web environment.
    • Final cutover planned for June 3^rd, 2007.
  • Magic 9.0 Upgrade:
    
    • Set up SQL 2005 db on cluster
  • SharePoint Implementation
    • A formal project team have begun to meet and define the ITS implementation of SharePoint including how to provision SharePoint, who can participate in the various rollout stages, etc.
    • Worked with Microsoft on resolving an issue with users receiving an “Access is Denied” error message whenever they go to ttps://sharepoint.its.vanderbilt.edu
  • CSM Implementation:
    • Implemented Network Load Balancing for Webmail
    • Planned Network Load Balancing dates for:
      • June 10^th – blogs.vanderbilt.edu
      • June 17^th – Misc/Vanity sites & Student Organizations
      • June 24^th – ITS website
      • July 8^th – SSL Websites
      • July 22^nd – www.vanderbilt.edu
  • SCOM:
    • Developed notifications and installed a VU SSL certificate on the web console so our passwords are not sent in the clear.
    • Installed the SCOM agent on 25 production ITS windows servers. The remaining are scheduled for early June.
    • Email notifications to work.
    • Configured different levels of notifications.
    • Install of consoles on 2 monitoring machines (Apphosting Management).
• Virtual Infrastructure
  • Migrated 23 servers from Owen Graduate School of Law to ITS Virtual Infrastructure
  • Migrated 5 servers from Law School of Law to ITS Virtual Infrastructure
  • Restored service to nde.vanderbilt.edu (crashed server) by implementing VM
  • Migrated VICC (Ingram Cancer Center) to a Virtual Machine.
  • Created a “VUNetID” replacement server as a VM. This is now in a migration analysis status with iDev.
  • Implemented Mediasite in Virtual Machines
  • Implemented Sharepoint test environment
  • Migrated Apphost1a to a virtual machine running. The databases were moved to the highly available Microsoft SQL 2005 cluster
  • Discussed Linen Services / Textraxx (Jennifer Keen)
  • Discussed DB1 web & app servers
  • Discussed Blair Virtual Machine Collocation
• Administrative
  • Evan Saltmarsh attended EMC Clariion Performance Tuning course.

April, 2007 WINS

·         Operational:

o   Blog Services: There are now 98 blogs on the server. 

o   Redhat VM patch: Applied the VMware recommended RPM for the MPT Fusion driver on all Redhat Linux VM’s.  This will prevent the Linux file systems from becoming read-only in the event of high IO or SAN fabric failover.

o   Apphost1a Migration:  Migrated single server called Apphost1a running Windows Server 2003, ColdFusion 6.1MX, and SQL Server 2000 to a virtual machine running Windows Server 2003 and ColdFusion 6.1MX and the databases were moved to the highly available Microsoft SQL 2005 cluster.

o   Mail routing for math.v.e is now routing through the proofpoint appliances.

o   359 magic tickets processed / closed.

·         Storage:

o   Upgraded Legato to 7.3, Legato branded version.

o   Implemented a second Linux storage node

o   Implemented NDMP for the Celerra NAS device.

o   Performed a storage upgrade to support Owen and Law migrations.

o   Racked second Cisco 9509 Director and planned migration.

o   Performed the first ILM movement of data across tiers of storage without an outage required on the user’s part. (The data store for snf-be was moved from tier 2 to tier 1 storage.)

o   Performed a ILM filesystem analysis for a customer.

·         Server:

o   The HP c-class blade chassis is powered on, and the onboard administrator has been configured.

o   LDAP1 and AUTH1 physical migration. This freed up rack space to allow the data center to reclaim a rack. 

o   VMware ESX security vulnerability patching – a couple of security vulnerabilities were discovered in VMware ESX Server 3.0.x.  The vulnerabilities were assessed and the patches were applied to 2 test servers to insure there would be no induced instabilities.  After testing, the patches will be rolled out into the production environment.

·         DNS:

o   NETID separation planning, designing, risk assessment

·         ESM / Metrics

o   All of the SAN infrastructure was put into Cacti this month for metering SAN bandwidth usage.

o   Created monthly Email metrics report.

·         Projects:

o   Owen / Law Migration: Designed, validated, and implemented the facilities and network components of this project.  Validated breath of scope and created timelines for server migrations.

o   Web Architecture Migration: 34% of the web content has been migrated, redirected, or deleted. 13% of the sites have been copied over for testing.  The pace of the migrations has picked up, and status updates are supplied on a tracking page . http://its.vanderbilt.edu/support/migrations.php

o   SharePoint Implementation – Created a real project for this effort.  Worked with Microsoft on resolving an issue with users receiving an “Access is Denied” error message whenever they go to https://sharepoint.its.vanderbilt.edu

o   Altiris Project:  Redo Stockroom tables, add and change fields

o   Started the MAGIC UPGRADE project   

o   The ePolicy application server (ePO) is now ready to be used, ePO agent still needs to be deployed to the Windows Servers.

o   Fax Server project:  completed the requirements doc for IDEV’s development portion.

o   CSM Implementation:  The CSM’s are completely ready for production use.  Webmail is scheduled to be the first customer later in May.

o   MOM 2007: The server is build and the Pilot is underway.

o   Infoblox DNS/DHCP Enhancement Project:  Progress was made in the DNS/DHCP Enhancement Project with the go-ahead to proceed to taking the project to governance.  The DNS portion is extremely stable and is pretty much good to go. 

·         Virtual Machine Discussions

o   Mediasite (Cliff Wilson)

o   Linen Services / Textraxx (Jennifer Keen)

o   Paul L – DB1 web & app servers

o   Sharepoint –6 VM’s requested

·         Administrative

o   Kevin executed 12/13 evals!

o   Evan attended Legato Networker training.

o   Guy Shepperd attended Proofpoint training.

o   Roland attended HP Insight Manager conference.

o   Kevin & Kenon attended Blade  Insight conference.

o   Kevin started using eProcurement instead of Yeller sheets!

o   SML Presentation for Matt and Fran.

• Operational:
  • DST Patches
    • 200+ Servers
    • 30+ Databases
    • Several Applications
  • VUExchange retirement (Be01)
• Storage:
  • Finalized backup enhancement strategy – EMC Networker upgrade to second storage node.
  • Purchased NDMP for backing up NAS shares
  • Ordered storage to support Owen and Law hosting
  • Training – Evan on Clariion
• Process Improvement
  • PASC definitions for Storage and Proofpoint
  • Started Virtual Server Build Process Improvement effort.
• Server:
  • ND&E syslog server deployed
  • Replacement dbdev-1 built
  • Added RHEL 5 to RHN
  • Began snapshot management for the virtual infrastructure

     

• Website Migrations to new architecture
  • PAVE (converted to SSL site)
  • PlantOps
  • Summer Sessions
  • Master of Liberal Arts and Science
  • Womens Studies
  • Registrar Intranet
  • Middle TN Chapter Society for Neuroscience (migrated to VUMC server)
  • Brain Institute (migrated to VUMC server)

   

• DNS:
  • Finalized Infoblox setup, received new 1550’s, beginning final test steps
• ESM
  • IBM “Director” – server built and tested
    • Windows hardware monitoring Pilot underway
    • Linux testing in progress
  • HP Insight Manager – server built, no testing completed at this time
• Projects:
  • Fax Server project, successfully finished upgrade testing from our current version to the new version, and we’re roughly 90% completed with the requirements doc for IDEV

     

• Kevin completed 13 evals!

One of the ITS System Administrators has put together a very nice environmental diagram of the ITS Storage environment. Here it is saved as a JPG file.

Last week, we presented our findings to Matt Hall on the responses for SMTP delays this summer. You can read the report here. SMTP Response Delays Fall/Winter 2006-2007 So far, we have executed the following recommendations:

• Applied Proofpoint Performance Related Patches (#343, #406)
• Replaced Mailgate03 with P800 hardware (with 4G RAM) on Friday, January 19th, 2007
• Planned install of memory for mailgates on Wednesday, Jan 31st, 2007
• Scheduled the enforcement of SPF (client side) change for Feb 7th, 2007.
• Started recommended operational changes.

Additional Notes:

Jan 23, 2007:     It is recommended that we apply a configuration setting for the conversion daemon to stop core dumps.

Jan 24, 2007:    The memory expansions for all mailgates arrived today.  Gary Howard is in possession.  

Jan 25, 2007:    Ryan and Nifin from Proofpoint met with the Vanderbilt team onsite today. 

1. Storage
a. EMC Celerra configured / deployed.
b. Upgraded the BlueArc firmware upgrade.
c. Reconfigured the BlueArc SMU
d. Decommissioned the HDS 9970
e. Kicked off the BURA Project
f. R&D Around SVC

2. Servers
a. Applies OS patches
i. ip-srv1
ii. ip-srv2
iii. nidora1
iv. list-srv1
b. Installed Altiris Agents on 86 Unix / Linux Servers
c. Installed Altiris Agents on 63 Windows Servers
d. WSUS workstation patching (409) servers (26)
3. Security
a. Applied IPF Rules to list-srv1
b. Pilot web application authenticated via LDAP.

4. Projects
a. Content Switching Module: tested and operations defined in lab.
b. DNS project planning continued.
c. IdM: Deployed a new LDAP server & iplanet 5.2
d. Nagios rebuild: Started project.
e. Video Conference Implementation (Polycom)
f. Amcom:
i. Assisted Amcom consultants in on-site implementation
ii. Identified several security implications in Amcom implementation
g. Altiris:
i. VUnet query for active ITS STAFF
ii. VUnet query for DEPARTMENTS that ITS has any interest such as DOA
iii. Design how COST Centers will be displayed.
iv. Made ND&E location data upper case and consistent. Divided data between buildings and buildings+rooms
v. Created import scripts for all imports into database
vi. Imported Vendor data from Accounting’s FilePro export.
vii. Designed RUNBOOK fields in Altiris (still in progress)
viii. Imported some RUNBOOK data. Cleaned up data. ALL computers needed serial numbers. I queried the NOC database for their Audited list of serial numbers and merged those into the RUNBOOK db.
ix. Added fields such as VTAG, cellphone fields, many other fields.

5. Training:
a. Derek Miller: Cisco MDS 9000 (EMC Subscription)
b. Julie Catellier: Oracle DBA
c. Peter, Julie, Tony: Sun IdM training (Onsite)

6. Email
a. Exchange DR environment set up and documented.
b. Collected data for Six Sigma analysis of Proofpoint Updates correlated to webmetrics alerts and the impact of performance of the mailgates.
c. Researched spambot attack impact on mailgate performance.
d. Migrated Law Faculty from ITS-EXCHBE01 to MAILBE16
e. Research/Started implementation of change in Exchange Routing to resolve Exchange Black Listing Issues

7. Magic Support
a. Coordinated 135 Magic Tickets for Applications Hosting.
b. Completed 35 Magic Tickets.
c. Designed 3 custom Magic Helpdesk reports for ITS-PARTNER.

8. Database Administration
a. Copy and built development 10.1.0 VUnet database for iDev.
b. Attempted to apply the Oct Critical Patch on dbdev-1.
c. UPGRADE DB-1, DB-2, DB-3.
i. Roll back 10.2 critical patches and apply fix and repatch for VUAPPS on DB-2 and the standby on DB-3.
ii. Upgrade and patch the standby databases for VUNET on DB-2 and DB-3.
iii. Update dozens of user profiles and procedures to point to the new oracle directory on DB-1, DB-2 and DB-3
d. Created a list of Databases in Application Hosting that need administration (32 defined).

9. Virtualization
a. Gig Vmotion Network Setup
b. NFS Share for software (from celerra) implemented.

10. Web Architecture
a. Four more web servers have been put into production, bringing the total number of new web servers to five.
b. The first which was deployed last month is the new home for its.vanderbilt.edu, and has been running smoothly without incident. The time to render a page has been reduced approximately 40%.
c. The blog server is now receiving attention. The entire App Hosting team now has an individual blog, and a couple of VU organizations have requested services.