Planet AppHosting

I feel like a broken record….

1.  BT INS aka DiamondIP aka DIP - Test environment still not up.  Having multiple issues getting the database to work on the test environment.  Half tempted to blow the thing away and totally rebuild this test environment.

2.  Auth4 aka redundant kerb server - Built out a new kerb server to go to the hospital.

3.  Gave the Med Center an overview of DIP and the functionality.  They were made aware of the ups and downs of the product as well as given an insight on how it works.

4.  Finally got CA to get me uncorrupted patches for Spectrum One-Click.  Installed the patches and everyone is now happy that Report Manager works as it should.

5.  Lended a hand on a little forensic work for IRT.  Ended up taking much more time that I thought it would but was very much an eye opening experience.

6.  Fixed a ton of issues on RHN.  Purged a lot of non-responding servers, registered a bunch more, and finally got all the channels resync’d.

7.  Did a metric boatload (as opposed to Imperial or Standard boatloads) of DNS requests/changes/fixes/etc

8.  Managed to break DNS attempting to get BIND Views working…. GO ME!

Installing MOSS 2007 on SQL 2005 Instance on a non-default port:
In order to use an alias for SQL you must have the following installed on your MOSS 2007 servers:
1. Connectivity Tools
2. SQL Management Studio

On each MOSS 2007 server:
1. All Programs -> Microsoft SQL Server 2005 -> Configuration Tools -> SQL Server Configuration Manager -> SQL Native Client Configuration -> Aliases
2. Create a new alias. I used the IP address of the SQL instance because it’s on a cluster and seemed to work a little better.
3. I also used a FQDN for my alias name and created a DNS CNAME record for that name. (i.e. ALIASNAME.COMPANY.COM)

Identity Management: The new LDAP servers used in the IDM project currently suffer from some stability problems due to issues with the Directory Proxy Server (DPS). The DPS is being used to ensure that the correct SSL certificate is presented to the client. While the vendor is working to correct the problem, the IDEV and AppHosting teams are working on an alternate solution. I built two new development LDAP servers to be used by the IDEV team. These new servers more closely resemble the architecture and configuration of the production servers. We are currently testing the use of stunnel as a DPS replacement. While stunnel does not offer the same feature set as the DPS, it does provide the base SSL functionality and is considerably more light weight. We are also evaluating F5 LTM appliances in the test environment to see how well they integrate. We currently have a pair of evaulation appliances that we will use to front-end the new test LDAP servers that I built.

IMAP-to-Exchange Migration Tool: I have scripted the necessary tasks to prepare IMAP accounts for import into Exchange. The process essentially makes a duplicate of the user accounts so that the original is unmodified. I still need to create documentation to detail the process for the rest of the team.

Search Engine Replacement Project: I have attended several meetings and a couple of product demonstration. I am passing this project to Troy Osborn.

ITS Website Redesign: My involvement in this project has been relatively minimal this month while the vendor continues development efforts on the site. My activities this month included adjusting ModSecurity rules,  re-importing a database for the vendor after a failed upgrade,and identifying application errors in the logs.

Operational Issues: The majority of the operational issues for this month have been relatively routine, and most of the web requests were related to creating redirects and such. I did create a new virtual host for a website that was being publicized the next day in a national magazine.

1.) Coordinated 149 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 32 Magic Tickets. This majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry, mailing list owner changes and moving email from a user’s deleted account into their new account. This number was done some due to being out for training for a week.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continued work on the PAVE program. I reset 210 PAVE accounts to reauthorization status prior to the arrival of the PAVE students. Due to changes brought about by the IDM system, this task has been more or less turned over to the IDEV team until new tools are provided for the IDM system. I am still involved in a consultant status. This program is an annual task to create and maintain accounts for prospective engineering students each year.

5.) I have scheduled the upgrade of IPF on the IMAP and SMTP servers. This will be done over three weekends in July.

6.) Continued project to move ILO ports for HP and IBM servers to the admin network. Administrative usernames and passwords have now been changed. There are just a few odds and ends to finish now.

7.) The IMAP Cleanup Project is in maintenance mode at the moment and may be extended.

28 Closed Magic Tickets
1 Security Incidents
36 Imap -> Exchange Mail Migrations
3 Exchange -> Exchange Mail Migrations

Exchange

Email Account Bouncing Cleanup after IDM cutover
Worked with IDEV to identify and cleanup issues that occurred with new exchange mail enabling after the switchover to IDM. There were a couple of different issues identified. In a few cases the exchange delivery address and only the exchange delivery address was not populated to the AD account. This caused mail to bounce back and forth between exchange and the proofpoints, this issue was assumed resolved by putting the ad and exchange loaders back on the same machine. I am not sure this was proven as the fix or not. There were also a couple of accounts who seemed to have been provisioned properly on all systems but Exchange would not deliver the mail to the mailbox it seemed to think that the mail should go external to the Exchange environment. The AD object for the user had all the correct email addresses on it and you could look on the Exchange General tab to see where the mailbox existed. SMTP had the correct exchange delivery address for the user also. The resolution for this issue was a deletion of the Exchange account and a clearing of all mail attributes then recreating the mailbox.

Setup new Enterprise Postmaster account for the Exchange Environment
The old postmaster account was Greg Marlin’s 3 letter account which had been deleted. When uninstalling Exchange one of the checks is to see if the Enterprise Postmaster account is housed on any of the Mailstores on that server. The issue that I suspected we were running into was that the environment didn’t know where the mailbox lived since the underlying account was deleted. After making this change the removal of MAILFE05 was a success.

MAILBE16 System Attendant account cleanup
During the change last month to reconfigure the mailstores the System Attendant account was left on a Mailstore that we were not wanting to keep. I had to change the location of the account in adsiedit and restart the Information Store so that we could delete this superfluous exchange database.

Stealthbits evaluation
Evaluating Stealthbits Monitoring for exchange to initially provide monthly metrics numbers for the University Exchange environment. This can also be used much more extensively with some time and work to assist in usage projections as well as troubleshooting and configuration management. I am currently working with Stealthbits to create a custom report to mimic the one we are currently getting out of Messagestats.

Exchange Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches

Configure and Document procedure for removing specific attachments from Exchange
Utilizing Trend Scanmail setup a spam rule to replace a specific file name with a text file saying this file is against policy.

Assist in writing Documentation detailing the levels of Shadow Mailboxes that we can provide
This is to allow us to have a proper request for security incidents.

General Windows

Monthly Microsoft Server Patches
Monitored, verified and tested services after the monthly Microsoft patches for our Windows Server environment

Sharepoint forensics for outage on 6/23/2008
Investigate Sharepoint outage and resolve the issue. Move the servers to the new ESX machines also in the proces

June 2008 Monthly Activitiy Report

1)  Worked on an assortment of odd, challenging, and familiar helpdesk tickets.

2)  Lifecycle replacement of mailgate01 was completed.  Upgrade scheduled for June 18th was rescheduled due upgrade preparation issue.  Worked with vendor in order to resolve issue.  Upgrade has been rescheduled for July 9th.

3)  User reported issue with VUWebmail related to QuickSave 2.3 plugin.  Determine that bug existed that is resolved by installing QuickSave 2.4 or better.  Relayed information to Peter who assigned task of upgrading QuickSave plugin to Troy.

4)   Newlightimaging.com - Have worked with VUMC NCS in regards to messaging routing for the newlightimaging.com domain.  Attended several meetings and exchanged several emails in order to perform fact gathering.  Once it was determined that concerned parties at VUMC wished to route email for the domain permanently, contacted  required parties in ITS in order to assure tasks required for message routing are completed.

5)  Project for mailing list replacement.   Meetings continue to determine what functionality will be implemented, migration strategy, loader development, test methodology, etc.

6)  Project to implement smtp and e-mail authentication.   Nothing new to report. 

7) Worked in conjuction with ITS Security on several incidences e.g. scams, threats, compromised hosts, etc.

8)  Managed abuse@v.e and postmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Created or edited Proofpoint rules in order to deter several threats.  Investigated over 150 spam false negatives. 

9)  Perform daily management of mail queues on mailgates.  Remove hundreds of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

10)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Attended the Red Hat class in Atlanta. Received an RHCT certification. I’ll get the RHCE next time..

Worked with Frank Kyle on the replacement VUNETID project. Bluestem has now been heavily modified from the base installation to run from a single directory tree. This allows the application to run with SELinux enabled and enforcing, as well as making it much easier to port to a new machine later on. Documentation is now in the directory tree along with the installation which explains the modified structure and locations of important files. We have been awaiting the correct ColdFusionMX7 license and clearance from ITS Security in order to move the replacement to a public IP address for pre-production testing.

Worked with Chris Marshall, ND&E, and the NOC to migrate a couple machines to the ITS Test Network.

Identified a problem with the .115 network while working on the bluestem replacement. The machine was a VM and the root problem was discovered to be inconsistent configuration on the switch ports which the VM host was attached. After reporting the issue I was directed to evaluate the entire ESX environment for this same type of problem. Only one other production problem was found and corrected, all others are non-production and will be resolved during normal maintenance for each affected host.

Performed upgrade of Helix Server for streaming media to version 12. Initial report from John in Streaming Media was that everything appeared to be working and the upgrade was a success. A compatibility issue was identified by John the next morning for encoding from the WRVU server. The upgrade had to be reverted to version 11 to enable encoding/streaming for WRVU until their server software can be upgraded to a compatible version.

Coordinated a migration from single 1Gb fiber to etherchannel (dual) 2Gb fiber for several production vlans in two of the Hill 148 High Density racks. Dan Raymer handled migrating VMs, Roland Serman handled MSSQL failover, and Barry McCurry performed reconfiguration of the Hill_Router and Hill_6513 for this change.

Awaiting coordination with the windows team to re-install the OSSEC client on a couple of their test machines to move forward with evaluation. The two SharePoint servers previously being used have been rebuilt and are no longer available for testing. OSSEC has been successfully reconfigured to utilize a mySQL backend for log aggregation, though local logs are still maintained for the WebUI.

The SWE for CSM monitoring is still on hold until an snmptrap handling solution has been installed and configured for the Nagios host.

SharePoint Lessons Learned

This month I’d like to discuss a few things we’ve learned over the past year while trying to deploy MOSS 2007.

•  ·       Our SharePoint deployment is more of an ASP model; each business unit that utilizes our SharePoint deployment wants their own unique URL.  One of the things that we’ve discovered is when you create a Web App, (and create a new application pool for the web app) MOSS doesn’t give the application pool the appropriate permissions on the SharePoint Servers, which leads to a series of DCOM errors in the event log (Event ID 10016), to resolve this issue we’ve done the following.

o   Add the application pool account to the local WSS_WPG group on each SharePoint server.

o   Also, you need to Add the WSS_WPG group to the ‘IIS WAMREG admin Service’ in ‘Component Services’ browse to ‘Computers/My Computer/DCOM Config’ right click, go to properties, and click edit under ‘Launch and Activation Permissions’ add the account, and give ‘Allow’ to ‘Local Launch’, ‘Remote Launch’, ‘Local Activation’, and ‘Remote Activation’.

o   You can either follow this route, or explicitly add each individual application pool account to the IIS WAMREG admin Service.

o   Also, in our installation, we’ve removed anonymous access from the ‘Access this computer from the network’ in the local security policy, if you do the same, then you will either need to allow the WSS_WPG group, or the individual application pool accounts to this as well.

• ·         Setting up Host Header for SSL, this is another one that took a bit of research.  Ultimately what we found was that there is a vb script in the Inetpub/AdminScripts directory called adsutil.vbs which allows you to define the host header for the site.  You simply run the following command:

o   adsutil.vbs set /w3svc/[identifier]/SecureBindings “IP:443:[host header name]”

o   So for example you’d run:

§  Adsutil.vbs set /w3svc/123456789/SecureBindings “192.168.1.10:443:mysharepointsite.com”

o   The [identifier] is the unique IIS Id of the virtual server.

• ·         We’ve also run into some crawling issues.  Our WFE’s sit behind a hardware based load balancer (Cisco CSM).  Currently our load balancer will not allow traffic to pass from behind it and then back in again.  So basically when SharePoint tries to do a crawl, it does a DNS query for all of the SharePoint sites it needs to crawl.  It then updates the HOSTS file with that information.  So then when it attempts to do a crawl it tries to crawl via the Load Balanced IP address instead of the local address, which in turn our Load Balancer won’t allow.  In order to work around the issue, we had to edit the HOSTS file on all of our SharePoint servers and add all of our URL’s with the actual local IP address instead, and then set it to read only otherwise every time SharePoint attempted to do a crawl it would overwrite our manual changes.   Of course now we get hundreds of errors a day in the event log because SharePoint can’t edit the HOSTS file, go figure.

On a positive note, we’ve managed to resolve our SP1 issues, though not by actually fixing the problem.  After 5 failed attempts based on Microsoft supports recommendations, we decided to completely rebuild the farm, and restore all the content manually.  At the same time we migrated to 64bit hardware.  We circumvented the SP1 issues by slipstreaming SP1 into our SharePoint and Project installs.

Monthly Activity Report
June 2008 

1. AIMWorX – Supported all AIMWorX users varying requests.  Copied all SMDR and CDR backup files to CDR for archive.  Processed the weekly update of cost centers and student information.  Ran a monthly process to remove any duplicate calls from the database prior to a billing cycle.
2. MA4000 project – MA4000 service stopped working sometime during the week of 6/16/08.  Working with NEC to resolve problem.
3. Virtual snapshots — Setup automatic snapshot service for VULS and VUSM.  VULS has 5 snapshots created on Sunday evenings and VUSM has one snapshot created Sunday mornings.  Created a one time snapshot of colo server ‘Flanders’ for Owen prior to them upgrading software on the server.  After the upgrade and Owen was satisfied the upgrade worked properly, the snapshot was removed.
4. Virtual Desktop project — Spent many hours researching & meeting with different vendors solutions for a virtual desktop.  The project team met with representatives in person or over the phone of Dell, Microsoft, Sun, VMware and HP.  Also met with OGSM and reviewed their VMware VDI installation.  We are in the process of setting up a pilot of the Sun virtual desktop solution.
5. Virtual ESX server — Completed setup of ‘Mammoth’ ESX server (its-hcvm12.an).  Setup ESX cluster between its-hcvm11 & its-hcvm12.  Removed all virtual servers from its-hcvm01 and reloaded ESX software using its-hcvm03-ts host name.  This server was placed in the development ESX environment.
6. Virtual Servers — Created 9 virtual servers in support of the following projects: UC2K7, Sharepoint, & Virtual Desktop.
7. AIMWorX & VPD — The AIMWorX alarm client stopped working for VPD on 6/2/08.  Worked with Terry Cavendar to make firewall changes to correct this problem.  Terry found encryption errors between the ITS and VPD firewalls that needed correcting.
8. DOAWeb — Attempted to convert the DOAWeb physical server to a virtual server.  Multiple attempts failed from the 194 network and the App Hosting Admin network.  Currently researching the error that caused the failure.
9. Operator Services — Every July the Medical Center has hundreds of new residents & clinical fellows that have to updated/added to AIMWorX.  This year they requested an auto load into AIMWorX.  I created a Perl script to read the file and populate AIMWorX accordingly.  This script will be run in the production database the morning of July 1.
10. AWAPI update — MIS moved the files used to update AIMWorX to their new secure ftp server.  The ITS firewall was changed to all the AIMWorX server access to this server without the use of Secure Remote.  Also, the automatic Perl script was changed to login to the new server and download the new files.
11. New AIMWorX Server VLAN — Move AIMWorX3 server to the new VLAN for testing prior to the production server move.  Had problems with WINS not being updated with the new IP address but was resolved once the firewall rules were changed to allow the WINS ports outbound traffic.  However, outside the VU IP range access does not work.
12. Windows Server 2003 sp2 upgrade to Magic-DB2 server.

This month we made a decision on a new tape library and have purchased it.  The new library is a Quantum I2k and should be here within a week.  In preparation for the arrival of the new hardware we have arranged installs of new HBA’s in our existing storage nodes and developed a strategy as to how the new device will be attached to our environment.  The addition of the new library to our backup environment will complete the life cycle replacement of our Sun STK L700 as well as give us a lot of additional capacity in our backup environment.  The main concern will be cloning speed and this should be greatly enhanced as well.  The addition of a VTL or similar technology will most likely be necessary to fully keep up with our cloning operations.

I have authored another Perl script which will keep track of all of our backup clients and act as the definitive list for clients configured in our backup environment.  This script launches every night at midnight and creates a master list of clients used for reporting purposes.  A second Perl script uses this master file to create three seperate lists used for cloning.  Cloning is now accomplished one third of our environment at a time over the course of a month.  I will most likely attempt to do the entire environment in a single week once the new library is in place.

Other than that it’s been a lot of operational work over this month.

1. Wrote script and crontab for spectrum mysql db optimize and maintenance programs.
2. Added and autoextended tablespace for RHN.  Also ran utilities to check fragmentation on tables, indexes, etc. 
3. Fixed a listener security vulnerability.
4. Refreshed the diamondip test data twice.
5. Initated and installed oracle 11G client on Nagios for the check_oracle plugin.  This plugin checks the tablespace size and db status. 
6. Installed Oracle 10GR2, upgrades and patches on the test dipdb 2 standby server for the test DIP dataguard configuration.

May MARS

Well, after taking a much needed 3 weeks off in April and the first part of May, it was time to get back to the grindstone.  Now my nose is painfully sore… I blame Kevin!

So, without further delay, here are the "Wins" for May:

1.  Moved the VCMS database off of the VCMS server to the Linux Oracle server cluster  - Instead of timing out when attempting to do any performance reports greater than 3 days, we can now query and receive our reports for the past year in less than 10 seconds.  This is extremely helpful in doing trend analysis of our virtual environment.

2.  Cloned and moved the NDE server to Stevenson - The NDE webserver was cloned and moved to the SC datacenter cluster to provide redundancy of services.  Now working on getting rsync over ssh working to insure data is correctly replicated.

3.  Spectrum patching - I attempted to patch our Spectrum environment to get the Business Objects Report Manager working correctly but one of the patches was corrupted.  This prevented me from finishing the patch process.  The vendor has yet to supply the patch again for download.

4.  Moved the Diamond IP test application server and rebuild - The test application server was moved and rebuilt.  The production data was loaded on the test database and I am in the process of removing the pointers to our production DNS and pointing them towards the test application/DNS service.

5.  Develop automated Virtual Environment Billing Script - Started work on writing a script that will automatically gather the billable data (procs and ram amounts) from the VMX files for co-located VM’s.  Currently fighting what I call "Special Character Hell" to get around the multitudes of parenthesis, dashes, spaces, and slashes in VM names/directories.

6.  Continued work/enhancement of Diamond IP Production environment - Work continues on the production DIP environment to insure stability and to get external LDAP authentication working.  Additionally, RFC1918 DNS preparation continues.

Last and not least… this is the new main priority in my life right now…

Please give a warm welcome to Zealy Caitlin!

This month was mainly work on the NEC Hydrastor in an attempt to stabilize the environment and improve performance. Kenon and I have worked extensively with NEC to identify bottle necks in our backup environment in an effort to reduce our backup window. Reclamation on the Hydrastor continues to be an issue, the read only phase is very difficult to work around, especially with the very long backup times we have been seeing.

We added an additional two storage nodes to our environment and they should go production this evening. I am hopeful this may help some of our issues; however, some work will be needed on NEC’s part as well to improve the throughput to the Hydrastor. Some streamlining of the reclamation process will also be necessary.

I worked with EMC this past month to evaluate EBA. After using the product for that time we have decided that as the product currently stands it is not of much use to us. It gathers available logs from several pieces of the storage environment and rolls them up to a single pane for review. However, it consumed over 40% of the processor on our primary backup server. The functionality it provided was not worth the performance impact to our systems.

This past weekend I attempted to migrate a large remaining portion of our storage environment to the admin network. Unfortunately, despite information from EMC that only the RM server for Exchange needed to communicate to the Clariions, the RM application was broken by the move. We learned during this process that the Exchange servers still need to communicate to the Clariions as well. We will be working with the mail team to try and get the Exchange servers dual homed and resolve an application issue preventing it.

I also developed scripts this month to pull CO-LO usage information from our Clariion arrays. This information was requested by management to help determine what ITS storage resources are dedicated to other departments.

Finally this month we reviewed several tape libraries for life cycle replacement of our current Sun StorageTek L700. We have made a decision and presented it to management.

Monthly Activity Report
May 2008 

1. AIMWorX – Supported all AIMWorX users varying requests.  Copied all SMDR and CDR backup files to CDR for archive.  Processed the weekly update of cost centers and student information.  Ran a monthly process to remove any duplicate calls from the database prior to a billing cycle.
2. AWAPI — Changed run times of AWAPI update to 3:25am.  This allowed the process to run without being blocked by another AIMWorX process.  This is running automatically again.
3. MA4000 project – Upgraded the MA4000 software to the beta release of version 7.0.1.  This caused a problem with the software connecting to the database.  Discovered this changed the ports used by Microsoft DTC.  After talking with NEC, they suggested setting the default ports used by MS DTC to 5000-5020.  This was done by a registry change on both the MA4000 server and the Mom-SQL server where the database is stored.  These ports were also opened on the ITS firewall.
4. Virtual snapshots — Restored a snapshot of itsproject01.  Once the snapshot was done the server would not boot.  After much research, it was found that the restore of the snapshot renamed vmdk files.  These were renamed back to thier appropriate names and the server booted.
5. Virtual Desktop project — It was determined that since NEC only used RDP to connect to the virtual machines the security did not meet VU’s standard and the pilot ws stopped before it started.  Now working with Dell and VMware on virtual desktop pilot.
6. Virtual ESX server — Completed setup of ‘Behemeth’ ESX server (its-hcvm11.an).  Setup in Nagios.  Discovered this server had a different chipset that other IBM ESX servers, vmotion will not work between IBM servers of different chipsets.  This server was setup in it own cluster and an additional server with same chipset was ordered.  Created a new virtual server for the sharepoint project.
7. Software Store project — Uninstalled and re-installed the RPEG software using the latest version supplied by e-Academy. 
8. Runbook — Updated install dates and locations of servers on Rick Williams list of incomplete server data for life cycle replacement.  Used websites of Dell & IBM to find warrenty expiration and used this date to determine install date.
9. OS updates — installed Windows 2003 server service pack 2 on Mom-SQL & MagicTest servers.

1.) Coordinated 119 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand. I was assisted with this task for a week by my co-workers while I was out for vacation.

2.) I resolved 27 Magic Tickets. This majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry, mailing list owner changes and moving email from a user’s deleted account into their new account. This number was done some due to being out for training for a week.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Created 197 accounts for the PAVE program. This program is an annual task to create and maintain accounts for prospective engineering students each year.

5.) I have been researching the upgrade of the version of IPF on the IMAP servers. This task should be completed in June.

6.) Continued project to move ILO ports for HP and IBM servers to the admin network. Administrative usernames and passwords have now been changed.

7.) Spent time assisting the Asset Management team in preparation for an equipment audit.

Helix Server Hardware Migration: I moved the Helix virtual machines from the Intel resource pool over to the AMD resource pool. These two particular VMs had better performance on the AMD hardware due to a system call issue. I was also involved in changing the content NFS mounts over to a new network.

Web MySQL Patching: The BEVS01 server was patched to current RHEL4 standards. I also implemented a script to reset the client error counts at periodic intervals to alleviate issues with incorrect code and occassional vulnerability scanners.

Sun Identity Management: I am providing support for various aspects of the IDM projects, and this has consumed the majority of time for this month. I performed the CSM configuration for IDM gateway servers. This configuration is slightly different from our typical least connections load balancing method in that all connections are routed to the primary server until it becomes unavailable, and then connections are routed to the secondary. I also built the eight virtual machines for the LDAP component of the project. These were a complete rebuild from the current ELDAP01 and CLDAP01 since the file systems needed to be resized to avoid Nagios warning errrors when local database backups are being run. The VMs were resized from 10GB to 18GB. It took approximately two days to get everyone to agree on an acceptable file system layout. I also spent some time ensuring that each application on the LDAP servers could be started and stopped as a non-privileged user. As the LDAP servers were being configured by the IDEV team, I installed and configured the ESX servers at the VUH and Sungard data centers. I now have the VMs moved over to their respective servers, and I am in the process of verifying that all necessary connectivity is in place. I am still in the process of updating the server commissioning documents for these VMs.

Documentation: I spent a couple of nights documenting some procedures so that work can be shared among the team. The design specification for automating the IMAP to Exchange migrations; however, it needs to be tested and verified. I am also in the process of documenting a break fix procedure our Diamond IP management application.

CSM Support: I have also performed some intermittent CSM support for other App Hosting team members.

HYDRAstor
Since moving to the HYDRAstor, our time to backup completetion has expanded by nearly 100%. Backups that were previously taking 4 hours are now completing in approximately 8 hours. Once ITS identified this issue, we began looking at metric data to identify bottlenecks. In doing so we realized that we should be able to push more data to the HYDRAstor. ITS notified NEC about this issue, and they sent an engineer from Japan to collect information about our setup. They are currently reviewing the data provided by the engineer.
ITS also moved some large client full savesets to run continuously over the weekend, in an effort to assist in allowing us a window to run the HYDRAstor reclamation process. We have set up a model that we may or may not be able to use depending on how long it takes the read-only portion of reclamation to run after a week of data growth. It has been hard for us to identify this time frame, because we have had to constantly make changes to the environment which can significantly impact the time it takes to complete reclamation.
ITS created new filesystems from the HYDRAstor to support expanding our backup storage nodes by two. We initially had 12 different filesystem mount points spread across two Legato networker storage nodes. Now there are 40 different filesystem mount points spread across four Legato Networker storage nodes (only 32 of these filesystems are usable, the other 12 still have production data until our onsite backup retention policy expires in two weeks, then we will remove these 12 filesystem mount points).

Library Replacement
ITS has begun looking into replacing our current backup eL700 robot library. We are currently comparing the different vendor products to see which one will suite ITS needs best. This is a very important decision as the library is how we clone all of our full savesets for offsite disaster recovery copies of production data. ITS is investigating the different achievable throughputs to assist us in making the correct decision, including growth considerations.

Backup Environment
ITS has purchased two new servers to license them as additional Legato storage nodes. Theses additional storage nodes and licenses will enable us to push more backup clients simultanesly as we will increase our server parallelism value from 64 simultaneous backup streams of savesets to 128 streams. This allows ITS to consider such implementations as allowing more clients to start at the same time, which should decrease our complete datacenter backup time to completion.
ITS has also begun moving client backup times around to facilitate the completion of reclamation process to our disk back end solution with the HYDRAstor.

Graduation
In order to ensure the success of the graduation streaming media initiative, ITS identified network congestion issues associated with live streaming. Once we identified these, we took the necessary steps to corrent these congestion locations. The first step was creating a new isolated network in front of our main firewall, but inside the perimeter firewall. We then moved the primary network connection to the new network. We also created a new isolated network to facilitate the streaming mount points for archived media, and added this network to the Celerra. The access list for IP address that are allowed to connect to the Celerra streaming media share was modified to grant the new IP address on the new isolated network to see the archived media files.

IMSP Tools
In order to facilitate the retirement of the IMSP server that stores user options and address books for Mulberry users, ITS was asked to provide a web enabled tool that users to authenticate to and export their Mulberry address book that is stored on the IMSP server. ITS has to first establish a file transfer mechanism (SSH), and installed the software for the transfer mechanism. ITS then developed a tool that transfers an authenticated users configuration directory, which includes the address book, and parses for valid email addresses. Once the parse is complete, it then either does direct MySQL insert statements and loads a users VUwebmail address book, or it generates two files, one that a user can import directly into Microsoft Outlook. The decision on what output is generated is based on user selection after authentication.

1. Move Runbook to Altiris
   1. I created an import rule to automatically import rack and vtag information from NOC database into Altiris
   2. I created an import rule to automatically import server information from current Runbook into Altiris
   3. Created a ASP.NET web page to query Altiris and return server name, OS, OS Service Pack, Tier Level, Service Status, Primary Admin, Resource Status, and Responsibility
      1. Created a ASP.NET datagrid to edit server name, Tier Level, Service Status, Primary Admin, Resource Status, and Responsibility

22 Closed Magic Tickets
3 Security Incidents
26 Mail Migrations

Exchange

HRSurvey user creation and setup for Rich
Utilized for an employee survey that is being performed.

Reconfigure of MAILBE16 Storage Groups and Mail Stores
This was done to assist in resolving performance issues. The reconfigure lowered the IOPS overhead caused by multiple mail stores on the same drive causing the drive heads to move back and forth between them. This also allowed us to balance the Mail Stores between MAILBE16 and MAILBE17.

Communicate concerns on IMAP -> Exchange Migration tool
Communicate concerns and push for fixing of the IMAP-Exchange .NET migration tool to not have a dependency upon the user/password list. This is to avoid issues once user provisioning moves to IDM. Also pushing to have the Unix side of our new migration process automated.

Cleanup of MAILBE16 reconfigure
I was here for the start of the reconfigure and assisted in getting the drives configured and setup for the mailbox moves. I was gone for 5 days following this at a school camp with my son. When I got back I took over cleanup and finishing the last vestiges of the move.

Stealthbits evaluation
Evaluating Stealthbits Monitoring for exchange to initially provide monthly metrics numbers for the University Exchange environment. This can also be used much more extensively with some time and work to assist in usage projections as well as troubleshooting and configuration management.

Troubleshoot/change primary delivery address for conference rooms
There was an issue with conference room auto-accepts and the UC2k7 environment that was caused by the its-exch delivery address not being accepted as an acceptable address on the proofpoints. Resolved it by changing the primary smtp address on our conference rooms to intemail.email.vanderbilt.edu.

RMSE -> RM upgrade
Finished at the beginning of the month. This is the backbone of our DR environment for Exchange.

Troubleshoot Exchange Backup times
Assist in trying to determine the issue causing Exchange backup times to take an inordinate amount of time on Hydrastore.

MISC

Assist in reviewing AD changes for Sharepoint Group Mail

May 2008 Monthly Activitiy Report

1)  Worked on an assortment of odd, challenging, and familiar helpdesk tickets.

2)  Lifecycle replacement of mailgate02 and mailgate03 with new hardware was completed.

3)  Lifecycle replacement of mailgate01.  Former mailgate03 has been prepared for testing of the process to migrate the PPS master.    Testing of process has been tentatively scheduled for June 2nd or 3rd timeframe.

4)  Google Apps Mail project begins pilot phase in June 08. 

5)  Project for mailing list replacement.   Continue to test LISTSERV eval.  Was able to resolve issues with masqerading, message delivery, and logging.  Continue testing to verify functionality, document differences between systems, and familarize myself with the product.

6)  Project to implement smtp and e-mail authentication.   Nothing new to report. 

7) Worked in conjuction with ITS Security on several incidences e.g. scams, threats, compromised hosts, etc.

8)  Managed abuse@v.e and postmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Created or edited Proofpoint rules in order to deter threat.  Investigated and reported over 75 spam false negatives. 

9)  Perform daily management of mail queues on mailgates.  Remove hundreds of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc.

10)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Completed installation and configuration of two storage nodes for Kenon. Learned how to setup bonding. Found work-around for problem with bonding Intel + Broadcom NICs. Worked with Barry McCurry in ND&E to configure the network ports and test initial connectivity.

Assisted Joy Saunders and Scott Hogan in locating hardware and documenting information for asset tracking. Submitted updated information to the NOC to update their inventory tracking database.

Worked with ND&E, Peter Woods, and Guy Sheppard to categorize the impact of migrating an old uplink for the High Density racks in Hill 148. Documented the impact analysis and submitted a change request for the proposal which was approved for implementation.

Worked with Kiran from the iDev team and Peter Woods to migrate the Iolan IP/Serial appliance to the .41 network and reconfigure Jtest1 to utilize the device. Made needed firewall updates to host-based firewall and submitted needed changes to ITS Security for the managed firewalls.

Worked with Chris Marshall to help plan and implement a new Apphosting Test Network. Worked with Kenny Elmore and Victor Herbert in ND&E to push the new vlan to all needed ports/switches, as well as Terry Cavender with ITS Security to get the firewall rules in place.

Migrated the OSIS-DEV server to the new AppHosting Test Network. Worked with Rick Williams and members of the iDev team to reconfigure firewall rules and permissions set for use.

Worked with other Apphosting Admins and iDev staff to migrate needed hosts to the new Apphosting Test Network. Coordinated cabling, firewall configuration, hardware relocation, and

Aided Lee Brewer from iDev with testing of the new IDM application servers. Reported findings to Peter Woods who acted on them to properly configure services to run within the needed guidelines for the project.

Found and corrected an issue with time discrepancies that was reported by Lee Brewer. When Selinux permissions were incorrectly set on the configuration file, and was therefore being denied access by the ntpd daemon. Reported findings to Peter Woods who checked other problem hosts for the same problem.

Verified communication from the HILLCSM to nagios.its for SNMPTraps being sent by the CSM module for monitoring. Awaiting time with Peter (probably after the IDM roll-out) to configure the server to monitor the incoming traps.

Implemented a test for an OSSEC server, 2 linux agents, and 2 windows agents for testing. Have been utilizing a book on OSSEC as well as the online wiki to tweak and configure the test setup. Testing is still ongoing.

1. Created vcms database on production dip oracle server.  Now, admins can get more than a weeks worth of stats in seconds.  Stats collection has been tested for up to 1 year successfully.  I increased the redo log size and defragmented the indexes and tables for the stats, history and events.  Set up rman backups.  Configured database for automatic storage management (ASM). 
2. Documented DiamondIP database, install and maintenance for test and production servers..  Documented VCMS database install and maintenance for test and production servers.
3. Refreshed the DiamondIP test database with the current production data. 
4. DiamondIP application had a runaway task.  I worked with DNS admins and Tech support to stop and cleanup the mess.  I worked to find the exact tables that were effected which in turn pointed to a runaway scheduled task.
5. Presented Enterprise level Service Desk solution to the Governance Board. 

• Virtual snapshots -

  ITS has released the Virtual Machine snapshot service. The "Snapshot Service" provides a snapshot of a real-time live image of a guest O/S system disk (or Hard Disk Drive) for a Virtual Machine. The frequency of the snapshot is recommended to be weekly at a cost of $150 yearly. The snapshot service is only offered to Virtual Machines. ITS System Administrators have created virtual server snapshots for multiple customers, including several for the Law School. These snapshots have proven beneficial in projects such as Sharepoint, where restoring from a snapshot has saved the project hours of rebuilding effort in re-establishing the test environment. The primary usage of snapshots is reverting to a previous state of the host operating system. This is useful for recovery from a bad change or a intrusion.

• Nettracker Database Upgrade

  ITS Database Administrators upgraded the Nettracker web server oracle client and Nettracker Oracle database versions applying several patches that will increase the security of the data kept on this system.

• Streaming Media Infrastructure - Enhanced

  ITS Staff have moved the Media Streaming Infrastructure components outside of the ITS Firewalls. This move should provide improved network bandwidth to the Media Streaming Infrastructure which better prepares Vanderbilt University for the upcoming commencement ceremonies. These servers are now protected by host based firewalls instead of the previous setup.

• Cohosted Web Server Patching

  The cohosted web servers were patched to current RedHat Enterprise Linux 4 standards. These are instrumental in enhancing the security and performance of these servers.

• Identifying web applications with security vulnerabilities

  ITS hosts web sites for many different departments with various levels of expertise in web site development and maintenance. As a result of an incident investigation, our administrators identified 987 sites that were open to vulnerabilities, and notified their owners (45). We also upgraded one user application that had a vulnerability and shut down 2 sites that could not be fixed and made the environment for all web site owners vulnerable.

• Operational and Administrative updates:
  • 151 Magic Tickets were processed in Applications Hosting.
  • 11 IMAP to Exchange Mailbox Migrations
  • 1 staff obtained Microsoft Certified Technology Specialist (MCTS)
  • 1 staff upgraded Microsoft Certified Systems Engineer to 2003.

Moved the encoding servers outside the firewall, in an attempt to resolve firewall performance issues.

Determined that our ISA deployment issues are directly related to the way our AD DNS is (or isn’t) configured.  Ultimately, we will not be able to deploy a multiple server ISA Array until the way we handle DNS is changed.

Determined that we have a processor bottleneck on our SharePoint deployment, the big question I keep running into is. Is the bottleneck there because we’re running in VM’s?  I think so, our biggest issue is excessive processor queue length.

Octel Replacement

We have had demonstrations by the top three candidates for replacing the Octel voice mail system. Each one had its good qualities, and each one had its bad. Over all each one will be a good fit with the university. It is hard with today’s technologies to find a comparable workhorse like the Octel, but for a 5 year solution they are all comparable.

Exchange 2007     

The Exchange 2007 project has come to the next phase, we are redoing the architectural diagrahm to get more users onto the system. The new architecture will be a mixture of Virtual Machines and Physical machines. This should give the best performance. We are also looking at different types of clustering that would be applicable for a 5,000 user deployment.

Office Communications Server

The requirements for this project have mandated more research on tying this implementation into the current pbx system. There were a couple of options, one is to build a Linux sip gateway, and the other would be to buy a pre-built gateway. Even though the first option gives more flexibility, the second option comes with configuration and pretested compatibility.

Proofpoint New Hardware Deployment

After purchasing new hardware last month, we have spent this month planning out how to implement the new hardware, and get the cluster upgraded to 5.03. This planning is due to the two upgrading attempts that had to be backed out. The new plan takes a cautious, more methodical approach, while getting the new hardware in place as quickly as possible.

Exchange DR

This has been one of the operation issues that have been open for most of the month. RMSE was the product that we were using, we now must upgrade to RM. This is due to a Fliar upgrade on the Clariion. This has had to be coordinated with EMC and their support, due to the different parts that have had to be upgraded. This issue is still open, while engineering works out one of their bugs.

Certifications

I passed the MCSE 2003 Upgrade Exam, Yes, that means that I am now totally certifiable. or is that certified. Now I will be looking to find a more generic Messaging certification to round out my skills.

1.) Coordinated 151 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand. I was assisted with this task for a week by my co-workers while I was out for training.

2.) I resolved 50 Magic Tickets. This majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry, mailing list owner changes and moving email from a user’s deleted account into their new account. This number was done some due to being out for training for a week.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing with project for IMAP Account Cleanup. The data for the accounts eligible to me moved has been removed from then accounts and set aside for removal in 30 days.

5.) I have been requested to evaluate and upgrade the version of IPF on the IMAP servers. This task should be completed in May.

April 2008 Monthly Activitiy Report

1)  Filter daemon issues have improved considerably after working with PPS.  Patches were deployed, PDR was enabled, some rules were cleaned up but the biggest impact was changing the maxsize rule to limit various properties of message attachments.

2)  Since deploying PDR, message volumes and spam volumes have decreased although vendors continue to report increased spam volumes.  Virus volumes continue upward trends due to several new virus outbreaks that continued beyond 1Q08. 

3)  Another attempted PPS upgrade of the production cluster failed once again.  Continue to work with PPS in order to resolve issues.  A new upgrade strategy has been adopted and folded into the project for life cycle replacement of mailgates. 4)  Google Apps Mail project is nearing pilot phase.  Completed configuration for inbound and outbound email routing, gateways, and configuring SMTP over TLS.  Executed suite of test messages in order to test message size limits, attachment limits, routing to/from test mail accounts to/from other VU mail systems and other domains, etc.

5) Resolved several issues reported directly by VUMC e.g. issue sending to med.wayne.edu, issue receiving messages from cardinal.com, issues sending messages to pfizer.com, etc.

6)  Project for mailing list replacement.  Nothing new to report.

7)  Project to implement smtp and e-mail authentication.   Nothing new to report. 

8) Worked in conjuction with ITS Security on several incidences e.g. scams, threats, compromised hosts, etc.   Created script to search all filter logs and return date, time, sender, recipient, and subject for any mesage sent/received from specific email address.

9)  Managed abuse@v.e and postmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Created or edited Proofpoint rules in order to deter threat.  Investigated and reported over 50 spam false negatives. 

10)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Vacation
I was out of the state in DC/Virginia for 8 working days this month as well as 3 weekends limiting the available time for changes/work.

Exchange DR RM
Been working diligently with EMC onsite trying to get our Exchange DR issues resolved. The flare upgrade on the Clariion from 24 to 26 caused the RMSE product to not be supported despite direct concern and questions to EMC of this. EMC has provided the RM product for us to upgrade to. Initial runs of the product were successful in both the test and production environment. Subsequent incremental runs in the production environment have not been successful and efforts to identify the issue causing this have been for naught thus far. Fifty plus hours have been spent this week by me alone working with EMC directly to keep the pressure on and working to attempt to resolve the issue.

Exchange DR Replistor
There was a vulnerability identified in the Replistor version 6.1 SP2 that we had installed. Previous installation of 6.1 SP5 crashed MAILBE13 and the change was aborted. This was a follow up of that with an upgrade to 6.2 SP4. Upgrade went smoothly with no issues. Replication of the Exchange Database Logs is working without problems again. This is one part of our Exchange DR strategy. This portion keeps the transaction logs synchronized between data centers. With any current copy of the database these logs can be applied to the database to bring us to a specific point in time.

DPM for Sharepoint
The DPM server is built and has been tested in backing up a file share. Test Sharepoint farm has been moved to the 10.2.170.x network to allow testing of DPM sharepoint backups. Agents have been loaded on the test sharepoint servers (itssharept01, itsproject01 and its-hcwnap32-ts) and registered with the DPM server. We are currently waiting on the test SQL cluster to be moved to the 10.2.170.x network also. Once this is done agents will be installed on these also. This will allow us to move forward with testing DPM’s capability to backup sharepoint at a high level and restore granularly.

Chancellor Public Folder Fix
Created chancellor@vanderbilt.edu public folder for all incoming mail to be delivered to. There is an application that is utilized to sort and manage these messages. It was not able to manage messages that were stored in Note format. Default behavior for Exchange 2003 public folder messages being delivered from external Exchange cloud is storing the message in IPM.Post format. After consulting with the MC Exchange team to ensure that our change would not affect them I applied a registry change to our Public Folder Back End servers that changed the behavior of incoming public folder messages so they would be stored as IPM.Note format (or look like mail messages rather then notes). This resolved the issue for the application managing the folder.

PRT1 Spooler fix and scramble to avoid crash
While oncall a space warning appeared for PRT1. In investigating this I noticed that there was a print job spooling to the data drive that eventually grew to over 3G in size. Quick work moving data off the C drive as the job grew kept the OS drive from filling up and crashing the server. Subsequent change that evening added a second drive to the box and moved all the print queues to this drive.

April Microsoft Patches
Applied to test servers and workstation groups in proper order for testing. Scheduled change for ITS production servers and Roland monitored the application of them while I was on vacation. Applied same patches to the Exchange Servers the following week. Adding one Hotfix associated with the DR RM project/work effort.

Magic Tickets
16 Magic Tickets closed
4 worked on and forwarded
Various AD tickets forwarded to the AD team. After a discussion I had with Brian Britt on the AD team last month before I went on vacation I have started forwarding any AD related tickets onto the Directory Services team.

DPM test environment setup
Built Windows 2003 x64 DPM server to test the System Center Data Protection manager 2007 ability to perform high level backups and granular restores. The server has been built and tested for backing up files. Waiting on Sharepoint test farm with recovery farm server to be built to continue testing

Brainstorm/Test IMAP migration alternative
Worked with Peter to figure out/test an alternative solution for IMAP to Exchange migrations. Migrations on the 18th of March were completed utilizing this method. This is to allow us to continue doing these migrations without having to have the users password. Moving forward with Sun IDM the process to get the users password will no longer be available, and having the users password in plain text was not a desirable solution anyways.

Performance Tuning for Exchange
Various changes were made to the University Exchange environment in an attempt to help alleviate performance issues that some users are experiencing. More plans are in the works to reconfigure the storage groups to a more optimal setup and rebalance the users across the storage groups.

Sun IDM/Microsoft Exchange/AD patch testing/Loader Code
Tested the need for a piece of AD loader code that was implemented in Aug of 2004 to resolve errors associated with exchange mailboxes. Testing was unable to reproduce the errors that the code was implemented to resolve. Determined that the Exchange patch to resolve the errors was thus not needed.

Manage Magic Tickets for Apphosting Group

22 Closed Magic Tickets

Cursory cleanup of apphost Vu Group Space

Assist w/uc2k7 setup and troubleshooting

NEC HYDRAastor
THe NEC HYDRAstor was promotod into a production fashion at the beginning of this month with 4 accelerator nodes and 8 storage node. Each accelerator node has a 2GB bonded 802.3ad uplink to provide NFS network mounts into the product, with an additional 2GB balance-rr uplink for internode communication within the grid to other accelerator and storage nodes.
It started out performing fine for about a two week period before we noticed some process flaw both internally as well as with the product intself. There is a READ-ONLY process in the product that identifies the amount of data that has been delted from filesystems, and upon completion, begins to reclaim the physical disk space. This process directly relates the amount of data that has been identified as deleted from the filesystems and needs to be reclaimed, to the amount of real time it takes to complete this process. Essentially the more data deleted from filesystems, the more time it will take to identify and remove it. This is very intrusive considering you can’t write to the filesystems during the time the product is identifying data to remove from the system. Furthermore since we reclaim about 800GB of data daily, it will take about 2-3 hours if we were to run this process daily. We are currently researching a “sweet spot” that will put ITS in the posistion of having enough space to continue moving forward with backups while the system is physcially reclaiming disk space, which may mean only running this process two times a week.
Once this issue was identified with our architecture and current usage patterns, we consulted NEC about some of the differences in understanding we had about the product. In an effort to increase our backup window, and decrease the amount of time to run backend HYDRAstor processes, NEC shipped four additional storage nodes with should increase our internode bandwidth by 50%, giving more processing power to backend processing like reclaimation, without sacrificing front end performance. However, until these four additional nodes are in the system and “stable”, NEC wanted us to have no production functionality for the product. We are looking at putting it back into production on Wednesday.

EMC RM
ITS upgraded the flare code on our Clariion SP’s. However, even after being assured by EMC support that this would not break our RM/SE application for our exchange disaster recovery scenario, it actually did break this application. EMC support informed us that we needed to updat our navisphere agent and our navisphere command line interface application that runs on the exchange server and works in tandum with RM/SE. However, upon attempting this installation it actually crashed our exchange back end server. We then called EMC back, and they noticed we needed to apply to non-standard patches. These patches got the application to install, however we still could not create a disaster recovery copy of our exchange environment. Opened another call with EMC, and they informed us that the flare code version we upgraded to on our Clariion is not supported with EMC’s RM/SE, only RM. So we adjusted our priorites to upgrading to RM. EMC sent two individuals and one RM specialists to assist with our upgrade. Upon completion we got a successful copy, but the next day it was broken again. It has consistenly breaking for a week straight with the same error. EMC has been on site for five days straight, and we will be here again today investigating root causes, and attempting to provide fixes.

SUN L700
The SUN Storagetek L700 has been having a whirlwind of phyiscal device failures. First the gripper on the robot failed, and SUN came out and replaced it. Ths L700 ran without flaw for about 4 hours before it was offline again. Opened another case with SUN, and they came out and replaced the MPC board for the unit. Once the new MPC board was in place, we had to replicate the configuration for the L700 on the new MPC board. Once we had all configurations in place, the L700 returned to normal operations. Later in the month a tape drive failed, and SUN replaced that as well.
It seems as if the stability our L700 is minimal at best. This seems to be caused by normal usage over the lifetime of this device. It may be time to look into life cycle replacement for this unit.

IMSP Retirement
ITS has agreed to develop tools to assist in the retirement of the IMSP service. This tool is an on demand address book export. It provides the capability for a user to authenticate and choose whether to export their email contacts out of IMSP into either the webmail database or a TAB Delimited file ready for import to exchange. This tool is ~ 70% complete. It currently has the ability to authenticate a user and get the indivudal email contacts. There are two major task left to complete this effort. 1) Automated file copy - the LINUX team is getting me an SSH server built on the IMSP server to assist in an automated file copy process to get the users flat text file address book from IMSP to the web server running the tool. 2) further code modifications to the tool that allow the export of distribution lists as well as individual contacts.

I took the SNIA SCSP test this month and passed.

Hydrastor Implementation:

We purchased the NEC Hydrastor this month and placed it into production. This involved doubling all of the hardware components in our existing eval unit and then doing several application side configuration changes to move the data being stored to a production media pool.

During our first week of production use we discovered that the read only portion of the reclamation process caused our backups to hang. We have had numerous conversations with NEC on the impact of their reclamation process on our environment. They have provided us with four additional storage nodes and some application tunes that may help mitigate the issue. However, we will still have to address a 3-4 hour daily read only window in our backup environment. We have several clients that take more than a full day to backup and the read only window causes these clients to abort. Possible solutions consist of temporary landing space for these clients and various modules to speed up the clients backup times. The most obvious solution, sending the slowest clients to tape, would unfortunately cause a contention for tape devices in our backup and cloning operations.

EBA Evaluation:

I have been working with EMC to conduct another evaluation of the EBA product. I have experienced several issues implementing it in our environment. The software was unable to monitor any of our network connections without a patch that they had to develop for us during this process. We have also been unable to monitor our fiber ports on the backup servers. It appears the cause of this problem is the absence of some SNIA libraries. Unfortunately, these libraries are included with vendor drivers for the HBAs and we currently use the Redhat provided drivers built into the kernel instead of vendor drivers. I am reluctant to switch to vendor drivers now that we have resolved several scsi issues we were having in the past within our tape library environment. We have also been unable to monitor our tape library with this product. The issue appears to be an SNMP problem and we are still trying to ascertain what is causing this issue. Finally the final issue is that the product doesn’t appear to be giving me the level of detail I had hoped to see for problem identification and resolution. Our main reason for evaluating this product again was for the ability to identify, locate, and resolve issues within our backup environment. Perhaps we will see more of this type of utility once we get some of the other issues resolved.

BlueArc Performance issues:

We have seen a drastic change in our usage pattern on the BlueArc NAS appliance. We went from a max utilization of around 50% to being maxed out almost continuously for several days at a time. The culprit appears to be an increase in usage from the ACCRE cluster, but I am still working with the vendor to make sure this is the actual cause and not a system problem of some kind.

Backup Operations:

Finally, backup has been going pretty well this month. The numerous changes to the environment last month appear to have stabilized the application and the tape library maintenance has addressed the stability of our library. This month has been primarily spent addressing bottle necks within our backup infrastructure. Hopefully when the Hydrastor is back in production we’ll see a vast improvement in our backup operations and we’ll be able to stay current with our disaster copies of our backup data. We also have two new servers on site to be configured as additional storage nodes. These two new machines should enable us to better utilize the Hydra and more evenly distribute the network and processing load of our backups.

Monthly Activity Report
April 2008 

1. AIMWorX – Supported all AIMWorX users varying requests.  Ran AWAPI manually several times a week since this process was broken with the SQL sp4 update.  Copied all SMDR and CDR backup files to CDR for archive.  Processed the weekly update of cost centers and student information.  Ran a monthly process to remove any duplicate calls from the database prior to a billing cycle.
2. AIMWorX templates — The ‘Hotline’ template was modified to correct an error.  On the backup/test server, the install template was modified so the template would not program the PBX.  This was requested by Taj Wolff and Dave Mathews.  Currently, this is only on the backup server until the modifications have been tested.  Taj, Dave, and the group were notified of the completed request.  The header template for data ports was modified to include the short description from the work order to give the customer more information as to what the order will be doing.  This was requested by Taj & Lisa after a department received multiple emails for port orders and they did not know which order went with which building request.
3. AIMWorX billing – It was discovered that the AT&T switched calls were imported multiple times.  Unfortunately, this was discovered after the billing cycle was run.  Created a SQL script to pull only the unique call records for this type of call and sent the resulting file to MIS for processing.
4. MA4000 project – Worked through the issues with the MA4000 server.  The first problem was the networks were setup on the incorrect NIC.  This was corrected by putting the correct static IP address on the correct NIC card.  It was also determined there was a problem with the ITS firewall letting DTC requests out but not back from the SQL server.  After the specific ports were opened, Marcus was able to login and run a PBX sync on the student PBX network.  Another problem has been found that the port being used for DTC request is not static.  Attempting to determine the range of ports used by DTC so the firewall can be setup accordingly.
5. Virtual snapshots — Created virtual server snapshots for multiple servers, including 2 for VULS.  Also fullfilled a request to restore snapshots for sharept & project virtual servers.
6. Virtual Desktop project — Research various virtual desktop software packages.  Specifically, VMware VDI.  NEC will be sending hardware & software to ITS for a pilot of thier virtual desktop solution.  Working on scheduling a meeting with Dell to discuss their virtual desktop solution.
7. Virtual ESX server — Worked with Kendra on installing ESX software on the new IBM server.  After several attempts at installing version 3.0.2.  Kendra discovered that the model IBM server we have must use version 3.0.2 update 1.  After downloading this version of the ESX software the install proceeded.  Now waiting on the storage group to complete the zoning on the SAN.  Server name: its-hcvm11.an.
8. Bluetooth on XP — The bluetooth technology does not natively work on a Windows XP laptop.  Did some research on the Microsoft and other websites.  Determined that the installed bluetooth drivers had to be unloaded and a different driver installed.  This was accomplished and a bluetooth headset was setup and worked with OCS.
9. Perl Script — Created a Perl script for Rick Carlton to download UCD station peg couts from AIMWorX.
10. Old Server — Clean the drives of the old AIMWorX2 server.  This server will be placed in the test lab for use by the AD group.  This server is out of warrenty.
11. Sotware Store project — Worked with ELMS software vendor to gain acces to their ftp site for cost center uploads.  The suggested using FileZilla ftp software after WinSCP did not work.  After setting FileZilla to the vendor specifications, connection was not optained.  Sent this information to ELMS vendor and requested assistance.  Waiting on their reply.
12. Certifications — Passed Microsoft SQL Server 2005 - Implementation & Maintenance exam

Sun Identity Management: I am making arrangements to get the hardware installed in the VUH and Sungard data centers.

Nagios Enhancement: The new Nagios server has been very stable, and we are continually adding new devices and services to be monitored.

Web Security Incidents: I spent nearly three whole days investigating, cleaning, and fixing various web applications on the primary web servers. Due to the open nature of this post, I won't divulge specifics of what occurred, but needless to say, there was an opportunity for some site owners to secure their websites. The situation was corrected with minimal impact to users and no impact to the core sites.

Cohosted Web Server Patching: The cohosted web servers were patched to current RHEL4 standards.

Streaming Servers Move: HelixA and HelixB have been moved to a new network for performance reasons. This was also used an opportunity to re-evaluate the local firewall rules.

Blog Services: There are now 55 organizational and 105 personal blogs on the blogs servers. I have started the testing to begin the upgrade of the blogs to WordPress 2.5.

PBX Pager Phase II: The Iolan TCP serial port was removed from Jprod1 and Jprod2. It will be moved to the Jtest1 server for development use.

Team Dynamics: Now that Troy has officially started for the team, I've started transitioning some of my operational work to his near-empty list. Troy is now the primary administrator for the Jprod servers, the webmail servers, the Sitemason servers, and the development Bluestem servers. We are also getting him acclimated to being on the other side of the operational tasks.

Primarily getting accustomed to the transfer from the NOC to the Unix Team by reading online documentation on policies and procedures and exploring the use of online tools available.

I have been granted access to more and more machines I’ve been trying to somewhat familiarize myself with the services, etc on each.

Assisted Roland with anti-virus integration to the new epolicy server by creating and setting up access for a test repository on the Linux back end.

I just finished up my change to remove the Trueport serial device from the Jprod servers. It took just a few seconds to shutdown the software, remove the device files, and disable the startup script. I also took the time to update my Pidgin client since I was up.

• I completed 70-291 and I now am an MCSE w/Security. Yay!

Things I figured out in SharePoint

• Connect to a WebDav SharePoint folder on a non-standard SSL port:
  • \\FQDN@SSL:PORTNUMBER\DavWWWRoot or \\server.test.com@SSL:9566\DavWWWRoot
• Map a drive to a SharePoint WebDav folder on a non-standard SSL port:
  • Net use DRIVELETTER: \\FQDN@SSL:PORTNUMBER\DavWWWRoot or net use S: \\server.test.com@SSL:9566\DavWWWRoot\
• How to create a UDC: http://dotnetbuzz.spaces.live.com/blog/cns!F3599E60D988EEB0!249.entry

Submitting InfoPath 2007 form data to SQL 2005 database: I created an InfoPath form to add/edit data in a simple SQL 2005 database. But after I published the form to SharePoint, I received this error message:

InfoPath cannot connect to the data source.

Access is denied.

Access is denied.

Thanks to this blog,http://www.eggheadcafe.com/software/aspnet/30510648/access-is-denied-but-win.aspx, I changed the form options to Full Trust. Here are the instructions on how to change an InfoPath 2007 form to Full Trust:

1. Tools Menu -> Form Options
2. Security and Trust
3. Unselect “Automatically determine security level”
4. Select Full Trust (The form has access to files and forms on the computer)

I was able to submit the data locally, but when I tried to submit the data from SharePoint, it told me I need a certificate. Urggh!

Altiris Unable to remove Windows Operating System Data Class from the Computer Resource - https://kb.altiris.com/display/1/articleDirect/index.asp?aid=38247&r=0.3199579

Today was my first day that I was down two on the Unix team.  I'm out one to position transfer and one to a family addition. The next few weeks are definitely going to be fun. I've already got a 7AM-4AM day going on Wednesday. Luckily my Magic queue is at zero.  I'm trying to get used to the new DNS admin tool that Raymer implemented before taking off. After many years of vi, it takes a bit to get used clicking through the new tool.  The on-call rotation is much quicker too.

I think that this is the first time since I've been in the Magic system that I've had no tickets in my queue.  I had to get a screen snippet to show off.

1. Added 11G datafile to Nettracker database.
2. Created Test Primary server for DiamondIP…twice.
3. Upgraded the ojdbc to version 5 and changed java parameters on the Diamondip app server  and db server to fix a jdbc error.
4. Added another oracle datafile to VCMS.
5. Upgraded Nettracker web server oracle client and Nettracker Oracle database.  Added security patches.
6. Moved datafiles on RHN to larger drive to fix diskspace warning issue.  Fixed a wait issue also.
7. Created a test database for VCMS on the dipdb test server.
8. Created a Service Desk Request to implemet ITS Collaboration requests into BMC Service Desk.  This required creating staff, groups, subjects, business rules, queries for 8 requests to include Groove, Centra, Video Conferencing and Collaboration Services Demo.

• New Nagios 2.10 in production!
  

  The AppHosting team migrated the existing hosts and service configurations from the old Nagios to the new version, and it went live on Friday March 28^th, 2008. The quantity and quality of the checks is a great enhancement to ITS and Vanderbilt. The server is performing very well considering that it is an extremely trim server compared to the current server. Below is some data from the migration.
  

  Old            New
  

  Hosts Monitored        319            329
  

  Services Monitored        657            884
  

  Service Groups        16            35

• New DNS Infrastructure in Production!
  

  Application Hosting rolled out the new DNS architecture. The new system is running a fully integrated DNS/DHCP/IP Management solution and went in with zero downtime. There were only six errors reports in over 29,000 records throughout 430 separate domains that were converted. This includes an upgrade from BIND8 to BIND9, a redesign of how the DNS topography was layed out, and introduction of a new management layer – using Diamond IP.

  Currently, ITS is serving up over 1.5 million DNS queries per hour without any hiccups. The transition was transparent to the vast Vanderbilt community, and the entire Internet.
  

• New Mailgate Feature Deployed!
  

  Application Hosting staff deployed Proofpoint Dynamic Reputation (PDR) this month. PDR uses a combination of local, predictive behavioral data and globally-observed reputation-analyzed by powerful machine learning algorithms-to block incoming connections from malicious IP addresses. This process will help decrease the amount of SPAM that makes it through the mail gateways.

• Chancellor Announcement
  

  Application Hosting staff participated in the new Chancellor announcement with responsibilities such as removing the redirect to the interim chancellor site and ensuring that the new content was web accessible.

• MS Project 2007 Server deployed
  

  As part of the Sharepoint rollout, MS Project 2007 is now online and appears to be working as described. The only functionality not deployed was some configuration related settings which will be addressed during the “enhancement phase” of the SharePoint project.
  

• Sitemason Disk Space Addition
  

  Additional disk space was added to the virtual machine providing the web service for the Sitemason frontend. This was a relatively easy task involving the creation of another virtual disk and addition to the web content volume.

• Server Decommissioning
  

  The following physical servers were powered off and removed from the Hill Data Center to be prepped for disposal. The services of most of these machines have been turned into “Virtual Machines.”

1. meru2
2. samsara
3. sitemsn-fe
4. sitemsn-be
5. nde-syslog
6. apps3

• Hera Decommissioning
  

  The Projector and Port Block applications have been migrated over to the main ITS website after the power supply failure. The Xserve is now merely providing a redirect page to the current locations. All existing links to these two applications have been found and corrected. The change to remove the Xserve hardware has been scheduled for Apr 7^th, 2008.

• MC Intranet
  

  Application Hosting Staff added another aliased virtual interface to VICC virtual machine to handle the new MC intranet website., and created another Apache virtual host as a placeholder for the content.
  

• Operational and Administrative updates:
  
  • 150 Magic Tickets were processed in Applications Hosting.
  • 14 IMAP to Exchange Mailbox Migrations
  • 1 additional staff obtained the VCP Certification (VMWare Certified Professional)
  • 1additional staff attended the “Legato Networker” training
  • 1 Staff attended SNIA Storage Networking Foundations course
  • 1 Staff obtained the “2003 MCSE+Messaging” certification
  • 1 Staff obtained the MCSA for 2003 certification
  • Annual Staff Evaluations were finalized and delivered.

Well… where to start….

I am proud to say that this should be the last month where I claim my main priority was DNS.

1.  BT INS Diamond IP successfully deployed - After some false starts, some anger, some frustration, and a whole lot of fatigue, we FINALLY rolled out our replacement DNS architecture.  The new system is running a fully integrated DNS/DHCP/IP Management solution and went in with Zero Downtime.  Out of over 29000 records and over 430 separate domains, I have received word of only 6 individual resource record errors.  Not too shabby.  Currently, we are serving up over 1.5 million queries an hour without a hiccup and the transition was transparent to the community.  Now I get to focus on getting BIND Views up and running and getting co-workers trained up.

2.  VMware Certified Professional - Yeah… it fell through the cracks and I got gigged on failing to take it prior to my review… I’ll accept that.  I will also accept that I took the test and PASSED.  Now, I need to figure out which additional alphabet soups I can append to my title… (RHCE, VCP, AEIOU, etc).

3.  The Solaris Oracle environment was patched up to current revs.  Of course, during one of the patch sessions, SunSolve decided to send its bandwidth out to lunch and make a 4 hour patch cycle take almost 10 hours.  Thanks Sun!  Also, DB-1 and DB-2 received some additional space so we don’t have to get called every time a backup of the database is kicked off.

4.  I really to tidy up some of my operational tasks/duties now that DNS is (mostly) done.

I would like to take this time to bid farewell to Kenon Ewing as he decides to play traitor and head over to the storage team from the vastly superior Unix team.  Just Kidding… I’m just jealous they are getting him and we are losing him.  He will continue to excel and his presence will be sorely missed.  Yeah, I know… he’s close enough to toss stuff at him, but still…

I would also like to take this time to give a heads up for next months MARS… it will be very incomplete/sparse… the baby is less than 3 weeks away!

See you all on the flip side!

I had a brief chat with one of the Network Security guys regarding a project that they are working one.  It's an issue that every IT shop has to deal with- password management. The biggest problem that most organizations face is propagating the administrative passwords to key people in a safe and efficient manner. In my experience, the big questions that need to be answered are:

• How are password changes to administrative accounts propagated to key staff?
• Are there at least two people that have each password? 
• Are there procedures in place for emergency account additions/removals/changes?
• Are appropriate measure in place to monitor how the administrative passwords are used?
• Has everyone tested to make sure that their access to appropriate resources works?

It's an issue that has hit home now that one of the Unix/Linux admins on my team is moving over to the storage.  A different role means different access.  Hence, we'll need to go through all of his systems and make sure that his access is appropriate for his new position. We'll be having what our team calls the "password party" in the near future.

Identity Management
The IDM project has scheduled for go live in March. In preparation for the go live, ITS has currently deployed two C-LDAP servers to provide directory services for Medical Center applications and an E-LDAP server to provide directory services for the general University community. ITS also has deployed IDM gateway servers for active directory provisioning events. ITS also increased the security of authentication services by ensuring encrypted network communication channels.

Storage Administration
ITS personnel changed, as I move from the system administration team to the storage administration team. This change requires knowledge transfer, which encapsulated time during this month.

Virtual Stabilization
Two new VLANS were trunked into the virtual infrastucture to support some new PBX applications that will run on virtual machines.
System Resource Reports - ITS spent time metering and gathering statistics about our virtual infrastucture depicting the host physical configuration and utilization as well as all virtual machine location and utilization.

Vacation
Personally, I was about one week this month for my first vacation. Upon return, I was fully revitalized and ready to hit the ground running.

Certifications

I recently found out that Microsoft was going to retire their upgrade exams on March 31^st, finding this out I broke down and took the time to study for the exam to upgrade my MCSA to a current certification. Panicking thinking that I had until the 31^st to get my other exam taken, I have been cramming to prepare for my last test. When I called to schedule the exam, I was told I couldn’t get in until the 1^st of which was a day late. Noting this to the person on the phone, she informed me that Microsoft has extended the retirement date, so I have been given a reprieve, but I have renewed my MCSA from a MCSA 2000 to a MCSA 2003, I will be upgrading my MCSE in April.

Exchange 2007 Prototype

This month has been more of a refinement of issues on the Exchange 2007 environment. After putting a second person on the system, Scott Hogan, he was able to help find some of the inconsistencies that were between the systems. Then a feature of Outlook 2007 of looking for Autodiscovery in DNS when it first starts up, had a brief issue of people on 2003 getting a login prompt. This was fixed with the removal of the A record for Autodiscovery. Vanderbilt.edu. This left only the dns entry for autodiscover.uc2k7.vanderbilt.edu. From this addition we also have concluded we need to move the VM’s off the test cluster onto a more dedicated system, or physical hardware, then we can proceed to move the rest of application hosting onto the 2007 system.

Office Communications Server/Office Communicator

With the start of the prototype for Office Communicator the server side was finally stable enough to get the whole department onto it. This allowed for instant messaging to stay within our network. Some of the features that will be useful is the tying it into outlook to set free busy, and being able to do PC to PC voice communications. Some of the issues that have been noted, and are configuration changes needed, is the ability to change passwords, the ability to have a preset group to pre-populate the contacts, transfer of files, and sending hyperlinks. This is still in its early stages of development, but shows a lot of promise of being a useful tool.

Tying this into the current VOIP system will take some concerted effort.

Proofpoint renewal

This month we spent a considerable amount of time discussion with Proofpoint representatives the different offerings that were available to the university. After all was said and done, we were able to save the university a considerable amount of money, improve the service by adding Dynamic Reputation Service, and refreshing the hardware to accommodate a large growth in traffic .

Octel replacement

The project is in to the phase of narrowing it down the candidates, and scheduling demos of their proposed solutions. After we see the demos we can better judge wich system will be the right fit for Vanderbilt.

Evaluation time

March also brought in with it the need to finish up team evaluations. This was made easier since we did a six month evaluation, and let people know where improvement was needed. This also made an improvement in the overall scores. A noticeable effort was seen from the members of the teams to take the necessary steps to improve their evaluation scores, which in the long run has made a better, more dynamic team.

Nagios Replacement: The AppHosting team is still entering hosts and services into the new Nagios server. The quantity and quality of the checks will be a great benefit to the team. The server is still very snappy considering that it is an extremely trim server compared to the current server.

IDM Rollout: My interaction on the IDM project have been relatively minimal until this point; however, team transitions have forced me to become more involved. I am now attending project meetings with Kenon so that the transition from Kenon to me will be relatively smooth.

Chancellor Announcement: I participated in the launch of the chancellor announcement. My key duties involved removed the redirect to the interim site and ensuring that the new content was web accessible.

Sitemason Disk Space Addition: Additional disk space was added to the virtual machine providing the web service for the Sitemason frontend. This was a relatively easy task involving the creation of another virtual disk and addition to the web content volume.

SSL Certificate Replacement: The certificates for www4.vanderbilt.edu, pave.vanderbilt.edu, and swdist.vanderbilt.edu were renewed for another year.

Hera Decommissioning: The Projector and Port Block applications have been migrated over to the main ITS website after the power supply failure. The Xserve is now merely providing a redirect page to the current locations. All existing links to these two applications have been found and corrected. The change to remove the Xserve hardware has been scheduled for Apr 7.

MC Intranet: I added another aliased virtual interface to VICC virtual machine to handle the new MC intranet website.  I also created another Apache virtual host as a placeholder for the content.

Website Migration Cleanup: Some of the DNS names used during the transition from the Solaris platform were removed from service.

CGI Functionality Fix: The Apache rewrite rules were modified to fix a problem with the CGI functionality in the test and production web environment. The previous rules were performing a "greedy" match and attempting to send some CGI requests to the Sitemason server. The rules have been corrected.

Server Decommissioning: The following servers were powered off and removed from the Hill Data Center to be prepped for disposal:

• meru2
• samsara
• sitemsn-fe
• sitemsn-be
• nde-syslog
• apps3

Team Member Evaluations: I finalized my evaluation input for the Unix/Linux team members. I'm very proud of my team performance. I think we have done very well as a team.