Planet AppHosting

GNAV upgrade/life-cycle should now be complete.  New host has been in production since Dec without issue. The old backup server has been decommissioned and removed from the datacenter and the now replaced old production host has assumed the role of backup.  MySQL replication has been setup between the primary and backup so that the backup is much closer to a ‘hot’ spare than ever before.

VUSpace replacement solutions are being tested for replacing the current Solaris hosts with Linux hosts.  Web server rewrites have been configured and successfully tested for replacing the ‘people.vanderbilt.edu’ websites.  Two solutions are being explored for the SMB/CIFS shares for vuspace-personal.  Samba on the Linux hosts has been configured and tested for AD authentication.  The ‘user’ share has also been configured and successfully tested on the Linux hosts.   Implementation of the ‘user’ share from the Celerra has been configured, but has not yet been tested.  The EMC implementation may require changing the way users connect to their personal space, but this has not yet been verified.

The new solution for intended to replace ‘people.vanderbilt.edu’ using Wordpress’ multi-site capabilities has been implemented on a test server as ‘my-test.vanderbilt.edu’ and will be pushed to production as ‘my.vanderbilt.edu.’  Testing is currently being done by the Public Affairs webteam.

A new RHEL6 host has been built and is being used for testing of the new ‘my.vanderbilt.edu’ service offering.   This is the first RHEL6 server built by ITS to test implementing the standard security and authentication methods normally used on our current RHEL4 and RHEL5 hosts.  Some differences in the LDAP configuration were noted and with the exception of mod-security for web services, no issues have been found in following our normal policies in server configuration.

The iDev test environment is almost fully migrated to the proper ‘test’ networks for admin and production interfaces.  A change is scheduled to add a new interface to the backup environment to allow backing up these servers over the test admin network by request of the iDev team.  Once testing of migrated hosts and backups has been completed, the final few osts will also be migrated.  Completion of this initiative will bring the test/dev servers more into a state of mirroring production that they have been in the past.

MyVU now has a new home on the vanity servers.  Public affairs requested that the redirects be updated on 3.28 at 6PM.   Testing and approval of the new site was performed by Lacy with the Public Affairs webteam once the changeover was made.

DNS configuration in DiamondIP has been updated to allow recursive lookups for the MIS/VUMC BigIP GTM appliances.  This change now allows for sites managed by the GTMs to be accessed from off-campus with the full capabilities of the load-balancing features.  The domains immediately effected by this are vandyworks.com and myhealthatvanderbilt.com.

Attended Red Hat 6 on-site training class.  Passed RHCE test for certification.

Continuing to work with the new guys to get the acclimated to services and systems in our environment.

Submitted requests for moving the www2 replacement test and production hosts to behind the LTM appliances.  The other DB-# replacements will also be moved to corresponding IP once this has been completed.

GNAV1 and GNAV2 have been removed from service.  Decommissioning has been completed on GNAV2 for its retirement.  Reinstallation of GNAV 6.1.20 and initial configuration has been completed on GNAV1 to make it the new backup to the current production system its-brlnap01.  Once final configuration has been completed this host will be placed back into production as its-brlnap02.

1.) Coordinated 142 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 23 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day. This also involves coordination with the Library Annex retrieving tapes for file restoration as needed.

4.) Continued project for Group Mail Mailbox Decommission.

5.) Continued project for Personal VUSpace Decommission/Replacement.

Monthly Activity Report
February 2011

1.  AIMWorX – Supported all AIMWorX users varying requests.  Processed the update of cost centers and student information.  Updated Area Code/Exchanges in AIMWorX with missing numbers.
2. ESX — Received seven new HP servers for lifecycle of current ESX servers.  Four of these servers have been racked, cabled, and ESX vSphere v4.1+ installed.  Three of the four have have the ESX Hill LUNs presented to them.  One of the four has issues connecting to the LUNs because it reused an IP from another server before those LUNs were deleted.  Kenon is working on this issue.  two of the remaining three servers are waiting on ordered switched to arrive.  The last server is racked but have been told by Peter it need to be moved.  Waiting for it to be moved.
3. ESX DRS servers — The two ESX servers in the “Tulane” blade were re-installed with vSphere v4.1.  A VCMS server was being created when it was discovered that there was no license for it.  This is on hold until a decision has been made to purchase a licens or not.
4. vSphere vCenter Server — The vCenter Server software and database has to be upgraded to v4.1 before the new HP servers can be setup into a cluster for DRS/HA.  A virtual server was built for testing.  After three days of multiple issues an upgrade path was found.  The production VCMS server was updated to vCenter v4.1 on 2/24/2011.
5. AIMWorX Backup Server — This server has had hardware and software issues.  Two drives have been replaced and the server rebuilt to correct a ‘Defunct logical drive error’.  Also, all drivers and firmware were updated.  The server was error free for one week.  After a power outage in two Bryan building racks the error is back.  IBM has been contacted.
6. AVST Servers — Due to a power loss to the rack that houses three AVST servers, on the morning of 2/23/11, the System server lost power.  This caused a database table ‘crash’ that had to be resolved by re-syncing the CallXpress servers to the System server.  This was executed through an emergency change at 7:00pm.  During the morning outage users could not retrieve messages but messages could be left.  The database table ‘crash’ caused users to not be able to change thier passwords or greetings.  This was corrected with the emergency change.
7. Decommissioned Servers – Four servers had thier hard drives wiped and removed from the data center:  its-hcvm08, its-hcvm02-ts, Apphost2, & Apphost2-Dev.
8. Documenation — Updated the documents used by Business and Planning to execute a billing cycle in AIMWorX.

Added additional Mediation server to OCS 2007 R2 environment for testing with the Sprint Abrazo-E device.

Patched OCS 2007 R2 Application software.

Upgraded the Nextplane XMPP software to R3.2

Opened 2 tickets with Microsoft:

Case# 111022564830435 Cannot connect to live meeting (This appears to be resolved but additionally testing is needed)

Case# 111021170414593 MOC client auto-updater error (There have been reports that clients started updating on 2/28 but additional information is needed).

Installed Lync Front-end to a pilot server in production. Tested IM, Lync-to-MOC, incoming/outgoing to/from Lync. The OCS 2007 R2 audiocodes gateway is being used while in the co-existence phase.

1.) Coordinated 79 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 14 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day. This also involves coordination with the Library Annex retrieving tapes for file restoration as needed.

4.) Continued project for Group Mail Mailbox Decommission.

5.) Working project to lifecycle DB servers. The two production servers have now been handed over to IDEV team for testing. Also have a request now to build a 3^rd test server. This server will be a VM.

6.) Collected data for project in progress to replace or remove current personal VUSpace environment.

Monthly Activity Report
January 2011

1.  AIMWorX – Supported all AIMWorX users varying requests.  Processed the update of cost centers and student information.  Completed the process to remove Octel programming from AIMWorX.  Created some reports for AIMWorX using SQL Reporting Services.  Updated Area Code/Exchanges in AIMWorX with missing numbers.
2. SSL Certificates — Install new SSL certificates on Apphost1a, & OOI Pertrac.
3. SCCM – Installed SQL Server 2008 R2 on its-hcwnap29 (SCCM).
4. Training — Attended two training classes.  Microsoft Technical Center – Lync (2 days) and AudioCodes Gateway in a Microsoft Lync Envirnonment (4 days).

1.) Coordinated 87 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 19 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day. This also involves coordination with the Library Annex retrieving tapes for file restoration as needed.

4.) Continued project for Group Mail Mailbox Cleanup.

5.) Working project to lifecycle DB servers. Servers have been built and two test servers have been handed over to IDEV team. IDEV is still in a testing phase.

6.) Completed project to assist Asset Management team with server inventory.

Monthly Activity Report
December 2010

1.  AIMWorX – Supported all AIMWorX users varying requests.  Processed the update of cost centers and student information.  Migrated AIMWorXBE02 SAN drives to the new NS960 SAN.  Started the process to remove Octel programming from AIMWorX, approximately 8000 voice mail boxes have been removed so far.
2. ESX Servers – Roland and myself migrated all 243 production virtual servers to the new NS960 SAN over two weekends.  There was an issue with the NAS shares presented to the ESX servers that caused a VULS server to crash.  A change was executed by Storage to change the IP address of the NAS.  Snapshots are copied to a NAS share, when the IP change was made, a VULS server was in the middle of a snapshot.  This was the cause of the crash.  The servers VMX file was modified to not use the NAS share and the server was brought back online.  However, the NAS shares mount points were no longer consistant between the ESX servers.  After some research, all NAS mounts were removed from all ESX servers and then remounted via the VI client.  Removing and then re-adding from the Linux kernal did not completely solve the problem, but using the VI client did.  This stablized the mount points and snapshots were resumed.
3. Virtual Servers — Created four virtual servers to test Blackberry Express.  Loaded Windows 2008 R2 on each server.  Moved all VULS virtual servers from the Colo ESX cluster to the Intel Hill ESX cluster to provide space on the Colo ESX cluster for DR.  After moving the VULS virtual servers, the drivers/firmware were updated so the new SAN hardware could be used.  Modified the VULS virtual servers VMX files for self-snapshot service after the storage move.
4. Apphost1a Software Install — Installed Master Calendar for use by Dean of Students.  There were some issues but with the assistance of Josh Culley and Dean Evans & Associates (vendor), these issues were corrected.
5. Server Lifecycle — Apphost2 & Apphost2-dev were lifecycled to new hardware, its-hcwnap80 & 80-ts.  The old servers are now in a waiting state before they are cleaned and removed from the rack.
6. OSIS-CR02 amber light — As recommended by the hardware vendor, attempted to update drivers and firmware.  This process failed because the server would not boot from a USB drive.  However, shutting down the server in the process cleared the amber light.
7. Documentation — As requested by a user, documented the AIMWorX work order process for Application Hosting charges (Storage, virtual servers, etc…).