Planet AppHosting

January was a fairly busy month especially with regards to EAS Archive. One of the things that we needed to do was get an accurate count of messages, compressed and uncompressed data size with regards to what is mark for deletion after a folder synch with delete is performed. For all you EAS Administrators you probably have an idea with what I’m talking about.

I have listed a very useful SQL query that should prove useful to all you EAS Gurus.

The SQL Query below will give you an accurate count of all messages that will be restored using the EAS Admin console.

The part of the query that is important is refer.folderid > 0. All messages within the refer.folderid,  that are marked for deletion after the “folder sync with delete is run” are changed from a positive to a negative number. So for instance if I had stub called “Joe message” with a number of 3 and then I deleted the stub. After a folder synch with delete is performed, that number would change to  -3.This is how EAS knows not to restore the message when we restore(un-archive)  messages  directly to an Exchange mailbox or a PST file using the EAS Admin Application.

Email Consolidation

Vmail has been up and running for 6 months, and we just went over 20,000 users. This is a great step into a single consolidated system. Two new mailstore servers were rolled into production this month; this brings our capacity to 30,000 mailboxes for general consumption. (We still have 4,000 to be rolled out in a Continuous Cluster Replication for HA)

New message sizes are now in place, currently the Max Message size is 30mb. This goes alongside the size limits offered by Google and Microsoft. With part of our population on the Edu version of their mail systems, aligning the message size limitations made since. (Even though it wasn’t me or my team http://Vshare.vanderbilt.edu went live this month as well. This will allow those wishing to send documents or files over 30mb and up to 700mb the ability to upload the files and send just a link.)

Validation scripting if coming along, we are now able to consistently check know custom configurations on the Mailstore servers. This allows us to validate nothing has changed after patching. Along with our Process Improvements, documentation and Health Checks, this has proved to help mediate inconsistencies that have been h us.

Research has started into am Exchange 2010 deployment scenario. Options available in the 2010 version are becoming more and more aligned in our critical path. Some of the new features or On-premise and hosted integration, Archiving, and of course the new 70% lower I/O needed for disk performance.

Closed ?? (The search filter is not working) Service Manager Tickets

Closed 13 Magic Tickets

Migrated Frank Rosato to Vmail on Jan 6th. Moved George Delong and Kathy Masters to Vmail on Jan 20^th.

Scheduled 1^st Microsoft certification 073-236 test for Feb 11^th.

Joined the Cisco IPT/ Microsoft Office Communicator trial integration team

Embracing OCS admin role by watching training videos and troubleshooting connecting issues, specifically peer to peer file sharing via Communicator

January 2010 Monthly Activity Report

1)  VUmailguard. Vendor finally released patches in order to resolve issue with messages being rejected if duplicate RCPT TO sent during SMTP protocol.  Patches were tested in lab in order to verify and then released to production where they were tested to verify.  Patches have resolved issue.

2)  VUmailguard.  Discovered MySQL configuration file had been misconfigured and was retaining a large amount of MySQL logs that was slowly consuming free disk space on the PPS partition.  Opened call with vendor and requested MySQL configuration file error to be corrected.  Resolved issue.

3) VUmailguard.  Preparing to test Proofpoint Release 6.0.2.  Have read all release notes and manuals in preparation for testing in vanderbilt_lab cluster.  Many changes in this release that require testing in order to determine risk or to determine how to best implement new features.

4) Vmail.  Verified Exchange 2007 server operation after Dec. & Jan. Microsoft security patches were applied to all e-mail servers.  Additonally verified operation of Exchange 2007 servers after teaming was reconfigured.  Also verified Exchange 2007 operation after firmware updates were applied to em02, em04, and em21.

5)  Vmail.  Determined how to configure Evolution client when in order to access shared Exchange mailboxes.

6)  Vmail.  Discovered bug where Activesync NTLM authentication was being disabled on Exchange 2003 servers.  Servers were patched and issue was resolved.

7)  Email.  Discovered issue where e-mail originating from organizations using Microsoft Forefront were bouncing when sent to Vanderbilt University.  Worked with Microsoft to identify the issue and with ITS i-Dev in order to implement a work around.

Email.  Determined that mesages to Yahoo domains are being deferred due to issues during Yahoo e-mail server lifecycle replacement.

9) Email.  team completed configuration of all e-mail servers in order to raise MaxMessageSize to 30 MB.

10) Email.  Led project to prepare e-mail environment for routing of e-mail for bonenjoint.com.

11) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.  

12)  Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Investigated over 100 spam false negatives. 

13)  Performed daily management of mail queues on mailgates.  Removed thousands of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

14)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Well, here we are. 2010 if you can believe it. Still no flying cars!!

We have completed and moved all non-archived Exchange 2003 mailboxes. As of today, 1.28.10 we have 20,625 on Exchange 2007 and 7,316 on Exchange 2003. A few users are offering to abandon their archived mail and be moved to EX07 but the bulk of users remain. We are still working with Symantec to come up with an archiving solution but believe Symantec cannot produce and present the proper tools and software for what we need to fit our environment. It’s still a work in progress but are possibly looking at other 3^rd party vendors for a solution.

Exchange environment –  We still have a nagging thorn in our side with regards to our CAS and HUB servers’ randomly rebooting or blue screening at any given time. The Windows admin team has been troubleshooting this and working with Microsoft to find the root cause but no fix has yet to be found. Though we have a “cluster” of CAS and HUB servers in a pool and is seamless to the end users, it’s a problem that needs to be corrected and fixed quickly.  Though for the most part our Exchange environment has been very stable with little or no downtime apparent to the end user.

Tidbits – As of last Monday the 18^thI am back over in Hill Center and with the guys again. It’s good to be back with them and getting back up to speed again in supporting the day to day operations and functions of Exchange.  I took the 70-236 exam just before Christmas but missed it by 70 points. Drats! I have it scheduled to take again on 2.2.10 – ground hogs day. Let’s just hope I don’t keep waking up on that day hearing Sonny and Cher’s “I got you babe” song and having to retake it over and over. Ha!!

See you February and enjoy the Super Bowl.  My prediction: 31-24 Colts.

Admissions websites were moved from the main www.vanderbilt.edu server to dedicated VMs.  The new hosts are being load balanced by the F5 LTM with a service point of “admissions.vanderbilt.edu.”  Replication has been configured to conform with the rest of the web environment.  Rewite rules were added to the www.vanderbilt.edu servers to direct traffic for admissions to these new servers.

The upgrade previously scheduled for the Google Search Appliances is still on hold pending a security fix being issued by Google.  There is currently no status from Google on a release date for this update.

VMWare Tools were upgraded on all linux hosts in the production environment.

Upgrade testing for Drupal on the main ITS web servers has begun.  The environment will be upgraded from the 5.7 to the current release of 6.15.  This upgrade needs to be performed to address security patches which have been released since the servers were placed in production.  The testing is going well and a streamlined upgrade path is almost complete which should minimize downtime required to effect this change.

Work is still progressing on the JPROD replacement project.  The physical JTEST1 replacement (JTEST3) and the Virtual host (JTEST2) are currently being tested for load balancing of applications.  New applications are added to this environment as they are imported from the current JTEST1 and tested.  The production replacement servers have been received and will be built out once more testing has been completed on the test servers and a definitive build specification has been developed.

Public Affairs has received orientation on the tools and commands in the web environment provided for ITS Partner.  All Public Affairs developers needing the elevated privileges have had their accounts added and/or reconfigured within the web environment to allow them access to these tools.

Updates were performed on the primary backup server which had been differed in a previous attempt.  The primary backup server has now been successfully upgraded to the latest kernel version and latest PowerPath release available.  This upgrade was performed at the request of the ITS Storage Team.

The MySQL server on the Nagios host has been reconfigured to allow connections from external hosts.  This change was made so that tools can be developed and run for the main purpose of availability tracking and reporting.

1.) Coordinated 93 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 32 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continued project to verify and eliminate potential vulnerabilities on Unix and Linux systems reported by the Security Team.

5.) Assigned and completed task of decommissioning LDAPA and LDAPB.

6.) Assigned project for IMAP Account Cleanup. Currently thoroughly identifying and removing closed IMAP accounts and commencing process of identifying users that have moved to the Exchange Servers that still have IMAP accounts.

Vshare: Vshare was one of my surprise projects for the month. It is a temporary web-based file store for things that are too large to send as an attachment.  It has been in production for about two weeks now, and it has been well received. The service is still relatively new so I don’t have enough data to forecast usage patterns.

Drupal Best Practices: I have been creating a document detailing our required and recommended practices for installing and maintaining Drupal sites on the Vanderbilt networks. Since the Drupal installation is actually one of the last phases of the document, it can really be used as a general best practices document.

NetTracker Decommissioning: The change requests have been scheduled for the service shutdown. After the required waiting period, the servers will be scrubbed and removed from the data center.

Trove/Horde Internal DNS Service Decommissioning: DNS services on the internal DNS servers used by AppHosting has been shutdown. One virtual server will be removed from inventory, and the other will remain in service.

1.  OpenDNS pilot concluded. All DHCP within Hill Center was used to assess the impact of using OpenDNS.  While some issues were noted, for the most part impact was minimal.  Results sent to leadership.

2.  The Diamond IP environment received an upgrade to 3.1.31 with minimal headaches.  A first for that environment.  DNS/DHCP remained 100% operational during the upgrade window.

3.  Diamond IP Callout Manager now generating alerts.  SNMP traps are still in progress but email alerts are going to be used in the mean time.  Base email alert functionality is working in testing.

4.  Library has been tapped for the next (and probably last) DNS/IPAM Self-Serve.  Training expected in February.

5.  Bonenjoint.com successfully transferred to Vanderbilt DNS

6.  The NDE Syslog server migrated out of 129.59.1.0

7.  The ancient Shibboleth server was decommissioned.

8.  Vandyworks.com has successfully performed multiple DR tests which required time sensitive DNS changes.

Below is the DNS statistics for the ITS hosted Vanderbilt DNS servers.

CY09 DNS Stats:
Total DNS Queries Answered    : 13,841,143,051
                IP-SRV1                        : 10,370,389,340
                IP-SRV2                        : 2,276,267,703
                IP-SRV3                        : 1,194,486,008

Total Average Daily Queries    : 37,962,338
                IP-SRV1                        : 28,446,349
                IP-SRV2                        : 6,239,242
                IP-SRV3                        : 3,276,747

Highest Month of Activity    : September – 1,877,470,808 total queries
Lowest Month of Activity    : May  – 774,095,515 total queries

And for 2009 – Facebook is the top DNS destination domain for Vanderbilt clients!

I’ve spent the last month working almost entirely on two different issues.  Primarily on trying to get the new Email Archive product functional, and when I had time, I’ve been working on troubleshooting the repeated reboots/bsod’s of our Exchange 2007 environment.

Email Archive

After spending 2 weeks locked in a room on conference calls with various vendor’s tech support engineers , we were able to prove that our Anti-Virus product was not playing well with the Archive Cluster.  Unfortunately by the time we got a competent support engineer for our AV product on the phone (this took 3 business days) the decision was made to just remove all AV software from the archive cluster. 

Removing our AV software solved our performance issues, but sadly the actual Archive application is still generating errors.  The vendor has been of little assistance, and we’ve had to resort to opening a ticket with Microsoft to try and troubleshoot their applications errors.  We are currently running a continuous network capture on one of the cluster nodes in the hopes it will be running the next time the infamous error occurs.  Sadly it consumes 2-300MB of disk space per minute, while the capture runs.  So we’ve been fighting disk space issues, and will have to constantly monitor the host 24 hours a day until the next occurrence of the error.

Exchange 2007

Well we’re still troubleshooting this one.  I’ve opened my third ticket on this particular issue.  After unexpected 89 reboots and 15 blue screens, I’m not sure we’re any closer to a resolution.  It is my opinion that the reboots/bsods across all of our exchange servers are related, and that the actual blue screens, are a symptom  of the problem and not the cause.

At this point the best recommendation from Microsoft is to install Windows 2008 SP2, and an unreleased hotfix that resolves issues with TCP Chimney Offload.  Unfortunately, we already have TCP Chimney Offload disabled on all of our Exchange 2007 servers, so I don’t see how this hotfix even applies.

Transporter Migration Script
Wrote script utilizing the Exchange 2007 Tranporter Suite command line functions that allows us to migrate VUMail mailboxes without having to have the individual user names and passwords. It also doesn’t require us to copy the mailbox contents to a separate account and bounce through Exchange 2003. This script migrates mail directly from VUMail to the Exchange 2007 mailbox. This was found while investigating automated options that Idev could use to pass encrypted user credentials for migrations. The command line transporter tool has options that are not available in the gui tool.

Transend
Investigated/tested migration solution to see if it was viable in our environment. It was decided that this tool could be used in our environment with some dev work but was abandoned once the Transporter solution was discovered.

EV Archive work
Extensive work installing/troubleshooting EV archiving on Windows Server 2008 cluster. Worked with Symantec and Microsoft troubleshooting the various issues that we have been experiencing. Resolved issues with the clustering setup that Symantec was not able to give us answers for.

Exchange Backup
Fixed multiple issues with Exchange backups and identified fixes for the remaining VSS issue that is causing the system backups to fail on many of the Exchange servers.

Exchange Scripting
Updated exchange script that pulls daily mailbox information from the Exchange 2007 and University Exchange 2003 environment to include all the new backend servers. The data files from this script can be utilized to get mailbox locations from a point in time or to get a histograph of mailbox sizes.

ADSExchange Resource
Participated in various meetings to discuss current use and requested requirements for the replacement of the ADSExch tool with one that will be utilized for creating shared mailboxes and setting the appropriate permissions for shared mailbox access and send as.

Exchange/AD Universal Groups
Identified via Script the groups that would need to be converted to Universal group to resolve the intermittent deliver issues experienced when the Forest Root Global Catalog server is queried to expand mail groups. Manually identified groups that are a sub member of builtin AD groups that cannot be set to Universal Group and passed the information on.

UM Server Patching
Patched the UM servers with December and January windows Patches

Migrated 500+ “unused” vumail mailboxes
“Migrated” (created mailboxes and set new delivery address) 500+ VUMail users to Exchange 2007 who had not accessed their mail in 3 months.

This month has almost entirely been work with E-mail archive.  The product has been deployed and we are working through environmental and application issues.

Other than archive I have been taking care of operations so that Kenon can put most/all of his time into working with the Archive product.

1. Built MySQL Cluster.  Manager and storage nodes work.  Configuring SQL Nodes now.
2. Refreshed VCMS test server.
3. created and updated auto-maintain queries.
4. Installed Oracle 11G on Windows 2008 server for an in-house app.

Monthly Activity Report
January 2010

1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Corrected issues with PBX synchronization where AIMWorX was not updated.  Purged calls from AIMWorX for date range 5/1/2009 thru 7/31/2009.
2. AIMWorX Backup server — Updated backup/test AIMWorX database with a current production SQL backup.
3. Training — Worked through Windows 2008 Server training online.
4. Perl Scripts — Modified Perl scripts for month end billing process to meet added requirements requested by MIS.
5. Decommission Servers — Decommissioned two HP servers from the ESX environment, cleaned the harddrives.  The servers are in the Oak room.
6. Billing Issues — The BellSouth charges journal had incorrect data because the some charges were imported with incorrect dollar amounts.  Worked with Buisness group to correct the file.
7. OCS Calls — Working with Marcus, it was determined that OCS was sending call records through to AIMWorX.  However, the extension number is coming through as a ten digit number instead of the expected five digits.  This is causing all OCS to reject because the extension is valid.  The extension being appearing in the call record and translated by AIMWorX is ’615875′.
8. Password Changes — Changed all administrative passwords for servers that I have been assigned in the runbook, except two.  The OSIS TAPI servers will have to be done through a change because the administrative account is used to run required services.
9. Virtual Servers — Migrated a Law School virtual server to the ITS ESX environment and setup a weekly snapshot.
10. AVST — New voice mail servers were placed into production but at times the response from servers is slow.  Each server was responsible for 72 ports but according to AVST, the servers were only rated for 48 ports each.  One T1 card was removed from each server and a third Call Manager server was shipped to ITS (at no cost to ITS).  This server is currently being built so each Call Manager server will be responsible for 48 ports.

Wow – like me you are wondering where another month went let alone another year. Well Happy New Year and hope 2010 brings you great things professionally and outside the office.

We are still moving along nicely with the migrations. We have taken the past two weeks off from moving mailboxes due to the holiday’s but that’s ok, as we were moving close to 2000 a week previously to that. As of today, 12.30.09, we have 19,520 mb’s on EX07 and only ~ 7378 remain on EX03. I am thinking about another 4-5 weeks and we will be VERY close to having all Exchange mailboxes on 2007. That’s very exciting. We built yet another mb server, ITS-HCWNEM07 to keep pace with the moves.  ITS-HCWNEM03 had to be completely rebuilt as it was the last of the “original” MB servers and needed to be tweaked to conform to our new build document and some things we found out from Tom Chambers when he was here in late November. 

Non-project related: This is my first Christmas at Vanderbilt and was pleasantly surprised when I got my free turkey just before Christmas. I think that is great that the Vanderbilt community does this to say thanks to all their staff and faculityand any remaining leftovers get donated to the  local soup kitchens. Glad to be a part of it all.

See you in 2010!!!

1.) Coordinated 95 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 21 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Assigned project to verify and eliminate potential vulnerabilities on Unix and Linux systems reported by the Security Team.

1. Gathered server data from 2004 through 2006 for Management stats report.
2. Worked with Mark on DNS DiamondIP issues.  Uploaded dmp files.  Ran scripts.  Still working out issues.
3. Wrote auto maintain query for Cardiology.
4. Assisted Grad student with helpdesk data collections.  Set up Magic account.  Wrote query to obtain data.  Exported data to excel.
5. Upgraded Oracle Databases from 10.2.0.3 to 10.2.0.4.  This included VCMS, DiamondIP and RedHat databases. 
6. VCMS had a permissions issue on the stats table.  Resolved this.
7. Attended MYSQL DBA class.  Passed Certification Exam for MySQL 5.1.
8. Wrote query and gathered management data for email accounts at Vanderbilt.

Monthly Activity Report
December 2009

1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Corrected issues with PBX synchronization where AIMWorX was not updated. 
2. AIMWorX Work Orders — Found work orders that had been sitting on a particular step for months.  Sent various users information on these old work orders and asked that they work or cancel the orders. 
3. AIMWorX Billing Scripts – Perl scripts were modified to meet new specifications requested by EAI and correct output errors.  These changes are part of the HP3000 migration project.  Other scripts were combined into one for ease of journal creation.
4. AIMWorX Report — Created a new report for PBX operators to list work orders that were executed by them based on a date range.
5. Snapshots — Updated snapshots of the Mediasite and SCALA servers.  Setup weekly snapshots of its-hclnwb10 (WWW) to run on Friday nights at 10:oopm.
6. Training — Attended sys admin and administrator training for the new voice mail system, AVST.  Continued online Microsoft training for Windows 2008 server.  Attended VMware vSphere4 training.

November was a busy month for the Messaging Team here at Vanderbilt University.
Flurries of email migrations have been completed. It’s nice to see the Exchange 2007 mailbox count up to 12455 when compared to 7324 at the beginning of November.
During this month, the majority of my time has been spent working on operational issues and working with the team on trying to keep incident numbers down to a manageable level.
I did discover that the information store on one of our Exchange 2007 SP1 servers (pilot-server) kept stopping like clockwork at 6:00 PM. It seemed to always coincide with backing up the VSS (Volume Shadow Copy Service) when using EMC’s networker. Thankfully, I discovered the following Microsoft KB article http://support.microsoft.com/kb/955429. It basically states that you should download and install Update Rollup 5 for Exchange 2007 SP1. I installed Update Rollup 8 for Exchange 2007 SP1 and that solved the problem.
I was also involved in an incident where mail destined for Exchange 2007 servers from all Exchange 2003 servers were stuck in a queue named “undeliverable destination”. When looking at the queues it was apparent that all Exchange 2003 servers were unable to bind via DNS to the bidirectional 2007-2003 Routing Group Connector.
I was able to run the Exchange Troubleshooting Assistant on one of our Exchange 2003 servers and it stated it was unable to find one of our Hub Transport servers via DNS. All servers that I logged into were unable to resolve the FQDN of one of our Transport Servers. I added the Hub Transport’s FQDN to a host file on an Exchange 2003 box and was then able to bind to the 2007-2003 Routing Group Connector. After the change was made to the host file, mail began to be delivered from the Exchange 2003 server to Exchange Server 2007.
I’m still not 100% sure why being able to resolve all other HUBS except for the one was causing the problem, but obviously it was. The DNS issues were fixed and all of the queued mail began to flow.
It looks as though this started after the E2k7 servers were rebooted after November Windows patching had been applied.

SMTP Replacement
Tested delivery of vanity domains in the SMTP replacement project. We also started looking into stress testing the virtual infrastructure that will replace the current physical SMTP structure. We have passed all information for the test to the ECS team for implementation.

Archive Project
ITS has been working on getting a test cluster environment for the archive project as a part of the exchange consolidation effort. ITS has been diligently learning more about the application during the implementation of this test environment. It is being deployed as a one node cluster on VMware ESX, which also has its own learning curve to implement a clustered VM.

BlueARC LDAP
ITS has been working on retiring our legacy LDAP authentication environment. The bluearc has a problem moving to the new infrastructure due to small changes in the LDAP implementation of the new environment. ITS has been working with bluearc support to resolve this issue. This month ITS patched the bluearc with the fixed firmware and is now utilizing the new LDAP authentication environment.

1.) Coordinated 111 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 38 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continued with leased equipment project. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time. All leased servers scheduled to be returned this calendar year have been returned.

5.) Completed assigned project of decommissioning the Spectrum servers and moving this service to VMWare as part of the leased equipment project. The new Spectrum platform is now running on VMWare using two Linux servers and one Windows server.

Hope everyone got their fill with turkey, all the fixings and football. November has been a very progressive months when it came to mailbox moves. We are getting in groove and seamlessly moving 1500+ mailboxes a week now. As of today, 11.27.09 we have ~ 12,450 on EX2007 and just over 13,000 still on VUMC. We are just near the 50/50 point and will surpass 50% of EX07 users next week. We built yet another MB server, ITS-HCWNEM06 to keep pace with the volume of users we are migrating.  We also successfully moved all ~4200 mailboxes from ITS-HCWNEM03 to the other servers as that server will be rebuilt. So currently in production is ITS-HCWNEM01, 02, 04, 05 and soon to be 06 after the pilot and pre-prod users have validated its stability.

We decommissioned a few 03 back end server’s and everyone is looking forward to getting the count down to zero on the remaining backend’s. Guy, Scott and I are over in the baker building so we can focus on the Exchange 2007 project. It’s a nice office and much easier to concentrate being away from operations but still miss the guys back at hill center.

We have Tom, so and so as his last name slips me coming in next week to give us a nice overview of our Exchange environment. He will look at our servers, build documents and anything and everything related to our environment. Looking forward to his feedback.

We had a nice fall potluck with the app hosting team and it was nice to partake in some good eats and talk to each other on “non-work” topics. It will be nice if we can meet up again for the up coming Holiday.

So again, hope everyone enjoyed the long weekend of turkey, football, shopping and some rest. December will be another busy month and hope to have another 6000+ users moved to Exchange 2007.  See you around Christmas!!!

November 2009 Monthly Activity Report

1)  VUmailguard bug discovered.  Discovered Sendmail bug in VUmailguard servers that causes rejection of messages with duplicate SMTP RCPT TO: addresses.  Opened call with vendor, they confirmed bug, and have escalated to engineering.  Greatest impact is on list which has owners who occasionally muck up their lists so the lists have duplicate subscriber addresses.  When a post is distributed to a list with duplicate subscriber addresses, this condition causes a backlog of messages to be queued and can only be resolved by editing mail files and removing the duplicate addresses.  A script has been created to identify lists with duplicate subscriber addresses.  Also am working on a solution to monitor via logs to alert when condition exists.

2)  Wrote scripts which provide an output file with quota info for a list of VUmail accounts provided in an input file.

3)  Vmail.  Discovered issue where one of the VM CAS, its-hcwnem76, is imapcted by CPU spike on ESX server to the point it becomes unresponsive.  Recommendation is to move the VM or remove it from the pool.  It constantly alerts in Nagios when under high utilization with POPs, IMAPs, etc. is unresponsive.

4)  Vmail.  Discovered issue where old mobile devices assoicated with AD accounts that have not been removed are actually still trying to perform ActiveSync sync.  Impact to performance upon the CAS is unknown at this time but logs indicated sync is occuring frequently throughout the day for every device.

5)  Gmail for Life.  The new provisioning process has been completed and is performing at a level equivalent to the previous configuration. 

6) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.   Note I have found that with current anti mailware provided by ITS, many rootkits are still able to be installed.  Recommend that ITS look into provided clients with a suite product e.g. McAfee Internet Security instead of just McAfee anti-virus.

7) Resolved a complicated issue related to the departure of Alan Baker and messages being sent for the Vanderbilt University Card Office.  Involved reconfiguring lists, resolving issues with vendor supported servers which could not send messages due to network configuration, resolving message loops due to Alan’s out of office, etc. 

8)  Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Investigated over 100 spam false negatives. 

9)  Performed daily management of mail queues on mailgates.  Removed thousands of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

10)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

IMAP Upgrade: I spent several hours upgrading the Cyrus IMAP services on the development server to the latest version. This is almost complete. I’m still debugging a few key functions that are necessary. I’m trying to find a good mesh of application and library versions that do not require us to modify the user mailboxes. I also assisted with several IMAP-to-Exchange migrations.

Dot 1 Network Cleanup: I have spent some time comparing our Diamond IP address reservations with actual system responses. The team has found a few IP addresses that were still identified as in-use; however, the servers that were utilizing them had already been decommissioned. We also been updating contact information to reflect staff changes.

F5 Load Balancer Stabilization: We disabled the AutoLastHop feature on our load balancers so that the F5 hardware could identify a firewall failover in a timely manner.

1.  OpenDNS pilot is in place for select wireless networks.  Once the initial pilot is completed, all of ITS will be involved in testing.

2.  Diamond IP has been upgraded from 3.0.71 to 3.1.31 in the test environment.

3.  The test database environment has been been patched and upgraded to Oracle 10.2.0.4

4.  RHN was upgraded from 5.0.1 to 5.3.  This allowed us to handle the increased clients coming from MIS/EAI and to migrate the database off Oracle 9.2.0.8 to 10.2.0.4 – freeing up some resources in the virtual environment.

5.  VUMC DNS administrators received Self-Serve DNS training and are now handling their own DNS requests.

6.  The NetID environment has been shutdown and is on-track for decommissioning the first week of December.

7.  ISIS has been scheduled for Self-Serve DNS/IPAM training.  This is the last major group identified for training at this time.  CSB & Library remain candidates and will be reviewed at a later time.

8.  Diamond IP still has some critical issues open with the vendor

• The appliances did not handle DST changes resulting in the failure of the NTP service.
• The DHCP servers w/ collection did not handle DST changes resulting in a massive database logging issue.
• Agent Appliances still lose connection with the Executives resulting in lost collections and ability to publish DNS/DHCP configurations.
•  Callout Manager & generated actions still remain to be configured to send SNMP traps on 70/80/90% DHCP subnet utilization

9.  Business hours DHCP Failover/Disaster Recovery Exercise was delayed and is now tentatively scheduled for the 2nd week of December.

* Built an OCS test environment (Front End, Client, SQL 2008 DB, Communicator Web Access, Edge Server)
* Once access to a test Exchange 2007 server is established, I’ll be able to test Live Meeting. We are currently having an issue with downloading content in a Live Meeting on the WAN. According to a Technet article (http://technet.microsoft.com/en-us/library/dd441312%28office.13%29.aspx), ISA was supposed to be configured with a Web Publishing rule for OCS. We were originally not requested to create a Web Publishing rule, so we have made the change on the test ISA server and I am waiting on firewall rules to validate content download now works. We will then update the production environment.

Messaging Work

Worked with the Exchange team to get tickets under control.

Identified multiple places where more QC was needed to bring the Exchange 2007 environment into a consistent configuration

* Recommended and implemented Autodiscover changes to resolve issues that were being experienced in the community.

Recommended changes to IMAP and POP configuration for calendar access

Wrote script to fix mail enabled Global Group issue then subsequently identified a problem with this. All the groups that a Universal Group is a member of has to be a Universal Group also. I then updated the script to recursively change the groups to Universal Group. There is at least one group that is recursively a member of a builtin group that cannot be made a Universal group.

Extensive troubleshooting of “Vanderbilt Heart” mail delivery issues. The problem ended up being many fold, lack of Universal Group settings, AD mail enabled groups that were not externally mail enabled, send to permissions for one of the groups being sent to, and local junk mail filing of the message.

Work on loader issue that was causing failed migrations and bounced delivery of mail because proxy addresses/alternate BCA’s were being lost. Wrote script to be utilized during migration to identify any users that were having the problem and fix it so migrations can continue but also flag the issue.

Identify issue with inconsistent quota configuration (quota’s under 1G) for Exchange 2007 users. Initiated request to have these fixed by idev after supplying list.

** Found a few users with mailboxes missing and one that had both account and mailbox missing from AD that existed in IDM/VUNet. Requested a reconciliation of accounts/known exchange mailboxes from IDM-> AD. Have not heard back yet

** Identified possible issue with Security Scanner and new IBM Director. Found established and close_wait ports on exchange backends. Still need to follow up to see what can be done about this.

** Identified inconsistent listening ports across Mail Backend servers in the Exchange 2007 environment and inconsistent IBM Director installations across the Mail Backend servers. Work still needs to be done to identify the cause of the issues.

* Identified configuration issue on EM01,EM02,EM04, EM05, DREM10, and DREM11 and implemented a change to resolve the problem setting the “HKLM\System\CCS\Services\MSExchangeSA\Parameters\TCP/IP NSPI Port” to 50920 to lock down the port utilized for MAPI connections to these servers to match the port that was allowed through the firewalls.

* Fixed problem with internal use of Live Meeting that was preventing viewing and sharing of documents. There is still an issue with external (outside Vanderbilt) use of these functions.

Windows Work

Write script to cleanup stuck print jobs on PRT1 on a weekly basis.

Research/Testing with Roland for LDAP login to RSA/IMM cards.

Continued work on PRT1 replacement.

VUgroupspace

Built migration box for VUgroupspace migrations

Identified issue with RAID group setup of NAS space causing lower then expected performance

Identified problem with VUGFS-01 and VUGFS-02. Cluster fails when attempting sync copies utilizing xxcopy or emcopy, causing rearchitecting of solution to migrate the service.

Over the past month, and then some, I’ve primarily been building servers for the Exchange 2007 project. I’d say over the past month I’ve built or rebuilt at least 10 servers for the ECS team.

I’ve also successfully deployed Windows Deployment Services (WDS).  Which has turned out to be a huge time savings.  Typically, it takes us 8 –12 hours to build a server, that’s the OS, our base OS settings, and all standard applications.  Now that we’ve deployed WDS, we can deploy the same server build in approximately 90 minutes.

There is still some work that needs to be done around WDS, mostly just expanding our list of prepared server installs.  Currently we only have images prepped and ready for Windows 2008 Standard and Enterprise 64bit.  Hopefully in the coming weeks I’ll be able to find the time to prep a 2008 32-bit install, as well as various installs for Server 2008 R2.

In talking with the AD team, they are interested in using our WDS server for future AD server builds, especially once they start to roll out Windows 2008 R2 Core installs.

1. Create new database on DIP DB server for RHN Satellite.  Move data from version 9 to version 10.
2. DST error on INC database caused infinite loop within the audit logs.  Stopped the error and purged the data before major effect on database.  Opened issue with BT INS so they can fix this in a patch
3. working with Mark and BTINS on a data issue.  Supplying BTINS with dmp files.
4. Created auto maintain list queries.
5. upgraded test dip db oracle server to 10.2.0.4 and patched with latest security patches.
6. modified pl/sql code for student/hr upload to magic.  Resolving a vanity email address issue.
7. added tablespace to ITS database for backup db reports.

Worked with ND&E to replace the old nde-mrtg server with a new VM host after a crash occurred on the current system which was up for life-cycle replacement.  The new system is running RHEL 5 whereas the old was on CentOS 3.   All historic data was migrated to the new host once completed and many web application and configuration changes were made in order to move the older data to the newer software releases now being used.    Requested enhancements were also made in coordination with Kenny  to provide expanded reporting which had not previously been implemented on the original server.

Continued working with iDev for the testing of JBoss on RHEL in order to better plan for life-cycle replacement of the JPROD and JTEST servers.  Quotes were requested and received from IBM for replacement hardware.  The hardware replacement for JTEST should arrive in early December which will allow for better testing of the final incarnation of this replacement project.  The test environment will consist of a VM and a physical host which will more closely mimic the production environment and allow for better testing of changes before full implementation.  Due to insistence on utilizing the newest release of the JBoss server,  this life-cycle activity will not be completed until the next release 1st quarter 2010.

Groundwork was done for granting elevated privileges to Vanderbilt Public Affairs within shared servers in the Web environment.    There will need to be training coordinated with them getting these privileges because the way they connect to servers and the interface they use in doing so will be changed in this process.

Worked with April Scott extensively to ensure that web content for CSEFEL was correctly replicating and fully working both before and during a small conference was held by that department.  Due to the size and types of content being added much of the conference material had to be manually added to each server after disabling the replication of their content.

Nearly completed the rebuild of the BOINC replacement server.  Many custom code changes have been required from the svn release of the software from Berkely in order to fill in gaps in functionality of the management interface as well as addition of enhanced security measures to thwart spam and abuse.   Testing of the project creation script done last month has continued as changes were made to both the code base and database schema.   Completion of the final pieces should be done the first part of December.

Attended meetings and vendor demonstrations on technical solutions for the web environment enhancement project.  No definitive decisions have been as of yet.

Continued testing of the replacement SMTP servers for the Legacy Email replacement project.  We are close to finalizing testing and hope to have new servers ready for production by the end of December.

Worked with developers for the new drupal site for the Blair School of Music.  Helped in procuring a resource VUNetID to implement  ePassword authentication within the new site so that content can now be added and maintained by Blair.   Met with Jim,  the primary developer, to configure and test authentication once the resource account was obtained from iDev.

Storage Arrays/SAN

• We have deployed several new DAE of storage into the dedicated Exchange CX4-480 array.
• We relocated the EMC EDL to the other side of the data center to provide expansion room for the CX4-480.

Archive

• We continue to work on hammering out all of the details associated with migrating from the Zantaz EAS environment to the Symantec Enterprise Vault environment.

Backup

• Several Exchange machines have been added to the backup environment as we receive requests from the E-mail team.
• We discovered an issue with udev on the primary backup server.  The Windows/Unix team is looking into it.

Projects

• VUspace groups replacement – we have created the new vuspacegroups name space on the EMC Celerra and we will be replicating the ITS share to the new service in a week.
• VUspace users replacement – I have been looking at a few different solutions to include Live Mesh, and Sugarsync.
• Web Replication – we are in the preliminary stages of looking at products that are capable of providing the functionality as dictated by the requirements of the project.
• Archive implementation – we are preparing for implementation of the archive migration environment.

Monthly Activity Report
November 2009

1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Corrected issues with PBX synchronization where AIMWorX was not updated. 
2. AIMWorX Work Orders — Found work orders that had been sitting on a particular step for months.  Sent various users information on these old work orders and asked that they work or cancel the orders. 
3. AIMWorX Web server — Ordered and installed a new one year certificate on the AIMWorX-web2 server.
4. ESX Servers — Reloaded ESX software on ITS-HCVM04-TS and renamed ITS-HCVM01.  Then upgraded it to the latest version (v3.5 update 4) for insertion into the Co-Lo cluster.
5. Virtual Servers — Created three virtual servers for iDev in support of varying projects.
6. Server Decommission — Decommissioned three Hewlett Packard servers in the ESX environment.  These servers were at or near end of service contracts.  Servers: ITS-HCVM01-TS, ITS-HCVM02-TS, & ITS-SCVM01.
7. AVST Servers — Received three physical servers from NewCastle for the Octel Replacement project.  Completed the Vanderbilt operating system setup and had a security scan performed on each server.  Worked with the PBX techs to rack one server in MCE and place on production networks.  The two servers in Bryan are waiting a new rack to be delivered and they were placed on the production network.  NewCastle technicians were on-site to verify and complete their setup of the three servers.  I worked with the NewCastle technicians and Dave Arkle to get each server talking to each other.
8. Software Store — Installed a software update to the RPEG transfer agent that was requested by the vendor.
9. Training — Worked through part of the Windows 2008 Server on-line training by Microsoft.

• We’ve worked with Symantec to develop a migration strategy and finalize the SOW for our engagement.
• We resolved an issue with a failed volume group on the BlueArc.  Due to a drive having communication issues on the fiber channel loops, two drives were marked as “failed” by the controller resulting in an entire volume group going offline.  Once the drive was identified that was causing the additional traffic, we were able to shut down the controller and re-seat the drive.  We were then able to replace both the actual failed drive and the drive causing the additional I/O on our fiber channel loops.
• I have created a Perl script in our backup environment to monitor all failed backup jobs and distribute an e-mail with a listing of any clients that failed and which savesets failed.
• We have connected the production Archive servers (its-hcwnar03,04,05) to the SAN and connected them to the Exchange 2007 CX4 Storage Array.  Storage has been presented to facilitate the production deployment of the E-mail Archive service.

Well, so another month has gone by and actually didn’t get a chance to write last month. We have accomplished quite a bit on the Exchang project the past 6 weeks. We have started our bulk migrations of the VUMC users and are currently moving around a 1000 a week with hopes of uping that to 1500+ a week. We currently have 4 new mb servers that we have built along with tweaking and smoothing out our scripts and build documents that go with it. We find out that we need to hard code this regkey on each mailbox server

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeSA\Parameters\TCP/IP NSPI Port

with port 50920 otherwise when users go to create an Outlook profile and manually type in it hangs and times out.

We are currently have ~ 7330 users on EX07 as of today and 18609 on EX03. I expect to very close to the half way point this time next month if we continue to move 1000+ or even 1500+ users a week. 

See you next month and Happy Halloween!!

1. Gathered Help Desk data for Law School and assisted with data associations so the Law School could set up their own help desk application.
2. The ITS DOA helpdesk web site was not compatible with IE8.  I got the Vandy templates and upgraded the site to work with IE8.
3. RE-wrote a nagios check for the DNS database tablespace.
4. Refreshed test DIP database for the DNS server. 
5. Cruise Director for App Hosting Fall Festival.
6. Wrote auto maintained query for Cardiology. 

1.) Coordinated 110 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 38 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Assisting with data collection for project to determine future of VUSpace.

5.) Continued with leased equipment project. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time. There will be 2 leased servers returned in late October.

6.) Continued with assigned project of decommissioning the Spectrum servers and moving this service to VMWare as part of the leased equipment project.

7.) Completed project of coordinating yearly maintenance agreement for Sun servers.