Planet AppHosting

• We’ve worked with Symantec to develop a migration strategy and finalize the SOW for our engagement.
• We resolved an issue with a failed volume group on the BlueArc.  Due to a drive having communication issues on the fiber channel loops, two drives were marked as “failed” by the controller resulting in an entire volume group going offline.  Once the drive was identified that was causing the additional traffic, we were able to shut down the controller and re-seat the drive.  We were then able to replace both the actual failed drive and the drive causing the additional I/O on our fiber channel loops.
• I have created a Perl script in our backup environment to monitor all failed backup jobs and distribute an e-mail with a listing of any clients that failed and which savesets failed.
• We have connected the production Archive servers (its-hcwnar03,04,05) to the SAN and connected them to the Exchange 2007 CX4 Storage Array.  Storage has been presented to facilitate the production deployment of the E-mail Archive service.

Well, so another month has gone by and actually didn’t get a chance to write last month. We have accomplished quite a bit on the Exchang project the past 6 weeks. We have started our bulk migrations of the VUMC users and are currently moving around a 1000 a week with hopes of uping that to 1500+ a week. We currently have 4 new mb servers that we have built along with tweaking and smoothing out our scripts and build documents that go with it. We find out that we need to hard code this regkey on each mailbox server

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeSA\Parameters\TCP/IP NSPI Port

with port 50920 otherwise when users go to create an Outlook profile and manually type in it hangs and times out.

We are currently have ~ 7330 users on EX07 as of today and 18609 on EX03. I expect to very close to the half way point this time next month if we continue to move 1000+ or even 1500+ users a week. 

See you next month and Happy Halloween!!

1. Gathered Help Desk data for Law School and assisted with data associations so the Law School could set up their own help desk application.
2. The ITS DOA helpdesk web site was not compatible with IE8.  I got the Vandy templates and upgraded the site to work with IE8.
3. RE-wrote a nagios check for the DNS database tablespace.
4. Refreshed test DIP database for the DNS server. 
5. Cruise Director for App Hosting Fall Festival.
6. Wrote auto maintained query for Cardiology. 

1.) Coordinated 110 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 38 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Assisting with data collection for project to determine future of VUSpace.

5.) Continued with leased equipment project. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time. There will be 2 leased servers returned in late October.

6.) Continued with assigned project of decommissioning the Spectrum servers and moving this service to VMWare as part of the leased equipment project.

7.) Completed project of coordinating yearly maintenance agreement for Sun servers.

Monthly Activity Report
October 2009

1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Corrected issues with PBX synchronization where AIMWorX was not updated and sent PBX tech information on issues that needed resolution in PBX.
2. Virtual Server – Moved Windows  virtual servers from AMD cluster to Intel cluster.
3. ESX Server — Upgraded 5 ESX servers to v3.5.0 update 4: its-hcvm01-ts, its-hcvm02-ts, its-hcvm05-ts, its-scvm03, & its-scvm04.  Inserted a third ESX server, its-scvm05, into the Intel-Stevenson cluster.
4. Virtual Snapshots — Compared snapshots being created against snapshots being paid for and found discrepencies.  Began creating snapshots for 3 virtual servers that were being paid for be not created. 
5. Perlscripts — Created two scripts to create text files for BellSouth charges & itemized calls journals.
6. HP3000 Decommission Testing — Transferred and /or created all billing files and journals to VIAFTP server for the month end billing in September 2009 for testing of process.
7. Higher Ground Backup NAS Share — The NAS share used for call recording backup, PBX-Recording, was alarming in Nagios because it was over 90% used.  It was decided to cut back on the amount of data moved to this share from 180 days to 150 days.  A ticket was entered with Amcom for Higher Ground to make the necessary changes to thier configuration files.  This process was completed and files removed that were older that 150 days.  However, this did not correct the over 90% issue for long.  It was discovered that the screen captures have increased in size from an average of 2.5GB of data to over 6GB of data per day.  This seems to be related to the opening of the internet to all PBX operator stations.  When the call begins the operator may be on a web page that contains graphics that increases the size of the capture file.

This is text

and below is an image:

September 2009 Monthly Activity Report

1)  Resolved several LISTSERV issues.  Messages sent from lists with duplicate subscriber addresses were being rejected after Proofpoint upgrade.  Appears that something changed in new version of Sendmail on Proofpoint that disallows duplicate recipient addresses.  Resolved by creatng script to identify lists with duplicate subscriber addresses and removing them.  Used similar script to identify lists with malformed subscriber addresses or invalid subscriber addresses and removed them.

2)  Upgraded Proofpoint from release 5.0.3 to lastest release.  This includues the addition of the Smart Search module and many bug fixes.

3)  Gmail for Life.  The new provisioning process for VUmailguard is progressing nicely and should be finished soon if no operation issues arise.

4) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.   Note the phishing scams seem to be on the rise.

5)  Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Investigated over 100 spam false negatives. 

6)  Performed daily management of mail queues on mailgates.  Removed thousands of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

7)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Worked on an assortment of odd and challenging helpdesk tickets

Rebuilt Exchange Servers a multitude of times
Created a SQL 2008 Cluster on Windows Server 2008 Ent
Created a partially functioning Windows Server 2008 Ent cluster for email archive
Build SQL 2008 SQL Reporting Services virtual machine running on Windows Server 2008 for email archive.
Supervised the upgrade of the test ESX Intel cluster to 3.5 update 4.
Scheduled vsphere 4 training

Enabled http tunneled streaming (RTMPT) of media from the Flash Media Servers.  This was done by request of the streaming media team to allow better access through firewalls.

Evaluated problems on some of the servers within the shared web environment in an attempt to improve stability and overall service availability.  Increased memory on several of the servers in order to accommodate the growing number of Content Management Systems (CMS) that are being deployed.  Reconfigured system logging do reduce ‘noise’ created by some services to increase the overall usability of the primary system logs.   New watchdog scripts have been put in place to monitor the size of logging files and rotate/compress the logs as needed to reduce storage utilization.  Installed and tuned the OSSEC client on the pair of servers handling the Vanderbilt Student Organization content.  The OSSEC client will be installed on all of the shared web servers once alerts tuning has been completed.   OSSEC is an open-source Host-based Intrusion Detection System (HIDS) with real-time log evaluation which reports to our OSSEC server.

All remaining shared web and MySQL servers have been migrated from the AMD cluster to the newer Intel 7100 cluster within the Hill Center ESX environment.  These were the last servers I am responsible for which needed to be migrated in preparation for decommissioning the AMD cluster.

Took over working with the outsource developers for the new Blair School of Music web site.  The site is being developed in Drupal.  Due to performance issues with the site, a dedicated virtual server has been provisioned and configured for use in this endeavor.  Having a dedicated server for this has been a great aid in determining the root causes for the performance problems.  These issues have been resolved and development is continuing to make progress on the project.  Once development is complete we will evaluate the feasibility of moving the site back within the shared web environment.

Six-month OS andf security patching of linux hosts has begun.  So far patching has been completed for one of the bastion hosts and the backup environment.  This patching is being done by service groupings for easier manageability  and will continue throughout October.

"Wins" for the month of September:

1. First up, the University and Medical Center now share a unified view of RFC1918 addresses, the reverse space for the Medical Center networks, and the foward zone for mc.vanderbilt.edu internally to the Vanderbilt community. This resolves a multitude of issues where data between the two organizations were out of sync causing conflicting name resolutions. Additionally, this supports the new secure relay for servers email implementation by providing proper reverse resolution for both VU and VUMC.

2. Additional departments were trained for self-serve IPAM and DNS. The Owen Graduate School of Management & the Vanderbilt University Law School now are empowered to administer their own IP and DNS space.

3. The Diamond IP environment was upgraded to to version 3.0.71 resolving some serious memory leaks present in the earlier version.

4. DHCP migrations from NetID to Diamond IP continue with the current migrations at 90% completed. This puts us well on track to have the NetID environment retired in November. That covers the major events of the month.

Most of my September was spent concentrating on replacing Vanderbilt’s current email archiving product with a new email archiving solution.
During one week of the month, I worked with a contractor who assisted with the implementation of the new archiving solution. Our plan was to set up the new application within an active/passive clustered windows server 2008 environment.
We were able to implement the new solution but were unable to take advantage of the clustered environment due to some unforeseen OS problems. We decided to carry on with a single server and were able to create a pilot group which included me, two other co-workers and two test user accounts.
There were some pitfalls along the way, but we were able to successfully provision and archive user’s email and retrieve the emails via OWA and the Outlook client add-in.
The last week of September was spent attending on-site training sessions that were geared towards installing, administrating and troubleshooting the new archive solution.
All in all I feel that Vanderbilt is headed in the right direction by replacing our legacy archiving system with the new archiving solution and I look forward to continuing this project and actually migrating users from our legacy system to the new.

EXCHANGE
Exchange CAS Build of ITS-HCWNEM77
Built new Exchange CAS in support of the MC FE collapse.
Troubleshoot Admin Network issues on ITS-HCWNEM76
Troubleshooting of intermittent admin network connectivity. It appeared to be a conflicting IP address but couldn’t get a ping back from another machine when this machine didn’t have the address configured. The issue ended up being a Cyclades switch that wasn’t responding to pings. Fixed the issue with a new ip on its-hcwnem76.
Assisted in testing MC exchange FE collapse
Helped the exchange team by taking part in the testing of the new MC Exchange CAS environment after the collapse.
Pushed for identification of Exchange Local FW rules for the new MC Exchange CAS’s
Windows Patches of UM environment
Patched the UM environment and troubleshoot resolve issue with ITS-HCWNEM75 CAS that was related to ASP.NET 1.1
Resolve SCOM Agent issues on Exchange boxes
Reinstalled SCOM agents after identifying local agent issues on EM01,EM04,EM75,EM76 and EM77
Memory Upgrade of 2007 Mailbox Servers
Upgraded memory on EM01-EM04 to 48Gb
Exchange Mailbox Switch Redundancy
Identified issues with mailbox servers teamed network switch redundancy. Documented the issues and put in changes to resolve them where possible.

WINDOWS
VUSpaceGroups Migration Project
Started documenting requirements, issues and procedure for migrating VUSpacegroups from Windows cluster to NAS.

*WINS1 Replacement
Replaced WINS1 production server with like replacement to resolve problems that were causing issues with our environment. Identified configuration difference between old WINS1 and WINS2 that was rectified in the replacement process. Documented replacement process.
Pertrac
Assist in troubleshooting/resolving issue with test pertrac server. Be available for when Pertrac needs connection to servers.
PRT1 replacement
Continued work on PRT1 replacement. All print queues have been created on the new print server minus the printers that the server does not have connectivity to. Working with Partner to identify LSP’s and contact them about the printers with no connectivity to ensure that they are not needed anymore to allow cleanup of them from the new server.

Antivirus Repository
Local testing of new Antivirus Repository has run into some issues that we are trying to identify the root cause of.
SCOM Rules
Identified issues with some of the custom SCOM rules that we had in place and reconfigured/documented the process for correctly limiting SCOM rules to a specific group or box.

PowerShell Scripting
Ping Test Script
Wrote a script to test ping connectivity. This was used to identify printers that were configured on PRT1 that had no connectivity.
Printer Information Script
Wrote a script to pull all the relevant information for each Print Queue remotely from PRT1 to facilitate the migration process to a new print server.
Assisted Roland with build doc automation script
Helped Roland with troubleshooting errors he was seeing in his base build doc automation script.

1. Researched and wrote scripts to purge logs and history from incontrol database.  These scripts are written and scheduled within the oracle database to purge these tables nightly up to a given number of days.
2. Updated the code on the IP Request web site by NDE's request.
3. Modified Service Desk Report for NOC Manager, Scott.
4. Purged mysql log tables on Spectrum One-Click.  This was needed becasue of space issues.
5. Added tablespace and flash recovery space to INCONTROL primary and standby databases.
6. Upgrading BMC Service Desk on the test servers.  Testing still continues.  BMC is currently looking into upgrade bugs that we found.

Over the past month I managed to upgrade our KMS server, so we can now license both Windows 2008 R2 and Windows 7.

Though I spent most of the month working on various issues/mini projects for the ECS team.  Ending the month with building the newly purchased em05, and rebuilding em04 so that it could utilize the additional RAM that was purchased… Gotta hate it when you build a server with 32 GB and only install Standard, then decided  you need more RAM.  Anyway, two mailbox servers down, eight or so go.

I spent quite a bit of time after hours setting up lab environment at the house, so that I can both work on my 2008 R2 skills, and to assist automating our build process.  So far I’ve gotten my 2008 R2 AD setup, and Windows Deployment services installed and configured.  Hopefully next month I can test doing some OS pushes, and then investigate incorporating our default settings, firewall rules, application installs etc, into an automated package.

Responsibility Transfer: I have been gradually moving out of my role as Unix Team Lead and System Administrator. Almost all of the daily operational duties are being performed by the System Administration team.

Exchange Support: The vast majority of my time this month has been spent work with the ECS team to support the new Exchange environment. The consolidation process is very complex, and it requires coordination from many people within the Vanderbilt community. The Exchange 2007 environment has been my re-introduction into the world of Microsoft Windows. I have been involved recently in monitoring server performance.

Enterprise Linux Reference Platform: We are continuing to meet and define the base RHEL standard.

1.) Coordinated 138 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 36 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail.

5.) Continued with project of identifying leased equipment along with monitoring status of this equipment. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time. There were 3 leased servers returned in September.

6.) Assigned project of decommissioning the Spectrum servers and moving this service to VMWare as part of the leased equipment project.

7.) Assigned project of coordinating yearly maintenance agreement for Sun servers as it nears time for renewal.

Monthly Activity Report
September 2009

1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Call records prior to 3/31/09 were purged.
2. Microsoft eLearning — Began online courses for Windows 2008 Server.
3. AIMWorX Work Order Template — Create a new template for Up/Downgrading OCS and added to the OCS template documentation.
4. Billing Perl Scripts — Created multiple scripts to create journal files for all types of calls as requested by MIS.  Sent test files to MIS.  This is part of the HP3000 dec0mmission project.
5. Virtual Server – Created a virtual server to replace the current WINS1 server.  This server was put into production without moving off the development ESX cluster.  Moved the new WINS1 server from development ESX cluster to production ESX cluster.  Created 3 virtual servers to replace 3 physical servers for Spectrum.
6. PBX Operators — Spent a couple hours with operators in an attempt to determine a cause for thier greetings failure.  Only on position had issues while I was there.  Ran multiple test to determine problem but came up with no answers.  However, there have been no problems reported for almost two weeks.
7. ESX Server — Received new physical ESX server.  Installed ESX version 3.5 update 3 on server, ITS-HCVM05-TS.  Moved servers from Intel-Dev cluster to new server test server.  Removed ITS-HCVM03-TS from Intel-Dev cluster and reloaded for production use, ITS-SCVM05.  Moved snapshot service to new test ESX server, ITS-HCVM05-TS.
8. AIMWorX Duplicate Billing — Created two files to credit calling card & switched calls that were billed twice on the August 2009 billing process.  These credits will be applied to the September 2009 billing process.

Exchange 2007:

The DR storage array a Clariion CX4-240 was configured and brought online this month.  The storage groups have all been configured and the dedicated SAN switches have been configured as well.  All that is left is for the Exchange host to be brought online and connected to the dedicated SAN switches.

An additional expansion of 6 DAE of storage to the Array dedicated to Exchange is scheduled for a week from this coming Sunday.

E-mail Archive:

We have spent a lot of time around the SOW provided by Symantec and have ironed out the process required for our implementation.

We have installed one archive server and we have a couple of admins using it for their mail.

Projects:

VUspace replacement projects are being ramped up for replacement of the groups and user services.  The groups service will be migrated to NAS storage.

Wow, what a fast month again. Our team saw some ups and downs within our environment. A lot of Exchange servers being put and pulled out of the environment for VM issues, physical issues and OS issues. I think we installed Exchange on probably 10 boxes and rotated out of production and back in just as many. We are currently in finished with the migration Exchange users but are catching our breath and doing some clean up on the backend before we really roll up our sleeves and begin to move the then some 22,000 VUMC mailboxes. In preparation of this move for VUMC, we had an issue when attempting to allow ITS-HCWNEM40 to be setup for 2003 public folders. Guy had to work with Microsoft one weekend day and in a nutshell, we are in a holding pattern to determine our best line of action to if any, add public folders within the 2007 environment.

A plethora of Magic and Service tickets still abound our queue as it seems when we resolve 2, 3 more come in.  They can be issues small in nature or great in size with visibility. We as a team do our best to complete them in a efficient and timely manner. One in particular which comes to mind is Kathryn Foote, Matt Halls assistant would accept a meeting request on his behalf then after a few minutes it would go tentative. We found a tracking option to uncheck on Matt’s client which corrected this but believe its caused by a MS patch that came out in June. I’m going to contact MS and determine if this is the correct behavior when both clients have this option checked. Hmm, I don’t think so but I’ll give MS the benefit of the doubt. sigh…….

So as August comes to an end, I have settled in much more than just 30 days ago. We even had a after work bowling get together with a few guys which was fun. Antwan is quite the bowler though. I’ll have to sharpen my skills before I bowl against him again.

See you next month   :-]

P.S. Can’t wait for the NFL season to start in just a couple of weeks.

1. Corrected Service Desk Business Rule code to eliminate monitoring errors.  Also fixed code to allow IPAM clients to get notification on ticket creation.
2. Worked with DNS admins on database issues to include creating export dmps, adding tablespace, adding flash recovery space and changing backup scripts.
3. Researched and downloaded BMC Service Desk upgrade application.  Rebuilding test environment.
4. Wrote SQL queries.
5. Organized desk.

We installed two new SAN blades into our existing Cisco 9509 directors.  This expanded our capacity by 24 ports and provided high bandwidth ports for the new CX4-480 storage array and Exchange 2007 mail stores.

We deployed a CX4-480 storage array for the purposes of supporting Exchange 2007 and e-mail archive.  We have presented storage to Exchange backend servers and MSSQL servers.  The ESX environment will also be getting some much needed additional capacity from this expansion.

Work continues to support the new Exchange 2007 and e-mail archive environment, additionally we have launched some initial project work into looking at VUspace lifecycle replacement.

I finished moving the SharePoint Test farm behind the F5, and setup SSL termination on the F5.  Some of hte service monitors are not ideal, but as soon as I have some more time to spend on it, I should be able to get things setup properly.  Then I can document the process, and start prepping to move the production SharePoint farm behind the F5.

I spent several days researching some blue screens that were generated on several of the ECS servers.  Initially we were not getting memory dumps.  We discovered two issues that were preventing the servers from creating memory dumps.

• The first issue is that the page file was not on the system partition. In order for the system to generate a User, or Kernel dump, an adequately sized page file must exist on the system partition.
• The second issue, was that the system partition was not large enough to place the entire page file on the system partition, so getting a User dump was not an option.  We could still get a Kernel dump, but that required us to make the following registry change, so that we could have a small (smaller than physical RAM) page file on the system partition:

1. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl\ Create a new DWORD named IgnorePagefileSize, with a value of 1.

After making the necessary changes we received our first memory dump on EM24.  After analyzing this dump, we were able to determine that an unkown driver had modified the PrintDbg routine.  Then I went through all the ECS servers looking for the same stop code, adn was able to determine that 3 different ECS servers had experienced 8 BSOD’s with the same stop code the first starting on 6/30/09.  Next we tried to track down what drivers were installed either on 6/30 or shortly beforehand.  We could not find any documented changes for driver updates on the three impacted servers. So the decision was to replace the 3 impacted VM’s with new VM’s.

New Position: I have recently been promoted to Senior System Administrator. As part of the transition, I am moving out of my operation role, and I am transferring those duties to the rest of the Unix team. I’m now supporting each of the AppHosting teams in my new role. I’m already diving into a couple of issues on the Windows platform, which is something I’ve not been heavily involved with for a while.  As such, I’ll be trying to absorb as much information as possible in the near future.

Exchange 2007 Support: I have been tasked with several items supporting the Exchange 2007 environment.

Sun Identity Management [SIDM]: The new SIDM LDAP services are live, available, and in-use by the general Vanderbilt community. The team is in the process ensuring that all clients are properly migrated to the new service.  Only a few clients are still connecting to the Solaris-based service, and we are working with the owners to move them over.

Enterprise Linux Reference Platform [ELRP]: The team has produced a server build with minimal packages to use as the base system.  We are in the process of determining the proper configuration for the installed packaging.

Wins for the month:

• First off, it’s was a year ago on July 31st that we began to actually gather metrics on our DNS servers…  Looking back over the past 365 days, here are some interesting tidbits of information you may or may not care to hear…

Total Queries
IP-SRV1                9,639,738,984
IP-SRV2                2,021,500,701    
IP-SRV3                420,262,993
Total                      12,081,502,678

Per Day Average Queries
IP-SRV1                26,372,205
IP-SRV2                5,524,468
IP-SRV3                1,811,722
Total                      33,055,400

Per Hour Average Queries
IP-SRV1                1,098,842
IP-SRV2                230,186
IP-SRV3                75,488
Total                      1,377,308

Per Minute Average Queries
IP-SRV1                18,314
IP-SRV2                3,386
IP-SRV3                1,258
Total                      22,955

Per Second Average Queries
IP-SRV1                305
IP-SRV2                56
IP-SRV3                21
Total                      383

12 BILLION queries answered…
383 every second of every day on average…

That’s a lot of “Who and where is this address,” requests!

• Speaking of DNS, we have started the process of presenting a unified internal view between the Medical Center and the University.  This will help clients in both institutions ti be resolvable within the Vanderbilt community.  In regards to email, this will go a long way to help us secure the environment and streamline our processes.  Initial testing has been successful and a roll out into production is scheduled for mid-September.

• The DNS servers were successfully patched to address CERT Vulnerability Note VU#725188.  There was no interruption of service to the community.

• 2 additional departments have been tapped for Self-Serve DNS/IPAM and will receive their training from ITS in the first week of September.  This is a part of the ongoing plan to empower selected departments to administer their own DNS and IP space while alleviating some of the load on ITS.

• DHCP migrations off the old NetID environment continue to be on pace with an earlier than expected completion.  We are currently at 66% of migrations completed with little impact to the customer base.

• The ESX environment will receive a nice boost in capability at the end of August with the addition of 8 more cores and 256GB of RAM.  The ESX development environment is slated for replacement with the current assets being redistributed to help out other production clusters.  This is an ongoing effort to increase our virtual hosting capacity and improve efficiencies.

Identified and resolved WINS connectivity issue with NCS Wins server
The issue appeared to be related to WINS server rename by the medcenter. Removal and readd of the WINS partner with correct name seemed to resolve the issue.

Resolve WINS database corruption
Identified wins corruption issue and executed break fix to bring it back online in a timely manner. Subsequently identified event viewer messages to key off of for SCOM and created new Crit/High alert that would appear in the NOC view of SCOM for notification.

Assisted in security investigation of WINS1

Decommission of EMC’s Replication Manager and Replistor
Decommision of RM server and uninstall of RM agents and Replistor software from Exchange 2003 Test and former production environments.

Patching of UM environment
Microsoft patching of the full messaging environment

Page file upate of UM environment
Standardized page file setup across the messaging environment servers.

Participated in buildout of new Exchange Machines in timely manner
Facilitating the build of the Exchange environment by building the new servers and ensuring consistency of the not only the software builds but the network setup of the trunked lines to minimize possible issues. Facilitated fixing of any network issues/differences found with the boxes.

Identified and resolved BIOS memory issue on rebuild of Exchange boxes
Identified BIOS issue on ITS-HCWNEM10,11 and 12 for adding of memory for re-use of these boxes. Installed BIOS and verified functionality of the servers.

Assisted in reworking of OWA rich web monitor conversion to work with Exchange 2007 environment
Assisted Peter in troubleshooting the old Exchange 2003 rich web monitor conversion to work with the Exchange 2007 environment.

Participated in Exchange Monitoring setup
Participated in meeting to ensure that monitoring of the current Exchange environment was satisfactory.

Troubleshot and resolved phantom drive issue on ITS-HCWNEM01,02 and 04
Worked with EMC to identify the issue that was causing the phantom drives to show up on these boxes. The issue was found to be the boxes fiber registration with the CX-340 without having any luns presented from that box. This caused the phantom drives to appear. The issue could be resolved by either presenting storage from that san or unregistering the boxes from that particular SAN. The latter was done leaving storage being presented just from the CX-380.

Cleanup of SCOM DB and change of default size/growth
Ran cleanup scripts and requested scheduling of these scripts on a weekly basis to keep database in a healthy state. Running of the scripts to cleanup the localizedtext tables keeps SCOM running more efficiently. We also increased the starting size of the database and increased the auto-growth to resolve size reporting issues of the SCOM db.

Identified issue with SCOM maintenance mode not working as desired
SCOM maintenance mode was not stopping all alerts from the associated objects with the client. It was only stopping alerts directly from the agent on the box. Found a powershell script that gives a gui tool to put a computer and all associated objects with that box in maintenance mode so no alerts are generated for it.

Put together SCOM documentation for NOC consumption
This documentation was generated for identifying how SCOM works. What the differences are between SCOM and Nagios, and also how to utilize the console to maximize our gain from it.

Decommissioned the old OSIS virtual machines which completed the project for migrating the OSIS environment to RHEL4 and from the AMD to the Intel ESX clusters.

Nagios has been fully migrated from a virtual machine to a physical machine.  This move was related to performance issues which arose after patching.  The new service has been upgraded to Nagios 3.0.6 and is also utilizing the NDOUtilities for logging to MySQL.   The migration went very smoothly overall.  The service has been stable again since the migration was completed.

Began work on building an additional Streaming Media Server.  The new server will be located on the VUMC network in order to better support their growing needs in streaming presentations, etc.

Groundwork has begun on planning for life-cycle of the JPROD servers.   To more completely mirror production services, the JTEST1 replacement will consist of two servers.   After discussing projected growth and comparing to existing system specifications,  a request for options and quotes has been sent to our vendor.

No work has been done recently to migrate the ND&E “weathermap” to a new server.

DHCP migrations are moving quickly for the DNS project.  Some minor feature changes were requested and completed for the reporting/scheduling page.

At the request of Network Security, the port-block pages have been slightly overhauled and fixed.  It was noticed that the pages were not working properly after being migrated to the new web servers.   So far no other problems have been reported.

Migrated a few smaller databases from mysql01 ro mysql02 in a continuing effort to retire the old MySQL4 server.

There have been no action items assigned in the GMail for Life project.

1.) Coordinated 146 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 45 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail.

5.) Assigned project of identifying leased equipment along with monitoring status of this equipment. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time.

* Created three new SQL Reporting Services reports for the ESX server metrics. I then created three ASPX SharePoint pages with page viewer web parts that reference these reports. I stored the SharePoint ASPX pages in the virtualization SharePoint site so that the entire team has access to the same information.

* I am now the Manager of System Administration in ITS.

Monthly Activity Report
August 2009

1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Call records prior to 1/31/09 were purged.
2. AIMWorX charge for backup – Setup new process in AIMWorX to charge for server backup.  This process consist of a Perl script to reformat the data from the Storage group and then an import into the charges table in AIMWorX.  This process was documented but has not yet been implemented.
3. Server Decommission — Had the old NEC Express server removed from the rack after the drives were cleaned.  This server was used as the AIMWorX SQL database server.  This server can be returned to the leasing company.  Two other IBM servers had their harddrives cleaned and can be returned to the leasing company.
4. PBX Operators & VUSearch — When searching the Vanderbilt website using the VUSearch from a PBX operator workstation, there was a 2 minute delay before data was returned.  Dave Arkle searched the firewall logs and then captured data and found that the VUSearch engine was looking to Google.com to record statistics.  The PBX operator workstations do not have access to the internet which is why the search took so long.  Peter Woods found the websites the search was trying to contact and the workstation host file was modified to point them to the local computer so the website lookup would fail immediately and return the search results.  This information was passed to Partner so the rest of the Operator workstations could be updated.
5. Server Setup & Modifications — Created a VM to replace EMC Management.  Added a NIC & RAM to requested Owen virtual servers.  Reloaded OS, installed new RAM and harddrives on two physical servers in support of the Exchange 2007 project.  Reloaded OS on two other physical servers in support of the Exchange 2007 project.  Turned off ESX DRS on three virtual servers to keep them from being VMotioned between servers.  These three virtual servers are now locked to different physical ESX servers.
6. PBX AIMWorX synchronizations — The AIMWorX sync with the PBX has stopped updating the database when the extension changes FPCs.  The log file from the days sync is now reviewed and extensions are manually moved.  During this process extensions were found to be programmed in multiple FPCs.  This list of extensions was sent to the PBX techs for correction.
7. VPD screen pops — On Saturday, August 15, I received a call from VPD stating they had lost thier e911 screen pops from AIMWorX.  After looking through the AIMWorX logs it was determined that AIMWorX was sending the alerts.  Setup the Alarm client on my desktop and received screen pops from AIMWorX.  Talking with VPD they said their computer had been getting harddrive errors for a long time and they had a new computer to replace the old one.  Even though I was not sure this was the problem I asked VPD to install the new computer.  I went to VPD and installed the AIMWorX Alarm client on the new computer, which did not resolve the issue.  I contacted the ITS security person on call to check the firewall.  Dave Arkle looked through logs and data being sent and blocked at the firewall.  Dave reset the link between the ITS & VPD firewall and reset the Alarm client.  This resolved the issue and e911 screen pops began working again.
8. MIS HP3000 Decommission — As part of the effort to decommission the old HP3000, new Perl scripts have been created to duplicate what MIS has done for years on this server.  Four of a possible six new script files have been created to summarize call charges to directly feed the G/L software and bypass the HP3000 processes.  These new files are in addition to the detail files we currently create.
9. Laptop conversion — Backed up all files from my laptop and then installed the Windows 7 operating system.

Closed 156 Service Manager Tickets

Closed 20 Magic Tickets

Exchange

Identified the types and number of resource accounts that reside on the Medical Center Exchange 2003 email servers.  Also provided documentation on how to move the different type of resource accounts (room, equipment and shared).

Part of Email Program - Legacy Email Systems team which is trying to reduce the physical imprint of servers via virtualization in Stevenson to alleviate heating issues.

Testing importing/exporting mail on its-hcwnem98.

Learned and started processing Exchange migrations for the previous two weeks.

Given OCS Admin role.

Windows

Requested a standard template build for a Windows 2008 (32-bit) Server VM named its-hcwnem98 and installed office 2007 and exchange management tools (32-bit) to be used for import\export functions on 2007 Exchange mailboxes.

Installed exchange management tools and OCS administrator on its-hcwnem99.  

Given task to install message stats on its-hcwnem98 and its-hcwnem99.

ITS

Configured Microsoft Roundtable for ITS conference room and provided documentation.

Met more ITS employees.

Introduced to ProofPoint.  

Misc

Received a Vanderbilt Medical Center USD 35 Award from Michael R Harris for perseverance, dedication, commitment and professionalism.

July was definitely learning month: Learning the environment, infrastructure, protocols, people and politics. I have at least to say we successfully migrated all Law users while dynamically modifying our script to move mailboxes. Everytime we modify the script(s) we decreased the time and increased the number of mailboxes moved. It’s still a work in progress but feel we are close in reaching out target goal of speed and efficiency as we have begun to also move the some 3000 VU mailboxes.

I also built two new Exchange servers and used the build template to follow and tweak. Note to self!! Make sure Windows UAC is disabled when trying to install Exchange otherwise you will be chasing a wild goose.

Here is a link from one of my favorite reference/knowledge sites Petri. http://www.petri.co.il/disable_uac_in_windows_vista.htm

Still tweaking the Mail flow Visio diagram which is a work in progress but a great learning tool that forces me to learn the messaging environment at a much better level.

I also went on call for the first time which I am still learning the in’s/out’s of the piece of the email team.

Overall it was a great first and quick month here. I have lots to learn but have some great co-workers around me to pick their brain and ask questions.

See you next month,

:-]

* Created Powershell script to automate the collection of VMware statistics.

* Created a Powershell script to send Kevin information on the number of operating systems we have in our environment for physical versus virtual computers.

* Coordinated and built 3 of the 9 new IBM servers for the Exchange project.

July has been a very busy month for the Exchange Team here at Vanderbilt. One of the largest aspects of moving forward with Exchange 2007 is migrating people from Exchange 2003 to Exchange 2007. After saying that, I have had the honor of migrating over 1300 people (with the help of powershell of course) this month alone. The aforementioned number not only included E2K3 users but also IMAP users.
I was also able to help introduce and implement the introduction of SMTP authentication services within the E2K7 environment, which by the way uses TLS port 587.
My next task for the month of July was to introduce E2K7 allowed relay services. This task was accommodated by introducing two network load balanced E2K7 Edge servers (behind an F5), but not Edge servers in the traditional sense. These Edge servers are being used strictly as relay services with the Edge’s having no knowledge of AD or the HUBs. To put it simply, the role was introduced, the receive connectors were set to allow anonymous users and were restricted based on IP. I then configured a send connector that smart hosts to a Proof Point appliance.
I was also involved with two companies that recently became part of the Vanderbilt Medical Group and whose email needed to be forwarded from their old email domain to the Vanderbilt.edu email domain. Not only that, but close to 150 email accounts had to be created in the E2K7 environment in order to accommodate all of their new email.
Finally, the archive project is finally coming into full swing. We are currently planning on migrating from a legacy email archive product to a new email archive product. This in its self will probably take up most of my time in the Month of August.

July 2009 Monthly Activity Report

1)  Resolved another LISTSERV issue.  Found that a configuration option value was carried over from the eval configuration.  With lists configured to require owner approval for subscriptions, the URL presented in the approval message contained a malformed email address.  This resulted in an invalid subscriber address being added to the list.  Issue has been resolved by correcting the configuration option value. 

2)  Created various scripts to gather inoformation related to the Exchange 2007 project. 

3)  Gmail for Life.  Nothing new to report.  Awaiting a source in order to begin testing of new provisioning process.

4) Legacy Email.  Provided assistance in order to configured the VM SMTP replacements and configuring a test environment.  Assisted with validation testing.

5) New Acquistion.  We are now routing email for the domains associated with the new VUMC acquisition, The CardiacCenter of Murfreesboro.  

6) New Acquistion.  As new employees from Franklin Bone and Joint activate their user ids, Exchange 2007 mailboxes are being created.  This has almost completed.  They have elected to initially forward email but at a date yet to be determined, they wish to route email for their domain also. 

7) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.   Note the phishing scams seem to be on the rise.

8)  Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Investigated over 100 spam false negatives. 

9)  Performed daily management of mail queues on mailgates.  Removed thousands of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

10)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

11) Worked on an assortment of odd and challenging helpdesk tickets

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

Google Search Appliances have been placed into production.  Old ultraseek fqdn has been replaced by a redirection page that collects referring pages which can be viewed through a limited access administrative interface for identification of sites which need to be updated to support the GSAs.  The old Ultraseek servers are currently being decommissioned.

Nagios performance problems remain an issue.  Due to the amount of resources the host is utilizing within the ESX environment, the service will most likely be migrated to a physical host.  Available hardware has been found and the initial build has been initiated.

The AmCom migration from 4.0.63 to 4.5 has been completed.  Fail-over to the new system proved to be a smooth transition.  The remainder of the OSIS environment has now been migrated to the new system.  Decommissioning of the old web/db servers will take place in the near future.

IMAP2 was renamed IMAP-DEV and moved from the production to the ITS-test network in order to test upgrading the cyrus-imapd packages.   Installation has been completed.  Currently testing migration and integration of the new cyrus-imapd build with the Solaris CSW bundled package.

Installed SLAMD client on several web enabled developement hosts for IDM load testing.  Setup firewall access for UDP ping utility which will be used to monitor network health between the individual LDAP servers during the load tests.  Worked with Roland to setup access to the UDP ping service on hosts behind the F5.

Worked to migrate databases from mysql01 to mysql02.   This is an ongoing effort working toward the eventual decommissioning of mysql01.

Closed 48 Service Manager Tickets

Closed 27 Magic Tickets

Migrated 212 law school accounts to Exchange 2007 on July 13^th.

Upgraded Message Stats version on its-hcwnap11 to 6.6
Backed up SQL DB on its-hcwnap11  and truncated log file that consumed the entire log drive. This had to be done prior to message stats install.

Set message stats default gathering tasks on all email servers. Also, successfully configured public folder gatherings. Currently in the process of configuring OCS and OWA statistics. There are two open case numbers with Quest Support, 739059 and 741003

Installed Terminal Server on its-hcwnem99. It has 5 concurrent licenses per device so more than 2 RDP sessions will be able to connect now on its-hcwnem99.

Joined the Rightfax deployment team with Chris Marshall and Erin Shelton.

Authored document to remotely wipe phone device from OWA.

Requested two VMs, its-scwnem27 and its-scwnem26.

Attended Nagios/Fruitty training.

Testing Jott application for iPhone.

Moved the test SharePoint farm moved behind the F5, though I still haven’t gotten it working yet. Mysites seem to work fine, but all the other sites, i.e. its, dar, etc don’t.  I’ve spent a large portion of my time learning the F5 this month, and have had quite a bit of assistance from Peter when he could spare the time.  Hopefully when I get back from vacation I can figure out why the core SharePoint sites on the test farm are unavailable through the F5.

I upgraded the ePolicy server to 4.0 Patch 5 to resolve a weak encryption vulnerability.  Only to discover a new more severe security vulnerability with patch 5.   I have an open ticket with McAfee on this, but have made very little progress.  They have escalated the ticket to tier 3 support, who in turn pushed it over to the security team.  I’m still waiting to hear back.

I opened a ticket with Microsoft about the recurring event 6614 that we’re getting on our production SharePoint farm.  Naturally once opening the ticket we have not seen the error.  Microsoft suggested that we remove some old application pools that are no longer in use, which we did, but ultimately the we have yet to receive the error again, so the ticket has been closed.

I applied Windows patches on all of the ECS servers, and in the process discovered several discrepancies, some of which have been resolved, and the rest will be resolved shortly. Hopefully next month we can do this as part of our regular WSUS patching schedule.

Built a few servers for the ECS guys, and during the process had to make several updates to our Windows 2008 build doc.  We are now deploying VSE 8.7 instead of VSE 8.5, as well as using a newer version of IBM director, pointed to our new IBMDir server.

I also finalized 4 new VSE installation packages for use on new server deployments.  Two for VSE 8.5 and two for VSE 8.7.  On all 4 builds the built in autoupdate task is disabled, which prevents some false alerts from behind sent out.  As we discovered recently the built in autoupdate tries to patch applications that don’t exist on the server, thereby generating Install/Upgrade failure notifications.  The other change in the 4 builds is that two of them install to our default installation path on the D drive.