Planet AppHosting

October 2009 Activity Report

2009-11-03T16:31:13+00:00

We’ve worked with Symantec to develop a migration strategy and finalize the SOW for our engagement.
We resolved an issue with a failed volume group on the BlueArc. Due to a drive having communication issues on the fiber channel loops, two drives were marked as “failed” by the controller resulting in an entire volume group going offline. Once the drive was identified that was causing the additional traffic, we were able to shut down the controller and re-seat the drive. We were then able to replace both the actual failed drive and the drive causing the additional I/O on our fiber channel loops.
I have created a Perl script in our backup environment to monitor all failed backup jobs and distribute an e-mail with a listing of any clients that failed and which savesets failed.
We have connected the production Archive servers (its-hcwnar03,04,05) to the SAN and connected them to the Exchange 2007 CX4 Storage Array. Storage has been presented to facilitate the production deployment of the E-mail Archive service.

October – 2009 leaves are falling

2009-10-30T21:20:58+00:00

Well, so another month has gone by and actually didn’t get a chance to write last month. We have accomplished quite a bit on the Exchang project the past 6 weeks. We have started our bulk migrations of the VUMC users and are currently moving around a 1000 a week with hopes of uping that to 1500+ a week. We currently have 4 new mb servers that we have built along with tweaking and smoothing out our scripts and build documents that go with it. We find out that we need to hard code this regkey on each mailbox server

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MSExchangeSA\Parameters\TCP/IP NSPI Port

with port 50920 otherwise when users go to create an Outlook profile and manually type in it hangs and times out.

We are currently have ~ 7330 users on EX07 as of today and 18609 on EX03. I expect to very close to the half way point this time next month if we continue to move 1000+ or even 1500+ users a week.

See you next month and Happy Halloween!!

OCT 2009 MAR

2009-10-30T14:05:35+00:00

Gathered Help Desk data for Law School and assisted with data associations so the Law School could set up their own help desk application.
The ITS DOA helpdesk web site was not compatible with IE8. I got the Vandy templates and upgraded the site to work with IE8.
RE-wrote a nagios check for the DNS database tablespace.
Refreshed test DIP database for the DNS server.
Cruise Director for App Hosting Fall Festival.
Wrote auto maintained query for Cardiology.

MAR – for October 2009

2009-10-27T19:40:46+00:00

1.) Coordinated 110 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 38 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Assisting with data collection for project to determine future of VUSpace.

5.) Continued with leased equipment project. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time. There will be 2 leased servers returned in late October.

6.) Continued with assigned project of decommissioning the Spectrum servers and moving this service to VMWare as part of the leased equipment project.

7.) Completed project of coordinating yearly maintenance agreement for Sun servers.

2009 Oct MAR

2009-10-26T15:49:00+00:00

Monthly Activity Report
October 2009

AIMWorX – Supported all AIMWorX users varying requests. Processed the weekly update of cost centers and student information. Corrected issues with PBX synchronization where AIMWorX was not updated and sent PBX tech information on issues that needed resolution in PBX.
Virtual Server – Moved Windows virtual servers from AMD cluster to Intel cluster.
ESX Server — Upgraded 5 ESX servers to v3.5.0 update 4: its-hcvm01-ts, its-hcvm02-ts, its-hcvm05-ts, its-scvm03, & its-scvm04. Inserted a third ESX server, its-scvm05, into the Intel-Stevenson cluster.
Virtual Snapshots — Compared snapshots being created against snapshots being paid for and found discrepencies. Began creating snapshots for 3 virtual servers that were being paid for be not created.
Perlscripts — Created two scripts to create text files for BellSouth charges & itemized calls journals.
HP3000 Decommission Testing — Transferred and /or created all billing files and journals to VIAFTP server for the month end billing in September 2009 for testing of process.
Higher Ground Backup NAS Share — The NAS share used for call recording backup, PBX-Recording, was alarming in Nagios because it was over 90% used. It was decided to cut back on the amount of data moved to this share from 180 days to 150 days. A ticket was entered with Amcom for Higher Ground to make the necessary changes to thier configuration files. This process was completed and files removed that were older that 150 days. However, this did not correct the over 90% issue for long. It was discovered that the screen captures have increased in size from an average of 2.5GB of data to over 6GB of data per day. This seems to be related to the opening of the internet to all PBX operator stations. When the call begins the operator may be on a web page that contains graphics that increases the size of the capture file.

This is a test from Microsoft Live Writer

2009-10-21T18:29:33+00:00

This is text

and below is an image:

September 2009 MAR

2009-10-06T16:29:49+00:00

September 2009 Monthly Activity Report

1) Resolved several LISTSERV issues. Messages sent from lists with duplicate subscriber addresses were being rejected after Proofpoint upgrade. Appears that something changed in new version of Sendmail on Proofpoint that disallows duplicate recipient addresses. Resolved by creatng script to identify lists with duplicate subscriber addresses and removing them. Used similar script to identify lists with malformed subscriber addresses or invalid subscriber addresses and removed them.

2) Upgraded Proofpoint from release 5.0.3 to lastest release. This includues the addition of the Smart Search module and many bug fixes.

3) Gmail for Life. The new provisioning process for VUmailguard is progressing nicely and should be finished soon if no operation issues arise.

4) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc. Note the phishing scams seem to be on the rise.

5) Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes. Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives. Investigated over 100 spam false negatives.

6) Performed daily management of mail queues on mailgates. Removed thousands of undeliverable messages daily in order to keep queues “clean”. Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

7) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Worked on an assortment of odd and challenging helpdesk tickets

September 2009 MAR

2009-10-02T19:03:24+00:00

Rebuilt Exchange Servers a multitude of times
Created a SQL 2008 Cluster on Windows Server 2008 Ent
Created a partially functioning Windows Server 2008 Ent cluster for email archive
Build SQL 2008 SQL Reporting Services virtual machine running on Windows Server 2008 for email archive.
Supervised the upgrade of the test ESX Intel cluster to 3.5 update 4.
Scheduled vsphere 4 training

SEP 2009 MAR

2009-10-02T14:19:03+00:00

Enabled http tunneled streaming (RTMPT) of media from the Flash Media Servers. This was done by request of the streaming media team to allow better access through firewalls.

Evaluated problems on some of the servers within the shared web environment in an attempt to improve stability and overall service availability. Increased memory on several of the servers in order to accommodate the growing number of Content Management Systems (CMS) that are being deployed. Reconfigured system logging do reduce ‘noise’ created by some services to increase the overall usability of the primary system logs. New watchdog scripts have been put in place to monitor the size of logging files and rotate/compress the logs as needed to reduce storage utilization. Installed and tuned the OSSEC client on the pair of servers handling the Vanderbilt Student Organization content. The OSSEC client will be installed on all of the shared web servers once alerts tuning has been completed. OSSEC is an open-source Host-based Intrusion Detection System (HIDS) with real-time log evaluation which reports to our OSSEC server.

All remaining shared web and MySQL servers have been migrated from the AMD cluster to the newer Intel 7100 cluster within the Hill Center ESX environment. These were the last servers I am responsible for which needed to be migrated in preparation for decommissioning the AMD cluster.

Took over working with the outsource developers for the new Blair School of Music web site. The site is being developed in Drupal. Due to performance issues with the site, a dedicated virtual server has been provisioned and configured for use in this endeavor. Having a dedicated server for this has been a great aid in determining the root causes for the performance problems. These issues have been resolved and development is continuing to make progress on the project. Once development is complete we will evaluate the feasibility of moving the site back within the shared web environment.

Six-month OS andf security patching of linux hosts has begun. So far patching has been completed for one of the bastion hosts and the backup environment. This patching is being done by service groupings for easier manageability and will continue throughout October.

MAR – Sept 2009

2009-10-02T13:08:18+00:00

"Wins" for the month of September:

1. First up, the University and Medical Center now share a unified view of RFC1918 addresses, the reverse space for the Medical Center networks, and the foward zone for mc.vanderbilt.edu internally to the Vanderbilt community. This resolves a multitude of issues where data between the two organizations were out of sync causing conflicting name resolutions. Additionally, this supports the new secure relay for servers email implementation by providing proper reverse resolution for both VU and VUMC.

2. Additional departments were trained for self-serve IPAM and DNS. The Owen Graduate School of Management & the Vanderbilt University Law School now are empowered to administer their own IP and DNS space.

3. The Diamond IP environment was upgraded to to version 3.0.71 resolving some serious memory leaks present in the earlier version.

4. DHCP migrations from NetID to Diamond IP continue with the current migrations at 90% completed. This puts us well on track to have the NetID environment retired in November. That covers the major events of the month.

September and archiving

2009-10-02T12:35:03+00:00

Most of my September was spent concentrating on replacing Vanderbilt’s current email archiving product with a new email archiving solution.
During one week of the month, I worked with a contractor who assisted with the implementation of the new archiving solution. Our plan was to set up the new application within an active/passive clustered windows server 2008 environment.
We were able to implement the new solution but were unable to take advantage of the clustered environment due to some unforeseen OS problems. We decided to carry on with a single server and were able to create a pilot group which included me, two other co-workers and two test user accounts.
There were some pitfalls along the way, but we were able to successfully provision and archive user’s email and retrieve the emails via OWA and the Outlook client add-in.
The last week of September was spent attending on-site training sessions that were geared towards installing, administrating and troubleshooting the new archive solution.
All in all I feel that Vanderbilt is headed in the right direction by replacing our legacy archiving system with the new archiving solution and I look forward to continuing this project and actually migrating users from our legacy system to the new.

September 09 MARS

2009-10-01T14:11:02+00:00

EXCHANGE
Exchange CAS Build of ITS-HCWNEM77
Built new Exchange CAS in support of the MC FE collapse.
Troubleshoot Admin Network issues on ITS-HCWNEM76
Troubleshooting of intermittent admin network connectivity. It appeared to be a conflicting IP address but couldn’t get a ping back from another machine when this machine didn’t have the address configured. The issue ended up being a Cyclades switch that wasn’t responding to pings. Fixed the issue with a new ip on its-hcwnem76.
Assisted in testing MC exchange FE collapse
Helped the exchange team by taking part in the testing of the new MC Exchange CAS environment after the collapse.
Pushed for identification of Exchange Local FW rules for the new MC Exchange CAS’s
Windows Patches of UM environment
Patched the UM environment and troubleshoot resolve issue with ITS-HCWNEM75 CAS that was related to ASP.NET 1.1
Resolve SCOM Agent issues on Exchange boxes
Reinstalled SCOM agents after identifying local agent issues on EM01,EM04,EM75,EM76 and EM77
Memory Upgrade of 2007 Mailbox Servers
Upgraded memory on EM01-EM04 to 48Gb
Exchange Mailbox Switch Redundancy
Identified issues with mailbox servers teamed network switch redundancy. Documented the issues and put in changes to resolve them where possible.

WINDOWS
VUSpaceGroups Migration Project
Started documenting requirements, issues and procedure for migrating VUSpacegroups from Windows cluster to NAS.

*WINS1 Replacement
Replaced WINS1 production server with like replacement to resolve problems that were causing issues with our environment. Identified configuration difference between old WINS1 and WINS2 that was rectified in the replacement process. Documented replacement process.
Pertrac
Assist in troubleshooting/resolving issue with test pertrac server. Be available for when Pertrac needs connection to servers.
PRT1 replacement
Continued work on PRT1 replacement. All print queues have been created on the new print server minus the printers that the server does not have connectivity to. Working with Partner to identify LSP’s and contact them about the printers with no connectivity to ensure that they are not needed anymore to allow cleanup of them from the new server.

Antivirus Repository
Local testing of new Antivirus Repository has run into some issues that we are trying to identify the root cause of.
SCOM Rules
Identified issues with some of the custom SCOM rules that we had in place and reconfigured/documented the process for correctly limiting SCOM rules to a specific group or box.

PowerShell Scripting
Ping Test Script
Wrote a script to test ping connectivity. This was used to identify printers that were configured on PRT1 that had no connectivity.
Printer Information Script
Wrote a script to pull all the relevant information for each Print Queue remotely from PRT1 to facilitate the migration process to a new print server.
Assisted Roland with build doc automation script
Helped Roland with troubleshooting errors he was seeing in his base build doc automation script.

Sept 2009 MAR

2009-10-01T13:34:34+00:00

Researched and wrote scripts to purge logs and history from incontrol database. These scripts are written and scheduled within the oracle database to purge these tables nightly up to a given number of days.
Updated the code on the IP Request web site by NDE's request.
Modified Service Desk Report for NOC Manager, Scott.
Purged mysql log tables on Spectrum One-Click. This was needed becasue of space issues.
Added tablespace and flash recovery space to INCONTROL primary and standby databases.
Upgrading BMC Service Desk on the test servers. Testing still continues. BMC is currently looking into upgrade bugs that we found.

MARS 09/09

2009-10-01T13:19:26+00:00

Over the past month I managed to upgrade our KMS server, so we can now license both Windows 2008 R2 and Windows 7.

Though I spent most of the month working on various issues/mini projects for the ECS team. Ending the month with building the newly purchased em05, and rebuilding em04 so that it could utilize the additional RAM that was purchased… Gotta hate it when you build a server with 32 GB and only install Standard, then decided you need more RAM. Anyway, two mailbox servers down, eight or so go.

I spent quite a bit of time after hours setting up lab environment at the house, so that I can both work on my 2008 R2 skills, and to assist automating our build process. So far I’ve gotten my 2008 R2 AD setup, and Windows Deployment services installed and configured. Hopefully next month I can test doing some OS pushes, and then investigate incorporating our default settings, firewall rules, application installs etc, into an automated package.

Sept MAR

2009-10-01T12:53:00+00:00

Responsibility Transfer: I have been gradually moving out of my role as Unix Team Lead and System Administrator. Almost all of the daily operational duties are being performed by the System Administration team.

Exchange Support: The vast majority of my time this month has been spent work with the ECS team to support the new Exchange environment. The consolidation process is very complex, and it requires coordination from many people within the Vanderbilt community. The Exchange 2007 environment has been my re-introduction into the world of Microsoft Windows. I have been involved recently in monitoring server performance.

Enterprise Linux Reference Platform: We are continuing to meet and define the base RHEL standard.

MAR – for September 2009

2009-09-28T18:35:43+00:00

1.) Coordinated 138 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 36 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail.

5.) Continued with project of identifying leased equipment along with monitoring status of this equipment. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time. There were 3 leased servers returned in September.

6.) Assigned project of decommissioning the Spectrum servers and moving this service to VMWare as part of the leased equipment project.

7.) Assigned project of coordinating yearly maintenance agreement for Sun servers as it nears time for renewal.

2009 Sept MAR

2009-09-28T12:20:16+00:00

Monthly Activity Report
September 2009

AIMWorX – Supported all AIMWorX users varying requests. Processed the weekly update of cost centers and student information. Call records prior to 3/31/09 were purged.
Microsoft eLearning — Began online courses for Windows 2008 Server.
AIMWorX Work Order Template — Create a new template for Up/Downgrading OCS and added to the OCS template documentation.
Billing Perl Scripts — Created multiple scripts to create journal files for all types of calls as requested by MIS. Sent test files to MIS. This is part of the HP3000 dec0mmission project.
Virtual Server – Created a virtual server to replace the current WINS1 server. This server was put into production without moving off the development ESX cluster. Moved the new WINS1 server from development ESX cluster to production ESX cluster. Created 3 virtual servers to replace 3 physical servers for Spectrum.
PBX Operators — Spent a couple hours with operators in an attempt to determine a cause for thier greetings failure. Only on position had issues while I was there. Ran multiple test to determine problem but came up with no answers. However, there have been no problems reported for almost two weeks.
ESX Server — Received new physical ESX server. Installed ESX version 3.5 update 3 on server, ITS-HCVM05-TS. Moved servers from Intel-Dev cluster to new server test server. Removed ITS-HCVM03-TS from Intel-Dev cluster and reloaded for production use, ITS-SCVM05. Moved snapshot service to new test ESX server, ITS-HCVM05-TS.
AIMWorX Duplicate Billing — Created two files to credit calling card & switched calls that were billed twice on the August 2009 billing process. These credits will be applied to the September 2009 billing process.

Activity Report September 2009

2009-09-25T15:04:26+00:00

Exchange 2007:

The DR storage array a Clariion CX4-240 was configured and brought online this month. The storage groups have all been configured and the dedicated SAN switches have been configured as well. All that is left is for the Exchange host to be brought online and connected to the dedicated SAN switches.

An additional expansion of 6 DAE of storage to the Array dedicated to Exchange is scheduled for a week from this coming Sunday.

E-mail Archive:

We have spent a lot of time around the SOW provided by Symantec and have ironed out the process required for our implementation.

We have installed one archive server and we have a couple of admins using it for their mail.

Projects:

VUspace replacement projects are being ramped up for replacement of the groups and user services. The groups service will be migrated to NAS storage.

August – 2009 summer is waning

2009-08-31T20:48:26+00:00

Wow, what a fast month again. Our team saw some ups and downs within our environment. A lot of Exchange servers being put and pulled out of the environment for VM issues, physical issues and OS issues. I think we installed Exchange on probably 10 boxes and rotated out of production and back in just as many. We are currently in finished with the migration Exchange users but are catching our breath and doing some clean up on the backend before we really roll up our sleeves and begin to move the then some 22,000 VUMC mailboxes. In preparation of this move for VUMC, we had an issue when attempting to allow ITS-HCWNEM40 to be setup for 2003 public folders. Guy had to work with Microsoft one weekend day and in a nutshell, we are in a holding pattern to determine our best line of action to if any, add public folders within the 2007 environment.

A plethora of Magic and Service tickets still abound our queue as it seems when we resolve 2, 3 more come in. They can be issues small in nature or great in size with visibility. We as a team do our best to complete them in a efficient and timely manner. One in particular which comes to mind is Kathryn Foote, Matt Halls assistant would accept a meeting request on his behalf then after a few minutes it would go tentative. We found a tracking option to uncheck on Matt’s client which corrected this but believe its caused by a MS patch that came out in June. I’m going to contact MS and determine if this is the correct behavior when both clients have this option checked. Hmm, I don’t think so but I’ll give MS the benefit of the doubt. sigh…….

So as August comes to an end, I have settled in much more than just 30 days ago. We even had a after work bowling get together with a few guys which was fun. Antwan is quite the bowler though. I’ll have to sharpen my skills before I bowl against him again.

See you next month :-]

P.S. Can’t wait for the NFL season to start in just a couple of weeks.

Aug 2009 MAR

2009-08-31T13:03:34+00:00

Corrected Service Desk Business Rule code to eliminate monitoring errors. Also fixed code to allow IPAM clients to get notification on ticket creation.
Worked with DNS admins on database issues to include creating export dmps, adding tablespace, adding flash recovery space and changing backup scripts.
Researched and downloaded BMC Service Desk upgrade application. Rebuilding test environment.
Wrote SQL queries.
Organized desk.

August 2009 Activity Report

2009-08-28T14:16:26+00:00

We installed two new SAN blades into our existing Cisco 9509 directors. This expanded our capacity by 24 ports and provided high bandwidth ports for the new CX4-480 storage array and Exchange 2007 mail stores.

We deployed a CX4-480 storage array for the purposes of supporting Exchange 2007 and e-mail archive. We have presented storage to Exchange backend servers and MSSQL servers. The ESX environment will also be getting some much needed additional capacity from this expansion.

Work continues to support the new Exchange 2007 and e-mail archive environment, additionally we have launched some initial project work into looking at VUspace lifecycle replacement.

MARS 08/09

2009-08-27T13:49:47+00:00

I finished moving the SharePoint Test farm behind the F5, and setup SSL termination on the F5. Some of hte service monitors are not ideal, but as soon as I have some more time to spend on it, I should be able to get things setup properly. Then I can document the process, and start prepping to move the production SharePoint farm behind the F5.

I spent several days researching some blue screens that were generated on several of the ECS servers. Initially we were not getting memory dumps. We discovered two issues that were preventing the servers from creating memory dumps.

The first issue is that the page file was not on the system partition. In order for the system to generate a User, or Kernel dump, an adequately sized page file must exist on the system partition.
The second issue, was that the system partition was not large enough to place the entire page file on the system partition, so getting a User dump was not an option. We could still get a Kernel dump, but that required us to make the following registry change, so that we could have a small (smaller than physical RAM) page file on the system partition:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl\ Create a new DWORD named IgnorePagefileSize, with a value of 1.

After making the necessary changes we received our first memory dump on EM24. After analyzing this dump, we were able to determine that an unkown driver had modified the PrintDbg routine. Then I went through all the ECS servers looking for the same stop code, adn was able to determine that 3 different ECS servers had experienced 8 BSOD’s with the same stop code the first starting on 6/30/09. Next we tried to track down what drivers were installed either on 6/30 or shortly beforehand. We could not find any documented changes for driver updates on the three impacted servers. So the decision was to replace the 3 impacted VM’s with new VM’s.

Aug MAR

2009-08-26T18:36:13+00:00

New Position: I have recently been promoted to Senior System Administrator. As part of the transition, I am moving out of my operation role, and I am transferring those duties to the rest of the Unix team. I’m now supporting each of the AppHosting teams in my new role. I’m already diving into a couple of issues on the Windows platform, which is something I’ve not been heavily involved with for a while. As such, I’ll be trying to absorb as much information as possible in the near future.

Exchange 2007 Support: I have been tasked with several items supporting the Exchange 2007 environment.

Sun Identity Management [SIDM]: The new SIDM LDAP services are live, available, and in-use by the general Vanderbilt community. The team is in the process ensuring that all clients are properly migrated to the new service. Only a few clients are still connecting to the Solaris-based service, and we are working with the owners to move them over.

Enterprise Linux Reference Platform [ELRP]: The team has produced a server build with minimal packages to use as the base system. We are in the process of determining the proper configuration for the installed packaging.

MAR – Aug 2009

2009-08-26T17:55:56+00:00

Wins for the month:

First off, it’s was a year ago on July 31st that we began to actually gather metrics on our DNS servers… Looking back over the past 365 days, here are some interesting tidbits of information you may or may not care to hear…

Total Queries
IP-SRV1                9,639,738,984
IP-SRV2                2,021,500,701
IP-SRV3                420,262,993
Total                      12,081,502,678

Per Day Average Queries
IP-SRV1                26,372,205
IP-SRV2                5,524,468
IP-SRV3                1,811,722
Total                      33,055,400

Per Hour Average Queries
IP-SRV1                1,098,842
IP-SRV2                230,186
IP-SRV3                75,488
Total                      1,377,308

Per Minute Average Queries
IP-SRV1                18,314
IP-SRV2                3,386
IP-SRV3                1,258
Total                      22,955

Per Second Average Queries
IP-SRV1                305
IP-SRV2                56
IP-SRV3                21
Total                      383

12 BILLION queries answered…
383 every second of every day on average…

That’s a lot of “Who and where is this address,” requests!

Speaking of DNS, we have started the process of presenting a unified internal view between the Medical Center and the University. This will help clients in both institutions ti be resolvable within the Vanderbilt community. In regards to email, this will go a long way to help us secure the environment and streamline our processes. Initial testing has been successful and a roll out into production is scheduled for mid-September.

The DNS servers were successfully patched to address CERT Vulnerability Note VU#725188. There was no interruption of service to the community.

2 additional departments have been tapped for Self-Serve DNS/IPAM and will receive their training from ITS in the first week of September. This is a part of the ongoing plan to empower selected departments to administer their own DNS and IP space while alleviating some of the load on ITS.

DHCP migrations off the old NetID environment continue to be on pace with an earlier than expected completion. We are currently at 66% of migrations completed with little impact to the customer base.

The ESX environment will receive a nice boost in capability at the end of August with the addition of 8 more cores and 256GB of RAM. The ESX development environment is slated for replacement with the current assets being redistributed to help out other production clusters. This is an ongoing effort to increase our virtual hosting capacity and improve efficiencies.

August 09 MARS

2009-08-26T14:41:35+00:00

Identified and resolved WINS connectivity issue with NCS Wins server
The issue appeared to be related to WINS server rename by the medcenter. Removal and readd of the WINS partner with correct name seemed to resolve the issue.

Resolve WINS database corruption
Identified wins corruption issue and executed break fix to bring it back online in a timely manner. Subsequently identified event viewer messages to key off of for SCOM and created new Crit/High alert that would appear in the NOC view of SCOM for notification.

Assisted in security investigation of WINS1

Decommission of EMC’s Replication Manager and Replistor
Decommision of RM server and uninstall of RM agents and Replistor software from Exchange 2003 Test and former production environments.

Patching of UM environment
Microsoft patching of the full messaging environment

Page file upate of UM environment
Standardized page file setup across the messaging environment servers.

Participated in buildout of new Exchange Machines in timely manner
Facilitating the build of the Exchange environment by building the new servers and ensuring consistency of the not only the software builds but the network setup of the trunked lines to minimize possible issues. Facilitated fixing of any network issues/differences found with the boxes.

Identified and resolved BIOS memory issue on rebuild of Exchange boxes
Identified BIOS issue on ITS-HCWNEM10,11 and 12 for adding of memory for re-use of these boxes. Installed BIOS and verified functionality of the servers.

Assisted in reworking of OWA rich web monitor conversion to work with Exchange 2007 environment
Assisted Peter in troubleshooting the old Exchange 2003 rich web monitor conversion to work with the Exchange 2007 environment.

Participated in Exchange Monitoring setup
Participated in meeting to ensure that monitoring of the current Exchange environment was satisfactory.

Troubleshot and resolved phantom drive issue on ITS-HCWNEM01,02 and 04
Worked with EMC to identify the issue that was causing the phantom drives to show up on these boxes. The issue was found to be the boxes fiber registration with the CX-340 without having any luns presented from that box. This caused the phantom drives to appear. The issue could be resolved by either presenting storage from that san or unregistering the boxes from that particular SAN. The latter was done leaving storage being presented just from the CX-380.

Cleanup of SCOM DB and change of default size/growth
Ran cleanup scripts and requested scheduling of these scripts on a weekly basis to keep database in a healthy state. Running of the scripts to cleanup the localizedtext tables keeps SCOM running more efficiently. We also increased the starting size of the database and increased the auto-growth to resolve size reporting issues of the SCOM db.

Identified issue with SCOM maintenance mode not working as desired
SCOM maintenance mode was not stopping all alerts from the associated objects with the client. It was only stopping alerts directly from the agent on the box. Found a powershell script that gives a gui tool to put a computer and all associated objects with that box in maintenance mode so no alerts are generated for it.

Put together SCOM documentation for NOC consumption
This documentation was generated for identifying how SCOM works. What the differences are between SCOM and Nagios, and also how to utilize the console to maximize our gain from it.

August 2009 MAR

2009-08-26T14:38:18+00:00

Decommissioned the old OSIS virtual machines which completed the project for migrating the OSIS environment to RHEL4 and from the AMD to the Intel ESX clusters.

Nagios has been fully migrated from a virtual machine to a physical machine. This move was related to performance issues which arose after patching. The new service has been upgraded to Nagios 3.0.6 and is also utilizing the NDOUtilities for logging to MySQL. The migration went very smoothly overall. The service has been stable again since the migration was completed.

Began work on building an additional Streaming Media Server. The new server will be located on the VUMC network in order to better support their growing needs in streaming presentations, etc.

Groundwork has begun on planning for life-cycle of the JPROD servers. To more completely mirror production services, the JTEST1 replacement will consist of two servers. After discussing projected growth and comparing to existing system specifications, a request for options and quotes has been sent to our vendor.

No work has been done recently to migrate the ND&E “weathermap” to a new server.

DHCP migrations are moving quickly for the DNS project. Some minor feature changes were requested and completed for the reporting/scheduling page.

At the request of Network Security, the port-block pages have been slightly overhauled and fixed. It was noticed that the pages were not working properly after being migrated to the new web servers. So far no other problems have been reported.

Migrated a few smaller databases from mysql01 ro mysql02 in a continuing effort to retire the old MySQL4 server.

There have been no action items assigned in the GMail for Life project.

MAR – for August 2009

2009-08-25T22:29:56+00:00

1.) Coordinated 146 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 45 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail.

5.) Assigned project of identifying leased equipment along with monitoring status of this equipment. I have been tasked with keeping track of this equipment for purpose of decommissioning and returning to vendor at the appropriate time.

August 2009 MAR

2009-08-25T16:23:03+00:00

* Created three new SQL Reporting Services reports for the ESX server metrics. I then created three ASPX SharePoint pages with page viewer web parts that reference these reports. I stored the SharePoint ASPX pages in the virtualization SharePoint site so that the entire team has access to the same information.

* I am now the Manager of System Administration in ITS.

2009 Aug MAR

2009-08-25T13:20:07+00:00

Monthly Activity Report
August 2009

AIMWorX – Supported all AIMWorX users varying requests. Processed the weekly update of cost centers and student information. Call records prior to 1/31/09 were purged.
AIMWorX charge for backup – Setup new process in AIMWorX to charge for server backup. This process consist of a Perl script to reformat the data from the Storage group and then an import into the charges table in AIMWorX. This process was documented but has not yet been implemented.
Server Decommission — Had the old NEC Express server removed from the rack after the drives were cleaned. This server was used as the AIMWorX SQL database server. This server can be returned to the leasing company. Two other IBM servers had their harddrives cleaned and can be returned to the leasing company.
PBX Operators & VUSearch — When searching the Vanderbilt website using the VUSearch from a PBX operator workstation, there was a 2 minute delay before data was returned. Dave Arkle searched the firewall logs and then captured data and found that the VUSearch engine was looking to Google.com to record statistics. The PBX operator workstations do not have access to the internet which is why the search took so long. Peter Woods found the websites the search was trying to contact and the workstation host file was modified to point them to the local computer so the website lookup would fail immediately and return the search results. This information was passed to Partner so the rest of the Operator workstations could be updated.
Server Setup & Modifications — Created a VM to replace EMC Management. Added a NIC & RAM to requested Owen virtual servers. Reloaded OS, installed new RAM and harddrives on two physical servers in support of the Exchange 2007 project. Reloaded OS on two other physical servers in support of the Exchange 2007 project. Turned off ESX DRS on three virtual servers to keep them from being VMotioned between servers. These three virtual servers are now locked to different physical ESX servers.
PBX AIMWorX synchronizations — The AIMWorX sync with the PBX has stopped updating the database when the extension changes FPCs. The log file from the days sync is now reviewed and extensions are manually moved. During this process extensions were found to be programmed in multiple FPCs. This list of extensions was sent to the PBX techs for correction.
VPD screen pops — On Saturday, August 15, I received a call from VPD stating they had lost thier e911 screen pops from AIMWorX. After looking through the AIMWorX logs it was determined that AIMWorX was sending the alerts. Setup the Alarm client on my desktop and received screen pops from AIMWorX. Talking with VPD they said their computer had been getting harddrive errors for a long time and they had a new computer to replace the old one. Even though I was not sure this was the problem I asked VPD to install the new computer. I went to VPD and installed the AIMWorX Alarm client on the new computer, which did not resolve the issue. I contacted the ITS security person on call to check the firewall. Dave Arkle looked through logs and data being sent and blocked at the firewall. Dave reset the link between the ITS & VPD firewall and reset the Alarm client. This resolved the issue and e911 screen pops began working again.
MIS HP3000 Decommission — As part of the effort to decommission the old HP3000, new Perl scripts have been created to duplicate what MIS has done for years on this server. Four of a possible six new script files have been created to summarize call charges to directly feed the G/L software and bypass the HP3000 processes. These new files are in addition to the detail files we currently create.
Laptop conversion — Backed up all files from my laptop and then installed the Windows 7 operating system.

June MARS

2009-08-03T23:40:10+00:00

Closed 156 Service Manager Tickets

Closed 20 Magic Tickets

Exchange

Identified the types and number of resource accounts that reside on the Medical Center Exchange 2003 email servers. Also provided documentation on how to move the different type of resource accounts (room, equipment and shared).

Part of Email Program - Legacy Email Systems team which is trying to reduce the physical imprint of servers via virtualization in Stevenson to alleviate heating issues.

Testing importing/exporting mail on its-hcwnem98.

Learned and started processing Exchange migrations for the previous two weeks.

Given OCS Admin role.

Windows

Requested a standard template build for a Windows 2008 (32-bit) Server VM named its-hcwnem98 and installed office 2007 and exchange management tools (32-bit) to be used for import\export functions on 2007 Exchange mailboxes.

Installed exchange management tools and OCS administrator on its-hcwnem99.

Given task to install message stats on its-hcwnem98 and its-hcwnem99.

ITS

Configured Microsoft Roundtable for ITS conference room and provided documentation.

Met more ITS employees.

Introduced to ProofPoint.

Misc

Received a Vanderbilt Medical Center USD 35 Award from Michael R Harris for perseverance, dedication, commitment and professionalism.

July – 2009

2009-08-03T18:55:41+00:00

July was definitely learning month: Learning the environment, infrastructure, protocols, people and politics. I have at least to say we successfully migrated all Law users while dynamically modifying our script to move mailboxes. Everytime we modify the script(s) we decreased the time and increased the number of mailboxes moved. It’s still a work in progress but feel we are close in reaching out target goal of speed and efficiency as we have begun to also move the some 3000 VU mailboxes.

I also built two new Exchange servers and used the build template to follow and tweak. Note to self!! Make sure Windows UAC is disabled when trying to install Exchange otherwise you will be chasing a wild goose.

Here is a link from one of my favorite reference/knowledge sites Petri. http://www.petri.co.il/disable_uac_in_windows_vista.htm

Still tweaking the Mail flow Visio diagram which is a work in progress but a great learning tool that forces me to learn the messaging environment at a much better level.

I also went on call for the first time which I am still learning the in’s/out’s of the piece of the email team.

Overall it was a great first and quick month here. I have lots to learn but have some great co-workers around me to pick their brain and ask questions.

See you next month,

:-]

July 2009 MAR

2009-08-03T18:25:09+00:00

* Created Powershell script to automate the collection of VMware statistics.

* Created a Powershell script to send Kevin information on the number of operating systems we have in our environment for physical versus virtual computers.

* Coordinated and built 3 of the 9 new IBM servers for the Exchange project.

Migrations, HUBs, Relays, Aquisitions and Archive

2009-08-03T12:35:54+00:00

July has been a very busy month for the Exchange Team here at Vanderbilt. One of the largest aspects of moving forward with Exchange 2007 is migrating people from Exchange 2003 to Exchange 2007. After saying that, I have had the honor of migrating over 1300 people (with the help of powershell of course) this month alone. The aforementioned number not only included E2K3 users but also IMAP users.
I was also able to help introduce and implement the introduction of SMTP authentication services within the E2K7 environment, which by the way uses TLS port 587.
My next task for the month of July was to introduce E2K7 allowed relay services. This task was accommodated by introducing two network load balanced E2K7 Edge servers (behind an F5), but not Edge servers in the traditional sense. These Edge servers are being used strictly as relay services with the Edge’s having no knowledge of AD or the HUBs. To put it simply, the role was introduced, the receive connectors were set to allow anonymous users and were restricted based on IP. I then configured a send connector that smart hosts to a Proof Point appliance.
I was also involved with two companies that recently became part of the Vanderbilt Medical Group and whose email needed to be forwarded from their old email domain to the Vanderbilt.edu email domain. Not only that, but close to 150 email accounts had to be created in the E2K7 environment in order to accommodate all of their new email.
Finally, the archive project is finally coming into full swing. We are currently planning on migrating from a legacy email archive product to a new email archive product. This in its self will probably take up most of my time in the Month of August.

July 2009 MAR

2009-07-31T21:39:20+00:00

July 2009 Monthly Activity Report

1) Resolved another LISTSERV issue. Found that a configuration option value was carried over from the eval configuration. With lists configured to require owner approval for subscriptions, the URL presented in the approval message contained a malformed email address. This resulted in an invalid subscriber address being added to the list. Issue has been resolved by correcting the configuration option value.

2) Created various scripts to gather inoformation related to the Exchange 2007 project.

3) Gmail for Life. Nothing new to report. Awaiting a source in order to begin testing of new provisioning process.

4) Legacy Email. Provided assistance in order to configured the VM SMTP replacements and configuring a test environment. Assisted with validation testing.

5) New Acquistion. We are now routing email for the domains associated with the new VUMC acquisition, The CardiacCenter of Murfreesboro.

6) New Acquistion. As new employees from Franklin Bone and Joint activate their user ids, Exchange 2007 mailboxes are being created. This has almost completed. They have elected to initially forward email but at a date yet to be determined, they wish to route email for their domain also.

7) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc. Note the phishing scams seem to be on the rise.

8) Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes. Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives. Investigated over 100 spam false negatives.

9) Performed daily management of mail queues on mailgates. Removed thousands of undeliverable messages daily in order to keep queues “clean”. Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

10) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

11) Worked on an assortment of odd and challenging helpdesk tickets

Hello world!

2009-07-31T18:15:53+00:00

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

July 2009 MAR

2009-07-31T17:03:19+00:00

Google Search Appliances have been placed into production. Old ultraseek fqdn has been replaced by a redirection page that collects referring pages which can be viewed through a limited access administrative interface for identification of sites which need to be updated to support the GSAs. The old Ultraseek servers are currently being decommissioned.

Nagios performance problems remain an issue. Due to the amount of resources the host is utilizing within the ESX environment, the service will most likely be migrated to a physical host. Available hardware has been found and the initial build has been initiated.

The AmCom migration from 4.0.63 to 4.5 has been completed. Fail-over to the new system proved to be a smooth transition. The remainder of the OSIS environment has now been migrated to the new system. Decommissioning of the old web/db servers will take place in the near future.

IMAP2 was renamed IMAP-DEV and moved from the production to the ITS-test network in order to test upgrading the cyrus-imapd packages. Installation has been completed. Currently testing migration and integration of the new cyrus-imapd build with the Solaris CSW bundled package.

Installed SLAMD client on several web enabled developement hosts for IDM load testing. Setup firewall access for UDP ping utility which will be used to monitor network health between the individual LDAP servers during the load tests. Worked with Roland to setup access to the UDP ping service on hosts behind the F5.

Worked to migrate databases from mysql01 to mysql02. This is an ongoing effort working toward the eventual decommissioning of mysql01.

July MARS

2009-07-31T16:41:19+00:00

Closed 48 Service Manager Tickets

Closed 27 Magic Tickets

Migrated 212 law school accounts to Exchange 2007 on July 13^th.

Upgraded Message Stats version on its-hcwnap11 to 6.6
Backed up SQL DB on its-hcwnap11 and truncated log file that consumed the entire log drive. This had to be done prior to message stats install.

Set message stats default gathering tasks on all email servers. Also, successfully configured public folder gatherings. Currently in the process of configuring OCS and OWA statistics. There are two open case numbers with Quest Support, 739059 and 741003

Installed Terminal Server on its-hcwnem99. It has 5 concurrent licenses per device so more than 2 RDP sessions will be able to connect now on its-hcwnem99.

Joined the Rightfax deployment team with Chris Marshall and Erin Shelton.

Authored document to remotely wipe phone device from OWA.

Requested two VMs, its-scwnem27 and its-scwnem26.

Attended Nagios/Fruitty training.

Testing Jott application for iPhone.

MARS 07/09

2009-07-29T15:09:35+00:00

Moved the test SharePoint farm moved behind the F5, though I still haven’t gotten it working yet. Mysites seem to work fine, but all the other sites, i.e. its, dar, etc don’t. I’ve spent a large portion of my time learning the F5 this month, and have had quite a bit of assistance from Peter when he could spare the time. Hopefully when I get back from vacation I can figure out why the core SharePoint sites on the test farm are unavailable through the F5.

I upgraded the ePolicy server to 4.0 Patch 5 to resolve a weak encryption vulnerability. Only to discover a new more severe security vulnerability with patch 5. I have an open ticket with McAfee on this, but have made very little progress. They have escalated the ticket to tier 3 support, who in turn pushed it over to the security team. I’m still waiting to hear back.

I opened a ticket with Microsoft about the recurring event 6614 that we’re getting on our production SharePoint farm. Naturally once opening the ticket we have not seen the error. Microsoft suggested that we remove some old application pools that are no longer in use, which we did, but ultimately the we have yet to receive the error again, so the ticket has been closed.

I applied Windows patches on all of the ECS servers, and in the process discovered several discrepancies, some of which have been resolved, and the rest will be resolved shortly. Hopefully next month we can do this as part of our regular WSUS patching schedule.

Built a few servers for the ECS guys, and during the process had to make several updates to our Windows 2008 build doc. We are now deploying VSE 8.7 instead of VSE 8.5, as well as using a newer version of IBM director, pointed to our new IBMDir server.

I also finalized 4 new VSE installation packages for use on new server deployments. Two for VSE 8.5 and two for VSE 8.7. On all 4 builds the built in autoupdate task is disabled, which prevents some false alerts from behind sent out. As we discovered recently the built in autoupdate tries to patch applications that don’t exist on the server, thereby generating Install/Upgrade failure notifications. The other change in the 4 builds is that two of them install to our default installation path on the D drive.

MAR – for July 2009

2009-07-27T18:18:46+00:00

1.) Coordinated 137 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 34 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail. I am currently working on cleaning up old data on IMAP1/IMAP3.

5.) Applied OS/Security patches to personal VUSpace file servers.

July 2009 Activity Report

2009-07-27T17:17:47+00:00

This month we worked primarily on storage infrastructure that will facilitate the Exchange 2007 deployment and merger.

We added 66 TB of storage to our EMC EDL. We had to upgrade the code on the array and the EDL engines in order to successfully install the new capacity.

We ordered and received two new storage arrays. One is currently in the Hill Center and the other will be placed in an offsite location for disaster recovery purposes. We configured and assigned the first batch of storage just this past week to the new Exchange 2007 database servers from the new CX4 at the Hill Center.

We have ordered new SAN blades which will be installed into our existing Cisco 9509’s to provide high bandwidth ports to the new Storage Arrays and performance intensive Exchange servers. Additionally, a smaller SAN switch was purchased to go in the offsite location for DR.

We are still working on a migration path in our e-mail archive environment from the old EAS environment to the new environment.

We have been working on planning to resolve some fiber length issues in our SAN environment since the arrays and SAN switches have been moved to better facilitate data center cooling. Along with these changes we’ll be coordinating installation of the new SAN blades sometime by early August.

The BlueArc storage we purchased a couple of months ago is now almost completely used. We have migrated some shares around to provide more space to the users on those shares. I have also restructured old storage pools to make better use of spindles underneath those file systems.

2009 July MAR

2009-07-27T11:46:13+00:00

Monthly Activity Report
July 2009

AIMWorX – Supported all AIMWorX users varying requests. Processed the weekly update of cost centers and student information. Imported yearly Housestaff pagers for PBX operators. SMDR for April, May, & June were backed up to CD and sent to an off site location. Call record prior to 12/31/08 were purged.
AIMWorX Templates – Completed new work order template for the OCS project to setup billing in AIMWorX and to request an OCS account. These templates were imported into the production database.
AIMWorX Data Purge — Purged data prior to 2006 from CHARGES, TRANS, Trouble Ticket tables in AIMWorX. Work Order purge failed due to ‘database error’. Research into this issue has found no errors in the AIMWorX database.
AIMWorX Work Order Scheduling — An issue was resolved that was causing work orders to not show up on the technician calendar. In work order there are two types of tasks: Group & Task. The group type tasks does not get populated with hours the order will be scheduled for on a particular date, that information is suppose to be on the task type. For some reason, on one order this information was populated on a group task, this was the cause of the issue. With help from Paul Lockaby, the incorrect work order was found and the hours information was removed.
SQL Reporting Services — Began the learning process of this report writing software and created five AIMWorX reports in the process.
OCS Reports — Requested reports for OCS license information were unable to be created that gave an accurate number. This was due to the way AIMWorX stores work order information. SQL Reporting Services was tried but ran into the same problems.
IBM Amber Light — There was an amber light found on its-hcwnap64 that was reported to IBM. The problem turned out to be a bad power supply which was replaced.
Virtual Servers — Created two new virtual servers (DC-10, ITS-HCWNAP66) and rebuilt a virtual server (DC-SB1) for iDev test environment.
Exchange Servers — Build 3 of 9 new physical server for Exchange 2007 environment.

June 2009 MAR

2009-07-16T23:40:08+00:00

June 2009 Monthly Activity Report

1) Working with L-Soft and other team members gathered and provided metrics in order to debug issues with LISTSERV performance. Based upon data L-Soft recommended demoing the LISTSERV high performance option (HPO). Tests of HPO demo license in dev server resulted in much improved performance. Tests of HPO demo license in production resulted in much improved performance as well.

2) Created script to provide ACLs for all group mailboxes and determine which mailboxes were accessed over the life of the logs files.

3) Created script to provide info about which IMAP mailboxes were accessed over the life of the log files.

4) Gmail for Life. Nothing new to report. Awaiting a source in order to begin testing of new provisioning process.

5) Legacy Email. Provided assistance in order to configured the VM SMTP replacements and configuring a test environment.

6) New Acquistions. VUMC has acquired The CardiacCenter of Murfreesboro and Franklin Bone & Joint. Began preliminary discussions concerning how email consolidation will occur, information required, etc.

7) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.

10) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

11) Worked on an assortment of odd and challenging helpdesk tickets.

June MAR

2009-06-30T12:19:56+00:00

I only have one major WIN this month…

PHP5/MySQL5 Web Server Migrations: The project to upgrade all of the shared web services to RHEL5 is essentially complete. All of the content has been migrated, and the sites are running smoothly. This migration gives the web community access to PHP5 and MySQL5. It also brings along a much more robust ModSecurity ruleset. So far the vast majority of the help desk tickets have been related to applications with the new ModSecurity rules.

June MARS

2009-06-29T21:01:55+00:00

18 Closed Magic Tickets

Exchange
5 Migrations to Exchange 2007

Windows

VUGroupspace
Identified issue with quota software being utilized to manage VUGroupspace. One of the shares is being reported as uitlizing much more space then is actually in use, as reported by Windows. The issue appears to be a bug in the Quota software. Change has been finished and testing is continuing.

Exchange
Assisted in troubleshooting and resolving various issues this month.

Archiving
Assisted in the process of choosing an archiving solution for the Exchange 2007 Environment. This has included attending various vendor presentations, providing system configuration information and challenging\questioning vendor configurations and abilities.

Cross Training
Continued cross training new messaging employees with current configuration, setup and procedures for the VU Exchange Environment.

Exchange 2007

OWA Vanderbilt Rebranding work
Passed on knowledge for the rebranding for management to Antwan on the Email team. Assisted in putting new logo on OWA Site.
F5 OWA traffic rules
Identified solution for F5 forwarding of vunetid url to delivery address url. This allows for less impact on users after the OWA consolidation for those using macs and those accessing resource URLs.

Exchange integration effort
Worked as the exchange resource for identifying current tools and desired tools for migration to the Exchange 2007 environment and ongoing tool requirements.

PowerShell Scripting

OCS User Enabling
Scripted solution automate the setting up of users for utilizing OCS. This saves considerable time when going through this process.

Cluster Resource Script
Scripted the stopping and starting of cluster resources. Still working through the permissions to be able to run the script remotely from SCOM.

MAR – Jun 2009

2009-06-29T18:20:11+00:00

"Wins" for June…

DNS BIND VIEWS ARE NOW IN PRODUCTION!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Sure, it took a LONG while to get in place, but it was implemented without any clients needing to change any of their configurations or any changes to all of the ICANN/ARIN registrations. The implementation itself was relatively transparent to the University as well. Initial RFC1918 entries are live on Internal views and will be pushing out more as the service is publicized to the University.

DHCP Failover with the BT INS appliances has been solved. Well, not solved, but a functional work around for the ISC failover bug has been tested and implemented. This will allow us to FINALLY start the migration onto the new DHCP environment and get rid of those old, clunky Sun v210s & v240s.

MARS 6/09

2009-06-29T15:37:57+00:00

Well this month we finally managed to get all of our Wintel servers upgraded to ePO 4.0, sadly we still have one server that is still running VSE 8.0. Hopefully I can get that one upgraded to 8.5 this month, so I can close out that SWE as well.

I spent most of the month, preparing for, testing and then deploying WSS 3.0/MOSS 2007 SP2. The upgrade in our test farm went smoothly, and for the most part it went smoothly in production. I only ran into one unexpected snag in production. After extracting both WSS SP2 binaries and the MOSS SP2 binaries, when I ran the setup wizard, it never prompted to start the wizard on the other servers, and then failed to install. The fix was to reboot the entire farm, and then run the wizard again. At this point, everything proceded as it was supposed to, and the upgrade was a success. (that is after going in and adding our license key again, would’ve thought Microsoft would’ve released an updated service pack to resolve that little issue by now).

I also finished updating our SharePoint installation documents, to include not only installing SharePoint and Project server, but creating the default SSP, mysites, etc. As well as all post SharePoint configuration changes/application installs that we’ve made to our production environment. I’m still working on merging those changes into our DR document, but will hopefully have that completed shortly as well. Anyway, the big positive here, is that we now have a step by step document for every step needed to reproduce our production environment with every addon, third party app, and custom configuration setting.

Once I finish updating the DR document, I hope I can find the time to polish our back up script. Currently everytime a new site or mysite is added, we have to manually add it to the backup script. I’m pretty sure, that we can completely automate this process, I just need to find the time to work on it. As it sits now, I update the script once a month, and again before any major change, but that does leave holes in our backup process.

MAR – for June 2009

2009-06-29T15:16:17+00:00

2.) I resolved 41 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail. I am currently working on cleaning up old data on IMAP1/IMAP3.

5.) Assisted project to upgrade Drupal. I am currently researching this task.

6.) Completed task of removing surplus equipment from inventory.

June 2009 MAR

2009-06-29T15:11:40+00:00

Rewrote virtual machine inventory script in Powershell – The old script utilized perl, and I rewrote the script using Powershell and the vSphere Power CLI. The data is written to a MySQL database and I utilized the ODBC drivers to connect to the MySQL database.
Normalized Runbook data with the NOC inventory data – Due to naming mismatches and servers located in datacenters not managed by the NOC, we had servers that were in the Runbook and not in the NOC inventory. In order to give accurate counts on our inventory, it was necessary to normalize this data.
Recreated Weekly Helpdesk ticket in SQL Reporting Services – Previously a sysadmin had to manually calculate the number of helpdesk tickets in the various states. Now the report automatically calculates this information.

June MAR 2009

2009-06-29T14:59:22+00:00

Google search appliances arrived and were setup. Public Affairs is working with the appliances now to ensure appropriate spider settings and also customize the user interface. Failover testing and a more open community test will be done in early July. Production date is scheduled for the second week of July.

AmCom came on-site to perform the replacement OSIS server installations. The vendor decided that they would prefer a clean install versus upgrading he current systems. Using their software specifications, 4 new servers were built and turned over to the vendor. Upgrades went smoothly once a few minor software dependency problems were resolved. The vendor continues to work on client-side application upgrades. Production is planned for the middle of July.

The web pages for supporting IP/DNS requests for subnets which have been migrated to the DiamondIP appliances have been re-written and are ready again available for use. The perl API supplied by the vendor stopped functioning after a recent update. The vendor was disinclined to support the perl module which they supplied because it is not part of their standard support package, so a new solution was found using their CLI API which is supported. The pages which utilized the perl API have now been rewritten in PHP and should be supported by future updates.

LISTSERV began to exhibit performance problems. Worked with the email team and the vendor to troubleshoot the issue. Neither the vendor nor ourselves were unable to identify a definite root cause of the performance problems. The vendor suggested that we try using their High Performance Option (HPO). After obtaining a trial key and testing to evaluate the performance differences it was decided to place the key on our production host. At the same time a new PHP login page was activated to replace the previous perl CGI script, which will give more feedback to users.

Work and testing continues on the SMTP replacements. So far we have had successful tests of inbound, outbound, and vacation routing and delivery. We hope to be ready for a production SMTP replacement very soon.

Still no tasks for me from the GMail for Life project. IDEV are continuing to work on their solution.

PBX Pager project will have some tasks for me soon as the iDEV group draw closer to being ready for production. Testing on JTEST1 continues and is going well.

The Nagios replacement has been put on hold due to higher priority projects. There are still some issues with getting the logging agent to correctly connect and log to mySQL.

wins 6/2009

2009-06-29T14:24:00+00:00

SMTP

ITS has been looking into replacing the hardware supporting our mail delivery system for the University. Currently their are 5 inbound servers, two outbound servers, and two auto-responder servers running on physical hardware. ITS will consolidate these physical boxes into virtual machines as a result of this replacement effort. ITS successfully completed a POC and is now looking into a small testing initiative to continue this effort.

Exchange Consolidation

ITS is consolidating the VUMC and the University exchange environments in an effort to reduce overhead eliminate unnecessary redundancies. The effort will involve deploying a new exchange 2007 environment with an email archive offering, and systematically migrate users from University and VUMC to it. This month ITS successfully provided an architecture for storage solution as well as the email archive platform from concept to vendor, and has begun initiating purchasing processes for both.

BlueARC Upgrade

This month ITS expanded the capacity of the HPC NAS environment by 35TB of RAW capacity. This enables ITS to take on additional customers in need of HPC NAS space for research endeavors. However, in order to facilitate the new request, ITS has had to perform a number of maintenance task involving moving file systems around to free contiguous space on storage frames. ITS has nearly completed all maintenance tasks needed and will be complete in the coming week or two.

Virtual Desktop

ITS has been evaluating the plausibly and functionality of a Virtual Desktop Offering to the community to assist in lowering TCO of desktop computing resourced. This month ITS successfully created a POC environment using OS and application streaming to alleviate unnecessary computing and storage resources from being oversubscribed, but underutilized.

2009 June MAR

2009-06-29T12:09:53+00:00

Monthly Activity Report
June 2009

AIMWorX – Supported all AIMWorX users varying requests. Processed the weekly update of cost centers and student information. Ran a mock billing cycle on AIMWorX3 to create a BSITM.gl file because this data was not imported into the production database for proper billing.
AIMWorX Templates – Completed new work order template for the OCS project to setup billing in AIMWorX and to request an OCS account. These templates were imported into the production database.
Accordent Trial – Worked with an Accordent technician to correct issues their recorder was having with the servers scheduling software.
Higer Ground backup Server — New OSIS-CR02 server was recieved from vendor and racked in Bryan Building. Setup server to meet VU server standards and put into production. Worked with Higher Ground technicians to complete setup of server. Copied some detail files from old server to facilitate a quicker setup. Retrieved log files for HG prior to cleaning the hard drives of the old OSIS-CR02 server and sent back to vendor.
NEC License Manager for MA4000 — Installed License Manager software and discovered that passwords were stored in clear text. Since the ’sa’ password was used during setup this software was disabled until this was changed. Opened a ticket with NEC and worked with a technician to remove the ’sa’ password and setup a new login that is specific to this database. Also, setup https for web interface. Once these issues were resolved the License Manager software was turned back on. The LM software looks to FPC01 for PBX verification. Since FPC01 has problems with connections, the LM software is not working. This causes MA4000 to not function properly because it thinks it is not licensed. The PBX technicians have an open ticket with NEC concerning FPC01 connection issues.
Sharepoint Servers — Rebuilt test Sharepoint server (its-hcwnap51-ts & its-hcwnap52-ts) per Roland’s request using the virtual server template.
AT&T BMS file changes — Business & Planning is moving some charges to a new billing structure that is requiring changes to the BMS process. Worked with Paul Lockaby on changes to these changes.
Virtual Servers — Created 3 servers (its-hcwnem98, its-hcwnem18, & its-scwnem19) for the Email team as requested and approved. Rebuilt template for Windows 2008 32bit server because it was corrupted. Updated OS on all Windows virtual templates.

June 2009 Activity Report

2009-06-25T14:45:20+00:00

This month has been mainly spent working on finishing the details associated with the E-mail Archive project. In addition to that, there has been a lot of work around the Exchange 2007 storage backend.
We installed 3 new DAE’s of storage into the BlueArc high performance NAS environment. We are already making use of the space by moving departments around and optimizing the storage underneath their file systems.

The backup environment has been relatively quiet. We have included growth room in the Exchange projects to handle the upcoming load we are expecting from the deployment of those services. Additionally, we are starting to see a need for additional storage in our NDMP pools to support NAS backup.

First offical week so far…

2009-06-01T12:56:08+00:00

Well my first official week with App Hosting started on Tuesday, May 26th, 2009. I finally have everything moved over from my ole’ home at NCS, and into my new abode here at the University. With regards to Acorns, so far so good.

There is plenty to do with regards to Exchange 2007/2003, OCS, Unified Communications and the prospect of a new email archiving application. Here’s to the future.

‘Nuff said.

MARS 05/09

2009-05-29T15:56:43+00:00

We discovered this month that by default hibernation is enabled on all Windows 2008 installs. Thereby potentially consuming large amounts of disk space depending on the hardware in the server. As we discovered with our SharePoint servers, we had an 8GB hibernation file on each server. Fortunately this is easy to fix, by simply opening an administrative command prompt and typing <powercfg.exe -h off> to disable hibernation.

Our SharePoint rollout should be complete and officially in production by the end of this month. So with that being said I spent some time updating all of the support documentation that goes with that, and discovered it hadn’t been updated since prior to our upgrade to 2008 on all of our SharePoint servers, so all of the instructions for setting the SSL host headers were wrong.

I prepped our test SharePoint farm to do some DR/backup/restore testing via Legato, only to discover that the current version isn’t supported if your backup server is running on Linux, so that appears to have been a wasted effort. Supposedly the next version will correct this issue, I guess we’re back to waiting for EMC to provide a viable backup and restore method for SharePoint.

I spent quite a bit of time working on ISA this month. ITS-HCWNAP61 is now officially in production supporting our OCS deployment. I rebuilt and prepped ITS-HCWNAP60 to join the array, and discovered several potential issues while testing the change in our test environment, and have had to defer my original change.
• It appears the issue that caused us to rebuild our ISA servers repeatedly, where whichever node didn’t house the CSS would start to deny all incoming requests, is actually due to how NLB utilizes unicast, and most switches aren’t configured by default to allow all traffic destined for a specific MAC to be sent to multiple switch ports. I’m currently testing a potential fix for this in our test environment.
• In researching the previously mentioned problem, I stumbled across several things we weren’t aware of when we initially deployed ISA. When using NLB, intra-array communication cannot pass on any interface using NLB. Also, it is recommended that your Intra-array communication reside on a private network, a daunting challenge in our environment, since this private network also requires AD access, and is heavily used for Kerberos authentication, as well as passing configuration back and forth between all array members.

MAY MAR

2009-05-29T14:42:50+00:00

Completed migrations of Nagios, Owl, and VUNetID from the AMD Cluster to the Intel 7100 Cluster. Nagios was moved earlier than planned due to problems within the AMD Cluster which we were never able to explain. This completes the migrations I needed to get done before June in preparation of decommissioning the cluster.

The AMP2 memory upgrade was completed. Verified that the system was properly addressing the additional 8GB of memory both in the BIOS and the OS after the upgrade. Critical firmware patching was completed at this time as well.

Worked with Jeff and Gary to troubleshoot problems with SMTP and IMAP servers after one of the servers were removed from service and the user accounts were migrated to other hosts. Most of the issues with the SMTP servers and VUWebmail were resolved quickly. Peter has continued working with Jeff to work through minor problems with some user accounts in the IMAP environment.

The proposal was finally submitted for the search engine replacement. The current solution’s licensing expires the end of June. Given the time constraint I looked into an at least temporary solution using the Google Search API. This was presented to members of ITS and Public Affairs and was deemed a good idea but was not a robust enough solution. After doing some other research I started testing an open source solution called ‘Sphider’ which is php/mysql driven and has its own web crawler. This solution was dropped due to concerns of using open source, though it would have been a nearly 1-to-1 replacement to our current search engine.

Judy was assigned the task of decommissioning the 2 Kiosks located in the hallways in the Hill Center. She requested help in doing so since I had done the original installation and setup of the Kiosks. I told her I would take care of it. She said to let her know if there was anything she could assist with and asked that the televisions remain mounted on the walls with CNN being shown. I enlisted the help of Warren and after removing the NEC Bluefire appliances it was discovered that the televisions did not have built-in tuners. We located 2 unused VCRs in the building and connected them to the televisions after obtaining permission to use them.

Continued meeting for the GMAIL for Life project. IDEV is still working on their parts of the project. No new activities have been assigned for me as of yet.

It was discovered that the OnSite::API perl module provided by BT Diamond IP is no longer working properly, most likely due to recent patches. After more detailed troubleshooting it was shown that the API is in fact working, but is returning an NULL array for the specific queries needed for integration into our website. A ticket has been opened with the vendor concerning this on May 21. We are still awaiting a response from the vendor on the ticket.

May MARS

2009-05-28T21:19:48+00:00

Moved from NCS to ITS

Closed final two Service Center tickets

Closed 31 Service Manager Tickets

Exchange

Personal account migrated to Exchange 2007. Currently experiencing OWA and mail-sending issues.

Completed three days training for class 5047 - Introduction to Installing and Managing Microsoft Exchange Server 2007

Have been assigned a task to identity the types of resource accounts that reside on the Medical Center Exchange 2003 email servers.

Windows

Requested a standard template build for a Windows 2008 Server VM named its-hcwnem99

ITS

Introduced to Sharepoint Exempt leave site

Introduced to other ITS employees

Excited to be working for the ITS app hosting team

May 2009 MAR

2009-05-28T19:03:55+00:00

May 2009 Monthly Activity Report

1) Network reconfiguration of vanderbilt_production mailgates was incomplete. This change was related to issues where adding hosts and DNS entries via the Admin GUI can cause the hosts entries to be deleted or adding DNS servers can lock up the host. The issues can be resolved by assuring NIC#1 is configured on the same VLAN as the gateway as proven in the vanderbilt_lab cluster. Ran into an issue with mailgate02. It would not save the new configuration when reconfigurng via the Admin GUI or console. Opened call with Proofpoint and provided logs. They have recommended removing the agent from the cluster and reconfiguring then adding the agent back to the cluster. Note maillgate02 and mailgate03 reconifguration was not completed during the change window and must be rescheduled.

2) Deployment and configuration of LISTSERV development server has been completed with one exception. DNS MX record must be created.

3) Found method to provide list of all group mailboxes, their ACLS, and the last time they were accessed. Have supplied data to project manager.

4) Gmail for Life. Met with iDev team in order to determine a strategy for populating the Proofpoint user repository used for sapm policy assigment and recipieint verification. Provided them with user repository export and configuration documentation so that options could be explored.

5) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.

6) Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes. Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives. Investigated over 100 spam false negatives.

7) Performed daily management of mail queues on mailgates. Removed hundreds of undeliverable messages daily in order to keep queues “clean”. Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

8) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

9) Worked on an assortment of odd and challenging helpdesk tickets.

May MAR

2009-05-28T17:12:34+00:00

Sun Identity Management: The networking issue for the virtual machines located in the VUH data center has been resolved. Our intermittent network issues were cause by a shared IP address between F5 LTMs and a Cisco router. This caused the VMs to lose outbound network connectivity when the IP jumped to the router. The IP address was removed completely from the rouer, and network connectivity has been restored. We are moving on the final testing phase. We are working through the final steps for getting SLAMD setup.

[WIN] ITS Website Redesign: The cutover to the new ITS website is scheduled for this weekend. The change will involve exporting the test database, cleaning up all of the links, and importing the data on the production server. I’ll also need to copy the necessary files over to the production server. The last step will involve moving the IP addresses from the old servers over to the new servers. The CSM will pass the incoming connections to the new servers.

PHP5/MySQL5 Upgrades: The cutover deadline for the cut over of the first round of server has been extended until mid-June. This round of migration is progressing slowly. There are 58 folders left on the WWW4 servers 29 virtual hosts left on the vanity domain servers. This remaining content is under review by the site owners and awaiting migration approval.

MAR – for May 2009

2009-05-28T16:50:31+00:00

1.) Coordinated 234 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 42 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission IMAP2. Accounts from IMAP2 have been moved to IMAP1 and IMAP3.

5.) Assisted project to assist with decommission of VUMail.

May MARS

2009-05-28T16:32:44+00:00

22 Closed Magic Tickets

Exchange
38 Migrations to Exchange 2007

Windows

Exchange

Cross Training
Assist in cross training new messaging employees with current configuration, setup and procedures for the VU Exchange Environment.

Exchange 2007

OWA Vanderbilt Rebranding work
Installed new test\production CAS for internal testing in the production environment before implementation. This CAS will also be utilized for testing OWA integration.
Finished the OWA Rebranding and put it into production. Continuing the documentation of the .css files on my time for easier modification in the future if the need arises. Modified the test CAS setup to allow for multiple images at the top of the OWA login page to accomodate MC and VU logo’s for the combined environment.

OWA timeout issue
Worked with Peter in identifying the issue that was causing our 2007 OWA timeout issues. Tested solution on the test/production CAS successfully. Peter is putting in a change for implementing ths change on the production OWA Load balanced ip.

OWA integration effort
Worked on identifying issue that is preventing OWA 2007 from connection to VUExchange 2003 mailboxes. Still have not identified the complete root cause, it appears as if it is because the default internal delivery address for the VUExchange users is different from the default proxy address for the Exchange Enterprise. It is still being researched as to why this is happening

OWA 2007 Change Password
Identified the method for disabling the change password option for OWA 2007. This is important so as to not cause confustion issues with end users since this functionality will not work since AD is not authoritative for passwords and current ad permissions do not allow for changing the password in this fashion.

UM
Identified issue with loader process overwriting UM and OCS proxy address information in AD.

PowerShell Scripting

Mailbox Migration
Scripted to help automate the process for larger scale migrations. The tool automatically picks the storage group with the least number of mailboxes currently on it and creates a mailbox move script for moving each user that is listed in the users.txt file that is populated with the users to be migrated from 2003 to 2007. The next portion of the script that I am working on is automating the start of each of the created user move scripts. The tool will run up to 10 migrations at a time and then prompt for the script to be run again. It keeps a list of all the users that were moved and the dates. Passed off portions of the script to Denny for possible use in the new Exchange 2007 loader script.

UM and OCS metrics
Scripted solution to pull a count of OCS users, Voice Users, Federation enabled users, Internet enabled users and UM users and mail out the information to interested individuals. Continuing work on automating this and writing the data out with date information to a database for ongoing automated metrics.