Tony’s Place

27 Closed Magic Tickets
2 Security Incidents
Exchange 2003
46 IMAP -> Exchange Migrations
Exchange 2003 Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches
Buildout of ITS-HCWNEM01,02,03
In preparation of quota increase to 1GB for VUExchange, extensive work to get these boxes to pass security scans. The boxes had to go through security scans at least a half dozen times. With some boxes passing one scan then failing another with no new configuration. Also boxes failing then passing after only a reboot.
Buildout and install of Exchange for ITS-HCWNEM12
In support of life cycle replacement of the Exchange Front End servers
Verify and set EPO settings for Exchange servers
Troubleshoot/resolve issues with Law School students OWA and outlook display defaulting to Chinese characters
Troubleshoot issue with ~30+ users configured to activesynch against vuexchange environment when they have no exchange mailbox
Information gathering in support of quota increase to 1Gb for VUExchange
Continue work with Microsoft to resolve w3svc errors when recycling the application pool.
Research IMO Legacy Exchange DN as well as all objects with its-exch.mail.vanderbilt.edu proxy addresses in 2003 AD
Identify/Troubleshoot loader AD

Exchange 2007
Setup\writeup procedures for using Apphost-resource account for accessing 2003 ITS resource accounts for users in 2007 environment
2007 Exchange Public Folders
Fix public folder permissions of ITS Folders
Exchange 2003 mailbox migrations to Exchange 2007
Create connector for email.mc.vanderbilt.edu to avoid bouncing of mail to contacts that don’t exist in SMTP.
Troubleshoot /resolve UC2k7 startup issues due to Power outage change.
Troubleshoot/resolve Zafars connectivity issues with his client to the Exchange 2007 Mail Servers
Work out solution to allow Zafar to send as NOC@vanderbilt.edu, since this address resides/belongs to Exchange 2003.
Research/Troubleshoot Windows Server 2008 backup of Exchange 2007
Resolve network issues with ITS-HCWNAP12-ts.uc2k7.vanderbilt.edu

22 Closed Magic Tickets
1 Security Incidents
34 Imap -> Exchange Mail Migrations

Exchange 2003

Exchange 2003 Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches including Windows 2003 SP2

Document Exchange Front End Installation for Exchange 2003 Production environement
Allows other Admins to build front-ends and documents our current configuration.

Document IMAP->Exchange 2003 migration process
Created step by step documentation that has been followed by Guy to perform IMAP->Exchange migrations.

Buildout of New Exchange Boxes for 2K3 environment
Its-hcwnem10 and 11 were built and put into production as front ends in the email.vanderbilt.edu NLB. These boxes are actually in AD DNS which is a first for us. There was a small issue resulting from AD DNS not being authoritative as reverse DNS which was resolved by putting the entries in IP-SRV1 and IP-SRV2.
Its-hcwnem12 has been built and is being utilized for testing new 3rd party firewall to more securely protect our windows boxes
Its-hcwnem01-03 have been built and will be introduced to the environment to continue the VUExchange capacity enhancement project.

Worked with Microsoft on Front End Application crashing issue
The actual crashing issue was resolved via Windows Server 2003 SP2. The underlying cause of the w3svc service timing out when closing is due to aspx support on our front ends. Certain clients are requesting an aspx page from an ews subsite that does not exist in Exchange 2003, It is a subsite that exists in 2007. To get around this the new Front-Ends were configured without the aspx support which has resolved the issue. We will be applying this same configuration to the current Front-Ends.

RM for exchange
Worked with EMC to recover the RM admin password after my laptop crashed.

Exchange 2007

Document Procedure for Migration of mailboxes from 2k3 -> 2k7 environment

Perform migrations for ITS Sysadmin Windows team to 2k7 environment
This is to allow for further testing of the environment and show proof of concept for UM and UC.

Extensive troubleshooting to resolve various client issues in the 2k7 environment
Resolved Outlook Integration Issue with OCS
Fixed Free/Busy synch between 2k3 and 2k7 (except for confrm’s still working on this one)
Identified DNS issues that were resolved.
Documented and opened firewall ports to resolve issues with both normal MAPI connectivity to Exchange 2k7 as well as UM.

General Windows

Update Windows Server 2003 Installation Document
Update new procedures. Remove IP addresses from old SNMP Settings. Add new IP’s for backup FW rules. General cleanup.

IBM Director
Worked with IBM to try to identify the error message that we have been receiving on MSSql-SRV2. They have requested that the firmware on the Raid card be upgraded to the latest level. Still waiting on this to be able to get back with IBM on the error message.

CiscoWorks
Worked with Victor in setting requirements and speccing out a server to replace the CiscoWorks server.

SCOM Agent SP1 upgrade and AD DNS
Identified issues with SCOM and lack of presence in AD DNS along with the boxes that it is monitoring. This issue is actually much larger with any of our boxes going into AD DNS with a disjointed name space (any dns name that isn’t ds.vanderbilt.edu) while cnaming these in the future will work changing the dns name on the box and on certain services (mail, scom) after install would cause major issues. This is being addressed in the VUExchange environment with the new boxes being added to AD DNS with the .ds.vanderbilt.edu namespace.

28 Closed Magic Tickets
1 Security Incidents
36 Imap -> Exchange Mail Migrations
3 Exchange -> Exchange Mail Migrations

Exchange

Email Account Bouncing Cleanup after IDM cutover
Worked with IDEV to identify and cleanup issues that occurred with new exchange mail enabling after the switchover to IDM. There were a couple of different issues identified. In a few cases the exchange delivery address and only the exchange delivery address was not populated to the AD account. This caused mail to bounce back and forth between exchange and the proofpoints, this issue was assumed resolved by putting the ad and exchange loaders back on the same machine. I am not sure this was proven as the fix or not. There were also a couple of accounts who seemed to have been provisioned properly on all systems but Exchange would not deliver the mail to the mailbox it seemed to think that the mail should go external to the Exchange environment. The AD object for the user had all the correct email addresses on it and you could look on the Exchange General tab to see where the mailbox existed. SMTP had the correct exchange delivery address for the user also. The resolution for this issue was a deletion of the Exchange account and a clearing of all mail attributes then recreating the mailbox.

Setup new Enterprise Postmaster account for the Exchange Environment
The old postmaster account was Greg Marlin’s 3 letter account which had been deleted. When uninstalling Exchange one of the checks is to see if the Enterprise Postmaster account is housed on any of the Mailstores on that server. The issue that I suspected we were running into was that the environment didn’t know where the mailbox lived since the underlying account was deleted. After making this change the removal of MAILFE05 was a success.

MAILBE16 System Attendant account cleanup
During the change last month to reconfigure the mailstores the System Attendant account was left on a Mailstore that we were not wanting to keep. I had to change the location of the account in adsiedit and restart the Information Store so that we could delete this superfluous exchange database.

Stealthbits evaluation
Evaluating Stealthbits Monitoring for exchange to initially provide monthly metrics numbers for the University Exchange environment. This can also be used much more extensively with some time and work to assist in usage projections as well as troubleshooting and configuration management. I am currently working with Stealthbits to create a custom report to mimic the one we are currently getting out of Messagestats.

Exchange Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches

Configure and Document procedure for removing specific attachments from Exchange
Utilizing Trend Scanmail setup a spam rule to replace a specific file name with a text file saying this file is against policy.

Assist in writing Documentation detailing the levels of Shadow Mailboxes that we can provide
This is to allow us to have a proper request for security incidents.

General Windows

Monthly Microsoft Server Patches
Monitored, verified and tested services after the monthly Microsoft patches for our Windows Server environment

Sharepoint forensics for outage on 6/23/2008
Investigate Sharepoint outage and resolve the issue. Move the servers to the new ESX machines also in the proces

22 Closed Magic Tickets
3 Security Incidents
26 Mail Migrations

Exchange

HRSurvey user creation and setup for Rich
Utilized for an employee survey that is being performed.

Reconfigure of MAILBE16 Storage Groups and Mail Stores
This was done to assist in resolving performance issues. The reconfigure lowered the IOPS overhead caused by multiple mail stores on the same drive causing the drive heads to move back and forth between them. This also allowed us to balance the Mail Stores between MAILBE16 and MAILBE17.

Communicate concerns on IMAP -> Exchange Migration tool
Communicate concerns and push for fixing of the IMAP-Exchange .NET migration tool to not have a dependency upon the user/password list. This is to avoid issues once user provisioning moves to IDM. Also pushing to have the Unix side of our new migration process automated.

Cleanup of MAILBE16 reconfigure
I was here for the start of the reconfigure and assisted in getting the drives configured and setup for the mailbox moves. I was gone for 5 days following this at a school camp with my son. When I got back I took over cleanup and finishing the last vestiges of the move.

Stealthbits evaluation
Evaluating Stealthbits Monitoring for exchange to initially provide monthly metrics numbers for the University Exchange environment. This can also be used much more extensively with some time and work to assist in usage projections as well as troubleshooting and configuration management.

Troubleshoot/change primary delivery address for conference rooms
There was an issue with conference room auto-accepts and the UC2k7 environment that was caused by the its-exch delivery address not being accepted as an acceptable address on the proofpoints. Resolved it by changing the primary smtp address on our conference rooms to intemail.email.vanderbilt.edu.

RMSE -> RM upgrade
Finished at the beginning of the month. This is the backbone of our DR environment for Exchange.

Troubleshoot Exchange Backup times
Assist in trying to determine the issue causing Exchange backup times to take an inordinate amount of time on Hydrastore.

MISC

Assist in reviewing AD changes for Sharepoint Group Mail

Vacation
I was out of the state in DC/Virginia for 8 working days this month as well as 3 weekends limiting the available time for changes/work.

Exchange DR RM
Been working diligently with EMC onsite trying to get our Exchange DR issues resolved. The flare upgrade on the Clariion from 24 to 26 caused the RMSE product to not be supported despite direct concern and questions to EMC of this. EMC has provided the RM product for us to upgrade to. Initial runs of the product were successful in both the test and production environment. Subsequent incremental runs in the production environment have not been successful and efforts to identify the issue causing this have been for naught thus far. Fifty plus hours have been spent this week by me alone working with EMC directly to keep the pressure on and working to attempt to resolve the issue.

Exchange DR Replistor
There was a vulnerability identified in the Replistor version 6.1 SP2 that we had installed. Previous installation of 6.1 SP5 crashed MAILBE13 and the change was aborted. This was a follow up of that with an upgrade to 6.2 SP4. Upgrade went smoothly with no issues. Replication of the Exchange Database Logs is working without problems again. This is one part of our Exchange DR strategy. This portion keeps the transaction logs synchronized between data centers. With any current copy of the database these logs can be applied to the database to bring us to a specific point in time.

DPM for Sharepoint
The DPM server is built and has been tested in backing up a file share. Test Sharepoint farm has been moved to the 10.2.170.x network to allow testing of DPM sharepoint backups. Agents have been loaded on the test sharepoint servers (itssharept01, itsproject01 and its-hcwnap32-ts) and registered with the DPM server. We are currently waiting on the test SQL cluster to be moved to the 10.2.170.x network also. Once this is done agents will be installed on these also. This will allow us to move forward with testing DPM’s capability to backup sharepoint at a high level and restore granularly.

Chancellor Public Folder Fix
Created chancellor@vanderbilt.edu public folder for all incoming mail to be delivered to. There is an application that is utilized to sort and manage these messages. It was not able to manage messages that were stored in Note format. Default behavior for Exchange 2003 public folder messages being delivered from external Exchange cloud is storing the message in IPM.Post format. After consulting with the MC Exchange team to ensure that our change would not affect them I applied a registry change to our Public Folder Back End servers that changed the behavior of incoming public folder messages so they would be stored as IPM.Note format (or look like mail messages rather then notes). This resolved the issue for the application managing the folder.

PRT1 Spooler fix and scramble to avoid crash
While oncall a space warning appeared for PRT1. In investigating this I noticed that there was a print job spooling to the data drive that eventually grew to over 3G in size. Quick work moving data off the C drive as the job grew kept the OS drive from filling up and crashing the server. Subsequent change that evening added a second drive to the box and moved all the print queues to this drive.

April Microsoft Patches
Applied to test servers and workstation groups in proper order for testing. Scheduled change for ITS production servers and Roland monitored the application of them while I was on vacation. Applied same patches to the Exchange Servers the following week. Adding one Hotfix associated with the DR RM project/work effort.

Magic Tickets
16 Magic Tickets closed
4 worked on and forwarded
Various AD tickets forwarded to the AD team. After a discussion I had with Brian Britt on the AD team last month before I went on vacation I have started forwarding any AD related tickets onto the Directory Services team.

DPM test environment setup
Built Windows 2003 x64 DPM server to test the System Center Data Protection manager 2007 ability to perform high level backups and granular restores. The server has been built and tested for backing up files. Waiting on Sharepoint test farm with recovery farm server to be built to continue testing

Brainstorm/Test IMAP migration alternative
Worked with Peter to figure out/test an alternative solution for IMAP to Exchange migrations. Migrations on the 18th of March were completed utilizing this method. This is to allow us to continue doing these migrations without having to have the users password. Moving forward with Sun IDM the process to get the users password will no longer be available, and having the users password in plain text was not a desirable solution anyways.

Performance Tuning for Exchange
Various changes were made to the University Exchange environment in an attempt to help alleviate performance issues that some users are experiencing. More plans are in the works to reconfigure the storage groups to a more optimal setup and rebalance the users across the storage groups.

Sun IDM/Microsoft Exchange/AD patch testing/Loader Code
Tested the need for a piece of AD loader code that was implemented in Aug of 2004 to resolve errors associated with exchange mailboxes. Testing was unable to reproduce the errors that the code was implemented to resolve. Determined that the Exchange patch to resolve the errors was thus not needed.

Manage Magic Tickets for Apphosting Group

22 Closed Magic Tickets

Cursory cleanup of apphost Vu Group Space

Assist w/uc2k7 setup and troubleshooting

33 Closed Magic Tickets
Exchange
Test & Implement Custom over and warning quota messages for University Exchange System
1 VUMC – University Exchange Migration
3 Exchange Restore
Exchange Distribution List Creation
8 Exchange Resource Account disable and delete
IMAP – Exchange Migrations
3 Mail Enable Public Folder
13 Manually Mail Enable Public Folder
Upgrade to McAfee 8.5 on University Exchange Servers
Reset of Public Folders on MAILBE17
Localize Perfmon location to MAILBE13 to remove processor and memory utilization on production University Exchange Servers
Fix Public Folder Permissions for Divinity
Resolve issue sending from Exchange test environment via host entries to bypass CMS spoofing rule
Localize Perfmon location to MAILBE13 to remove processor and memory utilization on production University Exchange Servers

4 Vuspacegroups Restore

ITS Sysadmin VUGroupSpace cleanup

IBM Server Inventory Work

4 Resource Account Creation

Root Password Changes

 44 Closed Magic Tickets
 23 Imap Mail Migrations
 Hardware Monitoring
 IBM Director reporting for ESX system to NOC for critical alerts and to Unix team for All.
 Troubleshoot issue with San Manager Fabric Install on Strawberry
 Upgrade Jump01 Memory to 4G for better performance on Bastion Host
 Troubleshoot Document Tree issue in Sharepoint
 Resolve ITS-MOM01 errors mailing out
 Assisted in diagnosing Mailgate04 issue after migration behind the CSM
 Tested and implemented Legato for Exchange Backup dropped our backup window from over 17 hours to around 4 in the Exchange environment.
 Lowered disk consumption on Exchange environment with change of backup. Returning 1Tb of Tier 3 disk from the backup drives on MAILBE16 and MAILBE17
 EXBPA (Exchange Best Practices Analyzer) run in production environement, looking to improve MAILBE16 performance.
 Installed HP SIM agents in Exchange environment
 Troubleshoot/resolve RMSE/VSS Exchange outage
 Fruity configuration for Exchange and Groupspace boxes
 Configured EPO exclusions for Exchange environment and 8.5
 Upgraded McAfee to 8.5 in Exchange Environement
 Assist with Pertrac installation
 Kathryn Foot and Rachel
 Resolve Bounced mail
 Resolve Phone synching issue
 Setup podcast room permissions
 Quest Message Stats
 Continued troubleshooting on Mailbox Content
 Mailbox Content fix has been delivered and testing has started to resolve the issues with full runs crashing when mailboxes have been deleted or moved after the run has started.
 Resolved Mailbox Account Properties Gather issue.
 Restores
 4 VUSpacegroups restores
 4 VUExchange restores
 DPM 2007
 Literary Review completed
 Tech documents presented for approval
 Creation of 7 Exchange Resource accounts for calendar/group mailbox use
 Managed Magic Tickets for first part of month while Jeff was out
 Resolve Exchange permission issue for Jason Bradley to create distribution lists within AD
 Exchange Test Environment
 Monthly Exchange patch testing
 HP Sim Agent Testing
 RMSE issue
 Temporary hotfix of mailflow issue with CSM VIP to VIP communication
 Tested changing Quota messages to users for Exchange
 5 VUGroupSpace permissions check/Changes
 4 Public Folder creation\permission changes
 3 VUGroupspace Quota checks
 9 AD OU creation\permission changes
 7 Exchange permission changes
 3 Exchange performance complaints on MAILBE16

• 24 Closed Magic Tickets
• 52 Imap Mail Migrations
• Hardware Monitoring
o IBM Director reporting for Windows system to NOC for critical alerts and to Windows team for All.
o Continued work with Unix team to get procedures and test machine setup for Linux/Esx boxes on IBM Director and HP Sim
• SCOM Project
o Continued cleanup of SCOM alerts to filter out “noise” for production consumption.
 Research threshold warnings that are appearing without appropriate data
 Research WINS error messages associated with corrupted packets (ended up being security scan related) Resolved issue by overriding the settings on this error to make them a warning instead of critical so it wouldn’t create a NOC consumable issue each week.
 Look into Mail Flow monitoring issues with MAILBE16MOM and MAILBE17MOM mailboxes
o Created documentation for NOC use of SCOM console.
• Virtualized Jump01 utilizing the P-V function of ESX
• Diversified University Exchange front end servers in the 148 Data Center to multiple racks to better protect the environment from specific rack power and network failures.
• Quest Message Stats
o Gave demonstration to the Sysadmin team of the types of reports that can be generated by Message Stats
o Continued troubleshooting on Mailbox Content and Mailbox Account Properties issues
 Mailbox Account Properties issue appears to be a communication problem with our PDC emulator in the AD environment
 Mailbox Content is not feasible to be used for large scale usage. A full run in our environment (just University side) would take more then a week if there were no errors. Doing this would also put a hold on the publishing of all normal gathers during the timeframe of the content gather. Also any mailbox changes (move of mailbox or delete of one) will cause the gather to fail when it hits that part of the content gather
• VUSpace replacement project
o Worked with John to fill out information on the various solutions for presenting
• Fix 30 day retention of logs on the University Exchange environment
• Replistor error issue researched and resolved by getting an exemption to the security run scans of the University Exchange environment for ports 7004 and 7005.
• Restores
o 4 VUSpacegroups restores
o 2 VUExchange restores
• Impromptu Training for Chuck Owen on Recover Deleted Items on the DOA team
• Creation of 12 Exchange Resource accounts for calendar/group mailbox use
• Troubleshoot corrupted mailbox appointments for Ronnie McDowell and Pat Jones that were causing reminders to pop up consistently for the same appointment that could not be cleared.
• WSUS troubleshooting issue with Jeff Youngblood that led to a discovery of an issue with WSUS 2.0 and subsequent migration of the service to WSUS 3.0

·         24 Closed Magic Tickets
·         90 Imap Mail Migrations
·         1 Exchange-Exchange Mail Migration
·         Quest Message Stats Installation/Testing
Installed OWA report pack to allow us a better view into the Exchange environment.  Opened 3 different tickets with Quest to work through issues we were having with then content analysis and mailbox account properties gathers.  Massage the content gathers manually in the meantime to attempt to get full numbers out of the Exchange environment based on aging and size of messages.  Setup automated gathers for the University Exchange systems.
·         Blade Comparison
Wrote up blade comparison of HP C7000 and IBM Blade Center Series H Blade Chassis
·         Continued VUSpace Replacement work
Attended Documentum presentation. Researched Northwestern Xythos-Blackboard addon.
·         Operational Training

Trained on manual creation of AD accounts.  This is utilized in times when accounts are needed for services and external access is not needed/desired.