I only have one major WIN this month…
PHP5/MySQL5 Web Server Migrations: The project to upgrade all of the shared web services to RHEL5 is essentially complete. All of the content has been migrated, and the sites are running smoothly. This migration gives the web community access to PHP5 and MySQL5. It also brings along a much more robust ModSecurity ruleset. So far the vast majority of the help desk tickets have been related to applications with the new ModSecurity rules.
18 Closed Magic Tickets
Exchange
5 Migrations to Exchange 2007
Windows
VUGroupspace
Identified issue with quota software being utilized to manage VUGroupspace. One of the shares is being reported as uitlizing much more space then is actually in use, as reported by Windows. The issue appears to be a bug in the Quota software. Change has been finished and testing is continuing.
Exchange
Assisted in troubleshooting and resolving various issues this month.
Archiving
Assisted in the process of choosing an archiving solution for the Exchange 2007 Environment. This has included attending various vendor presentations, providing system configuration information and challenging\questioning vendor configurations and abilities.
Cross Training
Continued cross training new messaging employees with current configuration, setup and procedures for the VU Exchange Environment.
Exchange 2007
OWA Vanderbilt Rebranding work
Passed on knowledge for the rebranding for management to Antwan on the Email team. Assisted in putting new logo on OWA Site.
F5 OWA traffic rules
Identified solution for F5 forwarding of vunetid url to delivery address url. This allows for less impact on users after the OWA consolidation for those using macs and those accessing resource URLs.
Exchange integration effort
Worked as the exchange resource for identifying current tools and desired tools for migration to the Exchange 2007 environment and ongoing tool requirements.
PowerShell Scripting
OCS User Enabling
Scripted solution automate the setting up of users for utilizing OCS. This saves considerable time when going through this process.
Cluster Resource Script
Scripted the stopping and starting of cluster resources. Still working through the permissions to be able to run the script remotely from SCOM.
(Curiosity is the new name for the Mars Science Labratory)
Well, first off, here are the "Wins" for June…
And that’s about it for the WINS…
For the LOSES…
1 gallbladder, some weight, my daughter’s tonsils & adenoids, and 1 uncle.
For the most part, it has been one exhausting month.
** EDIT ** And I need to learn how to spell…
Well this month we finally managed to get all of our Wintel servers upgraded to ePO 4.0, sadly we still have one server that is still running VSE 8.0. Hopefully I can get that one upgraded to 8.5 this month, so I can close out that SWE as well.
I spent most of the month, preparing for, testing and then deploying WSS 3.0/MOSS 2007 SP2. The upgrade in our test farm went smoothly, and for the most part it went smoothly in production. I only ran into one unexpected snag in production. After extracting both WSS SP2 binaries and the MOSS SP2 binaries, when I ran the setup wizard, it never prompted to start the wizard on the other servers, and then failed to install. The fix was to reboot the entire farm, and then run the wizard again. At this point, everything proceded as it was supposed to, and the upgrade was a success. (that is after going in and adding our license key again, would’ve thought Microsoft would’ve released an updated service pack to resolve that little issue by now).
I also finished updating our SharePoint installation documents, to include not only installing SharePoint and Project server, but creating the default SSP, mysites, etc. As well as all post SharePoint configuration changes/application installs that we’ve made to our production environment. I’m still working on merging those changes into our DR document, but will hopefully have that completed shortly as well. Anyway, the big positive here, is that we now have a step by step document for every step needed to reproduce our production environment with every addon, third party app, and custom configuration setting.
Once I finish updating the DR document, I hope I can find the time to polish our back up script. Currently everytime a new site or mysite is added, we have to manually add it to the backup script. I’m pretty sure, that we can completely automate this process, I just need to find the time to work on it. As it sits now, I update the script once a month, and again before any major change, but that does leave holes in our backup process.
1.) Coordinated 138 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.
2.) I resolved 41 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.
3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.
4.) Continuing process of assigned project to decommission VUMail. I am currently working on cleaning up old data on IMAP1/IMAP3.
5.) Assisted project to upgrade Drupal. I am currently researching this task.
6.) Completed task of removing surplus equipment from inventory.
Google search appliances arrived and were setup. Public Affairs is working with the appliances now to ensure appropriate spider settings and also customize the user interface. Failover testing and a more open community test will be done in early July. Production date is scheduled for the second week of July.
AmCom came on-site to perform the replacement OSIS server installations. The vendor decided that they would prefer a clean install versus upgrading he current systems. Using their software specifications, 4 new servers were built and turned over to the vendor. Upgrades went smoothly once a few minor software dependency problems were resolved. The vendor continues to work on client-side application upgrades. Production is planned for the middle of July.
The web pages for supporting IP/DNS requests for subnets which have been migrated to the DiamondIP appliances have been re-written and are ready again available for use. The perl API supplied by the vendor stopped functioning after a recent update. The vendor was disinclined to support the perl module which they supplied because it is not part of their standard support package, so a new solution was found using their CLI API which is supported. The pages which utilized the perl API have now been rewritten in PHP and should be supported by future updates.
LISTSERV began to exhibit performance problems. Worked with the email team and the vendor to troubleshoot the issue. Neither the vendor nor ourselves were unable to identify a definite root cause of the performance problems. The vendor suggested that we try using their High Performance Option (HPO). After obtaining a trial key and testing to evaluate the performance differences it was decided to place the key on our production host. At the same time a new PHP login page was activated to replace the previous perl CGI script, which will give more feedback to users.
Work and testing continues on the SMTP replacements. So far we have had successful tests of inbound, outbound, and vacation routing and delivery. We hope to be ready for a production SMTP replacement very soon.
Still no tasks for me from the GMail for Life project. IDEV are continuing to work on their solution.
PBX Pager project will have some tasks for me soon as the iDEV group draw closer to being ready for production. Testing on JTEST1 continues and is going well.
The Nagios replacement has been put on hold due to higher priority projects. There are still some issues with getting the logging agent to correctly connect and log to mySQL.
SMTP
ITS has been looking into replacing the hardware supporting our mail delivery system for the University. Currently their are 5 inbound servers, two outbound servers, and two auto-responder servers running on physical hardware. ITS will consolidate these physical boxes into virtual machines as a result of this replacement effort. ITS successfully completed a POC and is now looking into a small testing initiative to continue this effort.
Exchange Consolidation
ITS is consolidating the VUMC and the University exchange environments in an effort to reduce overhead eliminate unnecessary redundancies. The effort will involve deploying a new exchange 2007 environment with an email archive offering, and systematically migrate users from University and VUMC to it. This month ITS successfully provided an architecture for storage solution as well as the email archive platform from concept to vendor, and has begun initiating purchasing processes for both.
BlueARC Upgrade
This month ITS expanded the capacity of the HPC NAS environment by 35TB of RAW capacity. This enables ITS to take on additional customers in need of HPC NAS space for research endeavors. However, in order to facilitate the new request, ITS has had to perform a number of maintenance task involving moving file systems around to free contiguous space on storage frames. ITS has nearly completed all maintenance tasks needed and will be complete in the coming week or two.
Virtual Desktop
ITS has been evaluating the plausibly and functionality of a Virtual Desktop Offering to the community to assist in lowering TCO of desktop computing resourced. This month ITS successfully created a POC environment using OS and application streaming to alleviate unnecessary computing and storage resources from being oversubscribed, but underutilized.
Monthly Activity Report
June 2009
This month has been mainly spent working on finishing the details associated with the E-mail Archive project. In addition to that, there has been a lot of work around the Exchange 2007 storage backend.
We installed 3 new DAE’s of storage into the BlueArc high performance NAS environment. We are already making use of the space by moving departments around and optimizing the storage underneath their file systems.
The backup environment has been relatively quiet. We have included growth room in the Exchange projects to handle the upcoming load we are expecting from the deployment of those services. Additionally, we are starting to see a need for additional storage in our NDMP pools to support NAS backup.
We discovered this month that by default hibernation is enabled on all Windows 2008 installs. Thereby potentially consuming large amounts of disk space depending on the hardware in the server. As we discovered with our SharePoint servers, we had an 8GB hibernation file on each server. Fortunately this is easy to fix, by simply opening an administrative command prompt and typing <powercfg.exe -h off> to disable hibernation.
Our SharePoint rollout should be complete and officially in production by the end of this month. So with that being said I spent some time updating all of the support documentation that goes with that, and discovered it hadn’t been updated since prior to our upgrade to 2008 on all of our SharePoint servers, so all of the instructions for setting the SSL host headers were wrong.
I prepped our test SharePoint farm to do some DR/backup/restore testing via Legato, only to discover that the current version isn’t supported if your backup server is running on Linux, so that appears to have been a wasted effort. Supposedly the next version will correct this issue, I guess we’re back to waiting for EMC to provide a viable backup and restore method for SharePoint.
I spent quite a bit of time working on ISA this month. ITS-HCWNAP61 is now officially in production supporting our OCS deployment. I rebuilt and prepped ITS-HCWNAP60 to join the array, and discovered several potential issues while testing the change in our test environment, and have had to defer my original change.
• It appears the issue that caused us to rebuild our ISA servers repeatedly, where whichever node didn’t house the CSS would start to deny all incoming requests, is actually due to how NLB utilizes unicast, and most switches aren’t configured by default to allow all traffic destined for a specific MAC to be sent to multiple switch ports. I’m currently testing a potential fix for this in our test environment.
• In researching the previously mentioned problem, I stumbled across several things we weren’t aware of when we initially deployed ISA. When using NLB, intra-array communication cannot pass on any interface using NLB. Also, it is recommended that your Intra-array communication reside on a private network, a daunting challenge in our environment, since this private network also requires AD access, and is heavily used for Kerberos authentication, as well as passing configuration back and forth between all array members.
Completed migrations of Nagios, Owl, and VUNetID from the AMD Cluster to the Intel 7100 Cluster. Nagios was moved earlier than planned due to problems within the AMD Cluster which we were never able to explain. This completes the migrations I needed to get done before June in preparation of decommissioning the cluster.
The AMP2 memory upgrade was completed. Verified that the system was properly addressing the additional 8GB of memory both in the BIOS and the OS after the upgrade. Critical firmware patching was completed at this time as well.
Worked with Jeff and Gary to troubleshoot problems with SMTP and IMAP servers after one of the servers were removed from service and the user accounts were migrated to other hosts. Most of the issues with the SMTP servers and VUWebmail were resolved quickly. Peter has continued working with Jeff to work through minor problems with some user accounts in the IMAP environment.
The proposal was finally submitted for the search engine replacement. The current solution’s licensing expires the end of June. Given the time constraint I looked into an at least temporary solution using the Google Search API. This was presented to members of ITS and Public Affairs and was deemed a good idea but was not a robust enough solution. After doing some other research I started testing an open source solution called ‘Sphider’ which is php/mysql driven and has its own web crawler. This solution was dropped due to concerns of using open source, though it would have been a nearly 1-to-1 replacement to our current search engine.
Judy was assigned the task of decommissioning the 2 Kiosks located in the hallways in the Hill Center. She requested help in doing so since I had done the original installation and setup of the Kiosks. I told her I would take care of it. She said to let her know if there was anything she could assist with and asked that the televisions remain mounted on the walls with CNN being shown. I enlisted the help of Warren and after removing the NEC Bluefire appliances it was discovered that the televisions did not have built-in tuners. We located 2 unused VCRs in the building and connected them to the televisions after obtaining permission to use them.
Continued meeting for the GMAIL for Life project. IDEV is still working on their parts of the project. No new activities have been assigned for me as of yet.
It was discovered that the OnSite::API perl module provided by BT Diamond IP is no longer working properly, most likely due to recent patches. After more detailed troubleshooting it was shown that the API is in fact working, but is returning an NULL array for the specific queries needed for integration into our website. A ticket has been opened with the vendor concerning this on May 21. We are still awaiting a response from the vendor on the ticket.
May 2009 Monthly Activity Report
1) Network reconfiguration of vanderbilt_production mailgates was incomplete. This change was related to issues where adding hosts and DNS entries via the Admin GUI can cause the hosts entries to be deleted or adding DNS servers can lock up the host. The issues can be resolved by assuring NIC#1 is configured on the same VLAN as the gateway as proven in the vanderbilt_lab cluster. Ran into an issue with mailgate02. It would not save the new configuration when reconfigurng via the Admin GUI or console. Opened call with Proofpoint and provided logs. They have recommended removing the agent from the cluster and reconfiguring then adding the agent back to the cluster. Note maillgate02 and mailgate03 reconifguration was not completed during the change window and must be rescheduled.
2) Deployment and configuration of LISTSERV development server has been completed with one exception. DNS MX record must be created.
3) Found method to provide list of all group mailboxes, their ACLS, and the last time they were accessed. Have supplied data to project manager.
4) Gmail for Life. Met with iDev team in order to determine a strategy for populating the Proofpoint user repository used for sapm policy assigment and recipieint verification. Provided them with user repository export and configuration documentation so that options could be explored.
5) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.
6) Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes. Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives. Investigated over 100 spam false negatives.
7) Performed daily management of mail queues on mailgates. Removed hundreds of undeliverable messages daily in order to keep queues “clean”. Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.
8) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.
9) Worked on an assortment of odd and challenging helpdesk tickets.
Sun Identity Management: The networking issue for the virtual machines located in the VUH data center has been resolved. Our intermittent network issues were cause by a shared IP address between F5 LTMs and a Cisco router. This caused the VMs to lose outbound network connectivity when the IP jumped to the router. The IP address was removed completely from the rouer, and network connectivity has been restored. We are moving on the final testing phase. We are working through the final steps for getting SLAMD setup.
[WIN] ITS Website Redesign: The cutover to the new ITS website is scheduled for this weekend. The change will involve exporting the test database, cleaning up all of the links, and importing the data on the production server. I’ll also need to copy the necessary files over to the production server. The last step will involve moving the IP addresses from the old servers over to the new servers. The CSM will pass the incoming connections to the new servers.
PHP5/MySQL5 Upgrades: The cutover deadline for the cut over of the first round of server has been extended until mid-June. This round of migration is progressing slowly. There are 58 folders left on the WWW4 servers 29 virtual hosts left on the vanity domain servers. This remaining content is under review by the site owners and awaiting migration approval.
1.) Coordinated 234 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.
2.) I resolved 42 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.
3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.
4.) Continuing process of assigned project to decommission IMAP2. Accounts from IMAP2 have been moved to IMAP1 and IMAP3.
5.) Assisted project to assist with decommission of VUMail.
22 Closed Magic Tickets
Exchange
38 Migrations to Exchange 2007
Windows
VUGroupspace
Identified issue with quota software being utilized to manage VUGroupspace. One of the shares is being reported as uitlizing much more space then is actually in use, as reported by Windows. The issue appears to be a bug in the Quota software. Change is in place to update the software to the latest rev.
Exchange
Archiving
Assisted in the process of choosing an archiving solution for the Exchange 2007 Environment. This has included attending various vendor presentations, providing system configuration information and challenging\questioning vendor configurations and abilities.
Cross Training
Assist in cross training new messaging employees with current configuration, setup and procedures for the VU Exchange Environment.
Exchange 2007
OWA Vanderbilt Rebranding work
Installed new test\production CAS for internal testing in the production environment before implementation. This CAS will also be utilized for testing OWA integration.
Finished the OWA Rebranding and put it into production. Continuing the documentation of the .css files on my time for easier modification in the future if the need arises. Modified the test CAS setup to allow for multiple images at the top of the OWA login page to accomodate MC and VU logo’s for the combined environment.
OWA timeout issue
Worked with Peter in identifying the issue that was causing our 2007 OWA timeout issues. Tested solution on the test/production CAS successfully. Peter is putting in a change for implementing ths change on the production OWA Load balanced ip.
OWA integration effort
Worked on identifying issue that is preventing OWA 2007 from connection to VUExchange 2003 mailboxes. Still have not identified the complete root cause, it appears as if it is because the default internal delivery address for the VUExchange users is different from the default proxy address for the Exchange Enterprise. It is still being researched as to why this is happening
OWA 2007 Change Password
Identified the method for disabling the change password option for OWA 2007. This is important so as to not cause confustion issues with end users since this functionality will not work since AD is not authoritative for passwords and current ad permissions do not allow for changing the password in this fashion.
UM
Identified issue with loader process overwriting UM and OCS proxy address information in AD.
PowerShell Scripting
Mailbox Migration
Scripted to help automate the process for larger scale migrations. The tool automatically picks the storage group with the least number of mailboxes currently on it and creates a mailbox move script for moving each user that is listed in the users.txt file that is populated with the users to be migrated from 2003 to 2007. The next portion of the script that I am working on is automating the start of each of the created user move scripts. The tool will run up to 10 migrations at a time and then prompt for the script to be run again. It keeps a list of all the users that were moved and the dates. Passed off portions of the script to Denny for possible use in the new Exchange 2007 loader script.
UM and OCS metrics
Scripted solution to pull a count of OCS users, Voice Users, Federation enabled users, Internet enabled users and UM users and mail out the information to interested individuals. Continuing work on automating this and writing the data out with date information to a database for ongoing automated metrics.
This month has seen a lot of work around e-mail archive and Exchange 2007 storage planning.
We are currently evaluating 4 different vendor archive solutions and we are in the process of putting together our final technical recommendation to ITS management.
Operationally our backup environment continues to run smoothly and we are even completing all of our cloning within 2-3 days of processing. Since we have implemented the new backup infrastructure we have growen from ~157 TB of data backed up per month to 247 TB backed up monthly. We are growing at a rate of approximately 85% per year.
Exchange 2007 Deployment
The majority of the month has been working on project documentation, hardware requirements, and investigating different archiving solutions.
One of the things that I realized needs more documentation in the messaging community is Public Folders. There really is very little guidance on sizing hardware for dedicated Public Folder servers. Here is what Microsoft has written on Scalability. I know in Exchange 2010 they are being deemphasized even more and Sharepoint is probably a better place for them, but when you have over 8,000 Public folders; I think we might have them for a little while.
One of the issues that has been plaguing the deployment has been that after Exchange Rollup-6 was deployed, Entourage clients were unable to see any messages in the inbox. The client would connect, but no messages were displayed. The rollup-6 rollout was a red herring, which coincided with a server rebuild, and mailbox migration.
After numerous hours of troubleshooting, a call to Microsoft Premier Support was made. Here is what I learned. (thanks Pawan Kapoor!)
It seems that even though all other clients do their authentication on the Client Access Server, but Entourage authenticates and uses ASP.net. on the Mailstore role This means that you will need to add the ASP.net feature along with the integrated and windows authentication features on each mailstore server you deploy.
Here are the two KB articles that the Technician sent to me.
Configuring Authentication in IIS 7.0:
You cannot connect to your mailbox on Exchange Server when you use Entourage for Mac:
Office Communication Server Deployment
The pilot is now in full swing. The adoption rate of this product is astounding.
It is really amazing to see the ease of extemporaneous collaboration that the new R2 release has achieved. Just tonight, I was sitting working on this report, when a colleague asked me to look over a document before he published the final version. I was able to see his desktop inside my Communicator client, make some word change suggestions, point the mouse to where I was referring to, and then being a second set of eyes to verify accuracy, All 131 miles apart. Through on top of that, that we were able to talk over a peer to peer VOIP connection and discuss the changes. This is truly where Unified Communications comes together.
FaxServer Replacement
The hardware named FaxSrv1 was decommissioned, and removed from the data center. The newer hardware formally known as FaxSrv2 was formatted and is currently being reloaded with Windows 2008 Standard on it. There seems to be an issue finding RAID controller drivers, this is where we currently are and looking for alternatives to this dilemma.
SharePoint Implementation - Had a wrap up meeting for the project. We are actually going into production.
Windows Server 2008 Cluster - Still having issues with adding the second node to the cluster.
In my “spare” time - Began work on improving the server provisioning and decommissioning process by adding in workflows for the various tasks and creating helpdesk tickets for work needed by other teams.
SCOM Audible Alerts - Install SCOM, Powershell as well as the scripts that provided the audible alerts on several workstations in the NOC. I am currently using it on my machine without issue.
Monthly Activity Report
May 2009
1) Completed testing of mailgate upgrade process in vanderbilt_lab cluster. Includes inspection of policies and rules, LDAP imports, etc.
2) Started deployment and configuration of LISTSERV development server. Will be completed in early May 2009.
3) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.
4) Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes. Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives. Investigated over 100 spam false negatives.
5) Performed daily management of mail queues on mailgates. Removed hundreds of undeliverable messages daily in order to keep queues “clean”. Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.
6) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.
7) Worked on an assortment of odd and challenging helpdesk tickets.
This month has been a lot of work around the BlueArc, backing up the BlueArc, and e-mail archive.
I have architected a backup solution for the possible expanded BlueArc environment. The environment will include two dedicated tape libraries, a SAN switch, two storage nodes, 36 LTO-4 drives, and 6800 LTO-4 media. This solution will be able to produce monthly full backups and weekly incrementals of all of the data residing on the new service. Additionally, we will be producing monthly clones of all full data.
E-mail archive is in the research phase. We are looking at various vendors and their solutions to our archiving requirements.
1.) Coordinated 136 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.
2.) I resolved 36 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.
3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.
4.) Continued preparation of assigned project to decommission IMAP2.
5.) Assisted Asset Management crew with an inventory audit.
Memory upgrade requested by ND&E after recommendations from their software vendor for AMP2 has finally arrived. Due to the change moritorium during finals and graduation this will not be completed until mid May.
Worked with the vendor for the VEHS Radiation database. A working solution has been completed after nearly two weeks of trial-and-error and as well as 5 updated versions of the client software provided as problems were found. The client is successfully allowing admin password updates with a hashed/encrypted password, SSL connectivity to the MySQL database, and SSL connection and authentication against our Active Directory servers. Client installation package has been created and passed along to Joanne in Desktop Support to install on VEHS workstations. Joanne has provided a list of workstations which will be running the client and needing MySQL connectivity. Currently awaiting the go-ahead from Peter on the new MySQL 5 server for final deployment.
Began researching installation needs for replacing our current Nagios server. The replacement will be running Nagios v3 and will utilize a MySQL database for logging verses the standard flatfiles. SNMP Trap handling will also be incorporated for better monitoring of services such as the Cisco CSM load balancer.
Worked with the email and security teams in identifying webmail accounts which were found to have been compromised.
Finalized configuration of a new flash server located on the VUMC network and worked with VUMC Network Security to get appropriate access to the new server. This was deployed to handle streaming load on the VUMC network for their graduation ceremonies. ITS Streaming Services are in the process of testing the new server.
There are no updates on the Search Engine Replacement project. The SDM reports that the report is being finalized to be put forth for financial approval and governance. The contract for current Ultraseek Service will expire in the next 60 days.
Completed a couple of requests from iDev on the ‘GMail For Life’ project. No additional tasks have been presented to me while iDev works toward their intended solution.
Completed installation and initial configuration of the new HypericHQ server for testing. Requested basic firewall access from Network Security and setup accounts for the NOC Manager to be able to begin testing.
Began working on presenting the GNAV database for VUMC and MIS for data mining and reporting. In the process it was discovered that the vendor provided backup script appeared to have been working but was not properly performing backup operations. Instead of attempting to fix the exxisting script a new one was written. Backups are now compressed which allows for retention beyond a single daily backup. Replication of backups to the secondary server has also been incorporated into the script and tested. Any failures encountered by the script should trigger a notification email to the Unix Team.
Due to incompatabilities with archived Windows Media files, ITS Streaming Services will no longer be pushing for upgrading the Helix servers to v12. Per request, all previous revisions of v11.x have been installed on the test server. Installations have been made and access has been given to allow easily changing the running version on the server for testing. ITS Streaming hopes to find a preferred previous version with the highest number of working features and least amount of bugs to move the production servers to at a later date.
Exchange 2007 Deployment
The exchange 2007 deployment has started out pretty well, we have moved all of ITS over to the new system. There have been very few issues. One of the largest issues has been with the MAC Entourage client. Once we rolled out Rollup 6, the Entourage client is no longer able to see the contents or folder structure of the mailbox. This is all Entourage clients, even the new beta one.
On the positive side, there have been numerous other clients that have been able to attach to the new Exchange 2007 Systems, including two different versions of Thunderbird, and one version of Alpine. These are all using the IMAP settings, but seem to working as expected.
The next steps are to work through the issues presented by the entourage clients, and then we will continue to retire the VUexchange 2003 systems.
Office Communications Server 2007 R2 Deployment
This has been the month for OCS, we have been anticipating the Lighthouse engagement, and the rollout of Office Communication Server 2007 R2 since last October. We have built the infrastructure, 5 physical servers, an Enterprise front end, a SQL Backend, a mediation server, and Edge server, and a Unified Messaging server. This will allow us to implement presence, IM, federation, public Internet connections, voice mail, and remote enterprise phones.
The deployment has been successful; we have normalized our phone numbers to an E164 format, and have the SIP gateway doing the conversion that the PBX is expecting. Learned more about regular expressions while doing this, regular expressions are very robust and powerful once you understand some of the syntax.
We currently have 175 people provisioned on the Office Communications Server.
30 Closed Magic Tickets
Exchange
43 IMAP -> Exchange 2003 Migrations
69 Exchange 2003 -> Exchange 2007 Migrations
Exchange 2007
OWA Vanderbilt Rebranding work
The login and logoff pages are complete. Premium OWA pages are about 80% done. Built test CAS and mailbox server in MS-Test environment for facilitating the rebrand testing.
Disaster Recovery Drill
Executed Disaster Recovery Drill for ITS-HCWNEM01. Successfully rebuilt and restored mailboxes.
PowerShell Scripting
Size/Quota/Message/Item # Script and Mailbox Move Script
This script is utilized to pull mailbox size, location, quota and number of items in a mailbox. This is being utilized in conjunction with a move mailbox script that records the time it took to migrate a mailbox from 2003 to 2007. This is being utilized to try and normalize mailbox move times based on size/number of items.
[WIN] RHEL5/PHP5/MySQL5 Upgrades: The migrations for the FEVS02 (misc domains) and FEVS06 (www4) servers pairs have begun. The first attempt to do this was interrupted by another work activity, and they were rescheduled for Apr 26. The rescheduled migrations were completed without issues.
ITS Website Redesign: The final cutovef date has been scheduled.
Sun Identity Management: All of the LDAP virtual machines have been built and are accessible via our bastion hosts. The networking issues with the F5 LTMs have been resolved, and this appears to have revealed a possible issue with the ESX servers behind them. I have opened a ticket with the VUH helpdesk, and I am working with their staff to resolve the issue.
Security Investigations: I spent several days working with the Network Security team to identify and mitigate a security issue.
Operations
Streamlined our server build process, and updated our build documentation accordingly. For both Windows 2003 and Windows 2008 I wrote some PowerShell scripts that utilize netsh to set all of our default firewall rules.
Also for Server 2008, I wrote some additional scripts that install all the Windows Features that we install by default, such as PowerShell. As soon as I find the time I plan on attempting to completely automate our OS install to include all of our default settings and application installs, i.e. Anti-virus, smartcard, etc.
OCS
Not much to report on this one, other than that I prepped two of the OCS servers for our R2 rollout. I also prepped our ISA farm (added some new network interfaces) for OCS.
ISA
Well after months of troubleshooting and countless hours on the phone with Microsoft Premier support we’ve finally figured out a work around to our issue with editing documents in SharePoint when using ISA. In a nutshell clients running Windows Vista and Office 2007 would receive an error when trying to save a document back to SharePoint after editing. After spending a lot of time on the phone with and outstanding Support Engineer at Microsoft it was determined that it was a bug in WebDAV. Until the WebDAV team releases a fix we were able to find the following work around.
SharePoint
Deployed a new GPO to all of ITS staff that configures all settings needed for seamless use of SharePoint. The changes made by the GPO are as follows:
HKLM\SYSTEM\CurrentControlSet\Services\WebClient\Parameters
REG_MULTI_SZ “AuthForwardServerList”
IBM Director 6.1 Implementation – After moving the new IBM server to the production VLAN, I was able to successfully get several machines to “talk” to the new server. I have hardware inventories for both Windows and Linux machines. I will be putting in a change to put the new server into production sometime in May.
Shibboleth Project – I have had a quiet month this month on the Shibboleth project. I am still waiting on the IDev resource to complete testing on their tasks before I can clone the main virtual machine for DR purposes.
SCOM Audible Notifications – I was able to get SCOM to send audible notifications using Windows Media Player by harnessing the power of PowerShell and PSEXEC. I believe that after further testing, I should be able to put a list of servers in a text file and send the notifications to multiple machines. Once I have the code, I post it here.
SharePoint Rollout – I believe SharePoint is very close to going into production. Some important decisions have been made surrounding this application paving the way for an actual productions date. Whoo Hoo!
OCS Lighthouse – In addition to building one server for this project, I also configured several of the OCS lighthouse servers for teams using LACP. I was able to team two NICs and configure the team for two VLANs. It was a bit frustrating at first, but I’m glad I was able to accomplish this.
BlueArc upgrade
ITS is planning an upgrade of the bluearc high perfomance NAS environemtn. ITS will be expanding the capabilities from 6oTB to 750TB usable. ITS is also architecting a backup soluetion for the new 750TB environment. This will encompass 4 bluearc contollers and 2 i2k with 18 drives each. This provides enough throughput to perform full backups once a month and incrementals once a week.
SMTP Replacement
ITS has been working to replace the SMTP infrastucture from physical Sun servers to virtual Linux servers. ITS has completed the development work required for this effort, and is now looking into the deployment process. This will allow time to re-architect our email solution without running into server issues due to hardware problems.
1. BIND Views are finally working in DIP on the primary name server. There is still an issue with getting the views propagated to slave servers. Following the ISC instructions does not work and the vendor has been engaged.
2. The issue with pushing updates in the DIP environment has been resolved. Apparently ActiveMQ was refusing to play nicely. This paves the way for upgrades to the Sapphire Appliances and to the application.
3. Disk consolidation continues in the Virtualization environment. All “troubled” LUNs have been replaced. Additionally, prep work to retire the AMD ESX servers continue. When all is said and done, the ESX environment will drop from 20 hosts down to 12 hosts.
4. Work continues on resolving backup issues with a number of hosts moving off the .1 network to the Admin Network. There are still a number of hosts that needs this addressed.
That’s enough typing on the iPhone for blog purposes. Until next month…
Monthly Activity Report
April 2009