Planet AppHosting

June 30, 2009

Peter Woods

June MAR

I only have one major WIN this month…

PHP5/MySQL5 Web Server Migrations: The project to upgrade all of the shared web services to RHEL5 is essentially complete. All of the content has been migrated, and the sites are running smoothly. This migration gives the web community access to PHP5 and MySQL5.  It also brings along a much more robust ModSecurity ruleset.  So far the vast majority of the help desk tickets have been related to applications with the new ModSecurity rules.

by Peter Woods at June 30, 2009 12:19 PM

June 29, 2009

Tony Hortert

June MARS

18 Closed Magic Tickets

Exchange
5 Migrations to Exchange 2007

Windows

VUGroupspace
Identified issue with quota software being utilized to manage VUGroupspace. One of the shares is being reported as uitlizing much more space then is actually in use, as reported by Windows. The issue appears to be a bug in the Quota software. Change has been finished and testing is continuing.

Exchange
Assisted in troubleshooting and resolving various issues this month.

Archiving
Assisted in the process of choosing an archiving solution for the Exchange 2007 Environment. This has included attending various vendor presentations, providing system configuration information and challenging\questioning vendor configurations and abilities.

Cross Training
Continued cross training new messaging employees with current configuration, setup and procedures for the VU Exchange Environment.

Exchange 2007

OWA Vanderbilt Rebranding work
Passed on knowledge for the rebranding for management to Antwan on the Email team. Assisted in putting new logo on OWA Site.
F5 OWA traffic rules
Identified solution for F5 forwarding of vunetid url to delivery address url. This allows for less impact on users after the OWA consolidation for those using macs and those accessing resource URLs.

Exchange integration effort
Worked as the exchange resource for identifying current tools and desired tools for migration to the Exchange 2007 environment and ongoing tool requirements.

PowerShell Scripting

OCS User Enabling
Scripted solution automate the setting up of users for utilizing OCS. This saves considerable time when going through this process.

Cluster Resource Script
Scripted the stopping and starting of cluster resources. Still working through the permissions to be able to run the script remotely from SCOM.

by tony.hortert at June 29, 2009 09:01 PM

Dan Raymer

“Curiosity’s new home” for June 2009

(Curiosity is the new name for the Mars Science Labratory)

Well, first off, here are the "Wins" for June…

  • DNS BIND VIEWS ARE NOW IN PRODUCTION!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Sure, it took a LONG while to get in place, but it was implemented without any clients needing to change any of their configurations or any changes to all of the ICANN/ARIN registrations.  The implementation itself was relatively transparent to the University as well.  Initial RFC1918 entries are live on Internal views and will be pushing out more as the service is publicized to the University.
  • DHCP Failover with the BT INS appliances has been solved.  Well, not solved, but a functional work around for the ISC failover bug has been tested and implemented.  This will allow us to FINALLY start the migration onto the new DHCP environment and get rid of those old, clunky Sun v210s & v240s.

And that’s about it for the WINS…

For the LOSES…

1 gallbladder, some weight, my daughter’s tonsils & adenoids, and 1 uncle.

For the most part, it has been one exhausting month.

** EDIT ** And I need to learn how to spell…

by Daniel Raymer at June 29, 2009 06:20 PM

Roland Serman

MARS 6/09

Well this month we finally managed to get all of our Wintel servers upgraded to ePO 4.0, sadly we still have one server that is still running VSE 8.0.  Hopefully I can get that one upgraded to 8.5 this month, so I can close out that SWE as well.

I spent most of the month, preparing for, testing and then deploying WSS 3.0/MOSS 2007 SP2.  The upgrade in our test farm went smoothly, and for the most part it went smoothly in production.  I only ran into one unexpected snag in production.  After extracting both WSS SP2 binaries and the MOSS SP2 binaries, when I ran the setup wizard, it never prompted to start the wizard on the other servers, and then failed to install.  The fix was to reboot the entire farm, and then run the wizard again.  At this point, everything proceded as it was supposed to, and the upgrade was a success.  (that is after going in and adding our license key again, would’ve thought Microsoft would’ve released an updated service pack to resolve that little issue by now).

I also finished updating our SharePoint installation documents, to include not only installing SharePoint and Project server, but creating the default SSP, mysites, etc.  As well as all post SharePoint configuration changes/application installs that we’ve made to our production environment.  I’m still working on merging those changes into our DR document, but will hopefully have that completed shortly as well.  Anyway, the big positive here, is that we now have a step by step document for every step needed to reproduce our production environment with every addon, third party app, and custom configuration setting.

Once I finish updating the DR document, I hope I can find the time to polish our back up script.  Currently everytime a new site or mysite is added, we have to manually add it to the backup script.  I’m pretty sure, that we can completely automate this process, I just need to find the time to work on it.  As it sits now, I update the script once a month, and again before any major change, but that does leave holes in our backup process.

by roland.e.serman at June 29, 2009 03:37 PM

Jeff Sublett

MAR - for June 2009

1.) Coordinated 138 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 41 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission VUMail. I am currently working on cleaning up old data on IMAP1/IMAP3.

5.) Assisted project to upgrade Drupal. I am currently researching this task.

6.) Completed task of removing surplus equipment from inventory.

by jeff.sublett at June 29, 2009 03:16 PM

Kendra Thorpe

June 2009 MAR

  1. Rewrote virtual machine inventory script in Powershell – The old script utilized perl, and I rewrote the script using Powershell and the vSphere Power CLI. The data is written to a MySQL database and I utilized the ODBC drivers to connect to the MySQL database.
  2. Normalized Runbook data with the NOC inventory data – Due to naming mismatches and servers located in datacenters not managed by the NOC, we had servers that were in the Runbook and not in the NOC inventory. In order to give accurate counts on our inventory, it was necessary to normalize this data.
  3. Recreated Weekly Helpdesk ticket in SQL Reporting Services – Previously a sysadmin had to manually calculate the number of helpdesk tickets in the various states. Now the report automatically calculates this information.

by kendra.thorpe at June 29, 2009 03:11 PM

Troy Osborn

June MAR 2009

Google search appliances arrived and were setup.  Public Affairs is working with the appliances now to ensure appropriate spider settings and also customize the user interface.  Failover testing and a more open community test will be done in early July.  Production date is scheduled for the second week of July.

AmCom came on-site to perform the replacement OSIS server installations.  The vendor decided that they would prefer a clean install versus upgrading he current systems.  Using their software specifications, 4 new servers were built and turned over to the vendor.  Upgrades went smoothly once a few minor software dependency problems were resolved.  The vendor continues to work on client-side application upgrades.  Production is planned for the middle of July.

The web pages for supporting IP/DNS requests for subnets which have been migrated to the DiamondIP appliances have been re-written and are ready again available for use.  The perl API supplied by the vendor stopped functioning after a recent update.  The vendor was disinclined to support the perl module which they supplied because it is not part of their standard support package, so a new solution was found using their CLI API which is supported.   The pages which utilized the perl API have now been rewritten in PHP and should be supported by future updates.

LISTSERV began to exhibit performance problems.  Worked with the email team and the vendor to troubleshoot the issue.  Neither the vendor nor ourselves were unable to identify a definite root cause of the performance problems.  The vendor suggested that we try using their High Performance Option (HPO).   After obtaining a trial key and testing to evaluate the performance differences  it was decided to place the key on our production host.   At the same time a new PHP login page was activated to replace the previous perl CGI script, which will give more feedback to users.

Work and testing continues on the SMTP replacements.  So far we have had successful tests of inbound, outbound, and vacation routing and delivery.  We hope to be ready for a production SMTP replacement very soon.

Still no tasks for me from the GMail for Life project.  IDEV are continuing to work on their solution.

PBX Pager project will have some tasks for me soon as the iDEV group draw closer to being ready for production.  Testing on JTEST1 continues and is going well.

The Nagios replacement has been put on hold due to higher priority projects.  There are still some issues with getting the logging agent to correctly connect and log to mySQL.

by troy.osborn at June 29, 2009 02:59 PM

Kenon Ewing

wins 6/2009

SMTP

ITS has been looking into replacing the hardware supporting our mail delivery system for the University. Currently their are 5 inbound servers, two outbound servers, and two auto-responder servers running on physical hardware. ITS will consolidate these physical boxes into virtual machines as a result of this replacement effort. ITS successfully completed a POC and is now looking into a small testing initiative to continue this effort.

Exchange Consolidation

ITS is consolidating the VUMC and the University exchange environments in an effort to reduce overhead eliminate unnecessary redundancies. The effort will involve deploying a new exchange 2007 environment with an email archive offering, and systematically migrate users from University and VUMC to it. This month ITS successfully provided an architecture for storage solution as well as the email archive platform from concept to vendor, and has begun initiating purchasing processes for both.

BlueARC Upgrade

This month ITS expanded the capacity of the HPC NAS environment by 35TB of RAW capacity. This enables ITS to take on additional customers in need of HPC NAS space for research endeavors. However, in order to facilitate the new request, ITS has had to perform a number of maintenance task involving moving file systems around to free contiguous space on storage frames. ITS has nearly completed all maintenance tasks needed and will be complete in the coming week or two.

Virtual Desktop

ITS has been evaluating the plausibly and functionality of a Virtual Desktop Offering to the community to assist in lowering TCO of desktop computing resourced. This month ITS successfully created a POC environment using OS and application streaming to alleviate unnecessary computing and storage resources from being oversubscribed, but underutilized.

by k.ewing at June 29, 2009 02:24 PM

Scott Evans

2009 June MAR

Monthly Activity Report
June 2009

  1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Ran a mock billing cycle on AIMWorX3 to create a BSITM.gl file because this data was not imported into the production database for proper billing.
  2. AIMWorX Templates – Completed new work order template for the OCS project to setup billing in AIMWorX and to request an OCS account.  These templates were imported into the production database.
  3. Accordent Trial – Worked with an Accordent technician to correct issues their recorder was having with the servers scheduling software.
  4. Higer Ground backup Server — New OSIS-CR02 server was recieved from vendor and racked in Bryan Building.  Setup server to meet VU server standards and put into production.  Worked with Higher Ground technicians to complete setup of server.  Copied some detail files from old server to facilitate a quicker setup.  Retrieved log files for HG prior to cleaning the hard drives of the old OSIS-CR02 server and sent back to vendor. 
  5. NEC License Manager for MA4000 — Installed License Manager software and discovered that passwords were stored in clear text.  Since the ’sa’ password was used during setup this software was disabled until this was changed.  Opened a ticket with NEC and worked with a technician to remove the ’sa’ password and setup a new login that is specific to this database.  Also, setup https for web interface.  Once these issues were resolved the License Manager software was turned back on.  The LM software looks to FPC01 for PBX verification.  Since FPC01 has problems with connections, the LM software is not working.  This causes MA4000 to not function properly because it thinks it is not licensed.  The PBX technicians have an open ticket with NEC concerning FPC01 connection issues.
  6. Sharepoint Servers — Rebuilt test Sharepoint server (its-hcwnap51-ts & its-hcwnap52-ts) per Roland’s request using the virtual server template.
  7. AT&T BMS file changes — Business & Planning is moving some charges to a new billing structure that is requiring changes to the BMS process.  Worked with Paul Lockaby on changes to these changes. 
  8. Virtual Servers — Created 3 servers (its-hcwnem98, its-hcwnem18, & its-scwnem19) for the Email team as requested and approved.  Rebuilt template for Windows 2008 32bit server because it was corrupted.  Updated OS on all Windows virtual templates.

by Scott.Evans at June 29, 2009 12:09 PM

June 25, 2009

Derek Miller

June 2009 Activity Report

This month has been mainly spent working on finishing the details associated with the E-mail Archive project.  In addition to that, there has been a lot of work around the Exchange 2007  storage backend.
We installed 3 new DAE’s of storage into the BlueArc high performance NAS environment.  We are already making use of the space by moving departments around and optimizing the storage underneath their file systems.

The backup environment has been relatively quiet.  We have included growth room in the Exchange projects to handle the upcoming load we are expecting from the deployment of those services.  Additionally, we are starting to see a need for additional storage in our NDMP pools to support NAS backup.

by derek.miller at June 25, 2009 02:45 PM

May 29, 2009

Roland Serman

MARS 05/09

We discovered this month that by default hibernation is enabled on all Windows 2008 installs.  Thereby potentially consuming large amounts of disk space depending on the hardware in the server.  As we discovered with our SharePoint servers, we had an 8GB hibernation file on each server.  Fortunately this is easy to fix, by simply opening an administrative command prompt and typing <powercfg.exe -h off> to disable hibernation.

Our SharePoint rollout should be complete and officially in production by the end of this month.  So with that being said I spent some time updating all of the support documentation that goes with that, and discovered it hadn’t been updated since prior to our upgrade to 2008 on all of our SharePoint servers, so all of the instructions for setting the SSL host headers were wrong.

I prepped our test SharePoint farm to do some DR/backup/restore testing via Legato, only to discover that the current version isn’t supported if your backup server is running on Linux, so that appears to have been a wasted effort.  Supposedly the next version will correct this issue, I guess we’re back to waiting for EMC to provide a viable backup and restore method for SharePoint.

I spent quite a bit of time working on ISA this month.  ITS-HCWNAP61 is now officially in production supporting our OCS deployment.  I rebuilt and prepped ITS-HCWNAP60 to join the array, and discovered several potential issues while testing the change in our test environment, and have had to defer my original change.
•    It appears the issue that caused us to rebuild our ISA servers repeatedly, where whichever node didn’t house the CSS would start to deny all incoming requests, is actually due to how NLB utilizes unicast, and most switches aren’t configured by default to allow all traffic destined for a specific MAC to be sent to multiple switch ports.  I’m currently testing a potential fix for this in our test environment.
•    In researching the previously mentioned problem, I stumbled across several things we weren’t aware of when we initially deployed ISA.  When using NLB, intra-array communication cannot pass on any interface using NLB.  Also, it is recommended that your Intra-array communication reside on a private network, a daunting challenge in our environment, since this private network also requires AD access, and is heavily used for Kerberos authentication, as well as passing configuration back and forth between all array members.

by roland.e.serman at May 29, 2009 03:56 PM

Troy Osborn

MAY MAR

Completed migrations of Nagios, Owl, and VUNetID from the AMD Cluster to the Intel 7100 Cluster.  Nagios was moved earlier than planned due to problems within the AMD Cluster which we were never able to explain.  This completes the migrations I needed to get done before June in preparation of decommissioning the cluster.

The AMP2 memory upgrade was completed.  Verified that the system was properly addressing the additional 8GB of memory both in the BIOS and the OS after the upgrade.  Critical firmware patching was completed at this time as well.

Worked with Jeff and Gary to troubleshoot problems with SMTP and IMAP servers after one of the servers were removed from service and the user accounts were migrated to other hosts.   Most of the issues with the SMTP servers and VUWebmail were resolved quickly.  Peter has continued working with Jeff to work through minor problems with some user accounts in the IMAP environment.

The proposal was finally submitted for the search engine replacement.  The current solution’s licensing expires the end of June.  Given the time constraint I looked into an at least temporary solution using the Google Search API.   This was presented to members of ITS and Public Affairs and was deemed a good idea but was not a robust enough solution.  After doing some other research I started testing an open source solution called ‘Sphider’ which is php/mysql driven and has its own web crawler.   This solution was dropped due to concerns of using open source, though it would have been a nearly 1-to-1 replacement to our current search engine.

Judy was assigned the task of decommissioning the 2 Kiosks located in the hallways in the Hill Center.  She requested help in doing so since I had done the original installation and setup of the Kiosks.  I told her I would take care of it.  She said to let her know if there was anything she could assist with and asked that the televisions remain mounted on the walls with CNN being shown.   I enlisted the help of Warren and after removing the NEC Bluefire appliances it was discovered that the televisions did not have built-in tuners.  We located 2 unused VCRs in the building and connected them to the televisions after obtaining permission to use them.

Continued meeting for the GMAIL for Life project.  IDEV is still working on their parts of the project.  No new activities have been assigned for me as of yet.

It was discovered that the OnSite::API perl module provided by BT Diamond IP is no longer working properly, most likely due to recent patches.  After more detailed troubleshooting it was shown that the API is in fact working, but is returning an NULL array for the specific queries needed for integration into our website.  A ticket has been opened with the vendor concerning this on May 21.   We are still awaiting a response from the vendor on the ticket.

by troy.osborn at May 29, 2009 02:42 PM

May 28, 2009

Gary Howard

May 2009 MAR

May 2009 Monthly Activity Report

1)  Network reconfiguration of vanderbilt_production mailgates was incomplete.  This change was related to issues where adding hosts and DNS entries via the Admin GUI can cause the hosts entries to be deleted or adding DNS servers can lock up the host.  The issues can be resolved by assuring NIC#1 is configured on the same VLAN as the gateway as proven in the vanderbilt_lab cluster.   Ran into an issue with mailgate02.  It would not save the new configuration when reconfigurng via the Admin GUI or console.  Opened call with Proofpoint and provided logs.  They have recommended removing the agent from the cluster and reconfiguring then adding the agent back to the cluster.  Note maillgate02 and mailgate03 reconifguration was not completed during the change window and must be rescheduled.

2)  Deployment and configuration of LISTSERV development server has been completed with one exception.  DNS MX record must be created. 

3)  Found method to provide list of all group mailboxes, their ACLS, and the last time they were accessed.  Have supplied data to project manager.

4)  Gmail for Life.  Met with iDev team in order to determine a strategy for populating the Proofpoint user repository used for sapm policy assigment and recipieint verification.  Provided them with user repository export and configuration documentation so that options could be explored.

5) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.  

6)  Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Investigated over 100 spam false negatives. 

7)  Performed daily management of mail queues on mailgates.  Removed hundreds of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

8)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

9) Worked on an assortment of odd and challenging helpdesk tickets.

by gary.howard at May 28, 2009 07:03 PM

Peter Woods

May MAR

Sun Identity Management: The networking issue for the virtual machines located in the VUH data center has been resolved. Our intermittent network issues were cause by a shared IP address between F5 LTMs and a Cisco router. This caused the VMs to lose outbound network connectivity when the IP jumped to the router. The IP address was removed completely from the rouer, and network connectivity has been restored. We are moving on the final testing phase. We are working through the final steps for getting SLAMD setup.

[WIN] ITS Website Redesign: The cutover to the new ITS website is scheduled for this weekend. The change will involve exporting the test database, cleaning up all of the links, and importing the data on the production server. I’ll also need to copy the necessary files over to the production server.  The last step will involve moving the IP addresses from the old servers over to the new servers. The CSM will pass the incoming connections to the new servers.

PHP5/MySQL5 Upgrades: The cutover deadline for the cut over of the first round of server has been extended until mid-June. This round of migration is progressing slowly. There are 58 folders left on the WWW4 servers 29 virtual hosts left on the vanity domain servers. This remaining content is under review by the site owners and awaiting migration approval.

by Peter Woods at May 28, 2009 05:12 PM

Jeff Sublett

MAR - for May 2009

1.) Coordinated 234 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 42 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continuing process of assigned project to decommission IMAP2. Accounts from IMAP2 have been moved to IMAP1 and IMAP3.

5.) Assisted project to assist with decommission of VUMail.

by jeff.sublett at May 28, 2009 04:50 PM

Tony Hortert

May MARS

22 Closed Magic Tickets

Exchange
38 Migrations to Exchange 2007

Windows

VUGroupspace
Identified issue with quota software being utilized to manage VUGroupspace. One of the shares is being reported as uitlizing much more space then is actually in use, as reported by Windows. The issue appears to be a bug in the Quota software. Change is in place to update the software to the latest rev.

Exchange

Archiving
Assisted in the process of choosing an archiving solution for the Exchange 2007 Environment. This has included attending various vendor presentations, providing system configuration information and challenging\questioning vendor configurations and abilities.

Cross Training
Assist in cross training new messaging employees with current configuration, setup and procedures for the VU Exchange Environment.

Exchange 2007

OWA Vanderbilt Rebranding work
Installed new test\production CAS for internal testing in the production environment before implementation. This CAS will also be utilized for testing OWA integration.
Finished the OWA Rebranding and put it into production. Continuing the documentation of the .css files on my time for easier modification in the future if the need arises. Modified the test CAS setup to allow for multiple images at the top of the OWA login page to accomodate MC and VU logo’s for the combined environment.

OWA timeout issue
Worked with Peter in identifying the issue that was causing our 2007 OWA timeout issues. Tested solution on the test/production CAS successfully. Peter is putting in a change for implementing ths change on the production OWA Load balanced ip.

OWA integration effort
Worked on identifying issue that is preventing OWA 2007 from connection to VUExchange 2003 mailboxes. Still have not identified the complete root cause, it appears as if it is because the default internal delivery address for the VUExchange users is different from the default proxy address for the Exchange Enterprise. It is still being researched as to why this is happening

OWA 2007 Change Password
Identified the method for disabling the change password option for OWA 2007. This is important so as to not cause confustion issues with end users since this functionality will not work since AD is not authoritative for passwords and current ad permissions do not allow for changing the password in this fashion.

UM
Identified issue with loader process overwriting UM and OCS proxy address information in AD.

PowerShell Scripting

Mailbox Migration
Scripted to help automate the process for larger scale migrations. The tool automatically picks the storage group with the least number of mailboxes currently on it and creates a mailbox move script for moving each user that is listed in the users.txt file that is populated with the users to be migrated from 2003 to 2007. The next portion of the script that I am working on is automating the start of each of the created user move scripts. The tool will run up to 10 migrations at a time and then prompt for the script to be run again. It keeps a list of all the users that were moved and the dates. Passed off portions of the script to Denny for possible use in the new Exchange 2007 loader script.

UM and OCS metrics
Scripted solution to pull a count of OCS users, Voice Users, Federation enabled users, Internet enabled users and UM users and mail out the information to interested individuals. Continuing work on automating this and writing the data out with date information to a database for ongoing automated metrics.

by tony.hortert at May 28, 2009 04:32 PM

Derek Miller

Activity Report May 2009

This month has seen a lot of work around e-mail archive and Exchange 2007 storage planning.

We are currently evaluating 4 different vendor archive solutions and we are in the process of putting together our final technical recommendation to ITS management.

Operationally our backup environment continues to run smoothly and we are even completing all of our cloning within 2-3 days of processing.  Since we have implemented the new backup infrastructure we have growen from ~157 TB of data backed up per month to 247 TB backed up monthly.  We are growing at a rate of approximately 85% per year.

by derek.miller at May 28, 2009 04:12 PM

Guy Shepperd

May 2009 Monthly Activity Report

Exchange 2007 Deployment

The majority of the month has been working on project documentation, hardware requirements, and investigating different archiving solutions.

One of the things that I realized needs more documentation in the messaging community is Public Folders. There really is very little guidance on sizing hardware for dedicated Public Folder servers. Here is what Microsoft has written on Scalability. I know in Exchange 2010 they are being deemphasized even more and Sharepoint is probably a better place for them, but when you have over 8,000 Public folders; I think we might have them for a little while.

One of the issues that has been plaguing the deployment has been that after Exchange Rollup-6 was deployed, Entourage clients were unable to see any messages in the inbox. The client would connect, but no messages were displayed. The rollup-6 rollout was a red herring, which coincided with a server rebuild, and mailbox migration.

After numerous hours of troubleshooting, a call to Microsoft Premier Support was made. Here is what I learned. (thanks Pawan Kapoor!)

It seems that even though all other clients do their authentication on the Client Access Server, but Entourage authenticates and uses ASP.net. on the Mailstore role This means that you will need to add the ASP.net feature along with the integrated and windows authentication features on each mailstore server you deploy.

Here are the two KB articles that the Technician sent to me.

Configuring Authentication in IIS 7.0:

You cannot connect to your mailbox on Exchange Server when you use Entourage for Mac:

Office Communication Server Deployment

The pilot is now in full swing. The adoption rate of this product is astounding.
It is really amazing to see the ease of extemporaneous collaboration that the new R2 release has achieved. Just tonight, I was sitting working on this report, when a colleague asked me to look over a document before he published the final version. I was able to see his desktop inside my Communicator client, make some word change suggestions, point the mouse to where I was referring to, and then being a second set of eyes to verify accuracy, All 131 miles apart. Through on top of that, that we were able to talk over a peer to peer VOIP connection and discuss the changes. This is truly where Unified Communications comes together.


FaxServer Replacement

The hardware named FaxSrv1 was decommissioned, and removed from the data center. The newer hardware formally known as FaxSrv2 was formatted and is currently being reloaded with Windows 2008 Standard on it. There seems to be an issue finding RAID controller drivers, this is where we currently are and looking for alternatives to this dilemma.

by guy.shepperd at May 28, 2009 04:18 AM

May 27, 2009

Kendra Thorpe

May 2009 MAR

SharePoint Implementation - Had a wrap up meeting for the project. We are actually going into production.
Windows Server 2008 Cluster - Still having issues with adding the second node to the cluster.
In my “spare” time - Began work on improving the server provisioning and decommissioning process by adding in workflows for the various tasks and creating helpdesk tickets for work needed by other teams.
SCOM Audible Alerts - Install SCOM, Powershell as well as the scripts that provided the audible alerts on several workstations in the NOC. I am currently using it on my machine without issue.

by kendra.thorpe at May 27, 2009 09:17 PM

May 26, 2009

Scott Evans

2009 May MAR

Monthly Activity Report
May 2009

  1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Calls prior to October 31, 2008 were purged from the production AIMWorX database.  Reviewed work orders that were several months old, asked Analyst/CSR to look into each one or canceled tasks where work was already done to setup billing.  Promoted new IBM server to production database server to replace the unsupported NEC Fault Tolerant AIMWorXBE02.
  2. AIMWorX Templates — Working on a new work order template for the OCS project to setup billing in AIMWorX and to request an OCS account.  Updated ’No-PBX’ templates where problems were found from testers.
  3. Call Detail — Provided Business & Planning with call detail for multiple approved requests for information.
  4. Cubical Move — Gathered phone port information for those involved in the cubical rearrangement.  Worked with Phyllis Gray to get the work order setup properly.  Moved to new cubical.
  5. Servers — Setup new virtual server for the Accordent trial.  Worked with Accordent technician to install scheduling software.
  6. OSIS Servers — Worked with PBX operators in an attempt to get personized greetings and on-screen transfers working from OSIS-TAPI01.  Rebooted OSIS-TAPI01 approximately 10 times over the last two weeks in conjunction with PBX Techs making changes in the PBX.  Properly labeled OSIS-CR02 & OSIS-VAT02 in the Bryan building and requested NOC inventory be updated with correct information.
  7. Higer Ground backup Server — OSIS-CR02 unexpectedly shutdown multiple times.  The only way to bring server back was to unplug power and then power on.  In a conferance call with Higher Ground they stated that the VU version of hardware has had problems with the power supply.  Since the hardware does not offer a hot swappable power supply, Higher Ground is building VU a new server.  OSIS-CR02 is alarming in Nagios until this new server is put into production.
  8. PBX-Recordings NAS share — This share is used as backup to PBX operator call recordings, 6 months worth of records in 498GB and the share is 500GB.  It was decided to increase the share size by 100GB to 600GB, this cleared the Nagios alarm.
  9. Software Store — Installed latest RPEG software on ITS-HCWNAP21 server.

by Scott.Evans at May 26, 2009 12:21 PM

May 01, 2009

Gary Howard

April 2009 MAR

April 2009 Monthly Activitiy Report

1)  Completed testing of mailgate upgrade process in vanderbilt_lab cluster.  Includes inspection of policies and rules, LDAP imports, etc.

2)  Started deployment and configuration of LISTSERV development server.   Will be completed in early May 2009.

3) Worked in conjuction with ITS Security on several incidences e.g. daily scams, threats, compromised hosts and accounts, etc.  

4)  Managed abuse@v.e, postmaster@v.e., vumailguard-cmd@v.e., vumailguard-review@v.e. and listmaster@v.e. mailboxes.  Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives.  Investigated over 100 spam false negatives. 

5)  Performed daily management of mail queues on mailgates.  Removed hundreds of undeliverable messages daily in order to keep queues “clean”.  Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

6)  Created monthly Email metrics report for dashboard.  See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

7) Worked on an assortment of odd and challenging helpdesk tickets.

by gary.howard at May 01, 2009 09:56 PM

April 27, 2009

Derek Miller

April 2009 Activity Report

This month has been a lot of work around the BlueArc, backing up the BlueArc, and e-mail archive.

I have architected a backup solution for the possible expanded BlueArc environment.  The environment will include two dedicated tape libraries, a SAN switch, two storage nodes, 36 LTO-4 drives, and 6800 LTO-4 media.  This solution will be able to produce monthly full backups and weekly incrementals of all of the data residing on the new service.  Additionally, we will be producing monthly clones of all full data.

E-mail archive is in the research phase.  We are looking at various vendors and their solutions to our archiving requirements.

by derek.miller at April 27, 2009 03:46 PM

Jeff Sublett

MAR - for April 2009

1.) Coordinated 136 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 36 Magic Tickets. The majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continued preparation of assigned project to decommission IMAP2.

5.) Assisted Asset Management crew with an inventory audit.

by jeff.sublett at April 27, 2009 02:46 PM

Troy Osborn

APR MAR

Memory upgrade requested by ND&E after recommendations from their software vendor for AMP2 has finally arrived.  Due to the change moritorium during finals and graduation this will not be completed until mid May.

Worked with the vendor for the VEHS Radiation database.  A working solution has been completed after nearly two weeks of trial-and-error and as well as 5 updated versions of the client software provided as problems were found.   The client is successfully allowing admin password updates with a hashed/encrypted password, SSL connectivity to the MySQL database, and SSL connection and authentication against our Active Directory servers.  Client installation package has been created and passed along to Joanne in Desktop Support to install on VEHS workstations.  Joanne has provided a list of workstations which will be running the client and needing MySQL connectivity.   Currently awaiting the go-ahead from Peter on the new MySQL 5 server for final deployment.

Began researching installation needs for replacing our current Nagios server.  The replacement will be running Nagios v3 and will utilize a MySQL database for logging verses the standard flatfiles.  SNMP Trap handling will also be incorporated for better monitoring of services such as the Cisco CSM load balancer.

Worked with the email and security teams in identifying webmail accounts which were found to have been compromised.

Finalized configuration of a new flash server located on the VUMC network and worked with VUMC Network Security to get appropriate access to the new server.  This was deployed to handle streaming load on the VUMC network for their graduation ceremonies.  ITS Streaming Services are in the process of testing the new server.

There are no updates on the Search Engine Replacement project.  The SDM reports that the report is being finalized to be put forth for financial approval and governance.  The contract for current Ultraseek Service will expire in the next 60 days.

Completed a couple of requests from iDev on the ‘GMail For Life’ project.  No additional tasks have been presented to me while iDev works toward their intended solution.

Completed installation and initial configuration of the new HypericHQ server for testing.  Requested basic firewall access from Network Security and setup accounts for the NOC Manager to be able to begin testing.

Began working on presenting the GNAV database for VUMC and MIS for data mining and reporting.  In the process it was discovered that the vendor provided backup script appeared to have been working but was not properly performing backup operations.   Instead of attempting to fix the exxisting script a new one was written.   Backups are now compressed which allows for retention beyond a single daily backup.  Replication of backups to the secondary server has also been incorporated into the script and tested.  Any failures encountered by the script should trigger a notification email to the Unix Team.

Due to incompatabilities with archived Windows Media files, ITS Streaming Services will no longer be pushing for upgrading the Helix servers to v12.  Per request, all previous revisions of v11.x have been installed on the test server.  Installations have been made and access has been given to allow easily changing the running version on the server for testing.  ITS Streaming hopes to find a preferred previous version with the highest number of working features and least amount of bugs to move the production servers to at a later date.

by troy.osborn at April 27, 2009 02:33 PM

Guy Shepperd

April 2009 Monthly Activity report

Exchange 2007 Deployment

The exchange 2007 deployment has started out pretty well, we have moved all of ITS over to the new system. There have been very few issues. One of the largest issues has been with the MAC Entourage client. Once we rolled out Rollup 6, the Entourage client is no longer able to see the contents or folder structure of the mailbox. This is all Entourage clients, even the new beta one.

On the positive side, there have been numerous other clients that have been able to attach to the new Exchange 2007 Systems, including two different versions of Thunderbird, and one version of Alpine. These are all using the IMAP settings, but seem to working as expected.

The next steps are to work through the issues presented by the entourage clients, and then we will continue to retire the VUexchange 2003 systems.

Office Communications Server 2007 R2 Deployment

This has been the month for OCS, we have been anticipating the Lighthouse engagement, and the rollout of Office Communication Server 2007 R2 since last October. We have built the infrastructure, 5 physical servers, an Enterprise front end, a SQL Backend, a mediation server, and Edge server, and a Unified Messaging server. This will allow us to implement presence, IM, federation, public Internet connections, voice mail, and remote enterprise phones.

The deployment has been successful; we have normalized our phone numbers to an E164 format, and have the SIP gateway doing the conversion that the PBX is expecting. Learned more about regular expressions while doing this, regular expressions are very robust and powerful once you understand some of the syntax.

We currently have 175 people provisioned on the Office Communications Server.

by guy.shepperd at April 27, 2009 02:08 PM

Tony Hortert

April MARS

30 Closed Magic Tickets

Exchange
43 IMAP -> Exchange 2003 Migrations
69 Exchange 2003 -> Exchange 2007 Migrations

Exchange 2007

OWA Vanderbilt Rebranding work
The login and logoff pages are complete. Premium OWA pages are about 80% done. Built test CAS and mailbox server in MS-Test environment for facilitating the rebrand testing.

Disaster Recovery Drill
Executed Disaster Recovery Drill for ITS-HCWNEM01. Successfully rebuilt and restored mailboxes.

PowerShell Scripting
Size/Quota/Message/Item # Script and Mailbox Move Script
This script is utilized to pull mailbox size, location, quota and number of items in a mailbox. This is being utilized in conjunction with a move mailbox script that records the time it took to migrate a mailbox from 2003 to 2007. This is being utilized to try and normalize mailbox move times based on size/number of items.

by tony.hortert at April 27, 2009 02:05 PM

Peter Woods

April MAR

[WIN] RHEL5/PHP5/MySQL5 Upgrades: The migrations for the FEVS02 (misc domains) and FEVS06 (www4) servers pairs have begun. The first attempt to do this was interrupted by another work activity, and they were rescheduled for Apr 26.  The rescheduled migrations were completed without issues.

ITS Website Redesign: The final cutovef date has been scheduled.

Sun Identity Management: All of the LDAP virtual machines have been built and are accessible via our bastion hosts. The networking issues with the F5 LTMs have been resolved, and this appears to have revealed a possible issue with the ESX servers behind them. I have opened a ticket with the VUH helpdesk, and I am working with their staff to resolve the issue.

Security Investigations: I spent several days working with the Network Security team to identify and mitigate a security issue.

by Peter Woods at April 27, 2009 01:49 PM

Roland Serman

MARS 04/09

Operations

Streamlined our server build process, and updated our build documentation accordingly.  For both Windows 2003 and Windows 2008 I wrote some PowerShell scripts that utilize netsh to set all of our default firewall rules.

Also for Server 2008, I wrote some additional scripts that install all the Windows Features that we install by default, such as PowerShell.  As soon as I find the time I plan on attempting to completely automate our OS install to include all of our default settings and  application installs, i.e. Anti-virus, smartcard, etc.

OCS

Not much to report on this one, other than that I prepped two of the OCS servers for our R2 rollout.  I also prepped our ISA farm (added some new network interfaces) for OCS.

ISA

Well after months of troubleshooting and countless hours on the phone with Microsoft Premier support we’ve finally figured out a work around to our issue with editing documents in SharePoint when using ISA.  In a nutshell clients running Windows Vista and Office 2007 would receive an error when trying to save a document back to SharePoint after editing.  After spending a lot of time on the phone with and outstanding Support Engineer at Microsoft it was determined that it was a bug in WebDAV.  Until the WebDAV team releases a fix we were able to find the following work around.

  1. Require Check In/Check Out on all document libraries.
  2. Disable Basic Authentication for the the Web Client Service, open regedit, and browse to HKLM\SYSTEM\CurrentControlSet\Services\WebClient\Parameters
    1. Find the BasicAuthLevel key, and set the value to 0

SharePoint

Deployed a new GPO to all of ITS staff that configures all settings needed for seamless use of SharePoint.  The changes made by the GPO are as follows:

  1. Adds the SharePoint URL’s to the trusted sites list.
  2. Enables automatic login for trusted sites.
  3. Adds the VU Certificate Authority as a trusted root certificate authority.
  4. Adds the AuthForwardServerList registry key listing all the SharePoint URL’s via a startup script:
  5. HKLM\SYSTEM\CurrentControlSet\Services\WebClient\Parameters

    REG_MULTI_SZ  “AuthForwardServerList”

by roland.e.serman at April 27, 2009 01:37 PM

Kendra Thorpe

April 2009 MAR

IBM Director 6.1 Implementation – After moving the new IBM server to the production VLAN, I was able to successfully get several machines to “talk” to the new server. I have hardware inventories for both Windows and Linux machines. I will be putting in a change to put the new server into production sometime in May.

Shibboleth Project – I have had a quiet month this month on the Shibboleth project. I am still waiting on the IDev resource to complete testing on their tasks before I can clone the main virtual machine for DR purposes.

SCOM Audible Notifications – I was able to get SCOM to send audible notifications using Windows Media Player by harnessing the power of PowerShell and PSEXEC. I believe that after further testing, I should be able to put a list of servers in a text file and send the notifications to multiple machines. Once I have the code, I post it here.

SharePoint Rollout – I believe SharePoint is very close to going into production. Some important decisions have been made surrounding this application paving the way for an actual productions date. Whoo Hoo!

OCS Lighthouse – In addition to building one server for this project, I also configured several of the OCS lighthouse servers for teams using LACP. I was able to team two NICs and configure the team for two VLANs. It was a bit frustrating at first, but I’m glad I was able to accomplish this.

by kendra.thorpe at April 27, 2009 01:28 PM

Kenon Ewing

BlueArc upgrade
ITS is planning an upgrade of the bluearc high perfomance NAS environemtn. ITS will be expanding the capabilities from 6oTB to 750TB usable. ITS is also architecting a backup soluetion for the new 750TB environment. This will encompass 4 bluearc contollers and 2 i2k with 18 drives each. This provides enough throughput to perform full backups once a month and incrementals once a week.

SMTP Replacement
ITS has been working to replace the SMTP infrastucture from physical Sun servers to virtual Linux servers. ITS has completed the development work required for this effort, and is now looking into the deployment process. This will allow time to re-architect our email solution without running into server issues due to hardware problems.

by k.ewing at April 27, 2009 01:28 PM

Dan Raymer

MARS for May via the iPhone

1. BIND Views are finally working in DIP on the primary name server. There is still an issue with getting the views propagated to slave servers. Following the ISC instructions does not work and the vendor has been engaged.

2. The issue with pushing updates in the DIP environment has been resolved. Apparently ActiveMQ was refusing to play nicely. This paves the way for upgrades to the Sapphire Appliances and to the application.

3. Disk consolidation continues in the Virtualization environment. All “troubled” LUNs have been replaced. Additionally, prep work to retire the AMD ESX servers continue. When all is said and done, the ESX environment will drop from 20 hosts down to 12 hosts.

4. Work continues on resolving backup issues with a number of hosts moving off the .1 network to the Admin Network. There are still a number of hosts that needs this addressed.

That’s enough typing on the iPhone for blog purposes. Until next month…

by Daniel Raymer at April 27, 2009 12:50 PM

Scott Evans

2009 April Mar

Monthly Activity Report
April 2009

  1. AIMWorX – Supported all AIMWorX users varying requests.  Processed the weekly update of cost centers and student information.  Calls prior to August 30, 2008 were purged from the production AIMWorX database. 
  2. AIMWorX PBX synchronization — PBX synchronization does not run through all switches.  Since the SV8500 update, the synchronization fails at different points during the process.  PBX connection fails not only during the synchronizations but also during daily processes.
  3. AIMWorX Templates — Updated work order templates to send an email when a red phone is installed/removed to Rick Carlton, Dave Mathews and others so their red phone database can stay updated.  This change is now in production, email list was created.  All work order templates were backed up.
  4. AIMWorX Duplicate Extensions — Due to an unknown issue between work order and PBX synchronizations, some extensions were duplicated in AIMWorX.  All duplicate extensions were removed and data associated with the remaining record.
  5. AIMWorX Purge — Sent a list of data that could be purged from the AIMWorX database to responsible parties for thier input.  Data listed is from work orders, trouble tickets, billed charges, & traffic information all prior to calendar year 2006.
  6. Call Detail — Provided ITS Security with call detail for multiple approved requests for information.
  7. Server Decommission – Two servers, OpenWorX & AIMWorXFE01 physical, were decommissioned, hard drives wiped or removed and servers were put in the Oak room.  The AIMWorX server was leased and will be sent back to leasing company.
  8. OCS project – Installed OS and default software on 3 servers to operate within the OCS project.  (its-hcwnem51, its-hcwnem52, & its-hcwnem53)  Setup RSA connection.  One server, its-hcwnem51, had some hardware issues which IBM was contacted.  Several parts were replaced before the server was working properly:  RAID memory board, system board, power backplane, & two hard drives.  Ultimately, the problem was with the power backplane.  An issue with the RSA connection on its-hcwnem52, turned out to be a duplicate IP address.  Worked with Kendra & Barry to setup proper cabling and VLANs for teamed NICs on OCS servers.
  9. Nagios checks — The Nagios server IP address’ were added to all Windows firewall rules to allow for RDP check to each server.
  10. Virtual Desktop Project — Extened the evaluation license for VMware View.  Worked with VMware View software to provision a desktop and push MS Office  through Thin App.  So far without success.
  11. Snapshots — Worked with Kenon who modified the snapshot script to copy the current snapshot to a {servername}.old directory, create the new snapshot, then remove the .old directory.  This was to make sure the old snapshot was still there if the new snapshot failed.  This updated script was tested and put into production.  This was done at the request of Owen.  Updated snapshots of several ITS servers: its-scwnwb01 (Mediasite), its-hcwnap14 (SCALA), & its-hcwnap27 (MA4000). 
  12. Certificates — Updated the certificate on its-scwnap03 (BMC Magic) with a new 2 year certificate from GEO Trust.