Though I spent most of the month working on various issues/mini projects for the ECS team.  Ending the month with building the newly purchased em05, and rebuilding em04 so that it could utilize the additional RAM that was purchased… Gotta hate it when you build a server with 32 GB and only install Standard, then decided  you need more RAM.  Anyway, two mailbox servers down, eight or so go.

I spent quite a bit of time after hours setting up lab environment at the house, so that I can both work on my 2008 R2 skills, and to assist automating our build process.  So far I’ve gotten my 2008 R2 AD setup, and Windows Deployment services installed and configured.  Hopefully next month I can test doing some OS pushes, and then investigate incorporating our default settings, firewall rules, application installs etc, into an automated package.

I spent several days researching some blue screens that were generated on several of the ECS servers.  Initially we were not getting memory dumps.  We discovered two issues that were preventing the servers from creating memory dumps.

• The first issue is that the page file was not on the system partition. In order for the system to generate a User, or Kernel dump, an adequately sized page file must exist on the system partition.
• The second issue, was that the system partition was not large enough to place the entire page file on the system partition, so getting a User dump was not an option.  We could still get a Kernel dump, but that required us to make the following registry change, so that we could have a small (smaller than physical RAM) page file on the system partition:

1. HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControl\ Create a new DWORD named IgnorePagefileSize, with a value of 1.

After making the necessary changes we received our first memory dump on EM24.  After analyzing this dump, we were able to determine that an unkown driver had modified the PrintDbg routine.  Then I went through all the ECS servers looking for the same stop code, adn was able to determine that 3 different ECS servers had experienced 8 BSOD’s with the same stop code the first starting on 6/30/09.  Next we tried to track down what drivers were installed either on 6/30 or shortly beforehand.  We could not find any documented changes for driver updates on the three impacted servers. So the decision was to replace the 3 impacted VM’s with new VM’s.

I upgraded the ePolicy server to 4.0 Patch 5 to resolve a weak encryption vulnerability.  Only to discover a new more severe security vulnerability with patch 5.   I have an open ticket with McAfee on this, but have made very little progress.  They have escalated the ticket to tier 3 support, who in turn pushed it over to the security team.  I’m still waiting to hear back.

I opened a ticket with Microsoft about the recurring event 6614 that we’re getting on our production SharePoint farm.  Naturally once opening the ticket we have not seen the error.  Microsoft suggested that we remove some old application pools that are no longer in use, which we did, but ultimately the we have yet to receive the error again, so the ticket has been closed.

I applied Windows patches on all of the ECS servers, and in the process discovered several discrepancies, some of which have been resolved, and the rest will be resolved shortly. Hopefully next month we can do this as part of our regular WSUS patching schedule.

Built a few servers for the ECS guys, and during the process had to make several updates to our Windows 2008 build doc.  We are now deploying VSE 8.7 instead of VSE 8.5, as well as using a newer version of IBM director, pointed to our new IBMDir server.

I also finalized 4 new VSE installation packages for use on new server deployments.  Two for VSE 8.5 and two for VSE 8.7.  On all 4 builds the built in autoupdate task is disabled, which prevents some false alerts from behind sent out.  As we discovered recently the built in autoupdate tries to patch applications that don’t exist on the server, thereby generating Install/Upgrade failure notifications.  The other change in the 4 builds is that two of them install to our default installation path on the D drive.

I spent most of the month, preparing for, testing and then deploying WSS 3.0/MOSS 2007 SP2.  The upgrade in our test farm went smoothly, and for the most part it went smoothly in production.  I only ran into one unexpected snag in production.  After extracting both WSS SP2 binaries and the MOSS SP2 binaries, when I ran the setup wizard, it never prompted to start the wizard on the other servers, and then failed to install.  The fix was to reboot the entire farm, and then run the wizard again.  At this point, everything proceded as it was supposed to, and the upgrade was a success.  (that is after going in and adding our license key again, would’ve thought Microsoft would’ve released an updated service pack to resolve that little issue by now).

I also finished updating our SharePoint installation documents, to include not only installing SharePoint and Project server, but creating the default SSP, mysites, etc.  As well as all post SharePoint configuration changes/application installs that we’ve made to our production environment.  I’m still working on merging those changes into our DR document, but will hopefully have that completed shortly as well.  Anyway, the big positive here, is that we now have a step by step document for every step needed to reproduce our production environment with every addon, third party app, and custom configuration setting.

Once I finish updating the DR document, I hope I can find the time to polish our back up script.  Currently everytime a new site or mysite is added, we have to manually add it to the backup script.  I’m pretty sure, that we can completely automate this process, I just need to find the time to work on it.  As it sits now, I update the script once a month, and again before any major change, but that does leave holes in our backup process.

Our SharePoint rollout should be complete and officially in production by the end of this month.  So with that being said I spent some time updating all of the support documentation that goes with that, and discovered it hadn’t been updated since prior to our upgrade to 2008 on all of our SharePoint servers, so all of the instructions for setting the SSL host headers were wrong.

I prepped our test SharePoint farm to do some DR/backup/restore testing via Legato, only to discover that the current version isn’t supported if your backup server is running on Linux, so that appears to have been a wasted effort.  Supposedly the next version will correct this issue, I guess we’re back to waiting for EMC to provide a viable backup and restore method for SharePoint.

I spent quite a bit of time working on ISA this month.  ITS-HCWNAP61 is now officially in production supporting our OCS deployment.  I rebuilt and prepped ITS-HCWNAP60 to join the array, and discovered several potential issues while testing the change in our test environment, and have had to defer my original change.
•    It appears the issue that caused us to rebuild our ISA servers repeatedly, where whichever node didn’t house the CSS would start to deny all incoming requests, is actually due to how NLB utilizes unicast, and most switches aren’t configured by default to allow all traffic destined for a specific MAC to be sent to multiple switch ports.  I’m currently testing a potential fix for this in our test environment.
•    In researching the previously mentioned problem, I stumbled across several things we weren’t aware of when we initially deployed ISA.  When using NLB, intra-array communication cannot pass on any interface using NLB.  Also, it is recommended that your Intra-array communication reside on a private network, a daunting challenge in our environment, since this private network also requires AD access, and is heavily used for Kerberos authentication, as well as passing configuration back and forth between all array members.

Streamlined our server build process, and updated our build documentation accordingly.  For both Windows 2003 and Windows 2008 I wrote some PowerShell scripts that utilize netsh to set all of our default firewall rules.

Also for Server 2008, I wrote some additional scripts that install all the Windows Features that we install by default, such as PowerShell.  As soon as I find the time I plan on attempting to completely automate our OS install to include all of our default settings and  application installs, i.e. Anti-virus, smartcard, etc.

OCS

Not much to report on this one, other than that I prepped two of the OCS servers for our R2 rollout.  I also prepped our ISA farm (added some new network interfaces) for OCS.

ISA

Well after months of troubleshooting and countless hours on the phone with Microsoft Premier support we’ve finally figured out a work around to our issue with editing documents in SharePoint when using ISA.  In a nutshell clients running Windows Vista and Office 2007 would receive an error when trying to save a document back to SharePoint after editing.  After spending a lot of time on the phone with and outstanding Support Engineer at Microsoft it was determined that it was a bug in WebDAV.  Until the WebDAV team releases a fix we were able to find the following work around.

1. Require Check In/Check Out on all document libraries.
2. Disable Basic Authentication for the the Web Client Service, open regedit, and browse to HKLM\SYSTEM\CurrentControlSet\Services\WebClient\Parameters
1. Find the BasicAuthLevel key, and set the value to 0

SharePoint

Deployed a new GPO to all of ITS staff that configures all settings needed for seamless use of SharePoint.  The changes made by the GPO are as follows:

1. Adds the SharePoint URL’s to the trusted sites list.
2. Enables automatic login for trusted sites.
3. Adds the VU Certificate Authority as a trusted root certificate authority.
4. Adds the AuthForwardServerList registry key listing all the SharePoint URL’s via a startup script:

HKLM\SYSTEM\CurrentControlSet\Services\WebClient\Parameters

REG_MULTI_SZ  “AuthForwardServerList”

• Determined that if you disable WebDAV and require Check In/Check Out then you no longer get prompted to authenticate with Vista clients when accessing SharePoint via ISA.
• Developed PowerShell scripts to configure the Host Based firewall with all of our default settings for both Windows 2003 and Windows 2008.
• Assisted with the Exchange 2007 DR test, and have documented the OS side of the Exchange 2007 build for the Mail Team,  as well as developed a firewall script to apply the additional firewall config changes needed for Exchange.

At this point, I think we’re moving in the right direction.  It appears that we’ve solved the issue with some Vista clients not being able to save a document in SharePoint after editing it by using LDAP authentication to AD instead of AD Integrated.  I can’t explain it, but thats what we’ve discovered.

We still have two outstanding issues, one being login prompts for some users when opening a document, and the second being that our second ISA server decides to deny all incoming connections after an hour or two.

Oh yeah, SharePoint, and a bit of ePO stuff as well.

• Rebuilt the SharePoint test farm on the new Test Admin Network.
• Spent countless hours either on the phone with Microsoft troubleshooting our file edit issues when using ISA.
• Spent countless hours testing various scenarios and building and rebuilding Vista VM’s to accomodate Microsofts, testing requests.
• Assisted Partner with vairous issues with ePolicy Orchestrator, as they tested and then deployed it to all of ITS.
• Developed a login script for Partner so that they can easily deploy the ePO Agent to all of DOA.
• Removed, and retired ITS-HCWNAP54 from our SharePoint farm, the last Windows 2003 server (with the exception of the SQL Cluster) in our current production farm.
• Resolved an issue with ePO reporting, where none of the canned reports were showing up.

I’ve also spent quite a bit of time trying to figure out why some people have issues editing documents in SharePoint if they log in via ISA.  I currently have a ticket open with Microsoft on this, but we’ve made little if any progress as to determining the actual cause.