Planet AppHosting

August 2008 WINS

2008-08-29T19:30:25+00:00

Microsoft Unified Communications – Proof Of Concept update

ITS needed to develop a platform that brings communication convergence to the desktop to enhance/enable spontaneous communication and collaboration within and externally to Vanderbilt University. We now have forty-one users on the POC, all at varying levels of service. Sixteen of the users are full Unified Communications with their email living in Exchange 2007 and twenty-three have just instant messaging and presence.

The team produced a small video displaying the agility and spontaneity that can be gained from the use of Unfired Communications. This four minute video has been showed to different groups to gauge the interest in such a project. There seems to be a large interest in this technology.
Microsoft Unified Communications – Production Pilot

The other initiative is to get the schema changes into the Production Active Directory. To accomplish this we deployed the schema changes into the MS-Test environment. We also deployed a working OCS Standard server into that environment, to help identify the attributes that will be needed for the development team to automate activations.

After two weeks of life in the MS-test environment, we were able to make the schema changes into the Production Active Directory, there were some hurdles in this deployment, most of them have to do with the locked down security model of the root domain, and being able to add attributes into the child domain using the Enterprise Administration roles.

We have also set up meeting with Microsoft and one of their Partners to help validate our Design, and to gain us entrance into the MS Lighthouse program.
Adobe Flash Server upgrade

Application Hosting System Administration Upgraded the adobe 'Flash' server to the latest revision. This update was performed by request in order to provide expanded capabilities to end users of the flash service.
New Load Balancers for IdM on their way!

The original network load balancers used to balance the IdM LDAP authentication service deployed in June did not meet expectations. Since then, Application Hosting staff have evaluated and purchased lab equipment from F5 called BigIP Local Traffic Managers. Thorough testing has resulted in the procurement of an entire infrastructure based on this technology that is intended to be deployed in three data-centers throughout campus. The equipment is expected to arrive in late September.
ITS Owns the Change Management Application!

This month, ITS staff migrated the Change Management application from the VUMC server named "trutto" to the VU server named change.its. The medical center retired its use of this application last year, and has donated the code to the University. ITS staff have already customized the application adding in rich features and more business aligned data fields.
Extending the use of the VU Data Center

This month, ITS NOC personnel, Application Hosting administrators, and several co-located departments throughout campus worked together to complete Phase I of the "Data Center Re-alignment" project, which is underway in order to more efficiently cool the thousands of computing devices housed in the data center.
Lifecycle Replacement

ITS System Administration replaced the three existing VUExchange Mail Front End servers this month with brand new equipment. These servers are used for web email capability, all smartphone Activesync connections, and mail routing for the exchange environment.
Additional diskspace for research computing!

This month, ITS Storage administration staff upgraded the BlueArc Network Attached Storage appliance adding 64 SAN ports and 45 Terabytes of space. The Bluearc is a storage array that supports the Research community here at Vanderbilt University.
Operational:
- 175 Magic Tickets were processed in Applications Hosting.

46 VUmail to VUExchange Mailbox Migrations

Mars - Aug 2008

2008-08-28T21:15:41+00:00

VTL
This month the storage team has been working with EMC on deploying a disk back end evaulation Virtual Tape Library solution. We worked with EMC for three days to get the initial installation complete. EMC is on site Tuesday and Wednesday next week to finish the configuration. Once this is complete we will begin a one month evaluation to ensure that this product will perform as needed in out environment. This will entail reconfiguring the backup server running networker to use the new VTL that we will present from the EMC evaluation unit. We are looking at creating four Virtual Tape Libraries (one for each Legato Networker storage node) with 12 virtual tape drives. This should substantially increase our throughput for backup clients, which in turn should give us the capability to do more work in less time.

Backup Activity Reporting
I have completed phase I of a custom application that will report statistics about what files we are backing up. This visibility will provide the capability to begin taking steps to managing the data that we backup, which in turn can effect the amount of data we have stored for Vanderbilt University. For instance, there are some servers we have already seen that have extremely large files like database backups and keep rotatations of these backups for weeks on end. If we could identify a smaller number of these backups (say one week as opposed to four) we could decrease the time needed to complete our daily backups, as well as ensure we have capacity for upcoming projects, as opposed to having to purchase more storage.

Aug MAR

2008-08-28T05:07:36+00:00

Sun Identity Management (IDM): The team has decided to create another classification of servers to add some additional flexibilty into the development progress. The new classification will be called preproduction, and it will combine some of the elements of the existing test and production classifications. I also removed the Directory Proxy Server (DPS) configurations from the CSM. This caused a small problem with production transaction routing, and a temporary out-of-service serverfarm was created so that the server was still accessible.

Test F5 LTM Implementation: The two LTMs that we purchased are now ready to take up their roles as test hardware. I had to modify our server naming policy to accommodate the new device types and roles. I also renamed all of our initial testing configurations to meet the new standards.

Bastion Host Patching: I patched the two AppHosting bastion hosts up to current RHEL4 standards. During the patching process, remote access was available by using the alternate host.

mysql-srv1 Decommissioning: The Sun Solaris MySQL server has been officially decommissioned. It is now out of service and waiting for the disks to be wiped.

thecard.vanderbilt.edu: I created a virtual on the SSL servers for Dining Services and change the DNS to point to the new site. This new configuration allows Dining Services to offer additional services on their main site beyond those currently provided by the vendor.

change.its.vanderbilt.edu: I created a new virtual host on the ITS web servers for the new change management website. The new virtual host involves an additional vserver and serverfarm on the CSM as well as some modifications to the existing mod_security rules.

ldap.vunetid.vanderbilt.edu: I updated the SSL certificate use by the two Solaris LDAP servers.

August “Home of Phoenix, Spirit, Opportunity, Pathfinder, and the Viking twins”

2008-08-27T21:29:38+00:00

August has come and gone… and not a moment too soon, I say! This means it’s time to read Kim Stanley Robinson’s Mars Trilogy! Uh… I mean, it’s time for the Monthly Activity Reports (MARs).

Once again, my main focus for the month has been DNS with a healthy dose of DHCP.

The Diamond IP test environment was finally successfully upgraded to 3.0.53. This will allow us to move forward with the NetID import process and get ready for DHCP roll out.
Prerequisites Phase I, and Phase II steps for production migration successfully completed. We continue to run through the process to iron out any possible "gotcha’s" that may appear.
OMAPI was successfully implemented and works with Diamond IP.
DNS Views testing is halted while we finalize DHCP testing. The repeated and rapid database refreshes makes it extremely difficult to make any meaningful progress before being reset to 0.

DNS took an interesting turn this month as we finally started to do some analysis of the traffic and performance of the servers.

What started out as curiosity, performance reporting on the name servers has become an automated process. Daily total query counts and identification of the top query sources are reported to management. Looking through this information has allowed us to identify, with sometimes very surprising results, where the majority of the DNS traffic originates. It has also opened our eyes to some concerns of acceptable usage, too.
Next on the list is the identification of top requested domains and to automate that report as well.
Named stats are FINALLY being collected and are almost ripe for graphing and analysis. This will give us a good idea of how the server is performing in more granular levels.

Not to be stuck feeling left alone, our ESX virtualization environment decided to rear its head and demand some attention.

The final fault is our own, but a networking hiccup uncovered some serious design/implementation flaws in our ESX environment. The hiccup caused a cascade of network outages which made the ESX servers to think they were in an isolated state. High Availability did what it was supposed to and attempted to initiate migrations of the VM’s to other servers and to kill the running VM processes. Except… almost every ESX host thought the same thing at the same time resulting in some VM’s not moving, some moving repeatedly, some shutting down, and others just getting into a confused state.
Fault lies not with the software or even the network, but in the lack of paying attention to detail to insure that our virtual switches were redundant to separate physical switches and all VLAN’s were correctly tagged to their associated physical ports.
Identification and repair of the situation continues.

Other miscellaneous tasks filled up the rest of the time.

Power/rack moves during the weekends
Operational duties
Patched some of the standby and test/dev database servers.

Next month is patch happy heaven and a massive ramp up to DHCP migration.

MAR - for August 2008

2008-08-27T21:04:47+00:00

1.) Coordinated 175 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 45 Magic Tickets. This majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry, mailing list owner changes and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Started meeting and discussing project to upgrade SMTP09 and SMTP10.

5.) I spent a good portion of the month involved in data restoration.

6.) Spent more time with the PAVE project due to a problem with 31 user accounts.

August 2008 MAR

2008-08-26T22:19:28+00:00

August 2008 Monthly Activitiy Report

1) Worked on an assortment of odd, challenging, and familiar helpdesk tickets.

2) After upgrading Proofpoint vanderbilt_production cluster from version 4.0.7.67 to 5.0.3.103, there were several issues that had to be resolved. a) Custom Sendmail configuration had to be reinstalled. b) Maillog rotation had to be reconfigured. Note maillog rotation occurs by size instead of date as proetection against DOS attacks. Had to recalculate how many rotations could be retained based on average disk space consumed by log files in order to retain approximately 30 days of logs. c) Several issues had arisen with the databases. They include corruption in the log db that was repaired, tuning the database for better response, adjusting the expiration and optimization cron jobs so the databases do not grow too large, and purging old replication logs that had consumed more that 30 GB of disk space. d) Discover that Email Firewall rules based upon scanning message content became more expensive in the new version and were a source of frequent and numerous filterd restarts. Proofpoint advised making use of Policy Routes in rules since they are less expensive with less impact when messages are scanned.

3) Discovered issue where Webroot Spyware could cause issues sending email from a Windows or Mac clients. Issue was resolved by disabling the email firewall in Webroot Spyware. Communicated solution to Partner Support.

4) Several errors were being reported in the inbound SMTP server logs related to TLS. Resolved by configuring TLS for various domains/hosts. The errors do not prevent email delivery. The errors simply communicate that a server is offering TLS but the client has not been configured to require it. The reason the client does not require it is no one has requested that it be configured for the domains/hosts.

5) Project for mailing list replacement. Frank Kyle and I continued to meet regarding loader development. Frank was able to successfully create and delete lists, add and delete subscribers from the lists. All current loader functionality was tested successfully. Kate per Zafar and Cindy has requested loaders to be altered so that specific types of list e.g. moderated, unmoderated, announce-only can be provisioned automatically. Frank is determining work effort and time frame required for these changes. Working with Troy we resolved issue with notebooks not working. Continue to work seeveral tasks that need to be completed before going production. Continue to lead role with developing standards, security model, configuration and customization of server, and providing assistance with development and customization.

6) Project to implement smtp and e-mail authentication. Nothing new to report.

7) Worked in conjuction with ITS Security on several incidences e.g. scams, threats, compromised hosts, and SWEs, etc.

8) Managed abuse@v.e and postmaster@v.e. mailboxes. Monitored abuse@v.e. and vumailguard-review@v.e. for daily reports of spam false negatives. Created or edited Proofpoint rules in order to deter several threats. Investigated over 100 spam false negatives.

9) Perform daily management of mail queues on mailgates. Remove hundreds of undeliverable messages daily in order to keep queues “clean”. Review messaging reports daily in order to spot trends, abuse, etc. and took appropriate action to deter threats.

10) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

August MARS

2008-08-26T15:45:04+00:00

27 Closed Magic Tickets
2 Security Incidents
Exchange 2003
46 IMAP -> Exchange Migrations
Exchange 2003 Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches
Buildout of ITS-HCWNEM01,02,03
In preparation of quota increase to 1GB for VUExchange, extensive work to get these boxes to pass security scans. The boxes had to go through security scans at least a half dozen times. With some boxes passing one scan then failing another with no new configuration. Also boxes failing then passing after only a reboot.
Buildout and install of Exchange for ITS-HCWNEM12
In support of life cycle replacement of the Exchange Front End servers
Verify and set EPO settings for Exchange servers
Troubleshoot/resolve issues with Law School students OWA and outlook display defaulting to Chinese characters
Troubleshoot issue with ~30+ users configured to activesynch against vuexchange environment when they have no exchange mailbox
Information gathering in support of quota increase to 1Gb for VUExchange
Continue work with Microsoft to resolve w3svc errors when recycling the application pool.
Research IMO Legacy Exchange DN as well as all objects with its-exch.mail.vanderbilt.edu proxy addresses in 2003 AD
Identify/Troubleshoot loader AD

Exchange 2007
Setup\writeup procedures for using Apphost-resource account for accessing 2003 ITS resource accounts for users in 2007 environment
2007 Exchange Public Folders
Fix public folder permissions of ITS Folders
Exchange 2003 mailbox migrations to Exchange 2007
Create connector for email.mc.vanderbilt.edu to avoid bouncing of mail to contacts that don’t exist in SMTP.
Troubleshoot /resolve UC2k7 startup issues due to Power outage change.
Troubleshoot/resolve Zafars connectivity issues with his client to the Exchange 2007 Mail Servers
Work out solution to allow Zafar to send as NOC@vanderbilt.edu, since this address resides/belongs to Exchange 2003.
Research/Troubleshoot Windows Server 2008 backup of Exchange 2007
Resolve network issues with ITS-HCWNAP12-ts.uc2k7.vanderbilt.edu

MARS 08/08

2008-08-26T14:27:15+00:00

Host Header Site Collections over SSL

We’ve been working condensing the number of web applications that we’re hosting, and have turned to Host Header based site collections. Doing so over http is quite simple, but since we only host sites via SSL I ran into a few things that I couldn’t really find documented anywhere, and thought I’d share.

The first thing we discovered is that if you use a random port when creating your Web Application you will receive the following error every time you try to create a host header site collection:

"The port specified for the new host header site does not match any known bindings in the specified Web Application. The new site will not be accessible if the Web Application is not extended to an IIS Web Site serving this port."

This being said the host header site collection works, but if you try to restore content to it you receive the same error. What I found was that if you create your web application as http://site.myurl.com:443 and then change the alternate access mapping setting the default to https://site.myurl.com and intranet as http://site.myurl.com:443 you can create any number of host header based site collections error free.

Once your web application is created, creating the host header based site collection is quite simple, you just run the following:

Create Site

stsadm -o createsite -url https://hosted1.myurl.com -ownerlogin DOMAIN\username -owneremail username@myurl.com -sitetemplate STS#1 -hostheaderwebapplicationurl https://site.myurl.com

Create Site in New DB

stsadm -o createsiteinnewdb -url https://hosted1.myurl.com -ownerlogin DOMAIN\username -owneremail username@myurl.com -sitetemplate STS#1 -hostheaderwebapplicationurl https://site.myurl.com -databaseserver sqlsrv.mydomain.com -databasename WSS_Content-HHSite1

Note: you can use either -hostheaderwebapplication or -hhurl for the switch, both work, but currently (with SP1 installed) if you do a stsadm -help createsite it comes back with -hostheaderwebapplicationurl as the switch.

Once the site collection has been created, you now need to setup the SSL host header. This is rather simple as well, you just open a command prompt and navigate to C:\inetpub\adminscripts (or wherever you’ve placed your inetpub directory) and run the following command:

adsutil.vbs set /w3svc/[identifier]/SecureBindings [IP address]:443:[host header name]

So in example above of having the web app as site.myurl.com and the hosted site as being hosted1.myurl.com. you’d run the following:

adsutil.vbs set /w3svc/123456789/SecureBindings 192.168.1.10:443:site.myurl.com 192.168.1.10:443:hosted1.myurl.com

Note: You seperate each host header with a space.

You also need to ensure that you have a wildcard cert applied to IIS vserver that will cover all Host based site collections that you’re hosting on the associated web application.

SharePoint Powershell Warm Up Script

I take no credit for this, I found it at Kirk Hofer’s blog, and thought I would share it as well.

############################################################################
#Assumptions:
#-Running on machine with WSS/MOSS
#-C:Program FilesCommon FilesMicrosoft Sharedweb server extensions12BIN in path
############################################################################

$output=stsadm -o enumzoneurls
$x=[xml]$output
$x.ZoneUrls.Collection|foreach-object -process {$y=stsadm -o enumsites -url $_.Default;$sites=[xml]$y;$sites.Sites.Site}|foreach-object -process {write-host $_.Url; $html=[System.Net.WebRequest]::Create($_.Url);}

How to Schedule a Windows Powershell Script

I take no credit for this, I found it at SearchMarked.com posted by hstagner.

Basically all you need to do is create a batch file that calls your Powershell script and schedule the batch file. Your batch file should contain something similar to the following:

powershell -command "& 'MyScript.ps1' "

August 2008 MARS

2008-08-26T13:47:53+00:00

Windows Team Lead

Customized the ISA Login page that displays data from a mysql database

Activity Report August 2008

2008-08-25T20:23:34+00:00

We installed 2 new disk array enclosures and 2 new fiber channel switches into the BlueArc environment to expand existing capacity.
I optimized the cloning scripts for the i2k tape library data in order to speed up cloning time. Cloning is now organized such that each thread will completely clone all data available to clone on a single tape before moving onto the next tape. This should decrease time added to the cloning procedure by loading and unloading tapes.
We’ve been plagued by a couple of issues with our tape drives on the new i2k. A firmware upgrade appears to have fixed the issue.
We are continuing to work on the VTL evaluation. The hardware install has been completed. We are now coordinating with EMC to get software resources to complete configuration so that we can test the VTL in our environment.
I have continued to investigate any client failing backups within our environment and determining causes so that we can have as complete a backup of all of our systems as possible.
Operational activities have consumed the remainder of my time.

Aug 2008 MAR

2008-08-25T15:02:13+00:00

Completed audit of vlans on each trunk port versus the configuration of the vswitches on each of the ESX servers in Hill. Corrected problems found which had or may have impacted the production environment.

Closed the SWE for implementing a Samba server within VUMC. After trying to work through multiple problems and incompatibilities with integrating Samba and AD, the end user decided to switch to a windows solution for their new file server. This may be revisited later, but was ended due to time constraints with the start of the school year.

Worked with the NOC to implement LDAP authentication on the SmartCDUs used in the data centers. Received approval and setup an ftp repository for firmware updates to simplify updating the SmartCDUs to the supported revision. Demonstrated a working proof-of-concept to the Data Center Manager and provided instructions for installation and configuration of the solution.

Upgraded the ‘Flash’ server to the latest revision. This update was performed by request in order to provide expanded capabilities to end users of the service.

Continued to make progress on the Virtual Desktop project. Recently upgraded the Sun Global Desktop server software to the latest revision. This upgrade moves the server to more modern versions of apache and tomcat as well as provides additional functionality needed for the project.

Continued working with the project manager for the Ultraseek replacement project. Currently working with vendors to implement ‘pilots’ of their services over a subset of our web environment for testing and evaluation.

Signed up for the first ESX training class which will be held the first week of December in Nashville.

2008 August MAR

2008-08-25T14:31:51+00:00

Monthly Activity Report
August 2008

AIMWorX – Supported all AIMWorX users varying requests. Copied all SMDR and CDR backup files to CDR for archive. Processed the weekly update of cost centers and student information. Ran a monthly process to remove any duplicate calls from the database prior to a billing cycle. Setup charge information for new App Hosting monthly & onetime charges. During student return weekend it was discovered that Apphost2 did not have access to AIMWorX. Apphost2 runs a query against the AIMWorX database to give validated students their Vnet codes. When the AIMWorX servers were moved into thier new subnet, this firewall rule was not moved. Terry Cavender updated the firewall to allow the Apphost2 query.
AIMWorX Issues — The beginning of August brought problems with the OAI connections to both the Student & Admin PBX networks and AIMWorX. These problems originally showed themselves by a constant stream of screen pops for VPD stating there was a ‘line link error’ on the FPC17 route to VPD (170). After resetting AIMWorX and the OAI connection multiple times, the PBX techs and NECdecided that a restart of the Student PBX network NCN was needed. This corrected the problem. The next weekend the OAI problems presented themselves in the Admin PBX network. AIMWorX was rebooted several times and each time the problem bounced between VPD screen pops and Authorizer. Either Authorizer was not working or the ‘line link errors’ on the VPD route was received every minute. This problem was worsened because the CHECKSUM process in the PBX was not working to validate Vnets. The result was that no one could make long distance calls using a Vnet. It was decided that the Admin PBX NCN had to be rebooted. Again this resolved the problem.
MA4000 project – After installing the MA4000 software using the SQL test cluster the software was producing errors. The DTC registry hacks were checked on both the MA4000 & SQL test cluster servers. It was determined that a service restart did not always work, this was determined based on comments found while searching for a resolution to the DTC issues. The recommendation was always to reboot. So, the SQL test cluster servers were rebooted and MA4000 was able to make a connection to the database using DTC. During this process NEC was consulted and assisted in correcting a problem with the CAS software. The CAS software is used to validate logins with MA4000.
Virtual Desktop project — The three VISTA virtual desktops within VMware VDI lost trust with the domain. These three desktops were deleted and recreated to resolve the problem. Users are again testing. Went and talked with LPS about the Citrix Xendesktop solution. A pilot of this would require one physical server.
Virtual Servers — Created a virtual server template for Windows 2008 64 bit. Converted the DOA web server from physical to virtual.
OSIS servers — It was determined that the six windows servers did not have the security setup completed on them: IVR01 & 02, TAPI01 & 02, CR01 & 02. I have contacted Amcom and Higher Ground to determine what firewall rules are needed on the local host for their software to funtion. Once this is received the firewalls will be setup and turned on.
Higher Ground Server – On August 11, one of the Higher Ground call recording servers began sending error messages to the PBX operators every minute or so. They contacted me, on-call, and I called Amcom per our support agreement. Talking with Amcom support they stated that they did not support this server and that I should contact Higher Ground. I asked them to talk with Ian Campbell, Amcom support manager. They did and according to the tech support person he would get the server checked out. After a while the Amcom support tech called back and said that the Amcom software was working properly and that I should contact HG. Again I asked them to talk with Ian because my understanding was that Amcom would contact HG. I went ahead and contacted HG and opened a support ticket because the server had not been recording for several hours. HG resolved the problem in short order. I continued to email back and forth with Ian about the support arrangement. Ian stated that VU was suppose to contact Amcom when there are issues with the HG servers and that he needed to communicate this to his staff. In the next OSIS meeting, Ian stated that we should contact him with support for the HG servers until his staff is properly trained.
QueWorX Life Cycle replacement – The two IBM servers purchased for QueWorX were recieved and determined that they did not have the correct expansion ports. The Dialogic board being used for QueWorX was a PCIx board. The servers had two PCIx slots and two PCIe slots. The problem being the Dialogic board was a full length/full height board. Only one of the PCIx slots was usable. NEC was contacted and they determined that this board was only available in full length/full height. The install document states that PCIe slots are needed, but the board only comes in PCIx. This prompted a discussion on whether the servers received would work since there was no expansion capibility. During this discussion the current IVR server for node one, QueWorX is replacing this server, began having power supply problems. So, one of the two IBM servers was loaded with Windows 2003 sp2 and the QueWorX software and installed in the Bryan Building to begin the load process to replace the server going bad. No decision has been made regarding the second server.
Mediasite — Worked with Mike Vidal, of Sonic Foundry, to resolve an issue with MAC’s not being able to view presentation. This issue was corrected with the setup of http streaming on the Mediasite server.
SCALA players — Before student return, several of the SCALA players were not talking with the server over https. Tom Hunt, NEC, set the players to a http connection to allow the players to download their new content. Now it needs to be determined why https will not work. A new SSL certificate was installed on this server the week prior to the problems. An emergency sign process was tested and was successful. The process is very manual. NEC was then asked if there was an easier way to make this work. NEC came back with SCALA scripting that can be added at a cost to make the emergency process easy.
Server Decommission — The hard drives were erased or removed from four servers that need to be sent to supluss. The server information was sent to Joy Saunders to remove them from inventory. These servers came from the Bryan building: Two were used for MCS, one for Codemaster, and the other I do not know.

This Doesn’t Happen Very Often

2008-08-06T16:31:41+00:00

My Magic queue is entirely empty. Somebody submit something quick…

August 2008 Blog

2008-08-05T13:46:36+00:00

Wrote script on RHN to stop/stop/status for the oracle nagios listener.
Added a datafile to the nettracker database.
Extended VPX datafiles from 9GB to 15GB.
Wrote Nagios check oracle script to check Flash Recovery Area space usage for Oracle databases.
Upgraded the DiamondIP test database from 3.0.39 to 3.0.53.
Patched the nettracker database with the July security patch.
Wrote Nagios check oracle script to account for the 10G autoextend space.
refreshed the diamondip database and currently writing documentation and testing out sql refresh scripts.

July, 2008 Monthly Activity Report – App Hosting

2008-07-30T18:25:39+00:00

New Tape Library goes live!

ITS life-cycled the core component of the backup environment in July, 2008. The old L700 tape library has been replaced with a Quantum i2000 Scalar tape library which boasts a dozen LTO4 drives, and eleven-hundred tape slots. This new library is a foundational component for future business recoverability for many departments at Vanderbilt.
DNS Security Patch Applied

On July 8th, 2008, the Technical Cyber Security Alert TA08-190B announcing vulnerability within multiple DNS implementations to cache poisoning, including BIND was released. This is the protocol used here at Vanderbilt for Domain Name Resolution. Many technology vendors have released patches for their products to close this major vulnerability. ITS applied the patch to the primary internet-facing DNS servers on July 19^th, 2008.
Data Corruption Avoided

Earlier this summer, EMC contacted ITS Storage Administration about a bug in a particular version of a hard-disk drive deployed at ITS. This bug caused data corruption when specific conditions occurred. ITS quickly scheduled a Flare code patch that would identify corruption before it occurred and then subsequently replaced the disk drive that was susceptible to the error condition.
Mail Gateways Upgraded

The ITS Electronic Message Administration team has upgraded the Proofpoint Email Gateways to version 5.03. This upgrade provided a more granular reputation model, advanced administration GUI and reporting capability.
Web Application Scanning

In a coordinated effort with the Network Security tea, several hand selected websites on the main Vanderbilt web server were used as evaluation content for two web application scanners. These sites were selected because of suspicious log entries for requests to these sites. The scans involved a discovery phase, last minute database backup, an evaluation phase, and a reporting phase. In addition to evaluating the scanning productions, we were also able to pass the findings on to our customers so that they could fix their applications.
All Server Consoles moved to a protected VLAN

This month ITS System Administration completed the year long project to move Server Consoles for HP and IBM servers to a protected and private administration network. This configuration is much more secure than the previous.
Storage Upgrades in preparation of 1GB VUExchange Mailbox offering

This month ITS Storage Administration completed the expansion of the EMC Clariion and Cisco 9509 SAN Directors to support the 1GB VUExchange default mailbox size.
Vunetid server migrated to a Virtual Machine

ITS Staff migrated the vunetid server from an out-dated server and up-supported operating system to RedHat Enterprise Linux running on a virtual machine. This has removed dozens of security holes and properly aligned this service with modern infrastructure.
Operational and Administrative updates:
- 166 Magic Tickets were processed in Applications Hosting.
- 34 VUmail to VUExchange Mailbox Migrations
- Two employees attended F5 Load Balancer (Local Traffic Management) training

July MARS

2008-07-30T15:02:28+00:00

Well… take 2 on this…

1. DIP Test Environment - It’s up and running. DHCP Migration moving forward

2. BIND Views in DIP - Initial attempt to import a working BIND views configuration failed with tons of Java errors. Trying another method.

3. Moved a bunch of servers to new rack locations.

4. Fixed various maintenance issues with list-srv1, news-srv1, Napster, and other servers.

5. Patched the DNS vulnerability

6. Filled in for PW on IDM - ouch…

Primary DNS is now pumping out 28,500,000 queries in a 24 hour period.

(Original version of MUCH longer - due to a crash and a time limit, this is what you get)

July MAR

2008-07-29T21:47:38+00:00

Sun Identity Management: I have been involved in the various architectural and development processes for this project. We are re-architecting to work around the DPS which has caused availability issues with the LDAP services.

F5 LTM Training: I attended the F5 LTM Essentials and F5 LTM Advanced training courses in Atlanta. This training was needed for our proposed load balancing solution for the Sun IDM project.

Web Application Scanning: In a coordinated effort with the Network Security tea, several hand selected websites on the main Vanderbilt web server were used as evaluation content for two web application scanners. These sites were selected because of suspicious log entries for requests to these sites. The scans involved a discovery phase, last minute database backup, an evaluation phase, and a reporting phase. In addition to evaluating the scanning productions, we were also able to pass the findings on to our customers so that they could fix their applications.

Proofpoint Upgrade: I was an unexpected participant in the Proofpoint upgrades. The upgrade script did not perform as expected, and exited before installing all of the upgraded packages. This left the servers in a state where the kernel did not match the modules. I got the servers online to a point where the vendor could resolve the issues for us.

July MARS

2008-07-29T20:47:20+00:00

22 Closed Magic Tickets
1 Security Incidents
34 Imap -> Exchange Mail Migrations

Exchange 2003

Exchange 2003 Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches including Windows 2003 SP2

Document Exchange Front End Installation for Exchange 2003 Production environement
Allows other Admins to build front-ends and documents our current configuration.

Document IMAP->Exchange 2003 migration process
Created step by step documentation that has been followed by Guy to perform IMAP->Exchange migrations.

Buildout of New Exchange Boxes for 2K3 environment
Its-hcwnem10 and 11 were built and put into production as front ends in the email.vanderbilt.edu NLB. These boxes are actually in AD DNS which is a first for us. There was a small issue resulting from AD DNS not being authoritative as reverse DNS which was resolved by putting the entries in IP-SRV1 and IP-SRV2.
Its-hcwnem12 has been built and is being utilized for testing new 3rd party firewall to more securely protect our windows boxes
Its-hcwnem01-03 have been built and will be introduced to the environment to continue the VUExchange capacity enhancement project.

Worked with Microsoft on Front End Application crashing issue
The actual crashing issue was resolved via Windows Server 2003 SP2. The underlying cause of the w3svc service timing out when closing is due to aspx support on our front ends. Certain clients are requesting an aspx page from an ews subsite that does not exist in Exchange 2003, It is a subsite that exists in 2007. To get around this the new Front-Ends were configured without the aspx support which has resolved the issue. We will be applying this same configuration to the current Front-Ends.

RM for exchange
Worked with EMC to recover the RM admin password after my laptop crashed.

Exchange 2007

Document Procedure for Migration of mailboxes from 2k3 -> 2k7 environment

Perform migrations for ITS Sysadmin Windows team to 2k7 environment
This is to allow for further testing of the environment and show proof of concept for UM and UC.

Extensive troubleshooting to resolve various client issues in the 2k7 environment
Resolved Outlook Integration Issue with OCS
Fixed Free/Busy synch between 2k3 and 2k7 (except for confrm’s still working on this one)
Identified DNS issues that were resolved.
Documented and opened firewall ports to resolve issues with both normal MAPI connectivity to Exchange 2k7 as well as UM.

General Windows

Update Windows Server 2003 Installation Document
Update new procedures. Remove IP addresses from old SNMP Settings. Add new IP’s for backup FW rules. General cleanup.

IBM Director
Worked with IBM to try to identify the error message that we have been receiving on MSSql-SRV2. They have requested that the firmware on the Raid card be upgraded to the latest level. Still waiting on this to be able to get back with IBM on the error message.

CiscoWorks
Worked with Victor in setting requirements and speccing out a server to replace the CiscoWorks server.

SCOM Agent SP1 upgrade and AD DNS
Identified issues with SCOM and lack of presence in AD DNS along with the boxes that it is monitoring. This issue is actually much larger with any of our boxes going into AD DNS with a disjointed name space (any dns name that isn’t ds.vanderbilt.edu) while cnaming these in the future will work changing the dns name on the box and on certain services (mail, scom) after install would cause major issues. This is being addressed in the VUExchange environment with the new boxes being added to AD DNS with the .ds.vanderbilt.edu namespace.

Jul 2008 MAR

2008-07-29T20:29:42+00:00

Completed VUNETID replacement project. Minor issues with unknown applications were found after moving to production, but were all rectified in concert with Frank in iDev.

Samba integration into AD/LDAP is nearly complete for the Med Centre server PHM1. Host has been joined to the domain and appears to be properly authenticating with AD/LDAP security. Local admin still needs to create the dual data shares which are to be shared within the AD directory tree.

Provided support in getting the application for monitoring connectivity/latency between the university and gmail running on noc-apps.its.

Handled most all web environment oriented Magic tickets while Peter was on vacation and in training. Obtained an honorary promotion to the position of “Peter Junior”, with a later threat of being further promoted to a “Full Peter.”

Attempted to perform upgrade on flash.its from Flash Media Server (FMS) version 2 to version 3. Incorrect license key was provided for use in the upgrade. New key has been obtained and the upgrade is being rescheduled with the Streaming Media team.

Worked with Cindy Frank and other members of the group working on the ITS website redesign project. Was able to implement all requirements put forth including merger of the production site with the test site, configuring LDAP authentication, separation and security on the SSL implementation, inline email integration, etc.

Resolved a problem with interface bonding which appeared to be occurring on its-hclnbu03. Final testing was performed by Derek and Mark who verified proper operation. Not known why the problem was not seen until recently.

Helix project is still on hold awaiting upgrade of the WRVU server.

OSSEC project is still on hold awaiting further testing or instruction.

CSM monitoring project is still on hold awaiting time to work with Peter on snmptrap handling.

July 2008 MAR

2008-07-29T16:55:28+00:00

Put the new Apphosting Runbook in production
- New Features include: Export to Excel, Sort by Any Column
Migrated MOSS 2007 to SQL instance on non-default port
- Install SQL 2005 Management Studio and SQL Connectivity tools on all MOSS 2007 servers
- Install the latest service pack on all servers
- Edit the registry on each MOSS 2007 server with the following (Reference http://support.microsoft.com/default.aspx?scid=kb;%5bLN%5d;318432)
  
  [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\ConnectTo]
  
  “DSQUERY”=”DBNETLIB”
  
  “MyAlias”=”DBMSSOCN,VSERVER1\MyInstance,2433″
* Assume that your virtual instance of SQL Server is named VSERVER1\MyInstance. The virtual instance of SQL Server is listening on TCP/IP port number 2433. The name of the SQL Server alias that you want to create is MyAlias. If you open SQL Server Configuration Manager, you should see an alias with those settings. If you simply create the alias using SQL Server Configuration Manager, then it doesn’t add the REG_SZ entry for DSQUERY. BTW both entries should be REG_SZ.
- Run SharePoint Products and Technologies Configuration Wizard and use the alias you just created for the SQL Server name

July 2008 MAR

2008-07-29T14:25:11+00:00

July 2008 Monthly Activitiy Report

1) Worked on an assortment of odd, challenging, and familiar helpdesk tickets.

2) Upgrade of vanderbilt_production VUMailguard cluster was completed on July 9-10. It took approximately 22 hours due to various issues. After the upgrade an assortment of issues impacting performance continued over the next two weeks. Several bugs and configuration issues were discovered over the first week. Several patches were deployed to fix the bugs and configuration issues were resolved. Performance continued in a degraded state. A key issue impacting performance was discovered the second week after the upgrade. The vendor had changed how international characters are handled in rules in the new version. After disabling three rules that scanned message bodies for specific character strings, performance improved considerably and continues to operate at a much improved level. Additional patches are scheduled to be deployed on July 30 to address additional bugs and to assure the cluster is at a patch level equivalent to the next version. Note two calls remain open with the vendor regarding an issue where NTP is not working on VMs and slow queries for some reports due to database issues.

3) Certificates were renewed on all SMTP and IMAP servers totally 12 servers. Server roles were reviewed in order to access the need for a wildcard certificate. Thousands of dollars were saved by determining that six servers no longer required wildcard certificates.

4) Newlightimaging.com - Provided list of tasks and key personnel required to route email for this domain through the VU messaging system. Currently no time table has been set to complete this project due to contractual obligations with newlightimaging.com and their service provider.

5) Project for mailing list replacement. Frank Kyle and I have been meeting regularly in order to determine the best strategy for implementing provisioning. I’ve provided documentation on the current loader code and guidance on how to adapt the current loaders for LISTSERV. Guy and I are attending courses on LISTSERV management. Fully licensed software has been ordered and is ready to be installed.

6) Project to implement smtp and e-mail authentication. Nothing new to report.

7) Worked in conjuction with ITS Security on several incidences e.g. scams, threats, compromised hosts, etc.

10) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

MAR - for July 2008

2008-07-28T21:44:54+00:00

1.) Coordinated 166 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 48 Magic Tickets. This majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry, mailing list owner changes and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Completed the upgrade of IPF on the IMAP, Group Mail and SMTP servers. I will now document this procedure and this will be continued on the remainder of the Unix servers.

5.) Completed the project to move ILO ports for HP and IBM servers to the admin network. Administrative usernames and passwords have now been changed. There is just one system left to finish now. It is waiting input from a vendor.

6.) The IMAP Cleanup Project is in maintenance mode at the moment and may be extended.

How to Search for Connections using Netstat

2008-07-28T16:41:21+00:00

netstat -an |find /i “<<SearchTerm>>”

Examples:

Find all connections to/from TCP port 25
1. netstat -an |find /i “25″
Find all ESTABLISHED connections
1. Netstat –an |find /i “ESTABLISHED”
Find all ports the computer is listening on
1. Netstat –an |find /i “LISTENING”
Find all failed outgoing connections
1. Netstat –an |find /i “SYN_SENT”
Find all connections to/from IP 192.168.1.1
1. netstat -an |find /i “192.168.1.1″

Search using NETSTAT and return the PID netstat -ano |find /i “<<SearchTerm>>”

2008 July MAR

2008-07-26T22:07:41+00:00

Monthly Activity Report
July 2008

AIMWorX – Supported all AIMWorX users varying requests. Copied all SMDR and CDR backup files to CDR for archive. Processed the weekly update of cost centers and student information. Ran a monthly process to remove any duplicate calls from the database prior to a billing cycle. Upgraded McAfee to v8.5 and installed approved OS patches on AIMWorXFE01 and BE02.
MA4000 project – Talked with NEC about reasons MA4000 worked on week by then did not the next. Eventually, the project team decided to move the SQL database from the mom-sql server, which was having problems, to the new SQL test cluster. The MA4000 server was moved to the test network to remove the ITS firewall from the equation. So, the MA4000 & NECCAS software were unintalled and re-installed using the test SQL cluster.
Virtual Desktop project — Spent many hours trying to install the Sun Secure Global Desktop. It took two phone calls to a Sun technical person but have finally gotten a grasp on how it works. The problem being that to offer up a desktop requires a desktop, a 1 to 1 relationship. It basically gives access to your desktop on your desk via the internet. Also, SGD can offer up software running on a server to other desktops; keeping all data on the server. There was some good news. The VMware VDI was setup and has 5 virtual desktops ready for testing, 3 Vista and 2 XP clients. Servers in use: Virtual Desktop Manager, Windows Terminal Server, Vista golden image, XP golden image, Secure Global Desktop.
Virtual Servers — Created 3 virtual servers in support of the following projects: Virtual Desktop & Dreamspark.
Operator Services — Script written to update pager data in AIMWorX for the PBX Operators was run on July 1, 2008.
New AIMWorX Server VLAN — Moved AIMWorX production server to the new VLAN. Several firewall issues had to be worked through: secure remote connection to web2; access to backup environment; NTP using the DCs; and ftp connection to the MIS ftp server.
Business & Planning: A script was written to download monthend data files from the MIS server. This will assit B&P with research of past billing cycles.
SSL Certificates: New 2 year SSL certificates were installed for Mediasite and new ITS change management web site.
Change Management: Started the process of migrating the change management web site from the Medical Center to an ITS server. This web site was put on the Apphost1a web server. There are still a couple issues to work out with connecting to the web page.
Lifecycle Replacement: Worked with Rick Carlton on lifecycle replacement of two servers, IVR1 & 2. The new servers will use QueWorX for ACD menuing. Two IBM servers were ordered.

Activity Report July 2008

2008-07-25T17:59:58+00:00

This month saw many changes to our storage environment, here is a list:

Six 146G Fiber Channel DAE’s installed in the Hill Center CX3-80.
One 1TB SATA DAE installed in the Hill Center CX3-80.
One 1TB SATA DAE installed in the Stevenson Center CX3-40.
Cava upgraded on the two virus check servers for the Celerra.
Flare 26 patches applied to our Clariion service processors to address a bug.
New Quantum I2000 library installed into our backup environment.
BlueArc request pages added to the storage site to facilitate administrative requests from other departments.

In addition to these changes I have worked on the following:

Update of backup cloning scripts to support the I2k Library.
Setting up VTL evaluations.
Update storage diagram to include purchased equipment and removal of old equipment.

JULY 2008 MAR

2008-07-10T15:21:23+00:00

Updated diamondip test db with new import. Changed dns zones. Updated and completed documentation on how to create a test database. Created synonyms on Netid tables and worked with DiamondIP on scripts.
Created/refreshed standby test database for diamondip.
Wrote auto-maintain query for parking and created mailing list.
Created ITS test database on Oracle's test server
Created tables and helped Kenon with the Backup Reports Database and 11G client.
Wrote documentation for 11G client install.
Setup oracle_check plugin on Nagios to monitor oracle tablespace size, functionality for Oracle databases. Peter set up the environmental variables on the OS. 5 databases are now being monitored with this plugin. Also, updated documentation on how to configure this plugin to monitor a db.

Άρης για το μήνα Ιούνιο (MARS for the month of June)

2008-06-27T17:47:44+00:00

I feel like a broken record….

1. BT INS aka DiamondIP aka DIP - Test environment still not up. Having multiple issues getting the database to work on the test environment. Half tempted to blow the thing away and totally rebuild this test environment.

2. Auth4 aka redundant kerb server - Built out a new kerb server to go to the hospital.

3. Gave the Med Center an overview of DIP and the functionality. They were made aware of the ups and downs of the product as well as given an insight on how it works.

4. Finally got CA to get me uncorrupted patches for Spectrum One-Click. Installed the patches and everyone is now happy that Report Manager works as it should.

5. Lended a hand on a little forensic work for IRT. Ended up taking much more time that I thought it would but was very much an eye opening experience.

6. Fixed a ton of issues on RHN. Purged a lot of non-responding servers, registered a bunch more, and finally got all the channels resync’d.

7. Did a metric boatload (as opposed to Imperial or Standard boatloads) of DNS requests/changes/fixes/etc

8. Managed to break DNS attempting to get BIND Views working…. GO ME!

June 2008 MAR

2008-06-27T13:58:10+00:00

Installing MOSS 2007 on SQL 2005 Instance on a non-default port:
In order to use an alias for SQL you must have the following installed on your MOSS 2007 servers:
1. Connectivity Tools
2. SQL Management Studio

On each MOSS 2007 server:
1. All Programs -> Microsoft SQL Server 2005 -> Configuration Tools -> SQL Server Configuration Manager -> SQL Native Client Configuration -> Aliases
2. Create a new alias. I used the IP address of the SQL instance because it’s on a cluster and seemed to work a little better.
3. I also used a FQDN for my alias name and created a DNS CNAME record for that name. (i.e. ALIASNAME.COMPANY.COM)

June MAR

2008-06-27T12:54:14+00:00

Identity Management: The new LDAP servers used in the IDM project currently suffer from some stability problems due to issues with the Directory Proxy Server (DPS). The DPS is being used to ensure that the correct SSL certificate is presented to the client. While the vendor is working to correct the problem, the IDEV and AppHosting teams are working on an alternate solution. I built two new development LDAP servers to be used by the IDEV team. These new servers more closely resemble the architecture and configuration of the production servers. We are currently testing the use of stunnel as a DPS replacement. While stunnel does not offer the same feature set as the DPS, it does provide the base SSL functionality and is considerably more light weight. We are also evaluating F5 LTM appliances in the test environment to see how well they integrate. We currently have a pair of evaulation appliances that we will use to front-end the new test LDAP servers that I built.

IMAP-to-Exchange Migration Tool: I have scripted the necessary tasks to prepare IMAP accounts for import into Exchange. The process essentially makes a duplicate of the user accounts so that the original is unmodified. I still need to create documentation to detail the process for the rest of the team.

Search Engine Replacement Project: I have attended several meetings and a couple of product demonstration. I am passing this project to Troy Osborn.

ITS Website Redesign: My involvement in this project has been relatively minimal this month while the vendor continues development efforts on the site. My activities this month included adjusting ModSecurity rules, re-importing a database for the vendor after a failed upgrade,and identifying application errors in the logs.

Operational Issues: The majority of the operational issues for this month have been relatively routine, and most of the web requests were related to creating redirects and such. I did create a new virtual host for a website that was being publicized the next day in a national magazine.

MAR - for June 2008

2008-06-26T21:44:52+00:00

1.) Coordinated 149 Magic Tickets for Applications Hosting. This involves determining a priority for handling our trouble tickets as well as determining who would be best suited to handle each ticket based on individual expertise and the current time available of our staff on hand.

2.) I resolved 32 Magic Tickets. This majority of the tickets involved rebuilding and restoring personal and group mailboxes, group mailbox quota changes, VUNetID changes due to incorrect entry, mailing list owner changes and moving email from a user’s deleted account into their new account.

3.) Maintained the daily tasks of the backup system. This involves monitoring the ITS Backup Server to insure there are adequate tapes available for the backup process each day.

4.) Continued work on the PAVE program. I reset 210 PAVE accounts to reauthorization status prior to the arrival of the PAVE students. Due to changes brought about by the IDM system, this task has been more or less turned over to the IDEV team until new tools are provided for the IDM system. I am still involved in a consultant status. This program is an annual task to create and maintain accounts for prospective engineering students each year.

5.) I have scheduled the upgrade of IPF on the IMAP and SMTP servers. This will be done over three weekends in July.

6.) Continued project to move ILO ports for HP and IBM servers to the admin network. Administrative usernames and passwords have now been changed. There are just a few odds and ends to finish now.

7.) The IMAP Cleanup Project is in maintenance mode at the moment and may be extended.

June MARS

2008-06-26T20:48:03+00:00

28 Closed Magic Tickets
1 Security Incidents
36 Imap -> Exchange Mail Migrations
3 Exchange -> Exchange Mail Migrations

Exchange

Email Account Bouncing Cleanup after IDM cutover
Worked with IDEV to identify and cleanup issues that occurred with new exchange mail enabling after the switchover to IDM. There were a couple of different issues identified. In a few cases the exchange delivery address and only the exchange delivery address was not populated to the AD account. This caused mail to bounce back and forth between exchange and the proofpoints, this issue was assumed resolved by putting the ad and exchange loaders back on the same machine. I am not sure this was proven as the fix or not. There were also a couple of accounts who seemed to have been provisioned properly on all systems but Exchange would not deliver the mail to the mailbox it seemed to think that the mail should go external to the Exchange environment. The AD object for the user had all the correct email addresses on it and you could look on the Exchange General tab to see where the mailbox existed. SMTP had the correct exchange delivery address for the user also. The resolution for this issue was a deletion of the Exchange account and a clearing of all mail attributes then recreating the mailbox.

Setup new Enterprise Postmaster account for the Exchange Environment
The old postmaster account was Greg Marlin’s 3 letter account which had been deleted. When uninstalling Exchange one of the checks is to see if the Enterprise Postmaster account is housed on any of the Mailstores on that server. The issue that I suspected we were running into was that the environment didn’t know where the mailbox lived since the underlying account was deleted. After making this change the removal of MAILFE05 was a success.

MAILBE16 System Attendant account cleanup
During the change last month to reconfigure the mailstores the System Attendant account was left on a Mailstore that we were not wanting to keep. I had to change the location of the account in adsiedit and restart the Information Store so that we could delete this superfluous exchange database.

Stealthbits evaluation
Evaluating Stealthbits Monitoring for exchange to initially provide monthly metrics numbers for the University Exchange environment. This can also be used much more extensively with some time and work to assist in usage projections as well as troubleshooting and configuration management. I am currently working with Stealthbits to create a custom report to mimic the one we are currently getting out of Messagestats.

Exchange Monthly Microsoft Patches
Installed, verified and tested Monthly Microsoft Patches

Configure and Document procedure for removing specific attachments from Exchange
Utilizing Trend Scanmail setup a spam rule to replace a specific file name with a text file saying this file is against policy.

Assist in writing Documentation detailing the levels of Shadow Mailboxes that we can provide
This is to allow us to have a proper request for security incidents.

General Windows

Monthly Microsoft Server Patches
Monitored, verified and tested services after the monthly Microsoft patches for our Windows Server environment

Sharepoint forensics for outage on 6/23/2008
Investigate Sharepoint outage and resolve the issue. Move the servers to the new ESX machines also in the proces

June 2008 MAR

2008-06-26T20:28:34+00:00

June 2008 Monthly Activitiy Report

1) Worked on an assortment of odd, challenging, and familiar helpdesk tickets.

2) Lifecycle replacement of mailgate01 was completed. Upgrade scheduled for June 18th was rescheduled due upgrade preparation issue. Worked with vendor in order to resolve issue. Upgrade has been rescheduled for July 9th.

3) User reported issue with VUWebmail related to QuickSave 2.3 plugin. Determine that bug existed that is resolved by installing QuickSave 2.4 or better. Relayed information to Peter who assigned task of upgrading QuickSave plugin to Troy.

4) Newlightimaging.com - Have worked with VUMC NCS in regards to messaging routing for the newlightimaging.com domain. Attended several meetings and exchanged several emails in order to perform fact gathering. Once it was determined that concerned parties at VUMC wished to route email for the domain permanently, contacted required parties in ITS in order to assure tasks required for message routing are completed.

5) Project for mailing list replacement. Meetings continue to determine what functionality will be implemented, migration strategy, loader development, test methodology, etc.

6) Project to implement smtp and e-mail authentication. Nothing new to report.

7) Worked in conjuction with ITS Security on several incidences e.g. scams, threats, compromised hosts, etc.

10) Created monthly Email metrics report for dashboard. See \\vuspacegroups\ITS\common\dashboard\New Dashboard\Application Hosting.

Jun 2008 MAR

2008-06-26T17:18:59+00:00

Attended the Red Hat class in Atlanta. Received an RHCT certification. I’ll get the RHCE next time..

Worked with Frank Kyle on the replacement VUNETID project. Bluestem has now been heavily modified from the base installation to run from a single directory tree. This allows the application to run with SELinux enabled and enforcing, as well as making it much easier to port to a new machine later on. Documentation is now in the directory tree along with the installation which explains the modified structure and locations of important files. We have been awaiting the correct ColdFusionMX7 license and clearance from ITS Security in order to move the replacement to a public IP address for pre-production testing.

Worked with Chris Marshall, ND&E, and the NOC to migrate a couple machines to the ITS Test Network.

Identified a problem with the .115 network while working on the bluestem replacement. The machine was a VM and the root problem was discovered to be inconsistent configuration on the switch ports which the VM host was attached. After reporting the issue I was directed to evaluate the entire ESX environment for this same type of problem. Only one other production problem was found and corrected, all others are non-production and will be resolved during normal maintenance for each affected host.

Performed upgrade of Helix Server for streaming media to version 12. Initial report from John in Streaming Media was that everything appeared to be working and the upgrade was a success. A compatibility issue was identified by John the next morning for encoding from the WRVU server. The upgrade had to be reverted to version 11 to enable encoding/streaming for WRVU until their server software can be upgraded to a compatible version.

Coordinated a migration from single 1Gb fiber to etherchannel (dual) 2Gb fiber for several production vlans in two of the Hill 148 High Density racks. Dan Raymer handled migrating VMs, Roland Serman handled MSSQL failover, and Barry McCurry performed reconfiguration of the Hill_Router and Hill_6513 for this change.

Awaiting coordination with the windows team to re-install the OSSEC client on a couple of their test machines to move forward with evaluation. The two SharePoint servers previously being used have been rebuilt and are no longer available for testing. OSSEC has been successfully reconfigured to utilize a mySQL backend for log aggregation, though local logs are still maintained for the WebUI.

The SWE for CSM monitoring is still on hold until an snmptrap handling solution has been installed and configured for the Nagios host.

MARS 06/08

2008-06-26T17:03:40+00:00

SharePoint Lessons Learned

This month I’d like to discuss a few things we’ve learned over the past year while trying to deploy MOSS 2007.

· Our SharePoint deployment is more of an ASP model; each business unit that utilizes our SharePoint deployment wants their own unique URL. One of the things that we’ve discovered is when you create a Web App, (and create a new application pool for the web app) MOSS doesn’t give the application pool the appropriate permissions on the SharePoint Servers, which leads to a series of DCOM errors in the event log (Event ID 10016), to resolve this issue we’ve done the following.

o Add the application pool account to the local WSS_WPG group on each SharePoint server.

o Also, you need to Add the WSS_WPG group to the ‘IIS WAMREG admin Service’ in ‘Component Services’ browse to ‘Computers/My Computer/DCOM Config’ right click, go to properties, and click edit under ‘Launch and Activation Permissions’ add the account, and give ‘Allow’ to ‘Local Launch’, ‘Remote Launch’, ‘Local Activation’, and ‘Remote Activation’.

o You can either follow this route, or explicitly add each individual application pool account to the IIS WAMREG admin Service.

o Also, in our installation, we’ve removed anonymous access from the ‘Access this computer from the network’ in the local security policy, if you do the same, then you will either need to allow the WSS_WPG group, or the individual application pool accounts to this as well.

· Setting up Host Header for SSL, this is another one that took a bit of research. Ultimately what we found was that there is a vb script in the Inetpub/AdminScripts directory called adsutil.vbs which allows you to define the host header for the site. You simply run the following command:

o adsutil.vbs set /w3svc/[identifier]/SecureBindings “IP:443:[host header name]”

o So for example you’d run:

§ Adsutil.vbs set /w3svc/123456789/SecureBindings “192.168.1.10:443:mysharepointsite.com”

o The [identifier] is the unique IIS Id of the virtual server.

· We’ve also run into some crawling issues. Our WFE’s sit behind a hardware based load balancer (Cisco CSM). Currently our load balancer will not allow traffic to pass from behind it and then back in again. So basically when SharePoint tries to do a crawl, it does a DNS query for all of the SharePoint sites it needs to crawl. It then updates the HOSTS file with that information. So then when it attempts to do a crawl it tries to crawl via the Load Balanced IP address instead of the local address, which in turn our Load Balancer won’t allow. In order to work around the issue, we had to edit the HOSTS file on all of our SharePoint servers and add all of our URL’s with the actual local IP address instead, and then set it to read only otherwise every time SharePoint attempted to do a crawl it would overwrite our manual changes. Of course now we get hundreds of errors a day in the event log because SharePoint can’t edit the HOSTS file, go figure.

On a positive note, we’ve managed to resolve our SP1 issues, though not by actually fixing the problem. After 5 failed attempts based on Microsoft supports recommendations, we decided to completely rebuild the farm, and restore all the content manually. At the same time we migrated to 64bit hardware. We circumvented the SP1 issues by slipstreaming SP1 into our SharePoint and Project installs.

2008 June MAR

2008-06-25T14:52:18+00:00

Monthly Activity Report
June 2008

AIMWorX – Supported all AIMWorX users varying requests. Copied all SMDR and CDR backup files to CDR for archive. Processed the weekly update of cost centers and student information. Ran a monthly process to remove any duplicate calls from the database prior to a billing cycle.
MA4000 project – MA4000 service stopped working sometime during the week of 6/16/08. Working with NEC to resolve problem.
Virtual snapshots — Setup automatic snapshot service for VULS and VUSM. VULS has 5 snapshots created on Sunday evenings and VUSM has one snapshot created Sunday mornings. Created a one time snapshot of colo server ‘Flanders’ for Owen prior to them upgrading software on the server. After the upgrade and Owen was satisfied the upgrade worked properly, the snapshot was removed.
Virtual Desktop project — Spent many hours researching & meeting with different vendors solutions for a virtual desktop. The project team met with representatives in person or over the phone of Dell, Microsoft, Sun, VMware and HP. Also met with OGSM and reviewed their VMware VDI installation. We are in the process of setting up a pilot of the Sun virtual desktop solution.
Virtual ESX server — Completed setup of ‘Mammoth’ ESX server (its-hcvm12.an). Setup ESX cluster between its-hcvm11 & its-hcvm12. Removed all virtual servers from its-hcvm01 and reloaded ESX software using its-hcvm03-ts host name. This server was placed in the development ESX environment.
Virtual Servers — Created 9 virtual servers in support of the following projects: UC2K7, Sharepoint, & Virtual Desktop.
AIMWorX & VPD — The AIMWorX alarm client stopped working for VPD on 6/2/08. Worked with Terry Cavendar to make firewall changes to correct this problem. Terry found encryption errors between the ITS and VPD firewalls that needed correcting.
DOAWeb — Attempted to convert the DOAWeb physical server to a virtual server. Multiple attempts failed from the 194 network and the App Hosting Admin network. Currently researching the error that caused the failure.
Operator Services — Every July the Medical Center has hundreds of new residents & clinical fellows that have to updated/added to AIMWorX. This year they requested an auto load into AIMWorX. I created a Perl script to read the file and populate AIMWorX accordingly. This script will be run in the production database the morning of July 1.
AWAPI update — MIS moved the files used to update AIMWorX to their new secure ftp server. The ITS firewall was changed to all the AIMWorX server access to this server without the use of Secure Remote. Also, the automatic Perl script was changed to login to the new server and download the new files.
New AIMWorX Server VLAN — Move AIMWorX3 server to the new VLAN for testing prior to the production server move. Had problems with WINS not being updated with the new IP address but was resolved once the firewall rules were changed to allow the WINS ports outbound traffic. However, outside the VU IP range access does not work.
Windows Server 2003 sp2 upgrade to Magic-DB2 server.

Activity Report June 2008

2008-06-25T14:18:46+00:00

This month we made a decision on a new tape library and have purchased it. The new library is a Quantum I2k and should be here within a week. In preparation for the arrival of the new hardware we have arranged installs of new HBA’s in our existing storage nodes and developed a strategy as to how the new device will be attached to our environment. The addition of the new library to our backup environment will complete the life cycle replacement of our Sun STK L700 as well as give us a lot of additional capacity in our backup environment. The main concern will be cloning speed and this should be greatly enhanced as well. The addition of a VTL or similar technology will most likely be necessary to fully keep up with our cloning operations.

I have authored another Perl script which will keep track of all of our backup clients and act as the definitive list for clients configured in our backup environment. This script launches every night at midnight and creates a master list of clients used for reporting purposes. A second Perl script uses this master file to create three seperate lists used for cloning. Cloning is now accomplished one third of our environment at a time over the course of a month. I will most likely attempt to do the entire environment in a single week once the new library is in place.

Other than that it’s been a lot of operational work over this month.

June 2008 MAR

2008-06-18T19:41:54+00:00

Wrote script and crontab for spectrum mysql db optimize and maintenance programs.
Added and autoextended tablespace for RHN. Also ran utilities to check fragmentation on tables, indexes, etc.
Fixed a listener security vulnerability.
Refreshed the diamondip test data twice.
Initated and installed oracle 11G client on Nagios for the check_oracle plugin. This plugin checks the tablespace size and db status.
Installed Oracle 10GR2, upgrades and patches on the test dipdb 2 standby server for the test DIP dataguard configuration.

2008-05-29T16:50:50+00:00

May MARS

Well, after taking a much needed 3 weeks off in April and the first part of May, it was time to get back to the grindstone. Now my nose is painfully sore… I blame Kevin!

So, without further delay, here are the "Wins" for May:

1. Moved the VCMS database off of the VCMS server to the Linux Oracle server cluster - Instead of timing out when attempting to do any performance reports greater than 3 days, we can now query and receive our reports for the past year in less than 10 seconds. This is extremely helpful in doing trend analysis of our virtual environment.

2. Cloned and moved the NDE server to Stevenson - The NDE webserver was cloned and moved to the SC datacenter cluster to provide redundancy of services. Now working on getting rsync over ssh working to insure data is correctly replicated.

3. Spectrum patching - I attempted to patch our Spectrum environment to get the Business Objects Report Manager working correctly but one of the patches was corrupted. This prevented me from finishing the patch process. The vendor has yet to supply the patch again for download.

4. Moved the Diamond IP test application server and rebuild - The test application server was moved and rebuilt. The production data was loaded on the test database and I am in the process of removing the pointers to our production DNS and pointing them towards the test application/DNS service.

5. Develop automated Virtual Environment Billing Script - Started work on writing a script that will automatically gather the billable data (procs and ram amounts) from the VMX files for co-located VM’s. Currently fighting what I call "Special Character Hell" to get around the multitudes of parenthesis, dashes, spaces, and slashes in VM names/directories.

6. Continued work/enhancement of Diamond IP Production environment - Work continues on the production DIP environment to insure stability and to get external LDAP authentication working. Additionally, RFC1918 DNS preparation continues.

Last and not least… this is the new main priority in my life right now…

Please give a warm welcome to Zealy Caitlin!