1. OpenDNS pilot concluded. All DHCP within Hill Center was used to assess the impact of using OpenDNS. While some issues were noted, for the most part impact was minimal. Results sent to leadership.
2. The Diamond IP environment received an upgrade to 3.1.31 with minimal headaches. A first for that environment. DNS/DHCP remained 100% operational during the upgrade window.
3. Diamond IP Callout Manager now generating alerts. SNMP traps are still in progress but email alerts are going to be used in the mean time. Base email alert functionality is working in testing.
4. Library has been tapped for the next (and probably last) DNS/IPAM Self-Serve. Training expected in February.
5. Bonenjoint.com successfully transferred to Vanderbilt DNS
6. The NDE Syslog server migrated out of 129.59.1.0
7. The ancient Shibboleth server was decommissioned.
8. Vandyworks.com has successfully performed multiple DR tests which required time sensitive DNS changes.
Below is the DNS statistics for the ITS hosted Vanderbilt DNS servers.
CY09 DNS Stats:
Total DNS Queries Answered : 13,841,143,051
IP-SRV1 : 10,370,389,340
IP-SRV2 : 2,276,267,703
IP-SRV3 : 1,194,486,008
Total Average Daily Queries : 37,962,338
IP-SRV1 : 28,446,349
IP-SRV2 : 6,239,242
IP-SRV3 : 3,276,747
Highest Month of Activity : September – 1,877,470,808 total queries
Lowest Month of Activity : May – 774,095,515 total queries
And for 2009 – Facebook is the top DNS destination domain for Vanderbilt clients!
1. OpenDNS pilot is in place for select wireless networks. Once the initial pilot is completed, all of ITS will be involved in testing.
2. Diamond IP has been upgraded from 3.0.71 to 3.1.31 in the test environment.
3. The test database environment has been been patched and upgraded to Oracle 10.2.0.4
4. RHN was upgraded from 5.0.1 to 5.3. This allowed us to handle the increased clients coming from MIS/EAI and to migrate the database off Oracle 9.2.0.8 to 10.2.0.4 – freeing up some resources in the virtual environment.
5. VUMC DNS administrators received Self-Serve DNS training and are now handling their own DNS requests.
6. The NetID environment has been shutdown and is on-track for decommissioning the first week of December.
7. ISIS has been scheduled for Self-Serve DNS/IPAM training. This is the last major group identified for training at this time. CSB & Library remain candidates and will be reviewed at a later time.
8. Diamond IP still has some critical issues open with the vendor
- The appliances did not handle DST changes resulting in the failure of the NTP service.
- The DHCP servers w/ collection did not handle DST changes resulting in a massive database logging issue.
- Agent Appliances still lose connection with the Executives resulting in lost collections and ability to publish DNS/DHCP configurations.
- Callout Manager & generated actions still remain to be configured to send SNMP traps on 70/80/90% DHCP subnet utilization
9. Business hours DHCP Failover/Disaster Recovery Exercise was delayed and is now tentatively scheduled for the 2nd week of December.
In pursuant to HR Policy HR-025, all non-work related material has been removed from this blog.
"Wins" for the month of September:
1. First up, the University and Medical Center now share a unified view of RFC1918 addresses, the reverse space for the Medical Center networks, and the foward zone for mc.vanderbilt.edu internally to the Vanderbilt community. This resolves a multitude of issues where data between the two organizations were out of sync causing conflicting name resolutions. Additionally, this supports the new secure relay for servers email implementation by providing proper reverse resolution for both VU and VUMC.
2. Additional departments were trained for self-serve IPAM and DNS. The Owen Graduate School of Management & the Vanderbilt University Law School now are empowered to administer their own IP and DNS space.
3. The Diamond IP environment was upgraded to to version 3.0.71 resolving some serious memory leaks present in the earlier version.
4. DHCP migrations from NetID to Diamond IP continue with the current migrations at 90% completed. This puts us well on track to have the NetID environment retired in November. That covers the major events of the month.
1. BIND Views are finally working in DIP on the primary name server. There is still an issue with getting the views propagated to slave servers. Following the ISC instructions does not work and the vendor has been engaged.
2. The issue with pushing updates in the DIP environment has been resolved. Apparently ActiveMQ was refusing to play nicely. This paves the way for upgrades to the Sapphire Appliances and to the application.
3. Disk consolidation continues in the Virtualization environment. All "troubled" LUNs have been replaced. Additionally, prep work to retire the AMD ESX servers continue. When all is said and done, the ESX environment will drop from 20 hosts down to 12 hosts.
4. Work continues on resolving backup issues with a number of hosts moving off the .1 network to the Admin Network. There are still a number of hosts that needs this addressed.
1. DNS/DHCP
- The Diamond IP environment received an upgrade to 3.0.62 in a hope to solve some issues with zone publishing. While the software itself is stable, the problem was not solved. Oh well…
- The remaining 4 personnel in Application Hosting received their training on Self-Serve DNS/IPAM with the Diamond IP InControl software. I think everyone on both the AppHosting as well as the ND&E team can agree that this is a definite Win for both teams.
- RFC1918 subnets continue to be imported into the Diamond IP environment. Difficulties in DHCP failover w/ the supplied DHCP 3.0.6 version from BT INS keep us from really diving into migration of DHCP enabled subnets.
- I have exhausted my ideas on trying to get replicated DNS BIND Views implemented without using 72 hours of imports or significant name resolution service downtime. I escalated to BT INS but have yet to get an solid answer back from them.
- DNS survived the Great Power Outage of 08. Sure, service was degraded a bit with the Master down, but service never dropped completely off. YAY!
Next up… getting those Views implemented, NCS Self-Serve DNS Training, and Sapphire 3.0.72 upgrades.
2. The Virtual Environment
- Virtual Center upgraded to VC2.5-Update 3 – Kendra knocked it out of the park. Absolutely HAMMERED it. What an awesome job by her. I’m still wondering how some of the upgrade bugs escaped the VMware QA lab. We ran into the issue of vxpa corrupting on 3.0.2 hosts w/ VCMS 2.5u3. Took me a good portion of the night to figure out what was going on and how to fix it. While it took some time and effort, the capabilities now offerred with VC2.5u3 with ESX 3.5u3 have made our life soooo much better. And speaking of ESX 3.5u3….
- ESX Upgrades from 3.0.2 to 3.5 Update 3 – While it is not 100% complete (the prod AMD clusters remain to be upgraded), I am going to call this a WIN as over 1/2 of the total environment is upgraded and working extremely well. Storage VMotion has enabled us to FINALLY perform some much needed SAN consolidation. I also happen to love the new Health Condition report within the VI Client. I would love to say this upgrade was totally without downtime, but it was not meant to be so… of course, the downtime was pretty much our own fault. Putting servers on local storage, lack of VMware Tools, etc. Great job to Kendra and Scott E. for stepping up to help do the upgrades. BIG thanks!
- The Leviathan was pushed into production in a rushed manner to make up for the broken snapshots w/ the VCMS upgrade. Thanks to Kenon for the quick weekend work to get the storage presented and help get the service up and running. On a more positive note, it did push me rather forcefully into figuring out all the tricks with ESX 3.5u3 and well as getting the plugins working for VCMS. Nothing like a little pressure to make learning so much more satisfying.
3. RHN Upgrade
- RHN 5.1.1 has been pushed out the door and into production. 97 of the former ITS clients are re-registered and I hope to get the other departments to finally buy into what RHN Satellite can offer them in terms of ease of deployment via kickstarts, activation keys for RHEL5, easier/quicker patching, and a view into the health of their RHEL environments.
- RHN 5.2 was FINALLY released as well. It came a bit too soon to the 5.1.1 production date to put too much effort into it, but that is up on the slate soon. With Oracle 10g support (FINALLY), we can move this database to our existing, more robust clusters and gain some performance.
- The re-registration scripts worked for the most part and made it quite easy to register. Scripts are available for anyone in the Vanderbilt Community to take advantage of this service
Quite simple actually…
First, you need to get the HBA’s to issue a LIP and then a re-scan
- echo 1 > /sys/class/fc_host/<host #>/issue_lip
- echo "- – -" > /sys/class/scsi_host/<host #>/scan
Do this for every host path.
Now you just need to tell PowerPath to go do its normal discovery
If you do a display, you should see the new LUN.
That’s all there is to it…. go forth and fdisk!