System Patching: I patched all of the systems where I am listed as the primary admin. The most notable servers patched are: jprod1, jprod2, seeker-fe, seeker-be, nettracker-fe, nettracker-be, mysql-srv1, web-srv2, and helios. The patching on web-srv1 was not successful. Even though the same patches that were applied to web-srv1-preproduction, the system did not survive a reboot. The system had to be rebuilt.
Upon debugging the server rebuild, I discovered some items regarding the web migration. For starters, the existing environment is overly complex with multiple copies of the same libraries scattered through out the file system. Also, our IPF ruleset had to be slightly adjusted to accommodate the syntax for the latest version. Most surprisely, there are people that are not aware of the web migration project.
Web Migrations: swdist.vanderbilt.edu was migrated over to the ITS server. This move accomplished two things: 1) the site is now hosted on the new servers, and 2) the site was relaunched with the new SSL certificate. The site is now using LDAP for authentication. A SSL certificate has been added to the ITS website so that web content from various other servers can be consolidated into one website. The AppHosting (include the runbook) and NOC websites have been migrated to the new server.
There were no new virtual host migrations this month even though I have over 20 waiting for review and cutover approval. I have staged several departmental sites under the WWW URL, but I am still waiting the official word to cut them over. Student organizations are migrating rather smoothly. Where possible, I been making corrections to the web content so that it matches the current rendering on web-srv1. I've also been making some minor PHP and Perl modificatios if I notice them.
The "UTF-8 vs other character sets" appears to be resolved. I can convert web content to UTF-8 if the original character set is known. I've converted a couple of sites, and the owners have verified the conversion. This conversion even works for MySQL data.
Bastion Hosts: The App Hosting team is now using the new bastion hosts, and a few other ITS staff members are also testing connectivity to their servers. The Sitemason and PBX tech staff also have accounts on the bastions hosts for their remote access.
Jprod1 and Jprod2 are now patched to current levels, including the famed DST patch. I got a bit of a late start because cell phone connectivity appeared to be out in my area, and I could not call in the start of the change. Otherwise everything went pretty smoothly.
One of the side effects of migrating some sites to the new servers has been that some characters are not rendered correctly. This is basically due to the fact that the old environment only supported latin characters. The Redhat installations of Apache and MySQL use UTF-8 by default. When some content is moved over without conversion, the end result is usually a question mark or nothing at all. The most common characters affected by this are the copyright symbol, the apostrophe, and the mdash. Luckily converting data to UTF-8 is relatively easy using iconv. The only interesting part is determing what the original character set was. Some of the early development tools did not specify their output so the initial character set could be ASCII, Latin-1, MS-ANSI, or something entirely different. The data usually renders correctly after a couple of guesses.
The Solaris patches on the NetTracker systems went smoothly. I did have to delay the reboot until the logfile imports were complete. The Oracle patches will be done later this week.
I found this article over at the CIO website. It's a decent backgrounder on fighting spam, and it's even got a quote from David Linn over at VUSE.
Hmmm, the halt user is in the deny list…
Feb 14 00:03:03 server sshd[7305]:User halt not allowed because listed in DenyUsers
And it's not in the allowed list…
Feb 14 00:03:04 server sshd[7305]: PAM-listfile: Refused user halt for service sshd
And they attempted a password…
Feb 14 00:03:06 server sshd[7305]: Failed none for invalid user halt from ::ffff:218.38.214.115 port 51128 ssh2
Nothing to see here. Move along.
There were a few side effects with the swdist migration. For starters, the alerts for the swdist SSL certificate and web-srv2 disk utilization cleared in Nagios. The second less visible side effect was the remediation of the swdist disk alerts in the Nessus reports. No they didn't just move to the new server. They went away.
I've patched another set of servers. The new Linux web servers were the target this time, and they are now fully up to date. Each server is now running on the latest kernel with shiny new rebuilt VMware-tools and the daylight savings time patch.
The software distribution site was migrated this evening to the new servers. Everything went relatively smoothly. The site now uses LDAP for authentication (versus Kerberos). That's one site migrated off of web-srv2. Both sites are running in parallel until DNS cache expires. On Monday, I'll archive the site old site to CD, and remove it from the system.
We need one of these near the Hill Center. To tell you the truth, it's not the first revenue model that I would have thought of for a restaurant. They've had the domain since April 2006 so they must be doing something right. It would be interesting to have on near us, but it'd probably go under in a week.
