Toreishi.net ~Now with more SNAP~

A challenge when performing service refactoring is that you now need to start worrying a lot more about connection security. When everything's on one box, and you're connecting via ports on localhost, you don't generally need to worry much. However, once you start splitting out services, relying on trust and unencrypted connections becomes less viable.

Here are some notes regarding PostgreSQL, as an example:

• Your users should be already set up with encrypted passwords (you did, right?).
• Next, you need to prepare SSL certificates for PostgreSQL, assuming that your installation has SSL support (CentOS compiles it in by default), install them in the correct location, and enable them in postgresql.conf.
  • If your internal CA is based on Windows Server, it's a little trickier to generate the certificate and its private key. I prefer to generate the certificate request on the CA with an .inf file, import the certificate on the CA, export the public and private key from the CA, and then convert the .pfx file into a .crt and .key file on the target host.
• Once you have your SSL certificates up and running, the next step is to define PostgreSQL connection restrictions (using hostssl, of course).
• From there, it's a matter of adjusting your database connection strings appropriately.
  • PHP uses the sslmode parameter, although it's odd that the value list is in a comment and not in the documentation.
• JDBC uses the ssl parameter, although since Java keeps its own CA database, you may need to import your CA certificate into the Java truststore.

A few interesting things I've stumbled across while finalizing some of my migration preparations from my previous server to the new one

• If you have multiple network adapters, your default route might not be the one you want. At least in CentOS, you can configure that in /etc/sysconfig/network with GATEWAY and GATEWAYDEV. This could be the problem if you're finding that you can't connect to your system (ping, ssh, etc.) even when it's capable of reaching the outside world... and even more, that if you disable your internal network adapter, everything "magically works." netstat -r/route -e might reveal that your default gateway isn't the one you think it is.
• Brute force ssh attacks aren't fun, and they've been pretty common for a while. True, they're probably not going to hack into your box (You have disabled root ssh access, right? And you don't have any of the standard accounts they try to hack in with?), but they can be rather annoying. Fortunately, it's actually pretty easy to use iptables to limit the number of ssh connections per minute.

Yesterday, while installing some more software, the fans on my ESXi system spun up to high, then all my sessions halted. That's never a good thing.

I took a look at what was on the physical screen of my system, and I got to learn about the joys of the PSOD. Initially, I thought the issue was bad memory, so I ran MemTest86 for a little over 20 hours. Nope. Not the problem.

A bit more searching revealed this little gem of a VMware knowledge base article.

Really, VMware? You're going to have a patch available for 5.0 and 5.1, but not 5.5? What the hell?!

For the past several days, I've been working on getting a VPN working for my little cluster. There are several different options available for Windows Server. It didn't take too long to get PPTP working, which I would consider the base level. I'd prefer not to try to implement L2TP, as that would require installing client certificates on all systems, which is annoying.

As a result, I've been spending the past several days trying to get SSTP working instead, which looks pretty straightforward on paper. Unfortunately, there are all too many problems you can stumble across, and the one I've been fighting for a while has been error 0x80092013 (which lots of other people also seem to fight with). What's depressing is that Windows offers almost no logging, so I've been slowly working through all the issues, but I've now hit a dead end.

• The CDP and DeltaCRL locations now refer only to externally-accessible locations (verified via Enterprise PKI).
• The certificate's CRL distribution point is externally-accessible.
• The domain CA certificate is installed on the VPN client.
• IIS double-escaping has been enabled.

Disabling the CRL check does let SSTP work (which means that this is strictly a CDP issue), but that's far from ideal. What's particularly frustrating is, as I mentioned previously, there are several different places where this could be breaking, and Windows is being very opaque as to where. For example, the IIS server logs don't show any download attempts of the CRL or delta CRLs.

On the other hand, this isn't something I want to sink too much more time into, however, especially when I have several other things to work on.... I'll probably come back later to work with it more.

Edit: Predictably, I end up fixing this because I found I couldn't log in via SSTP any longer. As it turns out, it's pretty easy to have IIS SSL settings clobber your SSTP port binding (really?), so Microsoft has a knowledge base article on how to fix that. I ended up disabling the Default Web Site (largely to remove contention for the 0.0.0.0:443 binding, as I'll be using SNI for my SSL sites), which ended up fixing the CDP issue as well.

For what I'm building out, having a Windows Server installation (or two) can be quite useful.

As part of this process, I've found that one of the nice things they added back in Windows Server 2008 is a Server Core installation mode, which strips out almost the entire user interface, causing most of the interaction to occur via Computer Management and Windows PowerShell. An interesting development for an operating system that's been built up almost completely around its GUI.

Not surprisingly, they've had to add a couple of tools to make this work more smoothly. The two most useful ones are sconfig.cmd, designed to run on the host system for initial setup, and the Remote Server Administration Tools (RSAT), the most recent version of which only runs on Windows 8.1 (for Windows Server 2012 R2 support).

Oddly, though, the application for which most people would want to run Windows Server for, Microsoft Exchange, doesn't support running in Windows Server Core mode. From the system requirements documentation for Exchange 2013:

To compound matters, most management of Exchange 2013 is now performed via a web browser anyways... so why does Exchange require a full Windows Server installation? Hell if I know....

Edit: It's apparently because Exchange (mailbox role) requires the Windows Media Audio Voice Codec. This doesn't require just the GUI, it requires the Desktop Experience add-on. Microsoft really couldn't pull that dependency out separately...?

Additional notes

• Since I'm building out a replacement domain, and I'd prefer to use the same NetBios name (although a different domain name), using rendom is helpful. Feasible because it's only the NetBios name I'm changing, and not generally recommended for production.
• Dealing with Windows MAK activations and virtual machines can be... irritating. The Datacenter license allows for an unlimited number of activations on a single physical server, but if you're not running Hyper-V, the instances apparently don't know that they're all running on a single physical server. Add to this the lack of GUI in Windows Server Core mode, and you end up needing to use the command line to activate Windows instead.
• The problem with building out a system in Windows Server Core mode is that it becomes very painful to add major features (like a GUI) back in the longer you wait, as you end up needing to troll through the Windows Update Catalog to manually download all updates to the system.
• I was getting an odd error while promoting the system to be a domain controller in the Post-deployment Configuration via a remote Server Manager, as it was complaining about an permissions error, while unfortunately not leaving behind any logs to indicate what the problem was. I did, however, stumble across this blog article that uses Install-ADDSForest in PowerShell to create a new domain, and it's not a stretch to find the TechNet article from there for full details. Alternately, using Install-WindowsFeature Server-Gui-Mgmt-Infra works too.
• When dealing with entirely too many different networks, knowing whether or not the domain should be accessible (and why) can sometimes be tricky. In this situation, your DNS can often be the cause of the problem, and nslookup can be very helpful then.
• Manipulating disks on a Windows Server Core system can be rather difficult, unless you really like playing with diskpart. Fortunately, you can enable Disk Management remotely, although it requires changes both on the server and the client sides.
• Needing to enable the full blown GUI just so you can run the DirectAccess Getting Started Wizard bites. DirectAccess is cool... but you need to be running Ultimate or Enterprise editions of Windows. Ouch.
• User Profile Wizard is pretty darn awesome if you're migrating users between local/domain and local/domain. I've done it by hand before, and this makes life *so* much easier.

As noted previously, I'll be working on setting up an VMware ESXi system. I personally lean toward using CentOS for my systems, and there's an official CentOS EC2 AMI that's designed to be a secure, minimal profile. Unfortunately, contrary to its description, the CentOS AWS wiki page doesn't actually describe how they're built. There's been at least one person asking on the CentOS-virt mailing list about how they're created, but nothing on the list describes the procedure (at least back through January, 2013).

So, how to spec out the package list? Easy! Launch a simple EC2 instance. Here's the list from CentOS 6 (x86_64), version 6 (2013/05/27). Naturally, you should run yum update, but this should serve as a pretty good starting point.

What's particularly interesting is when you compare this against the list of RPMs installed by the CentOS 6.5 minimal CD.

A quick and dirty comparison of the packages shows these as the differences:

The necessity (or not) of these packages is left as an exercise for the reader. 😛

• Although, it is rather frustrating if I then end up hitting a kernel panic while trying to build my base CentOS image on my desktop....
• It also sucks when the minimal installation ISO fails to boot, and it takes you a while to figure out why.
• The image (at least temporarily) needs enough memory, otherwise you'll get the text installer, which has limited configurability.

It can be interesting learning on the fly. It's like being thrown into the deep end of the pool - you either sink, or you swim.

Several useful things to know about ESXi:

• Using SSH keys instead of passwords (since they have to do things in a non-standard way).
• Setting up port forwarding for vSphere Client (to go with the above).
• Patching manually (without vSphere Update Manager).

Additional notes to come as I stumble across them.

Additional notes

• Replacing the default ESXi SSL certificates with your own, officially signed certificates isn't very difficult, and is probably a good idea.
• Running ESXi on a USB flash drive allows for pretty quick operating system recovery/replacement, at the potential cost of USB flash drives being less reliable than magnetic platter drives with RAID1. This can, of course, be counterbalanced by having a spare USB flash drive with ESXi ready to go, but then you would want a configuration backup handy and ready to go. Even though vCLI's vicfg-cfgbackup doesn't work (since you need a client system), there is fortunately another tool that can be used instead, although you need to be running in evaluation mode (as it doesn't work with free ESXi licenses).
  • Alternately, you can perform a backup and restore by hand. Which works with free licenses as well.

Setting up XWiki to act exactly the way I wanted was... interesting. Even now, I would argue that not everything's perfect (although it's pretty close). Miscellaneous installation notes:

• I don't understand why Jetty (and I assume Tomcat) doesn't support the ability to have a WAR where you can maintain a separate directory with any changes you'd like to make on top of it. It looks like they had it in for a handful of point releases, and then removed it for 9.1. It was dropped out of their 9.0.6.v20130930 release as well, even though their source code tag claims otherwise (sloppy, imho). Why the development team couldn't have a replacement ready before removing it? Hell if I know....
  • See, the nice thing about this is that it lets you maintain your configuration changes separate from upstream, especially given that applications are often distributed as WARs. When the common use case is that I then have to extract it because I want to make a few configuration changes, and then have to hand-maintain those changes? Yeah... not so much. Which is why I'm just throwing the whole thing into a version source control system to manage. it'd be nice if I didn't need to, though.
• The Jetty startup script is... kinda sloppy, imho, from a system administrator's perspective. If you're not running on a Debian-based system (i.e. you don't have start-stop-daemon), it tries to run su. Except you should generally be running daemons as separate accounts with /sbin/nologin?

• If you want to build out start-stop-daemon yourself, here's a quick primer:

• It's always disappointing when the configuration implies you can do something, but it doesn't actually work. Case in point, xwiki.defaultweb, referenced on the Configuration documentation page as well as xwiki.cfg.

• Doesn't actually do anything unless you use xwiki.usedefaultweb, in which case it just does the wrong thing instead (sends you to Main.WebHome instead of the space you've picked). Searches tell you to use the Redirect macro instead.
• It's remarkable how much of a pita it is to have XWiki use short URLs. Although at least it works.

Update: Semi-mea culpa: So you don't actually need to use start-stop-daemon to run Jetty as a non-privileged user... although they don't do actively push it in front of you the way other daemons do. Jetty has a SetUID module... although the documentation is incorrect on the changes you need to make to start.ini. You need to add this instead (in addition to the change to jetty-setuid.xml):

Welcome!

This is the newly set up home for toreishi.net, which will serve both as a personal soapbox for the infrequent times I have something to say, as well as a reference site for users of the site (as I find time to document things, at least!). Things will be a bit cluttered for now, but bear with the mess - it'll get better!

Apologies for the short term use of reCAPTCHA for anonymous comments - I find that it tends to create some fairly ridiculous entries, but I'll need to get better at my Ruby hacking skills before I can replace it with are you a human, which I find much more tolerable.

I've been somewhat busy over the past few months, which has led to a definite dearth of updates, but I'm now at the point where I'm starting to work on a number of technical projects.

• As you can see, I've moved over the blog from using Wagn to using XWiki. A good piece of this has to do with the annoyance of trying to using Ruby Gems from a system administrator's perspective (running gem as root, which you normally want to do since it's installing files in /usr, then creates files that have bad permissions) and with dealing with Ruby app servers (for some reason, Passenger was magically working, and then equally magically broke). I don't love Ruby enough to fight with it, especially given how Gems are administratively unfriendly in how they're too much like the Wild West to be packaged in a native manner.
• I'm working on migrating from two physical servers to a single physical server running VMware ESXi with multiple virtualized hosts. This should allow me to better isolate systems from each other.
• Along with the above, I'm working on moving the physical server to be colocated, which should help buffer users from anything I may be doing with my connection.