Technical

Up until recently, I've been using RRAS (built into Windows Server) to handle my NAT/router needs on my VMware ESXi host. I had a couple of problems previously, and annoyingly, one of the things I had set up previously (subnet-to-subnet VPN) stopped working and I was unable to fix it, even after several days of kicking it around. So, I opted to replace it with a Linux option, using Libreswan (over Openswan due to the history) as my IPsec implemntation. One of the advantages (other than it being somewhat easier to debug odd issues) is that it's quite a bit simpler to add additional Linux systems to the subnet-to-subnet VPN, which I'm planning for later use.

The eventual plan is to combine the following into one system:

• NAT/router
• Subnet-to-subnet VPN (site-to-site)
• Host-to-subnet VPN (client)

This post will address the first two, and I'll cover the third a later time.

NAT/router

Bits and pieces taken from this page.

Naturally, if you'd like to set up your system as a NAT/router, you'll want to set your firewall software up; in this case, I'm operating under the assumption that you're using iptables. First off, you'll need to make sure that you enable IP forwarding. This entry should be in your /etc/sysctl.conf, so that it's set on each boot:

And then, on the shell:

This page assumes the following definitions:

You'll want entries similar to the following in your iptables configuration in the appropriate location (comments optional, of course):

At this point, you should be able to set up 192.168.0.1 as your gateway for the devices you'd like to put behind the NAT/router.

Subnet-to-subnet

Bits and pieces taken from this page and this page (particularly for the iptables setup).

Things get more interesting once you'd like to connect multiple private subnets. In my situation, I'm connecting my ESXi's virtual network to my home internal network so that I'm always on the "inside." Not surprisingly, I'd like to do this in a reasonably secure manner. As noted above, this setup uses Libreswan as its IPsec implementation. The following instructions apply to the systems on both sides of the tunnel.

Our example uses the following setup ("left" and "right" are essentially arbitrary, but should be consistent):

Installing Libreswan takes a little bit of effort on CentOS 6, as it's not in the default repositories. As a result, you'll need to grab Libreswan's repository file and put it in /etc/yum.repos.d (you'll also need to download their GPG key to /etc/pki/rpm-gpg), along with installing the Fedora Extra Packages for Enterprise Linux 6 RPM. At this point, you should be able to install Libreswan:

At this point, you now need to configure Libreswan. Starting with /etc/ipsec.conf, I only touch a couple of lines:

The virtual_private entry denotes which private subnets to allow/disallow; in this case, we're leaving the default except for explicitly disallowing the local subnet. This particular entry is for the 192.168.0.0 side; the 192.168.1.0 side should be updated appropriately. The last line is a personal preference; I like having configurations separated out on a per-file basis.

Next up is setting up the key for the connection:

You'll need the RSA key signature in the next step (your RSA key's key ID is obviously going to be different). The directory /etc/ipsec.d should have its permissions locked down, but if you're paranoid, you can lock down the signature file as well:

After this, you should set up your VPN configuration file, /etc/ipsec.d/vpn.conf:

You'll also need additional lines to your /etc/sysctl.conf:

And then, on the shell:

At this point, we can let Libreswan verify itself (everything should come up looking good):

Before we can start the VPN, however, we'll need to set up our iptables configuration appropriately (once again, comments optional):

At this point, you can start up the IPsec service.

You should now be able to ping from one subnet to systems on the other subnet.

While looking around to check if I might potentially be contributing to the DNS amplification problem, CERT pointed me at a pretty nifty site that checks a lot of other potential DNS issues: DNSInspect. It checks several different aspects related to nameservers (and a few besides), and generates a report that tells you what works, what doesn't, and what could potentially use some shoring up. It really likes IPv6 (although it doesn't hold it against you), but all in all, performs a considerable number of checks, including some I wasn't aware of (like that your MX records should be A records, and not, for example, CNAMEs). I highly recommend checking it out if you're setting up a domain.

The past few weeks have been pretty hectic, so I haven't had as much time to work on things as I'd prefer.

An important part of having a production-grade system is, naturally, having backups. it's remarkably possible to put together a semi-decent system with a little bit of effort. Some requirements:

• Cross-platform support
  • I have Linux and Windows systems with data I need to back up.
• Secure
  • No single system should have the ability to read the backups of all other systems.
• Redundant
  • As much as possible, the backup system should tolerate failure.
• Free
  • Okay, so I'm cheap. But this is a purely personal setup, so I'd like to minimize my investment (especially since a lot of the available options easily run over several thousand dollars!).

The layered system I've put together looks like this:

• Backup system: Greyhole
  • A Linux-based system that attempts to mimic Windows Home Server's Drive Extender capabilities. The advantage it provides is JBOD exported as a single logical volume, while not having to worry excessively about what happens when a single disk fails (you lose the contents of that disk, but the contents on other volumes remain intact). Honestly, it's a hack, but if it works....
• As a basic level of access control, different Samba accounts for the various systems should restrict access for one system to another's backups.
• Linux engine: tar through GnuPG
  • Piping tar through GnuPG allows for a simple system while keeping the encryption/decryption key on the client system.
  • Maintaining database consistency is incumbent on me.
• Windows engine: Windows backup through TrueCrypt
  • Windows Server's built-in backup works well, in that it ensures that databases stay consistent (in particular those for Active Directory and Exchange). But it doesn't support encryption. Setting up a TrueCrypt volume for Windows to mount alleviates this issue, and allows for retaining the encryption/decryption key on the client system.
• Secondary backup system: CrashPlan
  • Having "on-site" backups is great, but having "off-site" backups is also important. CrashPlan works well as a peer-to-peer backup system, allowing the backups to be duplicated elsewhere.

Putting together the Windows piece is actually a little bit more complex, due to the fact that although TrueCrypt declares itself as a local drive, it doesn't support all of the operations that Windows expects to be able to perform on a local drive, as a result failing with a not-so-helpful "Catastrophic failure" error message, with Event Viewer showing an error code of either 2147549183 or 8000FFFF (hex). There is a workaround described on Super User, but what's described doesn't lend itself well to automation. I've pulled together a batch script that handles everything reasonably well:

Naturally, make the following substitutions:

• NETHOST
  • The backup server that hosts the TrueCrypt volumes (accessed via SMB).
• NETHOST_USER/NETHOST_PW
  • The username and password for the backup server.
• TCPASSWORD
  • The password for the TrueCrypt volume.

A few assumptions made for general deployability:

• This script uses drives Y: and Z:. Change as needed.
• Each TrueCrypt volume goes to its own folder based upon the host's name (in case you want to have NETHOST_USER be per-host).
• Each TrueCrypt volume is named after the host's name.
• I chose not to rely on the administrative Y$ share, as it's not guaranteed to be available, and reliability >>> shorter code in this case.

I have an environment which mixes some users on Active Directory with some users who have straight Linux accounts. This doesn't always work well with authentication mechanisms, though, as going with standard mechanisms for one or the other has its own set of quirks (ignoring that going with one or the other would also leave out users). Authenticating against AD's LDAP can be a little strange, due to Windows Server's quirks. Authenticating against passwd/shadow requires either getting PAM working (which can be finicky) or running daemons as root (which is dangerous).

So instead, combine the two using a different tool! OpenLDAP supports use of multiple database definitions, of which one is a proxy. And on top of that, you can join them together by making one subordinate to the other so that both databases are searched.

Fortunately, this MSDN blog article has a lot of information that applies on connecting OpenLDAP to AD. But:

• The MSDN blog article is rather vague about how to set up LDAPS. There is a Technet article, but it refers to a specific MachineKey without explaining how to identify it.
• Figuring out the MachineKey for a certificate is described over here, on the Directory Services Team blog.
• Windows Server Core doesn't have Explorer to modify file/directory permissions, so you also need to know how to use icacls.
• Windows Server only passes back the client certificate, and unfortunately not the full certificate chain, so you end up needing to specify TLS_CACERT instead of just TLS_CACERTDIR.

The MSDN blog article primarily concerned about saslauthd access, though, rather than OpenLDAP connectivity, so this only gets us as far as having the infrastructure to query AD over LDAPS (which is valuable, though!).

If you put together several pieces of information out there, it's possible to now set up OpenLDAP with all the pieces we need. Inconveniently, public documentation seems to be largely out of date, and very fragmented. A lot of it refers to using slapd.conf, but that's been deprecated in favour of slapd.d. Yet oddly enough, users on openldap-technical seem to be perfectly okay with taking a slapd.conf, converting it to cn=config format, then shoving it in....

Notes from here on were gathered on CentOS 6.x - if you're following along, you may need to make allowances for your own distribution.

Pieces of documentation used to put together the OpenLDAP configuration (starting with slapd.conf, then converting to LDIF):

• Integrate Active Directory and OpenLDAP
  • Overall documentation on putting together slapd.conf, and particularly the pointer about subordinate.
• OpenLDAP Server
  • The Ubuntu OpenLDAP server tutorial, very useful for how to go about setting up the backend database, and part of the reason I selected HDB instead of BDB, which remains the default on CentOS.

Using these pages as a basis, I pulled together a slapd.conf looking roughly like this (if you want to use it as a basis for your own slapd.conf, you'll need to pay attention to all TODO entries):

You may have noticed that there's a reference to a microsoft.schema at the top. That's to help resolve some otherwise unresolved LDAP types. The file follows, with the first two entries from Apple, of all places, and the third being a localized (i.e. using a local Object Identifier) version of the standard NIS homeDirectory entry):

The rewrite/remap entries in slapd.conf for the AD LDAP proxy refer to a few non-standard properties. These are controlled via the Unix Attributes tab in AD, but in Windows Server 2012 R2, that's enabled by installing Identity Management for UNIX.

Once the slapd.conf is ready to go, it's pretty simple to generate a slapd.d directory:

This generates a series of LDIF files which can be massaged a bit more, which is where this documentation comes in very handy:

• LDAP for Rocket Scientists
  • The ZYTRAX guide to LDAP and LDIF was absolutely key to figuring out how to get slapd.d and LDIF working properly.

Combined with the above documentation, I generated three LDIF files (which, again, you'll need to change with your own information):

Importing the LDIFs is relatively straightforward, although cleaning up what we don't want is a good idea:

When you put your certificate files in /etc/openldap/certs, if you're running with SELinux (which you probably should), make sure that the file security contexts are correct, particularly for the CA certificate (otherwise, OpenLDAP won't be able to connect to your AD server):

Known Issues:

The connection to AD is unencrypted. I'm working on how to have it connect via ldaps://  instead. ldapsearch works without a problem, so it's puzzling why slapd fails. Further updates to come if/when I figure this out. Bitten by SELinux! See note above about restorecon.

Within the next few weeks, I expect that I'll be back on iOS. It's been a while since I've used it, but I've had experience with a number of different mobile operating systems at this point:

• Android (up through 4.2)
• DangerOS
• iOS (up through 5, occasional exposure up through 7)
• Windows Phone (7)
• Windows Mobile (up through 6.5)

Up until now, I've been using Android on a Samsung Note II. Which has gone reasonably well, but a few things have caused me to give up on Android:

• Replacing the home screen with Google Now.
• My device has been relatively stable... except for the past couple of months, it's been hard-locking repeatedly. Sometimes several times a day.
• I looked at a Note II because it was an attempt at trying something different with a mobile device. Unfortunately, several of the unique aspects just haven't worked out:
  • Stylus: I've barely used it.
  • Screen size: Larger than most phones, but not big enough to use for couch browsing.
  • Popup browser: Although active in Android 4.0, it was disabled in Android 4.2. Unfortunately, one of the things I liked.

In general, though, I've concluded that Android, at least at this point in time, isn't for me. At least some chunk of that is due to Google's business practices (Android is open... except when it's not). So back to a different walled garden for me - but at least it's one that doesn't pretend otherwise.

I'm... irritated. I've just spent several hours trying to figure out how to have Jetty use an SSL certificate generated wholly externally, all because the Jetty documentation is... less than complete.

The start line:

• One PKCS12 file with a X.509 certificate, its private key, and its chain of trust, encrypted with a password.
  • Generated from Windows Server, but the answer isn't "Don't use Windows". Yes, it's sometimes a perfectly valid answer. But not this time.

Things not obvious:

• jetty-ssl.xml refers to KeyStorePassword (easy enough), KeyManagerPassword (what's that?), and TrustStorePassword (huh?).
• The Jetty instructions for importing a PKCS12 file only talk about keystore passwords. But that leads to a situation where Jetty complains it can't read the necessary keys.
• If you'd like to leverage the default Java Jetty keystore (particularly if you have a secure system), you need to know the keystore's password. Which isn't provided anywhere... although an Internet search will tell you.
• After much digging around, maybe it's because for the key pair, the private key can have a password? Except having a blank password doesn't work, and setting it to the keystore's password doesn't work. Odd.
• The SSL documentation for Jetty 8 refers to a key pair password. What happened to that configuration key?
• So... somehow, "password" became "KeyManagerPassword". But what's the default password for that? It takes another Internet search to find that. Or maybe you're just supposed to look in the source code?

Spending several hours over this makes me not happy.

P.S. There's conveniently a command to import a PKCS12 file directly into a keystore. You can use it to rename the key in the process, since Jetty wants a key called... jetty.

A challenge when performing service refactoring is that you now need to start worrying a lot more about connection security. When everything's on one box, and you're connecting via ports on localhost, you don't generally need to worry much. However, once you start splitting out services, relying on trust and unencrypted connections becomes less viable.

Here are some notes regarding PostgreSQL, as an example:

• Your users should be already set up with encrypted passwords (you did, right?).
• Next, you need to prepare SSL certificates for PostgreSQL, assuming that your installation has SSL support (CentOS compiles it in by default), install them in the correct location, and enable them in postgresql.conf.
  • If your internal CA is based on Windows Server, it's a little trickier to generate the certificate and its private key. I prefer to generate the certificate request on the CA with an .inf file, import the certificate on the CA, export the public and private key from the CA, and then convert the .pfx file into a .crt and .key file on the target host.
• Once you have your SSL certificates up and running, the next step is to define PostgreSQL connection restrictions (using hostssl, of course).
• From there, it's a matter of adjusting your database connection strings appropriately.
  • PHP uses the sslmode parameter, although it's odd that the value list is in a comment and not in the documentation.
• JDBC uses the ssl parameter, although since Java keeps its own CA database, you may need to import your CA certificate into the Java truststore.

A few interesting things I've stumbled across while finalizing some of my migration preparations from my previous server to the new one

• If you have multiple network adapters, your default route might not be the one you want. At least in CentOS, you can configure that in /etc/sysconfig/network with GATEWAY and GATEWAYDEV. This could be the problem if you're finding that you can't connect to your system (ping, ssh, etc.) even when it's capable of reaching the outside world... and even more, that if you disable your internal network adapter, everything "magically works." netstat -r/route -e might reveal that your default gateway isn't the one you think it is.
• Brute force ssh attacks aren't fun, and they've been pretty common for a while. True, they're probably not going to hack into your box (You have disabled root ssh access, right? And you don't have any of the standard accounts they try to hack in with?), but they can be rather annoying. Fortunately, it's actually pretty easy to use iptables to limit the number of ssh connections per minute.

Yesterday, while installing some more software, the fans on my ESXi system spun up to high, then all my sessions halted. That's never a good thing.

I took a look at what was on the physical screen of my system, and I got to learn about the joys of the PSOD. Initially, I thought the issue was bad memory, so I ran MemTest86 for a little over 20 hours. Nope. Not the problem.

A bit more searching revealed this little gem of a VMware knowledge base article.

Really, VMware? You're going to have a patch available for 5.0 and 5.1, but not 5.5? What the hell?!

For the past several days, I've been working on getting a VPN working for my little cluster. There are several different options available for Windows Server. It didn't take too long to get PPTP working, which I would consider the base level. I'd prefer not to try to implement L2TP, as that would require installing client certificates on all systems, which is annoying.

As a result, I've been spending the past several days trying to get SSTP working instead, which looks pretty straightforward on paper. Unfortunately, there are all too many problems you can stumble across, and the one I've been fighting for a while has been error 0x80092013 (which lots of other people also seem to fight with). What's depressing is that Windows offers almost no logging, so I've been slowly working through all the issues, but I've now hit a dead end.

• The CDP and DeltaCRL locations now refer only to externally-accessible locations (verified via Enterprise PKI).
• The certificate's CRL distribution point is externally-accessible.
• The domain CA certificate is installed on the VPN client.
• IIS double-escaping has been enabled.

Disabling the CRL check does let SSTP work (which means that this is strictly a CDP issue), but that's far from ideal. What's particularly frustrating is, as I mentioned previously, there are several different places where this could be breaking, and Windows is being very opaque as to where. For example, the IIS server logs don't show any download attempts of the CRL or delta CRLs.

On the other hand, this isn't something I want to sink too much more time into, however, especially when I have several other things to work on.... I'll probably come back later to work with it more.

Edit: Predictably, I end up fixing this because I found I couldn't log in via SSTP any longer. As it turns out, it's pretty easy to have IIS SSL settings clobber your SSTP port binding (really?), so Microsoft has a knowledge base article on how to fix that. I ended up disabling the Default Web Site (largely to remove contention for the 0.0.0.0:443 binding, as I'll be using SNI for my SSL sites), which ended up fixing the CDP issue as well.