Toreishi.net ~Now with more SNAP~

I've decided to finally get around to set up kickstart configuration files for my system images, since I've started investigating migrating over to CentOS 7. Kickstart, if you're not familiar with it, is a method of automating Linux installation and configuration, and is largely centered around the Red Hat-based distributions. While I was setting all of this up, I decided to also investigate whether it was worth switching from CentOS to Ubuntu Server, seeing as Ubuntu usage has passed CentOS and RHEL usage according to W3Techs (I suppose I could have looked at Debian as well, but for whatever reason it doesn't particularly appeal to me - possibly because it's not as marketable a job skill, for all that I don't really do this for a career?).

That said, I decided not to, for a couple of reasons:

• The difference in how well they're documented is huge. Red Hat has a tremendous amount of documentation on setting up a kickstart installation compared to Ubuntu's preseed documentation (which seems to boil down to: take this undocumented file and it should work). Somewhat ironically, Debian has considerably better documentation.
• Red Hat's installation process generates a kickstart configuration file (/root/anaconda-ks.cfg) that you can immediately turn around and feed back into a kickstart installation to get the same result, whereas the Debian documentation notes that their equivalent (debconf-get-selections --installer; debconf-get-selections) doesn't actually quite work (and I would expect that Ubuntu would follow in the same footsteps).
• And, of course, I'm still better and more comfortable with Red Hat-based distributions than Ubuntu distributions, which matters for what is effectively a production deployment.

P.S. Yes, Ubuntu can also support kickstart installations, but it's a hacky process. I don't care that much.

I've now finished playing through all of 大図書館の羊飼い [WARNING: 18+ content], as well as 大図書館の羊飼い -Dreaming Sheep- [WARNING: 18+ content] (which is a fandisc of after stories), and I figured I would write down some of my thoughts (I haven't yet finished 大図書館の羊飼い～放課後しっぽデイズ～, which is a side story fandisc).

Story/Characterization

• The story for 大図書館の羊飼い just isn't as tightly written as that of Fortune Arterial. The base game, 大図書館の羊飼い, is designed in the standard form of a common main trunk, which then branches out into the separate character stories, with the True End route locked until certain requirements are fulfilled. Fortune Arterial [WARNING: 18+ content] is written such that each of the character arcs helps contribute knowledge towards the True End; in contrast, the primary character arcs in 大図書館の羊飼い don't, although (as expected) they do help flesh out each character. In an interesting twist, however, the True End route actually consists of multiple endings: one for Kodachi Nagi [WARNING: 18+ content] (which is more independent), and one for each of the other girls (which result in minor variations).
• The stories in 大図書館の羊飼い -Dreaming Sheep- consist of one after story for each character's ending in 大図書館の羊飼い (five main heroines, three secondary heroines), along with one ending for a secondary character (Takigawa Aoi [WARNING: 18+ content]) which attempts to explain her personality and motivation and one story arc, named the Toshobu arc, that covers a "blank" of a month that occurs during the True End routes.
  • In an unusual twist, Shirasaki Tsugumi's [WARNING: 18+ content] after story occurs after her True End route, while the other after stories occur after their normal endings. To further confuse players, although one of her appendix stories occurs after her True End route, the one she shares with her younger sister [WARNING: 18+ content] occurs after her normal ending (her other appendix story occurs sometime considerably later, and isn't tied to either).
  • Aoi's route is an interesting decision as, although it starts with an Aoi like in the other arcs, it ends in an Aoi that is substantially different from any of the other arcs.
  • The Toshobu arc is fluff. Entertaining fluff, but still fluff (obviously, as it was completely left out of the True End route).
• The various stories are, as expected from August, appropriately heartwarming, with a considerable amount of humour sprinkled throughout.

Such as this image, where Sakuraba Tamamo [WARNING: 18+ content] and Suzuki Kana [WARNING: 18+ content] are trying to figure out how to respond to a really bad joke by Tsugumi). For the most part, August doesn't seem to opt for considerable amounts of angst.

• Kana is awesome.

Nothing more needs be said. Although, this is also awesome.

• The main character, Kakei Kyoutarou [WARNING: 18+ content], has an odd choice of cell phone.

According to Japanese Wikipedia, this is an au Windows Phone IS12T. That's a particularly interesting choice, given the very small representation Windows Phone has in Japan.

• For what it's worth, the anime appears to have gone pretty much completely off the rails. The Fortune Arterial anime largely stayed pretty close to the source material (although it wove another Erika route), but the 大図書館の羊飼い anime has gone places the game never did.
  • As an entertaining cameo, 大図書館の羊飼い makes mention of a student council president, Sendou, from Shuuchikan Academy.

Gameplay

• As a standard visual novel, there's not a whole lot to say here. Although one interesting feature August added (which I wish more companies would add, now that I've seen it) is the ability, in the dialogue history, to jump back to any point. It's really nice for seeing some of the happenings "in context."

Artistic

• The character designs... well, they're August character designs (i.e. べっかんこう). You might love them, you might hate them, but they definitely don't change. There's even a "joke" gif that I've seen that takes characters from different August games, puts them in greyscale, then cycles through them, and you can't tell which character is which....
• August decided to take the art assets they had for character dialogues and use them in broader situations, which allowed them to have some interesting shots without needing new art.

They mostly tweaked lighting and occasionally focus. I think they did a good job with the additional images.

Technical

• Ah... the times I hate copy protection. I've purchased these games, and they still don't work reliably. August, like several other companies, uses SETTEC's Alpha-ROM, and it just doesn't work as well as it should. What bothers me most about copy protection is that I'm relying on a third party to remain in business - if they go upside down, then the games I've purchased become unplayable.

Of the overall Japanese electronic gaming market, a considerable chunk consists of what are called ren'ai games (恋愛ゲーム, lit. love games), which are games that revolve around love and romance (and, all-too-often for guys, sex). From a mechanics standpoint, a fairly large percentage are comprised of what the Japanese consider simulation games (シムレーションゲーム), which for those not familiar with the term, are essentially electronic versions of Choose Your Own Adventure books (or gamebooks, if you want to use the more general term). For the most part, these types of games don't exist in the Western market. From a theme standpoint, some of these are called nakige (泣きゲー, lit. crying games), where the goal is have a story so emotionally charged that the player is brought to tears. Again, for the most part, these types of games don't exist in the Western market. Standard examples of the cross-section of these two are games by Visual Art's/Key (Kanon]], Air, Clannad, リトルバスターズ！), as well as games in Giga (戯画)'s [WARNING: 18+ content if you do a search] maid cafe series (ショコラ ～maid cafe "curio"～, パ ル フ ェ ～ショコラ second brew～) and CIRCUS' [WARNING: 18+ content if you do a search] D.C. series (D.C.～ダ・カーポ～, D.C.II～ダ・カーポII～, D.C.III～ダ・カーポIII～), amongst others.

But what if you were to take some of these themes into a different environment? Such as, say, board gaming?

Then you might end up with a game like The Ravens of Thri Sahashri (三千世界の鴉).

• Official homepage (Japanese)
• BGG link
• BGG video link

I stumbled across the BGG coverage from Spiel 2014 by accident, having looked at it out of (perverse) curiosity from the mention in the BGG Ars Alchimia's video (where they describe The Ravens of Thri Sahashri as a work when the designer is mad).

The premise starts off with a not-too-unusual tragic romance base in a historical Japanese setting, where the family of Ren (四宮恋), the heroine, has died from an epidemic, and her relatives have sold her off to be a geisha. The hero, Feth (フェス), in Japan due to his father's trading career, falls in love with her while with his father in the red light district for a session of entertainment. He decides to free her, which he does with the help of one of the older geisha, but during the escape, the nameless older geisha is shot and killed, causing Ren to fall unconscious from shock. The game starts with Feth having returned to the foreign conclave with an unconscious Ren, where he wonders if this occasion is why he has the undesirable ability to enter other people's hearts, deciding to do so with a weakened Ren, with sunrise as his deadline. The mechanics are fairly involved, revolving around a very unusual two-player, asymmetrical, cooperative game with no allowed overt in-game communication. For each of three batches, the person playing as Ren draws four cards from the memory deck, keeping them out of view of the person playing as Feth. Each round, Feth draws as many cards as desired to make patterns on the board (referred to as the Atman); at the end of each round, Ren is able to take a card from the board, allowing Feth to use the power on that card from then on. A batch ends when certain conditions have been completed, and the players can win the game if they successfully complete the third batch. The specifics are complex enough such that I won't repeat them here.

The theme, however, is (imho) even stronger. Terms: (mostly) accurate translations. 三千世界 is an abbreviation of 三千大千世界, meaning trisāhasramahāsāhasralokadhātu (which thri hashari is also an abbreviation of). 鴉 would more correctly be translated as crows in this case, however, since crows are related to Mahākāla in Vajrayāna Buddhism, god of time and death. Rather appropriate in this case, given that in the game, crows attempt to take Ren's memories (presumably in preparation for her rebirth). For the playing field, the Japanese instructions refer to it as Limbo (辺獄), where the English instructions refer to it as the Ātman (self). Seeing as Limbo is a particularly Christian notion, it makes sense to look for something more applicable. The English description of the story from the BGG video unfortunately doesn't do the game justice... as the more you dig into the story, the more poignant it becomes. It's also written in the English instructions, but being a straight translation of the Japanese instructions, it doesn't have as much weight as the story at the bottom of the official homepage. It might only be a few paragraphs, but there's additional weight there. There's a sample translation of one of the poems available in this post. One things that disappoints me about the English instructions is that they don't describe the emotions behind the cards: hatred (red), remorse (green), grief (blue), isolation (white), terror (purple). There are the endings: much like a Japanese ren'ai game, there is a Normal, Good, and Happy ending (although I would argue that the last should actually be the True ending). The English translation is available, with the Japanese version being available at the end of the Japanese instructions.

Unfortunately? It was a limited print run, and may not ever be available again in the future. I'd love to get my hands on a copy, but it doesn't look likely at all. I've checked to see if I can source the original Japanese version, but I don't seem to be able to find that one either.

P.S. From a theme standpoint, I'm actually somewhat reminded of Navel's Really? Really! [WARNING: 18+ content], at least for the notion of sorting through somebody's memories to bring them back from an unconscious state... although it's a considerably less-depressing game, with frequent bouts of humour.

As mentioned in the previous post, I'm writing down some details concerning setting up a VPN, which can occasionally be quite useful, whether it be due to accessing the Internet from an insecure location or due to working around region restrictions. As before, I'm using Libreswan, along with xl2tpd and ppp. I'm also using winbind (part of the Samba project) in order to authenticate against a Windows domain.

VPN

Bits and pieces taken from this page.

As with the subnet-to-subnet configuration, you'll need to put the Libreswan repository file in /etc/yum.repos.d, put the the Libreswan GPG key in /etc/pki/rpm-gpg, and install the Fedora Extra Packages for Enterprise Linux 6 RPM. At this point, you can install the packages you need to move forward:

There are more moving pieces this time through, so the configuration is a bit more involved. Our example uses the following assumptions:

Libreswan

Configure /etc/ipsec.conf as noted in the previous article. We'll set up another configuration file for the dialin VPN, /etc/ipsec.d/dialin.conf. For the sake of simplicity (and compatibility), we'll do a pre-shared key (PSK) configuration:

To define the PSK, create /etc/ipsec.d/dialin.secrets:

As before, you can be paranoid with the secrets file:

And the same additional lines to /etc/sysctl.conf apply here:

Along with the verification step:

xl2tpd/ppp

xl2tpd and ppp are intertwined, so their configuration happens pretty much at the same time. Set up your /etc/xl2tpd/xl2tpd.conf roughly as follows, adjusting for the configuration above:

For now, set up /etc/ppp/options.xl2tpd as follows:

Set up usernames and passwords in /etc/ppp/chap-secrets, one per line (replace $username and $password as appropriate):

iptables

As before, there need to be additional entries in the iptables configuration:

Testing the VPN

At this point, you can set up the VPN from a client, with the following settings:

• Host: 1.2.3.4
• Type: L2TP/IPsec
  • Pre-shared key (not a certificate): the key defined above
  • Protocol: CHAP (not MS-CHAP v2)
• Username: a username defined above
• Password: a password defined above

You should be able to connect successfully, and connect to systems within the internal network at this point. If you are using a Windows system behind a NAT, you may need to set a registry setting and reboot your system prior to being able to connect to the VPN.

Connecting to a Windows domain

The way pppd works on Linux, authenticating against LDAP involves connecting the ppp daemon to RADIUS, then allowing RADIUS to validate credentials against LDAP. Unfortunately, RADIUS performs the validation itself, so it requires authenticating against the VPN server with PAP and storing plaintext passwords within LDAP, which is less than ideal. There is, however, a provision to allow pppd to authenticate against a Windows domain.

First, install winbind:

Set up your /etc/samba/smb.conf with the appropriate information (change the following settings to match those for your own domain):

Then, there are several commands to complete the domain setup:

You can clear out the entries from /etc/ppp/chap-secrets, as you won't be using the passwords stored in a text file. Update the following settings in /etc/xl2tpd/xl2tpd.conf:

And the following settings in /etc/ppp/options.xl2tpd:

If you have SELinux enabled, you'll need to generate a policy to allow ntlm_auth to run. You can follow the instructions on this page to figure out how to do so. Your template file should look something like this:

The end of the line

Ideally, we would go through and implement user authentication via domain username and password (as above) with machine authentication via certificates. Unfortunately, at least one of the targeted operating systems doesn't support machine certificates for L2TP/IPsec, so there isn't too much value in pursuing it further.

Up until recently, I've been using RRAS (built into Windows Server) to handle my NAT/router needs on my VMware ESXi host. I had a couple of problems previously, and annoyingly, one of the things I had set up previously (subnet-to-subnet VPN) stopped working and I was unable to fix it, even after several days of kicking it around. So, I opted to replace it with a Linux option, using Libreswan (over Openswan due to the history) as my IPsec implemntation. One of the advantages (other than it being somewhat easier to debug odd issues) is that it's quite a bit simpler to add additional Linux systems to the subnet-to-subnet VPN, which I'm planning for later use.

The eventual plan is to combine the following into one system:

• NAT/router
• Subnet-to-subnet VPN (site-to-site)
• Host-to-subnet VPN (client)

This post will address the first two, and I'll cover the third a later time.

NAT/router

Bits and pieces taken from this page.

Naturally, if you'd like to set up your system as a NAT/router, you'll want to set your firewall software up; in this case, I'm operating under the assumption that you're using iptables. First off, you'll need to make sure that you enable IP forwarding. This entry should be in your /etc/sysctl.conf, so that it's set on each boot:

And then, on the shell:

This page assumes the following definitions:

You'll want entries similar to the following in your iptables configuration in the appropriate location (comments optional, of course):

At this point, you should be able to set up 192.168.0.1 as your gateway for the devices you'd like to put behind the NAT/router.

Subnet-to-subnet

Bits and pieces taken from this page and this page (particularly for the iptables setup).

Things get more interesting once you'd like to connect multiple private subnets. In my situation, I'm connecting my ESXi's virtual network to my home internal network so that I'm always on the "inside." Not surprisingly, I'd like to do this in a reasonably secure manner. As noted above, this setup uses Libreswan as its IPsec implementation. The following instructions apply to the systems on both sides of the tunnel.

Our example uses the following setup ("left" and "right" are essentially arbitrary, but should be consistent):

Installing Libreswan takes a little bit of effort on CentOS 6, as it's not in the default repositories. As a result, you'll need to grab Libreswan's repository file and put it in /etc/yum.repos.d (you'll also need to download their GPG key to /etc/pki/rpm-gpg), along with installing the Fedora Extra Packages for Enterprise Linux 6 RPM. At this point, you should be able to install Libreswan:

At this point, you now need to configure Libreswan. Starting with /etc/ipsec.conf, I only touch a couple of lines:

The virtual_private entry denotes which private subnets to allow/disallow; in this case, we're leaving the default except for explicitly disallowing the local subnet. This particular entry is for the 192.168.0.0 side; the 192.168.1.0 side should be updated appropriately. The last line is a personal preference; I like having configurations separated out on a per-file basis.

Next up is setting up the key for the connection:

You'll need the RSA key signature in the next step (your RSA key's key ID is obviously going to be different). The directory /etc/ipsec.d should have its permissions locked down, but if you're paranoid, you can lock down the signature file as well:

After this, you should set up your VPN configuration file, /etc/ipsec.d/vpn.conf:

You'll also need additional lines to your /etc/sysctl.conf:

And then, on the shell:

At this point, we can let Libreswan verify itself (everything should come up looking good):

Before we can start the VPN, however, we'll need to set up our iptables configuration appropriately (once again, comments optional):

At this point, you can start up the IPsec service.

You should now be able to ping from one subnet to systems on the other subnet.

While looking around to check if I might potentially be contributing to the DNS amplification problem, CERT pointed me at a pretty nifty site that checks a lot of other potential DNS issues: DNSInspect. It checks several different aspects related to nameservers (and a few besides), and generates a report that tells you what works, what doesn't, and what could potentially use some shoring up. It really likes IPv6 (although it doesn't hold it against you), but all in all, performs a considerable number of checks, including some I wasn't aware of (like that your MX records should be A records, and not, for example, CNAMEs). I highly recommend checking it out if you're setting up a domain.

The past few weeks have been pretty hectic, so I haven't had as much time to work on things as I'd prefer.

An important part of having a production-grade system is, naturally, having backups. it's remarkably possible to put together a semi-decent system with a little bit of effort. Some requirements:

• Cross-platform support
  • I have Linux and Windows systems with data I need to back up.
• Secure
  • No single system should have the ability to read the backups of all other systems.
• Redundant
  • As much as possible, the backup system should tolerate failure.
• Free
  • Okay, so I'm cheap. But this is a purely personal setup, so I'd like to minimize my investment (especially since a lot of the available options easily run over several thousand dollars!).

The layered system I've put together looks like this:

• Backup system: Greyhole
  • A Linux-based system that attempts to mimic Windows Home Server's Drive Extender capabilities. The advantage it provides is JBOD exported as a single logical volume, while not having to worry excessively about what happens when a single disk fails (you lose the contents of that disk, but the contents on other volumes remain intact). Honestly, it's a hack, but if it works....
• As a basic level of access control, different Samba accounts for the various systems should restrict access for one system to another's backups.
• Linux engine: tar through GnuPG
  • Piping tar through GnuPG allows for a simple system while keeping the encryption/decryption key on the client system.
  • Maintaining database consistency is incumbent on me.
• Windows engine: Windows backup through TrueCrypt
  • Windows Server's built-in backup works well, in that it ensures that databases stay consistent (in particular those for Active Directory and Exchange). But it doesn't support encryption. Setting up a TrueCrypt volume for Windows to mount alleviates this issue, and allows for retaining the encryption/decryption key on the client system.
• Secondary backup system: CrashPlan
  • Having "on-site" backups is great, but having "off-site" backups is also important. CrashPlan works well as a peer-to-peer backup system, allowing the backups to be duplicated elsewhere.

Putting together the Windows piece is actually a little bit more complex, due to the fact that although TrueCrypt declares itself as a local drive, it doesn't support all of the operations that Windows expects to be able to perform on a local drive, as a result failing with a not-so-helpful "Catastrophic failure" error message, with Event Viewer showing an error code of either 2147549183 or 8000FFFF (hex). There is a workaround described on Super User, but what's described doesn't lend itself well to automation. I've pulled together a batch script that handles everything reasonably well:

Naturally, make the following substitutions:

• NETHOST
  • The backup server that hosts the TrueCrypt volumes (accessed via SMB).
• NETHOST_USER/NETHOST_PW
  • The username and password for the backup server.
• TCPASSWORD
  • The password for the TrueCrypt volume.

A few assumptions made for general deployability:

• This script uses drives Y: and Z:. Change as needed.
• Each TrueCrypt volume goes to its own folder based upon the host's name (in case you want to have NETHOST_USER be per-host).
• Each TrueCrypt volume is named after the host's name.
• I chose not to rely on the administrative Y$ share, as it's not guaranteed to be available, and reliability >>> shorter code in this case.

I have an environment which mixes some users on Active Directory with some users who have straight Linux accounts. This doesn't always work well with authentication mechanisms, though, as going with standard mechanisms for one or the other has its own set of quirks (ignoring that going with one or the other would also leave out users). Authenticating against AD's LDAP can be a little strange, due to Windows Server's quirks. Authenticating against passwd/shadow requires either getting PAM working (which can be finicky) or running daemons as root (which is dangerous).

So instead, combine the two using a different tool! OpenLDAP supports use of multiple database definitions, of which one is a proxy. And on top of that, you can join them together by making one subordinate to the other so that both databases are searched.

Fortunately, this MSDN blog article has a lot of information that applies on connecting OpenLDAP to AD. But:

• The MSDN blog article is rather vague about how to set up LDAPS. There is a Technet article, but it refers to a specific MachineKey without explaining how to identify it.
• Figuring out the MachineKey for a certificate is described over here, on the Directory Services Team blog.
• Windows Server Core doesn't have Explorer to modify file/directory permissions, so you also need to know how to use icacls.
• Windows Server only passes back the client certificate, and unfortunately not the full certificate chain, so you end up needing to specify TLS_CACERT instead of just TLS_CACERTDIR.

The MSDN blog article primarily concerned about saslauthd access, though, rather than OpenLDAP connectivity, so this only gets us as far as having the infrastructure to query AD over LDAPS (which is valuable, though!).

If you put together several pieces of information out there, it's possible to now set up OpenLDAP with all the pieces we need. Inconveniently, public documentation seems to be largely out of date, and very fragmented. A lot of it refers to using slapd.conf, but that's been deprecated in favour of slapd.d. Yet oddly enough, users on openldap-technical seem to be perfectly okay with taking a slapd.conf, converting it to cn=config format, then shoving it in....

Notes from here on were gathered on CentOS 6.x - if you're following along, you may need to make allowances for your own distribution.

Pieces of documentation used to put together the OpenLDAP configuration (starting with slapd.conf, then converting to LDIF):

• Integrate Active Directory and OpenLDAP
  • Overall documentation on putting together slapd.conf, and particularly the pointer about subordinate.
• OpenLDAP Server
  • The Ubuntu OpenLDAP server tutorial, very useful for how to go about setting up the backend database, and part of the reason I selected HDB instead of BDB, which remains the default on CentOS.

Using these pages as a basis, I pulled together a slapd.conf looking roughly like this (if you want to use it as a basis for your own slapd.conf, you'll need to pay attention to all TODO entries):

You may have noticed that there's a reference to a microsoft.schema at the top. That's to help resolve some otherwise unresolved LDAP types. The file follows, with the first two entries from Apple, of all places, and the third being a localized (i.e. using a local Object Identifier) version of the standard NIS homeDirectory entry):

The rewrite/remap entries in slapd.conf for the AD LDAP proxy refer to a few non-standard properties. These are controlled via the Unix Attributes tab in AD, but in Windows Server 2012 R2, that's enabled by installing Identity Management for UNIX.

Once the slapd.conf is ready to go, it's pretty simple to generate a slapd.d directory:

This generates a series of LDIF files which can be massaged a bit more, which is where this documentation comes in very handy:

• LDAP for Rocket Scientists
  • The ZYTRAX guide to LDAP and LDIF was absolutely key to figuring out how to get slapd.d and LDIF working properly.

Combined with the above documentation, I generated three LDIF files (which, again, you'll need to change with your own information):

Importing the LDIFs is relatively straightforward, although cleaning up what we don't want is a good idea:

When you put your certificate files in /etc/openldap/certs, if you're running with SELinux (which you probably should), make sure that the file security contexts are correct, particularly for the CA certificate (otherwise, OpenLDAP won't be able to connect to your AD server):

Known Issues:

The connection to AD is unencrypted. I'm working on how to have it connect via ldaps://  instead. ldapsearch works without a problem, so it's puzzling why slapd fails. Further updates to come if/when I figure this out. Bitten by SELinux! See note above about restorecon.

Within the next few weeks, I expect that I'll be back on iOS. It's been a while since I've used it, but I've had experience with a number of different mobile operating systems at this point:

• Android (up through 4.2)
• DangerOS
• iOS (up through 5, occasional exposure up through 7)
• Windows Phone (7)
• Windows Mobile (up through 6.5)

Up until now, I've been using Android on a Samsung Note II. Which has gone reasonably well, but a few things have caused me to give up on Android:

• Replacing the home screen with Google Now.
• My device has been relatively stable... except for the past couple of months, it's been hard-locking repeatedly. Sometimes several times a day.
• I looked at a Note II because it was an attempt at trying something different with a mobile device. Unfortunately, several of the unique aspects just haven't worked out:
  • Stylus: I've barely used it.
  • Screen size: Larger than most phones, but not big enough to use for couch browsing.
  • Popup browser: Although active in Android 4.0, it was disabled in Android 4.2. Unfortunately, one of the things I liked.

In general, though, I've concluded that Android, at least at this point in time, isn't for me. At least some chunk of that is due to Google's business practices (Android is open... except when it's not). So back to a different walled garden for me - but at least it's one that doesn't pretend otherwise.

I'm... irritated. I've just spent several hours trying to figure out how to have Jetty use an SSL certificate generated wholly externally, all because the Jetty documentation is... less than complete.

The start line:

• One PKCS12 file with a X.509 certificate, its private key, and its chain of trust, encrypted with a password.
  • Generated from Windows Server, but the answer isn't "Don't use Windows". Yes, it's sometimes a perfectly valid answer. But not this time.

Things not obvious:

• jetty-ssl.xml refers to KeyStorePassword (easy enough), KeyManagerPassword (what's that?), and TrustStorePassword (huh?).
• The Jetty instructions for importing a PKCS12 file only talk about keystore passwords. But that leads to a situation where Jetty complains it can't read the necessary keys.
• If you'd like to leverage the default Java Jetty keystore (particularly if you have a secure system), you need to know the keystore's password. Which isn't provided anywhere... although an Internet search will tell you.
• After much digging around, maybe it's because for the key pair, the private key can have a password? Except having a blank password doesn't work, and setting it to the keystore's password doesn't work. Odd.
• The SSL documentation for Jetty 8 refers to a key pair password. What happened to that configuration key?
• So... somehow, "password" became "KeyManagerPassword". But what's the default password for that? It takes another Internet search to find that. Or maybe you're just supposed to look in the source code?

Spending several hours over this makes me not happy.

P.S. There's conveniently a command to import a PKCS12 file directly into a keystore. You can use it to rename the key in the process, since Jetty wants a key called... jetty.