Puppet uses bidirectional SSL to protect its client-server communication. All of the participants in a Puppet system must have valid, signed certificates and keys to talk to one another. This prevents agents from talking to rogue masters and it prevents nodes from spoofing one another. It also allows the master and agents to establish secure communication channels to prevent eavesdropping. Puppet comes with a built-in certificate authority (CA) to make the management of all the keys, certs, and signing requests fairly easy.

But what if you already have a large, established Kerberos infrastructure? You’re probably already generating and managing keys for all of your trusted hosts. Wouldn’t it be great to leverage your existing infrastructure and established processes instead of duplicating that effort with another authentication system?

Enter kx509. kx509 is a method for generating a short-lived X.509 (SSL) certificate from a valid Kerberos ticket. Effectively, a client can submit its Kerberos ticket to a trusted Kerberized CA (KCA), which then copies the principal name into the subject field of a new X.509 certificate and signs it with its own certificate. There’s really no trickery to it: if you trust the Kerberos ticket, and you trust the KCA, then you can trust the certificate generated by it. Sounds great, except that there is virtually no documentation on kx509, and even when you do get it running, there are a couple of issues that prevent it from working with Puppet out-of-the-box.

I wanted to figure out how to get it working with the least number of changes possible. To do this, I set up my own clean Kerberos and Puppet environment in a couple of VMs (a client and a server). I am documenting the whole process here for my own benefit, but maybe it will be useful to others.

The Setup

• 2 virtual machines:
  • server.example.com (192.168.100.2)
  • client.example.com (192.168.100.3)
• OS: Ubuntu Server 12.04.1 LTS 64-bit
• Kerberos: Heimdal 1.5.2
• Puppet: 2.7.11

I also set up a DNS server containing entries for the two hosts. Kerberos and Puppet are a lot easier to work with when they can use DNS, and it will be required for the kx509 stuff which we’ll see later.

I chose Heimdal because it has a built in KCA (and that’s what we run at work).

Heimdal

I knew from testing with the Kerberos system at work that Heimdal refuses to generate X.509 certs for principals that contain slashes (like ‘host/client.example.com@EXAMPLE.COM’), saying “Principal is not a user.” Since this would require changes to the source code to fix, I built Heimdal from source in my VMs:

  Install dependencies for Heimdal
% sudo apt-get install build-essential libdb-dev libncurses-dev libssl-dev comerr-dev

  Then download, build, and install it
% wget http://www.h5l.org/dist/src/heimdal-1.5.2.tar.gz
% tar -xvzf http://www.h5l.org/dist/src/heimdal-1.5.2.tar.gz
% cd heimdal-1.5.2
% ./configure
% make && sudo make install

That will install Heimdal with kx509 support to /usr/heimdal. At this point I added /usr/heimdal/{bin,sbin} to my PATH.

I don’t want to go too crazy with documenting every little thing I did—suffice it to say I created a new realm, added a few user principals, got kadmind up and working, generated host principals for each of the VMs and added them to their local keytabs, and tested that everything worked properly. The Heimdal documentation does a good job of explaining how to do all of that.

kx509

kx509 requires a small client program to talk to the KCA. Seeing as the last release of the program was from 2003, I built it from CVS:

% cvs -z3 -d:pserver:anonymous@kx509.cvs.sourceforge.net:/cvsroot/kx509 co -P kx509
% cd kx509/src
% vi configure.in
  - Remove section about 'checking for proper OPENSSL library directory'
  - Remove references to libk5crypto (it is not used in the source)
% autoconf
% CFLAGS="-DHAVE_HEIMDAL" ./configure --with-krb5=/usr/heimdal
% make && sudo make install

That will install kx509 and kxlist binaries to /usr/local/bin.

KCA

Configuring the KCA is actually pretty easy, but good luck finding any documentation on it. First I created a self-signed CA certificate (it could also be signed by another CA):

server# hxtool issue-certificate \
    --self-signed \
    --issue-ca \
    --generate-key=rsa \
    --subject="CN=CertificateAuthority,DC=example,DC=com" \
    --lifetime=10years \
    --certificate="FILE:/var/heimdal/ca.pem"

Then I used the CA cert to generate a template certificate:

server# hxtool issue-certificate \
    --generate-key=rsa \
    --subject='CN=${principal-name},DC=example,DC=com' \
    --ca-certificate=FILE:/var/heimdal/ca.pem \
    --certificate=FILE:/var/heimdal/template.pem

The template cert’s subject could be practically anything you want, as long as it has the string ‘${principal-name}‘.

Then I edited /var/heimdal/kdc.conf to enable the KCA and tell it where to find the CA and template certificates:

and restarted the KDC. I also added SRV entries to my DNS server such that:

client% host -t SRV _kca._udp.example.com
_kca._udp.example.com has SRV record 0 0 9878 server.example.com.

With all of that done, I was able to get X.509 certs with my regular user:

client% kinit jtl
jtl@EXAMPLE.COM's Password: 
client% kx509
client% kxlist
Service kx509/certificate
 issuer= /DC=com/DC=example/CN=CertificateAuthority
 subject= /DC=com/DC=example/CN=jtl
 serial=23EA808B52A466F59E475846DD5E691D27CBA770
 hash=ef2e0acf

but not from a principal with a slash in it:

client% kinit jtl/admin
jtl/admin@EXAMPLE.COM's Password:
client% kx509
Timed out waiting on KCA

To fix this problem, I modified kdc/kx509.c from the Heimdal source as follows, and rebuilt the code:

Then it worked as you’d expect:

client% kinit jtl/admin
jtl/admin@EXAMPLE.COM's Password:
client% kx509
client% kxlist
Service kx509/certificate
 issuer= /DC=com/DC=example/CN=CertificateAuthority
 subject= /DC=com/DC=example/CN=jtl/admin
 serial=7F1C4AF8CA119B23069D2F6E098FA3F3D0F03142
 hash=b0b84785

Puppet Master

For the Puppet part of things, the first thing I did was to configure the master to use Apache Passenger (apt-get install puppetmaster-passenger), and make sure everything worked with the built-in CA. Best to start off with a known working state.

Then I created a certificate for the Puppet HTTP server:

server# rm -rf /var/lib/puppet/ssl
server# mkdir -p /var/lib/puppet/ssl/server
server# hxtool issue-certificate \
    --generate-key=rsa \
    --type="https-server" \
    --subject="CN=server.example.com,DC=example,DC=com" \
    --hostname="server.example.com" \
    --hostname="puppet" \
    --hostname="puppet.example.com" \
    --ca-certificate=FILE:/var/heimdal/ca.pem \
    --certificate=FILE:/var/lib/puppet/ssl/server/server.example.com.pem
server# openssl x509 -in /var/heimdal/ca.pem -out /var/lib/puppet/ssl/server/ca.pem

Here the cert’s CN should match the FQDN of the server, and you can optionally list other names by which the server can be accessed as --hostname parameters. Notice I saved the certificate portion of ca.pem next to my server cert as well.

I modified /etc/apache2/sites-enabled/puppetmaster to reflect these certificate locations, and also to remove the certificate revocation list line (kx509 certs have short expiration dates, so there is no need to revoke them). Here are the relevant SSL directives from the configuration file:

Before I restarted the Apache server, I disabled the internal Puppet CA by adding ca = false to the [master] section of /etc/puppet/puppet.conf. I also changed VALID_CERTNAME = /\A[ -.0-~]+\Z/ to VALID_CERTNAME = /\A[[:print:]]+\Z/ in /usr/lib/ruby/1.8/puppet/ssl/base.rb to workaround Puppet bug #15561. Finally, I modified the SSL subject matching code in /usr/lib/ruby/1.8/puppet/network/http/rack/rest.rb to properly parse out the hostname from my kx509 certificates:

Before: dn_matchdata = dn.match(/^.*?CN\s*=\s*(.*)/)
After: dn_matchdata = dn.match(/^.*?CN\s*=\s*host\/(.*)/)

Then I created a basic site manifest:

With all of that done, I could test the Puppet agent.

Puppet Agent

Now for the easy stuff. As I mentioned, each client should have a host key stored in its keytab. The creation of that looks something like:

client# kadmin
kadmin> add --random-key host/client.example.com
kadmin> ext host/client.example.com

Each client should also have the KCA’s certificate stored at /var/lib/puppet/ssl/certs/ca.pem. That only has to be done once.

The following should be done every time the Puppet agent runs:

1. Initialize a Kerberos ticket for the host principal:

   client# kinit -t /etc/krb5.keytab host/client.example.com
   client# klist
   Credentials cache: FILE:/tmp/krb5cc_0
           Principal: host/client.example.com@EXAMPLE.COM
   
     Issued                Expires               Principal
   Oct 25 15:45:48 2012  Oct 26 01:45:48 2012  krbtgt/EXAMPLE.COM@EXAMPLE.COM
   

2. Convert the ticket to an X.509 certificate and save the certs to a place Puppet expects to find them:

   client# kx509
   client# kxlist -o /var/lib/puppet/ssl/certs/client.example.com.pem
   client# kxlist -o /var/lib/puppet/ssl/private_keys/client.example.com.pem
   client# openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/client.example.com.pem | grep -A3 Validity
           Validity
               Not Before: Oct 24 19:48:59 2012 GMT
               Not After : Oct 26 05:45:48 2012 GMT
           Subject: DC=com, DC=example, CN=host/client.example.com
   

3. Run the Puppet agent and be amazed that it works!

   client# puppet agent --test
   info: Caching catalog for client.example.com
   info: Applying configuration version '1351194662'
   notice: I am client.example.com
   notice: /Stage[main]//Node[default]/Notify[I am client.example.com]/message: defined 'message' as 'I am client.example.com'
   notice: Finished catalog run in 0.03 seconds
   

Of course, these steps (kinit, kx509, kxlist) can and probably should be scripted. If I were actually deploying this in production, I would probably write a small wrapper around puppet agent to do it.

Conclusion

I guess that looks like a lot of stuff, but if you look again, and remove all of the setup stuff, it’s really just three small source code changes and a couple of commands that can be scripted. If you’re interested in this, it’s probably because you already have a lot invested in Kerberos, so there won’t be too much to the setup. You may already even have kx509 configured for other applications. Either way, I think for those people, doing a little bit of work like this upfront may be desirable compared to managing a separate CA for Puppet. Or, I don’t know…puppet cert --sign really isn’t that hard to run.