Unwinding the Stack

Benchmarking Duplog

May 20th, 2013

In my last post I introduced Duplog, a small tool that basically forms the glue between rsyslog, RabbitMQ, Redis, and Splunk to enable a highly available, redundant syslog service that deduplicates messages along the way. In order for Duplog to serve the needs of a production enterprise, it will need to perform well. I fully expect the deduplication process to take a toll, and in order to find out how much, I devised a simple benchmark.

One one side, fake syslog generators pipe pairs of duplicate messages into the system as fast as they can. On the other, a process reads the messages out as fast as it can. Both sides report the rate of message ingestion and extraction.

The Details

3 KVM virtual machines on the same host each with the same specs:
- 1 x Intel Core i5 2400 @ 3.10 GHz
- 1 GB memory
- Virtio paravirtualized NIC
OS: Ubuntu Server 12.04.1 LTS 64-bit
RabbitMQ: 2.7.1
Redis: 2.2.12
Java: OpenJDK 6b27

The Results

(My testing wasn’t very scientific so take all of this with a grain of salt.)

Duplog Benchmark

The initial results were a little underwhelming, pushing about 750 messages per second through the system. I originally expected that hashing would be the major bottleneck, or the communication with the Redis server, but each of those processes were sitting comfortably on the CPU at about 50% and 20% usage, respectively. It turned out that the RabbitMQ message brokers were the source of the slow performance.

I began trying many different settings for RabbitMQ, starting by disabling the disk-backed queue, which made little difference. In fact, the developers have basically said as much: “In the case of a persistent message in a durable queue, yes, it will also go to disk, but that’s done in an asynchronous manner and is buffered heavily.”

So then I changed the prefetch setting. Rather than fetching and acknowledging one message at a time, going back and forth over the network each time, the message consumers can buffer a configurable number of messages at a time. It is possible to calculate the optimum prefetch count, but without detailed analytics handy, I just picked a prefetch size of 100.

That setting made a huge difference, as you can see in the histogram. Without spending so much time talking to RabbitMQ, consumers were free to spend more time calculating hashes, and that left the RabbitMQ message brokers more free to consume messages from rsyslog.

Another suggestion on the internet was to batch message acknowledgements. That added another modest gain in performance.

Finally, I tried enabling an unlimited prefetch count size. It is clear that caching as many messages as possible does improve performance, but it comes at the cost of fairness and adaptability. Luckily, neither of those characteristics is important for this application, so I’ve left it that way, along with re-enabling queue durability, whose performance hit, I think, is a fair trade off for message persistence. I also reconfigured acknowledgements to fire every second rather than every 50 messages. Not only does that guarantee that successfully processed messages will get acknowledged sooner or later, it spaces out ACKs even more under normal operation, which boosted the performance yet again to around 6,000 messages per second.

So is 6,000 messages per second any good? Well, if you just throw a bunch of UDP datagrams at rsyslog (installed on the same servers as above), it turns out that it can take in about 25,000 messages per second without doing any processing. It is definitely reasonable to expect that the additional overhead of queueing and hashing in Duplog will have a significant impact. It is also important to note that these numbers are sustained while RabbitMQ is being written to and read from simultaneously. If you slow down or stop message production, Duplog is able to burst closer to 10,000 messages per second. The queueing component makes the whole system fairly tolerant of sudden spikes in logging.

For another perspective, suppose each syslog message averages 128 bytes (a reasonable, if small, estimate), then 6,000 messages per second works out to 66 GB per day. For comparison, I’ve calculated that all of the enterprise Unix systems under my group at UMD produce only 3 GB per day.

So as it stands now I think that Duplog performs well enough to work in many environments. I do expect to see better numbers on better hardware. I also think that there is plenty of optimization that can still be done. And in the worst case, I see no reason why this couldn’t be trivially scaled out: divide messages among multiple RabbitMQ queues to better take advantage of SMP. This testing definitely leaves me feeling optimistic.

Tags: deduplication, duplog, java, rabbitmq, redis, splunk, syslog
Posted in Programming, System Administration | No Comments »

Overengineering Syslog: Redundancy, High Availability, Deduplication, and Splunk

May 19th, 2013

I am working on a new Splunk deployment at work, and as part of that project, I have to build a centralized syslog server. The server will collect logs from all of our systems and a forwarder will pass them along to Splunk to be indexed. That alone would be easy enough, but I think that logs are too important to leave to just one syslog server. Sending copies of the log data to two destinations may allow you to sustain outages in half of the log infrastructure while still getting up-to-the-minute logs in Splunk. I think duplicating log messages at the source is a fundamental aspect of a highly available, redundant syslog service when using the traditional UDP protocol.

That said, you don’t want to have Splunk index all of that data twice. That’ll cost you in licenses. But you also don’t want to just pick a copy of the logs to index—how would you know if the copy you pick is true and complete? Maybe the other copy is more complete. Or maybe both copies are incomplete in some way (for example, if routers were dropping some of those unreliable syslog datagrams). I think the best you can do is to take both copies of the log data, merge them together somehow, remove the duplicate messages, and hope that, between the two copies, you’re getting the complete picture.

I initially rejected the idea of syslog deduplication thinking it to be too complicated and fragile, but the more I looked into it, the more possible it seemed. When I came across Beetle, a highly available, deduplicating message queue, I knew it would be doable.

Beetle Architecture

Beetle itself wouldn’t work for what I had in mind (it will deduplicate redundant messages from single sources; I want to deduplicate messages across streams from multiple sources), but I could take its component pieces and build my own system. I started hacking on some code a couple days ago to get messages from rsyslog to RabbitMQ and then from RabbitMQ to some other process which could handle deduplication. It quickly turned into a working prototype that I’ve been calling Duplog. Duplog looks like this:

Duplog Architecture

At its core, Duplog sits and reads messages out of redundant RabbitMQ queues, hashes them, and uses two constant-time Redis operations to deduplicate them. RabbitMQ makes the whole process fairly fault tolerant and was a great discovery for me (I can imagine many potential use cases for it besides this). Redis is a very flexible key-value store that I’ve configured to act as a least-recently-used cache. I can throw hashes at it all day and let it worry about expiring them.

One important design consideration for me was the ability to maintain duplicate messages within a single stream. Imagine you have a high-traffic web server. That server will probably be logging many identical HTTP requests at the same time. Those duplicates are important to capture in Splunk for reporting. My deduplication algorithm maintains them.

Looking at the architecture again, you will see that almost everything is redundant. You can take down almost any piece and still maintain seamless operation without dealing with failovers. The one exception is Redis. While it does have some high availability capability, it relies on failover which I don’t like. Instead, I’ve written Duplog to degrade gracefully. When it can’t connect to Redis, it will allow duplicate messages to pass through. A few duplicate messages isn’t the end of the world.

Fork Duplog on GitHub

Feel free to play around with the code, but know that it is definitely a prototype. For now I’ve been working with the RabbitMQ and Redis default settings, but there is probably a lot that should be tuned, particularly timeouts, to make the whole system more solid. I also intend to do some benchmarking of the whole stack (that’ll probably be the next post), but initial tests on very modest hardware indicate that it will handle thousands of messages per second without breaking a sweat.

Tags: deduplication, duplog, java, rabbitmq, redis, splunk, syslog
Posted in Programming, System Administration | No Comments »

Protecting Puppet with Kerberos

October 25th, 2012

Puppet uses bidirectional SSL to protect its client-server communication. All of the participants in a Puppet system must have valid, signed certificates and keys to talk to one another. This prevents agents from talking to rogue masters and it prevents nodes from spoofing one another. It also allows the master and agents to establish secure communication channels to prevent eavesdropping. Puppet comes with a built-in certificate authority (CA) to make the management of all the keys, certs, and signing requests fairly easy.

But what if you already have a large, established Kerberos infrastructure? You’re probably already generating and managing keys for all of your trusted hosts. Wouldn’t it be great to leverage your existing infrastructure and established processes instead of duplicating that effort with another authentication system?

Enter kx509. kx509 is a method for generating a short-lived X.509 (SSL) certificate from a valid Kerberos ticket. Effectively, a client can submit its Kerberos ticket to a trusted Kerberized CA (KCA), which then copies the principal name into the subject field of a new X.509 certificate and signs it with its own certificate. There’s really no trickery to it: if you trust the Kerberos ticket, and you trust the KCA, then you can trust the certificate generated by it. Sounds great, except that there is virtually no documentation on kx509, and even when you do get it running, there are a couple of issues that prevent it from working with Puppet out-of-the-box.

I wanted to figure out how to get it working with the least number of changes possible. To do this, I set up my own clean Kerberos and Puppet environment in a couple of VMs (a client and a server). I am documenting the whole process here for my own benefit, but maybe it will be useful to others.

The Setup

2 virtual machines:
- server.example.com (192.168.100.2)
- client.example.com (192.168.100.3)
OS: Ubuntu Server 12.04.1 LTS 64-bit
Kerberos: Heimdal 1.5.2
Puppet: 2.7.11

I also set up a DNS server containing entries for the two hosts. Kerberos and Puppet are a lot easier to work with when they can use DNS, and it will be required for the kx509 stuff which we’ll see later.

I chose Heimdal because it has a built in KCA (and that’s what we run at work).

Read the rest of this entry »

Tags: kerberos, puppet
Posted in System Administration | No Comments »

SuperSync

July 26th, 2012

I wrote in the about me blurb on this blog that I like writing little programs for myself. One of the programs I’m most proud of is called SuperSync.

Back in college when I started developing an interest in music, I got in the habit of only acquiring losslessly encoded files. FLACs mostly. It wasn’t long before my collection outgrew what I could store on my iPod. So I hacked together a little script which I called “Sync” to encode my music files to something smaller, like Ogg Vorbis. I wrote it in Java because that’s what I knew best at the time, and for the most part, it just worked. It kept a flat database of files and timestamps to know what to sync to the iPod without reencoding everything every time.

But unfortunately, as my music collection grew, there were times, like when MusicBrainz would have a minor update for all of my files, which would make Sync think that everything needed to be resynced again. It got to to a point where some syncs would take a week, doing one file at a time.

It got me thinking: I have 10 CPU cores in my house. If I could get them all working together on the problem, I could get those long syncs down to a day or two. And thus SuperSync was born.

Still written in Java, SuperSync adds a distributed client/server architecture and nice GUI over top of Sync. The program takes the same flat database, and when it sees a new or updated file in the source directory, it copies it to the destination directory. If the file is a FLAC, it broadcasts a conversion request to the network. Any server can then respond if it has a free CPU. The server reads the file from my network share and sends the encoded file back to the client where it gets written out to the destination. The whole setup relies on having a consistent global namespace for the source collection. In my case, all of my systems can access my fileserver mounted at /nest in the same way. I can’t imagine many people have such a setup, so I don’t think a formal SuperSync release would be worthwhile.

In any case, the process looks something like this in action:

At the end of the sync, the program can optionally read a song log from Rockbox and scrobble it to Last.fm.

I’m also really proud of the way SuperSync is written. I spent a lot of time upfront to define clean interfaces using good object-oriented style. Feel free to checkout the source code if you’re into that sort of thing. Just ask me first if you want to use any of it.

Now the times are changing, and with Subsonic allowing me to stream music to my phone, I haven’t had to sync my music as much recently. But my iPod still has its purposes, so I’m glad I have SuperSync to let me take my whole collection with me.

Tags: java, music
Posted in Programming | No Comments »

Retirement: Defined Benefit or Defined Contribution?

July 15th, 2012

Specifically, should employees of University System of Maryland institutions participate in the State Retirement and Pension System (SRPS) or the Optional Retirement Program (ORP)? That was one of the questions I had to answer for myself as I prepare to start a new job at the University of Maryland. The SRPS is a defined-benefit pension plan and the ORP is a defined-contribution 401(a)-like plan. The default is the SPRS, and it seems like they do everything they can to steer you to it (I suspect because your contributions are how the State funds current retirees), but is it a better deal?

First, some details: the SRPS requires employees to contribute 7% of their salary to the plan. In return, after 10 years of service, you can retire at age 65 and receive a monthly allowance following this formula:

$\frac{0.015\:\times\:\textrm{salary}\:\times\:\textrm{years of service}}{12}$

By contrast, the ORP is simple: the University will contribute a flat 7.25% of your salary to your choice of Fidelity or TIAA-CREF, and you can invest it however you want. The money is immediately vested. Additionally, you can take the 7% that you would have had to contribute to the SRPS and invest it on your own in a supplemental retirement plan or IRA. That’s a total of 14.25% of your salary going towards your retirement every year…comfortably within the 10-15% that experts recommend.

For me, considering it doesn’t vest until 10 years of service, the SRPS was right out. But as an experiment, I wanted to know which would be the better option if I worked for Maryland for 10 years. (The SRPS does allow you to withdraw your contributions compounded annually at 5% interest if you terminate employment before 10 years, but then you wouldn’t get the benefit of the State’s contribution to the plan, and you can almost certainly do better than 5% annually in the long-run by investing in a mix of stocks and bonds.)

I plugged in all of my numbers to the SRPS formula, and calculated an estimated withdraw rate for the ORP supposing a realistic inflation-adjusted growth rate. The results were clear: the ORP could provide me with about twice as much money during retirement. With results like those, I was curious whether the SRPS would be a good deal for anyone and under what circumstances that would be.

So I whipped up a little program to do the calculations for me. It has sliders for each of the input variables so the results can easily be compared for a wide variety of circumstances. The program works with your current salary and inflation-adjusted rates of return to give you a picture of what sort of spending power in today’s dollars you would have during retirement.

Consider a 35-year-old who makes $80,000 per year. If they expect to work for Maryland for 10 years and think they can earn around 5% in the market after adjusting for inflation (that’s a real annual return of 8% if you assume an average of 3% inflation), and intend to retire at 65 and expect to need income for 30 years in retirement, the ORP just barely comes out on top:

Indeed, that seems to be the turning point. Any older and you won’t have enough time to let those returns compound, and if you are any more risk-averse, then you won’t be able to generate the returns needed to outpace the pension system. In those cases, the SRPS would be a better choice for you, but only if you are in it for the long haul. You’d be wasting valuable investing time if you join the SRPS and leave before your contributions vest. Otherwise, read a book or two on investing, and do it yourself with the ORP.

But don’t take my word for it; do the math or put your numbers into my program and see what the better choice would be for you:

Run the Maryland Retirement Comparison Tool
Requires Java 5 or higher and Windows, MacOS X, or Linux

Of course, I make no guarantees that my program is accurate, but you can get the source code and check it out for yourself. Also consider investment risks and other factors such as plan benefits carefully.

Tags: java, retirement
Posted in Personal, Programming | No Comments »

Mounting Encrypted ZFS Datasets at Boot

May 31st, 2012

When ZFS encryption was released in Solaris 11 Express, I went out and bought four 2 TB drives and moved all of my data to a fresh, fully-encrypted zpool. I don’t keep a lot of sensitive data, but it brings me peace of mind to know that, in the event of theft or worse, my data is secure.

I chose to protect the data keys using a passphrase as opposed to using a raw key on disk. In my opinion, the only safe key is one that’s inside your head (though the US v. Fricosu case has me reevaluating that). The downside is that Solaris will ignore passphrase-encrypted datasets at boot.

The thing is, I run several services that depend on the data stored in my encrypted ZFS datasets. When Solaris doesn’t mount those filesystems at boot, those services fail to start or come up in very weird states that I must recover from manually. I would rather pause the boot process to wait for me to supply the passphrase so those services come up properly. Fortunately this is possible with SMF!

All of the services I am concerned about depend on, in one way or another, the svc:/system/filesystem/local:default service, which is responsible for mounting all of the filesystems. That service, in turn, depends on the single-user milestone. So I just need to inject my own service between the single-user milestone and the system/filesystem/local service that fails when it doesn’t have the keys. That failure will pause the boot process until it is cleared.

I wrote a simple manifest that expresses the dependencies between single-user and system/filesystem/local:

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='nest'>

<service
name='system/filesystem/nest'
type='service'
version='1'>

<create_default_instance enabled='true' />
<single_instance />

<dependency
name='single-user'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/milestone/single-user' />
</dependency>

<dependent
name='nest-local'
grouping='require_all'
restart_on='none'>
<service_fmri value='svc:/system/filesystem/local' />
</dependent>

<exec_method
type='method'
name='start'
exec='/lib/svc/method/nest start'
timeout_seconds='60' />

<exec_method
type='method'
name='stop'
exec=':true'
timeout_seconds='60' />

<property_group name='startd' type='framework'>
<propval name='duration' type='astring' value='transient' />
</property_group>

<stability value='Unstable' />

<template>
<common_name>
<loctext xml:lang='C'>Load key for 'nest' zpool</loctext>
</common_name>
</template>
</service>

</service_bundle>

and a script at /lib/svc/method/nest that gets called by SMF:

#!/sbin/sh

. /lib/svc/share/smf_include.sh

case "$1" in
'start')
if [ $(zfs get -H -o value keystatus nest) != "available" ]; then
echo "Run '/usr/sbin/zfs key -lr nest && /usr/sbin/svcadm clear $SMF_FMRI'" | smf_console
exit $SMF_EXIT_ERR_FATAL
fi
;;

*)
echo "Usage: $0 start"
exit $SMF_EXIT_ERR_CONFIG
;;
esac

exit $SMF_EXIT_OK

The script checks whether the keys are available, and if not, prints a helpful hint to the console. The whole thing looks something like this at boot:

SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Hostname: falcon

Run '/usr/sbin/zfs key -lr nest && /usr/sbin/svcadm clear svc:/system/filesystem/nest:default'
May 30 14:31:06 svc.startd[11]: svc:/system/filesystem/nest:default: Method "/lib/svc/method/nest start" failed with exit status 95.
May 30 14:31:06 svc.startd[11]: system/filesystem/nest:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)

falcon console login: jlee
Password: 
falcon% sudo -s
falcon# /usr/sbin/zfs key -lr nest && /usr/sbin/svcadm clear svc:/system/filesystem/nest:default
Enter passphrase for 'nest': 
falcon#

When I get to the console shell, I can just copy and paste the command printed by the script. Once the service failure is cleared, SMF continues the boot process normally and all of my other services come up exactly as I’d expect.

No, it’s not very pretty, but I’d rather have a little bit of manual intervention during the boot process for as infrequently as I do it, than to have to clean up after services that come up without the correct dependencies. And with my new homemade LOM, it’s not too much trouble to run commands at the console, even remotely.

Tags: smf, solaris, zfs
Posted in System Administration | No Comments »

My Homemade LOM

May 9th, 2012

In one of the final classes of my CS master’s program, Embedded Computing, we were required to complete a semester project of our choosing involving embedded systems. Like in previous semester projects, I wanted to do something that I would actually be able to use after the class ended.

This time around I chose to build a lights-out manager for my Sun Ultra 24 server (which this blog is hosted on). With a LOM I can control the system’s power and access its serial console so I will be able to perform OS updates remotely, among other things.

Since I already owned one and I didn’t want to spend a lot of money, I chose to develop the project on top of the Arduino platform. I like the size of the Arduino, and the availability of different shields to minimize soldering. It’s also able to be powered by USB, which is perfect because the Ultra 24 has an internal USB port that always supplies power.

To make things a little more challenging for myself (because the Arduino is pretty easy on its own), I chose to implement a hardware UART to communicate with the Ultra 24′s serial port. Specifically, I chose to use the Maxim MAX3110E SPI UART and RS-232 transceiver. Great little chip.

For communication with the outside world, I bought an Ethernet shield from Freetronics. It’s compatible with the official Arduino Ethernet shield, but includes a fix to allow the network module to work with other SPI devices (such as my UART) on the same bus. I started to implement the network UI using Telnet, but after realizing I would have to translate the serial console data from VT100 to NVT, I switched to Rlogin, which is like Telnet, but assumes like-to-like terminal types.

Lastly, for controling the system’s power, I figured out how to tap into the Ultra 24′s power LED and switch. Using the LED, I can check whether the system is on or off, and using the switch circuit and a transistor, I can power the system on and off. I managed to do this without affecting the operation of the front panel buttons/LEDs.

I’ll spare you all of the implementation details (if you’re interested, you can read my report). Suffice it to say, the thing works as well as I could have imagined. Here is a screenshot of me using the serial console on my workstation:

From my research, the serial and power motherboard headers are the same on most modern Intel systems, so this LOM should work on more than just an Ultra 24. If you want to build one of your own, my code is available on GitHub and the hardware schematic is in the report I linked to above.

Tags: arduino, hardware
Posted in Programming, System Administration | No Comments »

ITT I Try to Justify Upgrading My Camera

March 21st, 2012

I’ve said it before, and I’ll say it again: I am not a photographer. I am just a guy who enjoys taking high quality photos of the places I go and the things I enjoy doing. A couple years ago I picked up a Canon EOS Rebel T2i, a sharp 15-85mm f/3.5-5.6 zoom lens, and everyone’s favorite, the 50mm f/1.4, before a two-week trip to Japan. In the time since, I’ve taken thousands of photos and really enjoyed learning the technology and different photographic techniques. The T2i is really impressive, and is still a better camera than I am a photographer. That said, I’ve always known that I would want to upgrade at some point. The T2i is too small for my large hands to hold securely or comfortably, and I am not really fond of its plasticky build.

Meanwhile, I went through a couple hundred photos from a recent trip to Belgium with my girlfriend, and though I really like what I was able to capture, and I’m satisfied with the way they turned out, I can’t help but notice a distinct point-and-shoot quality about them. Obviously they can’t all be winners, but in the spur of the moment, it’s all too easy for me to let my zoom lens do all the work at the expense of good composition.

Looking back at my photos, I noticed that I like the ones I’ve taken with my 50mm lens the most. The fixed focal length forces me to really think about how I want the photo to look, and I have to move my feet in order to get it. Its large aperture allows me to take photos at a lower ISO level in low light, which means less sensor noise. And its shallow depth-of-field potential stimulates my creativity. I think Kai from DigitalRev explains it best:

However, with my T2i’s APS-C-sized sensor, the field-of-view of the 50mm lens is more like an 80mm lens on a full-frame sensor, which is really narrow for indoor shooting. I would have to stand all the way across the room to get anything more than just a headshot. So I started to look at what equivalent “normal” lenses I could get for my T2i.

I settled on the Canon EF 28mm f/1.8, and I’m generally very happy with it. It’s sharp, fast, and well made. However, the difference in the depth-of-field between the 50mm and 28mm is very noticeable, even at f/1.8. It’s much harder to get that nicely blurred background unless you’re within a couple feet of your subject with the 28mm. That is just the nature of wider angle lenses.

Meanwhile, Canon just released the brand-new full-frame EOS 5D Mark III and prices for the three-year-old 5D Mark II are dropping. I never would have considered getting a full-frame camera before (that’s just silly—I’m an amateur and full-frame cameras are for the pros, right?), but the prices are not much more than the 7D now. The 5D Mark II is still a great camera. One of the few things people complain about is its poor auto-focus performance. Fortunately, I learned early to use center-point and back-button focus, and I don’t shoot sports, so I couldn’t care less about auto-focus performance.

The 5D Mark II is better in almost every way compared to my T2i. It’s very well built and will fit my hands, so I’ll enjoy holding and using it. And as a bonus, the full-frame sensor will enable me to get the depth-of-field I’m used to on my 50mm lens with the field-of-view similar to my 28mm, which can open a whole new world of possibilities for me.

All of this is just to say: I think I’m going to upgrade to a 5D Mark II. I think now is the time. I am serious about improving my photography and I think sticking with only prime lenses for a while will help. It takes a huge variable out of the equation (focal length) so hopefully I can concentrate on the more important things. In fact, I’ve already sold my zoom lens. Between the money from the sale of my lens, the money I should be able to get from selling my T2i, and some credit card cash back, I will be able to pick up the 5D Mark II for a good price. It will be an early graduation present to myself. And if it turns out not to be right for me, photo gear keeps its value pretty well, so I can always sell it.

I’m not crazy for wanting to upgrade, am I?

UPDATE 05/09/2012: I managed to pick up a factory refurbished 5D Mark II during Canon’s friends and family sale for a ridiculously good price ($1596 after taxes and shipping). Canon’s refurbs, if you don’t know, are like new and mine was no exception. I sold my T2i and 15-85mm lens for $1000 after eBay and Paypal took their cut, and I allowed myself to buy a refurbed EF 100mm f/2 USM lens and Speedlite 580EX II during the same Canon sale.

And the result? Well, I feel like I’m already starting to make some good improvements. Shooting with primes often forces me to think more creatively about composition and perspective to get the shot I want. Then taking the shots into Lightroom helps me do minor white balance and color corrections to really make them pop. And finally, the flash is just a lot of fun.

Posted in Photography | 3 Comments »

Setting Up My Retirement Investments with TIAA-CREF

December 5th, 2011

This is a long post mostly so I can look back and remember what I did, but I’m posting it publicly in case anyone is in a similar position and could benefit from my research.

About a year ago I became eligible for my employer’s retirement plan, and at the time I was completely overwhelmed by this whole new world of mutual fund investments. Not only was I given a large selection of funds to choose from, but I was also given the choice of providers: Fidelity and TIAA-CREF. I did a little research, but couldn’t really decide what to do, so I just selected the defaults, Fidelity with the 2050 target fund, and let it sit.

Then, over the Thanksgiving weekend, I started reading the long-term investment thread at Something Awful where it quickly became apparent that (1.) choosing assets to invest in doesn’t have to be that hard, (2.) expense ratios (what a fund costs) matter a lot, and (3.) high-cost actively managed funds in general don’t perform better than their lower-cost index-based equivalents.

So I took another look at what was available to me in Fidelity, and I didn’t like what I saw. Two stock index funds, each tracking the S&P 500; one REIT index fund, and one bond index fund. The rest of the funds available were actively managed with expenses around 0.8% or more. Even the few Vanguard funds available to me through Fidelity seemed obscure and expensive.

Then I remembered about TIAA-CREF. I pulled up a list of their offerings, and it made me a little more optimistic. Not only are their options cheaper overall, but they also have index funds available for more market segments. Really the worst thing that could be said after an initial look is that they could have more international representation, but it would turn out that, for me, it doesn’t really matter.

I started reading TIAA-CREF’s literature and taking their asset allocation (AA) quizzes, you know the ones. Well, considering I’m only 25 and I have a good 40 years until retirement, I would consider myself more willing to take on risk than someone a little older. Their quiz would have me put 86% into equities, 9% in real estate, and 5% in bonds. Why 9% real estate? Why not ten, or eight? And can 5% of your portfolio really affect anything?

At this point, I should note that TIAA’s Real Estate Account (TREA) is a unique investment vehicle among providers in that it invests directly in commercial real estate, rather than in companies that manage real estate as the more risky REIT funds do. There is really nothing else like it, and that made it hard to reconcile with other popular AA tips I found on the internet, such as having a simple three-fund portfolio. I wanted to know whether I should include real estate, if I should follow TIAA-CREF’s advice for AA, why they chose the numbers they did, and also why the Bogleheads advocate slightly more conservatives allocations. I may be young, but I don’t want to turn the risk up to 11 just because I can. I want managed risk that I can understand.

So I picked up a copy of The Intelligent Asset Allocator by William Bernstein, hoping it would shed some light on my questions. To be honest, I was hoping it would have The Answer in it, that it would point me to The Optimal Asset Allocation. Thankfully, it did a whole lot better than that. It stated in no uncertain terms that there is no such thing as an optimal asset allocation, and anyone claiming to have one is conning you. The book started off by providing useful metrics for measuring the performance (in terms of annual return) and risk (in terms of standard deviation) of different asset classes. It was an easy, quick read that gave me a few techniques for understanding the behavior of different asset classes, and how they’ve historically interacted with each other in portfolios. Best of all, it gave me the confidence to tackle the same sort of research on my own.

Read the rest of this entry »

Tags: fidelity, investment, retirement, tiaa-cref
Posted in Personal | 3 Comments »

The Solaris 11 Experience So Far

November 15th, 2011

I have a system (a zone on which this blog is hosted) that has been running the same installation of Solaris since 11/11/2009, starting with OpenSolaris 2009.06. In the time since, it has seen every public build of OpenSolaris, then OpenIndiana, and finally Solaris 11 Express. Now, exactly two years later, I’ve updated it to Solaris 11 11/11, and I’d like to share my experience so far.

The update itself did not go smoothly. I was sitting at Solaris 11 Express SRU 8 and thought, like every update I’ve done in the past, that I could just run pkg image-update. Silly me, because when I did and then rebooted, the kernel panicked. No big deal, that’s what boot environments are for. I reverted to the previous boot environment and found some helpful documentation that told me to do exactly what I just did. It turns out that there is no way to update to SRU 13 using the support repositories because they already contain the Solaris 11 11/11 packages, and pkg tries to pull some of them in. And there is no way to update just pkg because the ips-consolidation prevents it, and trying to update the ips-consolidation pulls the entire package which breaks everything just the same. In short, Oracle bungled it. The only way to update to SRU 13 that I could see was to download the SRU 13 repository ISO from My Oracle Support and set up a local repository. Once I was on SRU 13, I could continue with the update to the 11/11 release. But there were more surprises in store for me.

First, it looks like pkg decided to start enforcing consistent attributes on files shared by multiple packages. Fine, I can understand that. As a result, I had to remove a lot of my custom packages (mostly from spec-files-extra) which I’ll have to rebuild. Second, pkg decided it doesn’t like the opensolaris.org packages anymore so I had to uninstall OpenOffice.org. Also fair enough.

Happily, after that, the updates got applied successfully and the system rebooted into the 11/11 release. Next came the zone updates. When I did the normal zoneadm -z foo detach && zoneadm -z foo attach -u deal, I was told I had to convert my zones to a new ZFS structure which more closely matches the global zone. The script /usr/lib/brand/shared/dsconvert actually worked flawlessly and the updated zones came up fine.

Unfortunately I couldn’t SSH into my zones because my DNS server didn’t know where they were. It seems that with the updated networking framework, DHCP doesn’t request a hostname anymore. (/etc/default/dhcpagent still says inet <hostname> can be put in /etc/hostname.<if> to request the hostname.) I found that you can create an addr object that requests a hostname with ipadm create-addr -T dhcp -h <hostname> <addrobj>, but NWAM pretty much won’t let you create or modify anything with ipadm, and there were no options for requesting hostnames with nwamcfg. As a result, I had to disable NWAM (netadm enable -p ncp DefaultFixed) and then I could set up the interface with ipadm. Why doesn’t Solaris request hostnames by default? Not very “cloud-like” if you ask me.

I have to say, I’m impressed by the way global zones and non-global zones are linked in the new release. Zone updates were an obvious shortcoming of previous releases. We’ll see how well it works when Solaris 11 Update 1 comes out.

What else…I lost my ability to pfexec to root. Oracle removed the “Primary Administrator” profile for security reasons so I had to install sudo. Not a big deal, I just wish they had said something a little louder about it.

Also, whatever update to pkg happened, it wiped out my repositories under /var/pkg. I had to restore them from a snapshot. Bad Oracle!

I’m also a little confused about some of the changes to the way networking settings are stored. For example, when I first booted the global zone, I found that my NFSv4 domain name was reset by NWAM. I set it to what it should be with sharectl set -p nfsmapid_domain=thestaticvoid.com nfs, but is that going to be overwritten again by NWAM? Also, the name resolver settings are now stored in the svc:/network/dns/client:default service, and according to the documentation, DHCP will set the service properties properly, but I have yet to see this work.

And the last problem I’ll mention is that the update removed my virtual consoles. I had to install the virtual-console package to restore them.

Overall, I’m happy that I was at least able to update to the latest release. Oracle could have cut off any update path from OpenSolaris. However, the update should have been a lot smoother. It doesn’t speak well of future updates when I can’t even update from one supported release (SRU 8) to another. I also wish Oracle were more open about upcoming changes (as in, having more preview releases or, dare I say it, opening development the way OpenSolaris was). Even to me, a long time pre-Solaris 11 user, the changes to zones and networking are huge in this release, and I would rather have not been so surprised by them.

Tags: network, nwam, packages, solaris
Posted in System Administration | 9 Comments »