This time around I chose to build a lights-out manager for my Sun Ultra 24 server (which this blog is hosted on). With a LOM I can control the system’s power and access its serial console so I will be able to perform OS updates remotely, among other things.

Since I already owned one and I didn’t want to spend a lot of money, I chose to develop the project on top of the Arduino platform. I like the size of the Arduino, and the availability of different shields to minimize soldering. It’s also able to be powered by USB, which is perfect because the Ultra 24 has an internal USB port that always supplies power.

To make things a little more challenging for myself (because the Arduino is pretty easy on its own), I chose to implement a hardware UART to communicate with the Ultra 24′s serial port. Specifically, I chose to use the Maxim MAX3110E SPI UART and RS-232 transceiver. Great little chip.

For communication with the outside world, I bought an Ethernet shield from Freetronics. It’s compatible with the official Arduino Ethernet shield, but includes a fix to allow the network module to work with other SPI devices (such as my UART) on the same bus. I started to implement the network UI using Telnet, but after realizing I would have to translate the serial console data from VT100 to NVT, I switched to Rlogin, which is like Telnet, but assumes like-to-like terminal types.

Lastly, for controling the system’s power, I figured out how to tap into the Ultra 24′s power LED and switch. Using the LED, I can check whether the system is on or off, and using the switch circuit and a transistor, I can power the system on and off. I managed to do this without affecting the operation of the front panel buttons/LEDs.

I’ll spare you all of the implementation details (if you’re interested, you can read my report). Suffice it to say, the thing works as well as I could have imagined. Here is a screenshot of me using the serial console on my workstation:

From my research, the serial and power motherboard headers are the same on most modern Intel systems, so this LOM should work on more than just an Ultra 24. If you want to build one of your own, my code is available on GitHub and the hardware schematic is in the report I linked to above.

Meanwhile, I went through a couple hundred photos from a recent trip to Belgium with my girlfriend, and though I really like what I was able to capture, and I’m satisfied with the way they turned out, I can’t help but notice a distinct point-and-shoot quality about them. Obviously they can’t all be winners, but in the spur of the moment, it’s all too easy for me to let my zoom lens do all the work at the expense of good composition.

Looking back at my photos, I noticed that I like the ones I’ve taken with my 50mm lens the most. The fixed focal length forces me to really think about how I want the photo to look, and I have to move my feet in order to get it. Its large aperture allows me to take photos at a lower ISO level in low light, which means less sensor noise. And its shallow depth-of-field potential stimulates my creativity. I think Kai from DigitalRev explains it best:

However, with my T2i’s APS-C-sized sensor, the field-of-view of the 50mm lens is more like an 80mm lens on a full-frame sensor, which is really narrow for indoor shooting. I would have to stand all the way across the room to get anything more than just a headshot. So I started to look at what equivalent “normal” lenses I could get for my T2i.

I settled on the Canon EF 28mm f/1.8, and I’m generally very happy with it. It’s sharp, fast, and well made. However, the difference in the depth-of-field between the 50mm and 28mm is very noticeable, even at f/1.8. It’s much harder to get that nicely blurred background unless you’re within a couple feet of your subject with the 28mm. That is just the nature of wider angle lenses.

Meanwhile, Canon just released the brand-new full-frame EOS 5D Mark III and prices for the three-year-old 5D Mark II are dropping. I never would have considered getting a full-frame camera before (that’s just silly—I’m an amateur and full-frame cameras are for the pros, right?), but the prices are not much more than the 7D now. The 5D Mark II is still a great camera. One of the few things people complain about is its poor auto-focus performance. Fortunately, I learned early to use center-point and back-button focus, and I don’t shoot sports, so I couldn’t care less about auto-focus performance.

The 5D Mark II is better in almost every way compared to my T2i. It’s very well built and will fit my hands, so I’ll enjoy holding and using it. And as a bonus, the full-frame sensor will enable me to get the depth-of-field I’m used to on my 50mm lens with the field-of-view similar to my 28mm, which can open a whole new world of possibilities for me.

All of this is just to say: I think I’m going to upgrade to a 5D Mark II. I think now is the time. I am serious about improving my photography and I think sticking with only prime lenses for a while will help. It takes a huge variable out of the equation (focal length) so hopefully I can concentrate on the more important things. In fact, I’ve already sold my zoom lens. Between the money from the sale of my lens, the money I should be able to get from selling my T2i, and some credit card cash back, I will be able to pick up the 5D Mark II for a good price. It will be an early graduation present to myself. And if it turns out not to be right for me, photo gear keeps its value pretty well, so I can always sell it.

I’m not crazy for wanting to upgrade, am I?

UPDATE 05/09/2012: I managed to pick up a factory refurbished 5D Mark II during Canon’s friends and family sale for a ridiculously good price ($1596 after taxes and shipping). Canon’s refurbs, if you don’t know, are like new and mine was no exception. I sold my T2i and 15-85mm lens for $1000 after eBay and Paypal took their cut, and I allowed myself to buy a refurbed EF 100mm f/2 USM lens and Speedlite 580EX II during the same Canon sale.

And the result? Well, I feel like I’m already starting to make some good improvements. Shooting with primes often forces me to think more creatively about composition and perspective to get the shot I want. Then taking the shots into Lightroom helps me do minor white balance and color corrections to really make them pop. And finally, the flash is just a lot of fun.

About a year ago I became eligible for my employer’s retirement plan, and at the time I was completely overwhelmed by this whole new world of mutual fund investments. Not only was I given a large selection of funds to choose from, but I was also given the choice of providers: Fidelity and TIAA-CREF. I did a little research, but couldn’t really decide what to do, so I just selected the defaults, Fidelity with the 2050 target fund, and let it sit.

Then, over the Thanksgiving weekend, I started reading the long-term investment thread at Something Awful where it quickly became apparent that (1.) choosing assets to invest in doesn’t have to be that hard, (2.) expense ratios (what a fund costs) matter a lot, and (3.) high-cost actively managed funds in general don’t perform better than their lower-cost index-based equivalents.

So I took another look at what was available to me in Fidelity, and I didn’t like what I saw. Two stock index funds, each tracking the S&P 500; one REIT index fund, and one bond index fund. The rest of the funds available were actively managed with expenses around 0.8% or more. Even the few Vanguard funds available to me through Fidelity seemed obscure and expensive.

Then I remembered about TIAA-CREF. I pulled up a list of their offerings, and it made me a little more optimistic. Not only are their options cheaper overall, but they also have index funds available for more market segments. Really the worst thing that could be said after an initial look is that they could have more international representation, but it would turn out that, for me, it doesn’t really matter.

I started reading TIAA-CREF’s literature and taking their asset allocation (AA) quizzes, you know the ones. Well, considering I’m only 25 and I have a good 40 years until retirement, I would consider myself more willing to take on risk than someone a little older. Their quiz would have me put 86% into equities, 9% in real estate, and 5% in bonds. Why 9% real estate? Why not ten, or eight? And can 5% of your portfolio really affect anything?

At this point, I should note that TIAA’s Real Estate Account (TREA) is a unique investment vehicle among providers in that it invests directly in commercial real estate, rather than in companies that manage real estate as the more risky REIT funds do. There is really nothing else like it, and that made it hard to reconcile with other popular AA tips I found on the internet, such as having a simple three-fund portfolio. I wanted to know whether I should include real estate, if I should follow TIAA-CREF’s advice for AA, why they chose the numbers they did, and also why the Bogleheads advocate slightly more conservatives allocations. I may be young, but I don’t want to turn the risk up to 11 just because I can. I want managed risk that I can understand.

So I picked up a copy of The Intelligent Asset Allocator by William Bernstein, hoping it would shed some light on my questions. To be honest, I was hoping it would have The Answer in it, that it would point me to The Optimal Asset Allocation. Thankfully, it did a whole lot better than that. It stated in no uncertain terms that there is no such thing as an optimal asset allocation, and anyone claiming to have one is conning you. The book started off by providing useful metrics for measuring the performance (in terms of annual return) and risk (in terms of standard deviation) of different asset classes. It was an easy, quick read that gave me a few techniques for understanding the behavior of different asset classes, and how they’ve historically interacted with each other in portfolios. Best of all, it gave me the confidence to tackle the same sort of research on my own.

I sat down and took another look at the asset classes available to me with TIAA-CREF:

TIAA Traditional is fixed annuity that guarantees at least 3% interest which, like TREA, is a unique offering that complicated things for me. More on that later. Money markets are, as I understand them, good places to keep cash for liquidity purposes, and they’re currently returning 0.0%. I could scratch that off my list right away—I’m in this for the long haul. The next step was to decide which assets were “high risk” and which ones were “low risk.” Bernstein says that you control the risk of your overall portfolio by your total allocation of low risk assets. Typically people consider equities as high risk and bonds as low risk. TREA was a bit of a mystery which required more investigation (and believe me, I read every document, article, and forums topic I could find about it), but ultimately my gut told me the answer. Anyone who owns a house or has watched the news in the past few years could tell you, real estate is a risky asset. And one look at the account’s growth in the past decade makes it clear:

All I had to do was ask myself, “if I were nearing retirement and that dip happened, would I be okay with it?” Because, if I classified it as a low risk asset, then in the future I would own more of it than I do today as one increases the total allocation of low risk assets to control risk approaching retirement. Clearly I would want to own decreasing amounts of real estate as I age, so it belongs in the high risk category:

Now I could take on the actual asset allocation by process of simplification. First I looked at domestic vs. international equities. As an aside, some people will further break down equities into the various classes of stock (large vs. small-cap, value vs. growth, etc.), and though Bernstein clearly shows that the different sub-classes have different behaviors and play different roles in a portfolio, he also makes it painfully clear that past performance is no guarantee of future performance, and that the behaviors of the different sub-classes have changed over time, so it makes the most sense to own the whole market as it is rather that weighting it. There are funds which track indices that represent the whole market, such as the Russell 3000 or MSCI EAFE. Makes things a lot simpler.

Back to the task at hand, I downloaded the annual returns for the Wilshire 5000 (an index similar to Russell 3000) and the EAFE all the way back to 1971 and plotted the risk vs. return for all of the different allocations of domestic and international equities:

The endpoints on the left of the curve represent portfolios of all domestic stock. The right endpoints represent portfolios of all international stock. Each point in between going from left to right represents replacing 10% of the domestic stock with international stock in the portfolio. The position of each point on the X-Y grid shows the amount of risk vs. return of that portfolio during the specified time period.

Looking at the first time period (in green), the plot would suggest I own a larger chunk of international stock for my level of risk tolerance; however, the next time period (in red) shows that that would have hurt me. Bernstein says that a lot of the change in the behavior of international stocks in the late 80s can be attributed to the rise and fall of the Japanese stock market, and emphasizes that if the past shows us anything, it’s that you can’t predict what will happen in the future, and thus, why not just split the difference? Looking at the long-term picture, a 50-50 allocation of domestic and international equities probably wouldn’t hurt you, and considering that is also about the ratio of the US to international market caps, it fits with the philosophy of “owning the market as it is.”

As another aside, Bernstein’s book has similar charts, and I can’t tell you how satisfying it was to see how much my own findings matched his.

With a 50-50 equities allocation in mind, I can replace two asset classes with one, simplifying my table a bit:

My next task was to determine how much, if any, real estate in the form of TREA I should own. This was a challenge for many reasons. First, as mentioned before, TREA is a unique offering that is hard to compare to other asset classes. Second, it was created in 1995, so it is hard to get an understanding of how it might behave long-term. And third, while the account does track a benchmark index, the REA Composite Index, which could help me gain some insight into its long term behavior, it systemically underperforms it due to the way the account’s properties are appraised. The result was a little guess work and hand waving, but that’s investing, I guess.

I downloaded the annual returns for TREA since its inception in ’95, and the returns for the components of the REA Composite Index which go back to 1978. (The iMoneyNet money market index component is proprietary, so I substituted it with returns from the Vanguard Prime Money Market Fund, which goes back to 1977, and tracks the same index.) Then I plotted the performance of all the various allocations of TREA, the REA Composite Index, and my chosen allocation of equities, after subtracting all of the associated costs (such as the 1.01% expense charge for TREA).

The first thing to note is how wildly different TREA (in green) is from its benchmark index (in red). They’re highly correlated from 1996 on with a correlation coefficient of 0.98, so I don’t think I’m out of line to use a linear approximation between the two to extrapolate what TREA might have looked like from 1978 (in teal). The second thing to note is how stable the REA Composite is. Look at how close together the two REA Comp endpoints are compared to the equities endpoints.

Anyway, this chart, along with a couple of other calculations, was revealing. Should I invest in TREA? Well TREA and my choice of equities have demonstrated a very low correlation coefficient of 0.16 since its inception, so it should be a good candidate for diversification. This is apparent in the graph above. If you had invested in 40% TREA, 60% global equities in 1996, you would have realized the same amount of returns by 2010 as someone who invested 100% in global equities without as many, or as large of ups and downs. The long-term model shows, however, that the benefits of real estate drop off very quickly. In the end, all this can tell me is what has happened, and tells me nothing about the future, but it also says that real estate can be a good diversifier in the bear economy we’ve had the past decade, and if I expect there to be more bad economies in my future, then TREA should have a place in it.

The next question, how much TREA, is a little easier. If I look at the past 15 years, I might be tempted to choose 40%, but I also expect (rather I hope) equities will return to historical levels of performance, so I should choose a much smaller allocation of real estate. Looking at the long term graph, by 20% TREA I would have sacrificed about 1% return compared to a portfolio of all equities. 10% seems to be a good compromise between the portfolio stabilization effect of TREA and the loss of returns. One of the most reassuring lines in Bernstein’s book is:

There is plenty of margin for error available in asset allocation policy. If you are off 10% or 20% from what in retrospect turned out to be the best allocation, you have not lost that much…sticking by your asset allocation policy through thick and thin is much more important than picking the “best” allocation.

So what does it mean to stick with your AA through thick and thin? Well first, I think it means having a rational basis for your choices in the first place, as I hope I’m establishing for my choices here. That should reduce the temptation to make major changes down the line. Second, I think it means that you should maintain your chosen allocations by rebalancing once a year.

Rebalancing is the process of moving money from assets that did well for the year to others that did not. If stocks did really well, it would weight your portfolio more towards stocks than originally planned. Rebalancing brings your portfolio back in line with your accepted level of risk and reinforces the idea of selling high and buying low.

That brings me to my comments about the TIAA Traditional Retirement Annuity, which I am eligible for under my 401(a) plan. As I said it’s a unique opportunity that delivers a guaranteed 3% interest (based on TIAA’s stellar claims-paying ability) and has delivered additional earnings over the guaranteed minimum every year since 1948. A lot of people really like this annuity, and for good reason. It delivers performance similar to that of an intermediate bond fund with even lower risk, historically. It also has a very low correlation with the performance of stocks, bonds, and real estate (a very rare thing), so it could make it an excellent diversifier, and one that I wanted in my portfolio. Unfortunately, that performance comes at the cost of reduced liquidity. Once your money’s in, for all intents and purposes, it’s in until you retire. And even that wouldn’t bother me, except that you can’t even do what you want with the interest. That means, you can’t rebalance its earnings into lower performing assets which makes it a non-starter as a diversifier.

The TIAA Traditional Group Supplemental Retirement Annuity, which I am eligible for under my 403(b) does not have the same liquidity restrictions, but it comes at a reduced interest rate. If I were to include any Traditional under my 403(b) plan, it would make it harder to analyze and compare to my 401(a). Also, because I can’t move money between the plans, the same limitations on rebalancing would apply.

Based on all of its complications, I decided not to include any TIAA Traditional in my portfolio. My table is looking a whole lot simpler now:

All that’s left is to figure out the ratio of my chosen high risk assets to bonds. For the purposes of my analysis, I use “bonds” to mean the entire bond market as tracked by the Barclays Capital US Aggregate Bond Index and available to me as the TIAA-CREF Bond Index Fund. Just as I want to own the entire stock market as it is, I want to own the entire bond market as it is.

I downloaded the annual returns of the bond index back to 1976 and generated a graph similar to the ones above, showing different allocations of bonds and my chosen mix of high risk assets.

I did this over two time periods because, for one thing, the recent data (in green) more clearly shows the behavior of the allocation in our crappy market, and the long-term data (in blue) includes that TREA extrapolation I did earlier. I just wanted to make sure I didn’t miss anything.

A lot of literature would suggest young investors go all in with high-risk equities, and that is the sentiment among a lot of posters at Something Awful. Bernstein suggested the same thing in the first edition of his book, but changed it to a 20% bond allocation in the second edition. The Bogleheads suggest having your age in percent of bonds (25% for me). My findings confirm the recommendations of Bernstein and the Bogleheads. Clearly, a person who invested more heavily in bonds the past 15 years was rewarded with the same returns and significantly lower risk than a person more in invested my chosen mix of high risk assets. But I should really be looking at the long term picture, where a 20-30% allocation of bonds can reduce the portfolio’s risk without affecting returns by much. Since I am more willing to take on risk, and I like nice round numbers, I will choose 20%.

Let’s take another look at my table:

Wow! It has been simplified down to one line. I think that means I’m done! Let’s blow it back up and see what we’ve got:

TREA itself contains up to 25% low-risk liquid assets, so in my portfolio there is more like 20 + 0.25 * 8 = 22.5% lower-risk assets, but that’s a technicality. I only mention it because it is exactly between Bernstein’s recommendation and the Boglehead’s recommendation of “your age in bonds,” and it was nice to arrive about the same figures independently.

Speaking of “your age in bonds,” that is something I will try to do every year when I rebalance. I will increase my percentage of low-risk assets by 1% per year. That way, by the time I retire, my allocation would theoretically look like:

Remember, the ratio of high-risk to low-risk assets is how you control risk, not the allocation of assets within those categories. If you think of that ratio as a volume knob, I would say I’m at an 8 right now, and I want to be around 3 or 4 when I retire.

One other thing I want to mention is that when I said the bond index represents the entire bond market, I lied. There is a fairly new invention of the US Treasury called the Treasury Inflation-Protected Security (TIPS) which is not represented by the bond index. It is a type of government bond which guarantees its principal will rise and fall with the consumer price index. They are a great way to hedge against the risk of inflation, but as they’re so new, and a bit complicated, I’m having a hard time understanding the way they work and how they might behave in the long term. For now I’ve excluded them from my portfolio (with my long time horizon and a large allocation of equities, I’m reasonably protected from inflation), but I hope to gain a better understanding of them soon to see where they might fit in.

Now that I’ve decided on an AA, let’s match them up to their low-cost index fund equivalents where possible:

My portfolio will cost me 0.31% per year which is less than half of the 0.80% I was paying Fidelity. To see just how much of a difference that makes, let’s assume I put in $10,000 per year for the next 40 years and average an annual return of 8% and that the costs of the funds stay the same. At the end of 40 years, I would have $2,388,152 with TIAA-CREF and $2,102,199 with Fidelity. That is a difference of $285,953! Expense ratios matter a lot.

So what did I actually get out of all this complicated analysis? Well, as I just demonstrated, my portfolio will have provably lower costs. Beyond that, it put me in control of my future. I now understand the behavior of different asset classes, the indices that represent them, and the funds that I am putting my hard-earned money into. Because I know how they’ve performed in the past, I have an idea of what to expect in the future. So when shit hits the fan again, I will be able to sleep soundly knowing that in the long-run it will work itself out, rather than endlessly fretting about whether I made sound investments. Isn’t that worth a few days of your time?

Stray Thoughts

• Obviously, none of this matters one bit if you aren’t contributing enough to meet your retirement goals. There are tons of calculators out there that will tell you how much to be saving based on how you want to live in retirement. TIAA-CREF has a good one. Thanks to my employer’s generous matching program, I am on track to meet my goal of retiring at 65. I could be doing better, but I think I’m probably doing better than most other people my age.
• Since this is a post about TIAA-CREF, I would be remiss not to mention how pleased I’ve been with their customer service so far as I’m transferring my meager sum from Fidelity. They’ve assigned me an advisor who helped me fill out all of the necessary forms, and who will keep me up-to-date on the status of the transfer until it is complete and then get out of my hair.
• The MATLAB code I wrote to generate the graphs is available from my GitHub account.
• This investing stuff is addictive. Maybe I’ll look into taking some classes next fall.
• And I have to say thanks to my girlfriend who barely saw me last week and dealt with me staying up until 2 or 3 AM every night researching this.

The update itself did not go smoothly. I was sitting at Solaris 11 Express SRU 8 and thought, like every update I’ve done in the past, that I could just run pkg image-update. Silly me, because when I did and then rebooted, the kernel panicked. No big deal, that’s what boot environments are for. I reverted to the previous boot environment and found some helpful documentation that told me to do exactly what I just did. It turns out that there is no way to update to SRU 13 using the support repositories because they already contain the Solaris 11 11/11 packages, and pkg tries to pull some of them in. And there is no way to update just pkg because the ips-consolidation prevents it, and trying to update the ips-consolidation pulls the entire package which breaks everything just the same. In short, Oracle bungled it. The only way to update to SRU 13 that I could see was to download the SRU 13 repository ISO from My Oracle Support and set up a local repository. Once I was on SRU 13, I could continue with the update to the 11/11 release. But there were more surprises in store for me.

First, it looks like pkg decided to start enforcing consistent attributes on files shared by multiple packages. Fine, I can understand that. As a result, I had to remove a lot of my custom packages (mostly from spec-files-extra) which I’ll have to rebuild. Second, pkg decided it doesn’t like the opensolaris.org packages anymore so I had to uninstall OpenOffice.org. Also fair enough.

Happily, after that, the updates got applied successfully and the system rebooted into the 11/11 release. Next came the zone updates. When I did the normal zoneadm -z foo detach && zoneadm -z foo attach -u deal, I was told I had to convert my zones to a new ZFS structure which more closely matches the global zone. The script /usr/lib/brand/shared/dsconvert actually worked flawlessly and the updated zones came up fine.

Unfortunately I couldn’t SSH into my zones because my DNS server didn’t know where they were. It seems that with the updated networking framework, DHCP doesn’t request a hostname anymore. (/etc/default/dhcpagent still says inet <hostname> can be put in /etc/hostname.<if> to request the hostname.) I found that you can create an addr object that requests a hostname with ipadm create-addr -T dhcp -h <hostname> <addrobj>, but NWAM pretty much won’t let you create or modify anything with ipadm, and there were no options for requesting hostnames with nwamcfg. As a result, I had to disable NWAM (netadm enable -p ncp DefaultFixed) and then I could set up the interface with ipadm. Why doesn’t Solaris request hostnames by default? Not very “cloud-like” if you ask me.

I have to say, I’m impressed by the way global zones and non-global zones are linked in the new release. Zone updates were an obvious shortcoming of previous releases. We’ll see how well it works when Solaris 11 Update 1 comes out.

What else…I lost my ability to pfexec to root. Oracle removed the “Primary Administrator” profile for security reasons so I had to install sudo. Not a big deal, I just wish they had said something a little louder about it.

Also, whatever update to pkg happened, it wiped out my repositories under /var/pkg. I had to restore them from a snapshot. Bad Oracle!

I’m also a little confused about some of the changes to the way networking settings are stored. For example, when I first booted the global zone, I found that my NFSv4 domain name was reset by NWAM. I set it to what it should be with sharectl set -p nfsmapid_domain=thestaticvoid.com nfs, but is that going to be overwritten again by NWAM? Also, the name resolver settings are now stored in the svc:/network/dns/client:default service, and according to the documentation, DHCP will set the service properties properly, but I have yet to see this work.

And the last problem I’ll mention is that the update removed my virtual consoles. I had to install the virtual-console package to restore them.

Overall, I’m happy that I was at least able to update to the latest release. Oracle could have cut off any update path from OpenSolaris. However, the update should have been a lot smoother. It doesn’t speak well of future updates when I can’t even update from one supported release (SRU 8) to another. I also wish Oracle were more open about upcoming changes (as in, having more preview releases or, dare I say it, opening development the way OpenSolaris was). Even to me, a long time pre-Solaris 11 user, the changes to zones and networking are huge in this release, and I would rather have not been so surprised by them.

The George Washington University (where I work and go to school) has recently implemented 802.1X to secure its wireless networks. 802.1X defines support for EAP over Ethernet (including wireless) and the WPA standards define several modes of EAP that can be used.

Solaris (I’m referring to version 11, OpenSolaris, OpenIndiana, and Illumos) supports WPA. It modified an early version of wpa_supplicant and called it “wpad“. However, they seemed to make a point of stripping out all EAP support in wpad.

So when my Network Security instructor said we had to do a term project of our choosing relating to network security, I decided I’d try to get 802.1X working in Solaris. To do this, I decided I could either add the EAP bits back into wpad, or add the Solaris-specific bits to the latest version of wpa_supplicant. wpad is based on very old code. It’s not even clear which version of wpa_supplicant it is based on, and there is no record of the massive amount of changes they made. It would be too hard for me to figure out where to plug EAP back in, and who knows how many bugs and security vulnerabilities were fixed upstream that we’d be missing out on.

Fortunately, wpa_supplicant is very modular, and reasonably well documented. I was able to graft the older Solaris code onto the newer interfaces. The result of my work is currently maintained in my own branch at GitHub. It’s not perfect, but it works (and I’ll explain how). Solaris has a very limited public API for wireless support and my goal was to get wpa_supplicant working without having to modify any system libraries or the kernel. I struggled to figure out some idiosyncrasies such as:

• Events (association, disassociation, etc.) are only sent to wpa_supplicant when WPA is enabled in the driver.
• Full scan results are only available when WPA is disabled in the driver.
• Scan results don’t provide nearly as much information as their Linux counterparts do, such as access point capabilities, signal strength, noise levels, etc. I was very worried I wouldn’t be able to fill out the scan results structure fully and wpa_supplicant would refuse to work without complete information.

Here is how you can get 802.1X support working on your Solaris laptop:

1. Install the wpa_supplicant package from my package repository:

   # pkg set-publisher -p http://pkg.thestaticvoid.com/
   # pkg install wpa_supplicant
   

2. Add the configuration for your protected wireless networks to /etc/wpa_supplicant.conf. Here is mine:

   ctrl_interface=/var/run/wpa_supplicant
   ctrl_interface_group=0
   ap_scan=0
   
   network={
       ssid="prey"
       key_mgmt=WPA-PSK
       psk="<network key>"
   }
   
   network={
       ssid="GW1X"
       key_mgmt=WPA-EAP
       eap=TTLS
       identity="jameslee"
       anonymous_identity="anonymous"
       password="<personal password>"
       phase2="auth=PAP"
   }

   The most important thing here is ap_scan=0. This tells wpa_supplicant not to do any scanning or association of its own. Those tasks will be handled by dladm and NWAM.

3. Backup /usr/lib/inet/wpad and replace it with this script:
   #!/bin/sh
   
   interface=`echo $@ | /usr/bin/sed 's/.*-i *\([a-z0-9]*\).*/\1/'`
   exec /usr/sbin/wpa_supplicant -Dsolaris -i$interface -c/etc/wpa_supplicant.conf -s &

Now connect to a wireless network with NWAM or dladm. When prompted for a network key, enter anything; it won’t be used. The actual keys will be looked up in /etc/wpa_supplicant.conf. Here is an example of me connecting to my 802.1X-secured network using dladm:

# dladm connect-wifi -e GW1X -s wpa -k nwam-GW1X iwh0
# dladm show-wifi
LINK       STATUS            ESSID               SEC    STRENGTH   MODE   SPEED
iwh0       connected         GW1X                wpa    excellent  g      54Mb

“-k nwam-GW1X” refers to a dummy key setup by NWAM. dladm will complain if it’s not supplied a key.

That should be it!

Future Directions

Obviously, the integration of wpa_supplicant and NWAM/dladm leaves a lot to be desired. If there is sufficient interest, I will start looking into how to modify the dladm security framework in Illumos to include EAP related configurations (keys, certificates, identities; it’s all much more complicated than the single pre-shared key that dladm supports now). My hope, though, is that Oracle is already working on this. Do you hear that Oracle?

Then I discovered autossh which does all the work of setting up and maintaining the tunnel for me. I coupled that with an executable autofs map to automatically start the tunnel just before trying to mount a share, like:

Using an executable autofs map allows me to avoid reconciling the differences between service managers like SMF and Upstart, offering a consistent way to start the tunnel exactly when it’s needed on both Solaris and Linux. When you ‘cd’ into a directory managed by autofs, autossh is started or woken up, then the share is mounted over the tunnel. If there is a network interruption or change (from wired to wireless, for example), ssh will disconnect after 15 seconds of inactivity and autossh will restart it. NFS is smart enough to resume its operation when the tunnel is reestablished.

autossh has built-in support for heartbeat monitoring, but I’ve found SSH’s built-in ServerAliveInterval feature to be more reliable.

With this setup I have very simple, robust, and secure remote access to my fileserver.

I’ve been working on a project off and on for the past year which uses the Spring Framework extensively. I love Spring for how easy it makes web development, from wiring up various persistence and validation libraries, to dependency injection, and brainless security and model-view-controller functionality. However, as the project has grown, I’ve become more and more frustrated with one aspect of Spring and Java web development in general: performance and resource usage. It’s so bad, I’ve pretty much stopped working on it altogether. Between Eclipse and Tomcat, you’ve already spent over 2 GB of memory, and every time you make a source code change, Tomcat has to reload the application which takes up to 30 seconds on my system, if it doesn’t crash first. This doesn’t suit my development style of making and testing lots of small, incremental changes.

So rather than buy a whole new computer, I’ve started to look for a new lightweight web framework to convert the project to. I really like Erlang and have wanted to write something big in it for a while, so when I found the Nitrogen Web Framework, I thought this might be my opportunity to do so. Erlang is designed for performance and fault-tolerance and has a great standard library in OTP, including a distributed database, mnesia, which should eliminate my need for an object-relational mapper (it stores Erlang terms directly) and enable me to make my application highly available in the future without much fuss. Nitrogen has the added benefit of simplifying some of the fancy things I wanted to do with AJAX but found too difficult with Spring MVC.

The thing I don’t like about Nitrogen is that it is designed to deliver a complete, stand-alone application with a built-in web server of your choosing and a copy of the entire Erlang runtime. This seems to be The Erlang/OTP Way of doing things, but it seems very foreign to me. I already have Erlang installed system-wide and a web server, Yaws, that I have a lot of time invested in. I’d rather use Nitrogen as a library in my application under Yaws just like I was using Spring as a library in my application under Tomcat.

Procedures

I start my new project with Rebar:

$ mkdir test && cd test
$ wget https://bitbucket.org/basho/rebar/downloads/rebar && chmod +x rebar
$ ./rebar create-app appid=test
==> test (create-app)
Writing src/test.app.src
Writing src/test_app.erl
Writing src/test_sup.erl
$ mkdir static include templates  # These directories will be used later

Now I define my project’s dependencies in rebar.config in the same directory:

These dependencies are taken from Nitrogen’s rebar.config. Next I write a Makefile to simplify common tasks:

I expect I’ll be tweaking this Makefile some more in the future, but it demonstrates the absolute minimum to compile the application. When I run make, four things happen the first time:

1. BASEDIR is defined as the current directory in include/basedir.hrl. We’ll use this later.
2. All of the Nitrogen dependencies are pulled from Git to the deps directory.
3. All of the code is compiled.
4. The static content from Nitrogen (mostly Javascript files) is symlinked into our static content directory.

Next I prepare the code for running under Yaws. First I create the Nitrogen appmod in src/test_yaws.erl:

This is taken from Nitrogen repository. I also modify the init/0 function in src/test_sup.erl to start the nprocreg application, similar to how it is done in Nitrogen proper:

Lastly, I add a function to src/test_app.erl which can be used by Yaws to start the application:

One other thing that I do before loading the application up in Yaws is create a sample page, src/index.erl. This is downloaded from Nitrogen:

I make sure to include basedir.hrl (generated by the Makefile, remember?) and modify the template path to start with ?BASEDIR. Since where Yaws is running is out of our control, we must reference files by absolute pathnames. Speaking of templates, I downloaded mine from the Nitrogen repository. Obviously, it can be modified however you want or you could create one from scratch.

Before we continue, I recompile everything by typing make.

Now the fun begins: wiring it all up in Yaws. I use my package for OpenSolaris which puts the configuration file in /etc/yaws/yaws.conf. I add the following to it:

Obviously, your paths will probably be different. The point is to tell Yaws where all of the compiled code is, tell it to start your application (where the business logic will be contained), and tell it to use the Nitrogen appmod. Restart Yaws and it should all be working!

Now for some cool stuff. If you run the svc:/network/http:yaws service from my package, or you start Yaws like yaws --run_erl svc, you can run yaws --to_erl svc (easiest to do with root privileges) and get access to Yaws’s Erlang console. From here you can hot-reload code. For example, modify the title in index.erl and recompile by running make. In the Erlang console, you can run l(index). and it will pick up your changes. But there is something even cooler. From the Erlang console, type sync:go(). and now whenever you make a change to a loaded module’s source code, it will automatically be recompiled and loaded, almost instantly! It looks something like:

# yaws --to_erl svc
Attaching to /var//run/yaws/pipe/svc/erlang.pipe.1 (^D to exit)

1> sync:go().
Starting Sync (Automatic Code Reloader)
ok
2>
=INFO REPORT==== 17-Feb-2011::15:03:10 ===
/docs/test/src/index.erl:0: Recompiled. (Reason: Source modified.)

=INFO REPORT==== 17-Feb-2011::15:04:20 ===
/docs/test/src/index.erl:11: Error: syntax error before: body

=INFO REPORT==== 17-Feb-2011::15:04:26 ===
/docs/test/src/index.erl:0: Fixed!

2> sync:stop().

=INFO REPORT==== 17-Feb-2011::15:07:17 ===
    application: sync
    exited: stopped
    type: temporary
ok

One gotcha that may or may not apply to you, is that Yaws should have permission to write to your application’s ebin directory if you want to save the automatically compiled code. In my case, Yaws runs as a different user than I develop as, a practice that I would highly recommend. So I use a ZFS ACL to allow the web server user read and write access:

$ /usr/bin/chmod -R A+user:webservd:rw:f:allow /docs/test/ebin
$ /usr/bin/ls -dv /docs/test/ebin
drwxr-xr-x+  2 jlee     staff          8 Feb 17 15:04 /docs/test/ebin
     0:user:webservd:read_data/write_data:file_inherit:allow
     1:owner@::deny
     2:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
         /append_data/write_xattr/execute/write_attributes/write_acl
         /write_owner:allow
     3:group@:add_file/write_data/add_subdirectory/append_data:deny
     4:group@:list_directory/read_data/execute:allow
     5:everyone@:add_file/write_data/add_subdirectory/append_data/write_xattr
         /write_attributes/write_acl/write_owner:deny
     6:everyone@:list_directory/read_data/read_xattr/execute/read_attributes
         /read_acl/synchronize:allow

ACLs are pretty scary to some people, but I love ‘em

Other Thoughts

You would not be able to run multiple Nitrogen projects on separate virtual hosts using this scheme. Nitrogen maps request paths to module names (for example, requesting “/admin/login” would load a module admin_login) and module names must be unique in Erlang. I think it would be possible to work around this using a Yaws rewrite module, though I haven’t tested it. I imagine if one virtual host maps “/admin/login” to “/foo/admin/login” and another maps it to “/bar/admin/login”, then Nitrogen would search for foo_admin_login and bar_admin_login, respectively, eliminating the conflicting namespace problem.

Now that I’ve gone through all the trouble of setting up Nitrogen the way I like, I should start converting my application over. Hopefully I’ll like it. It would be a shame to have done all this work for naught. I’m sure there will be posts to follow.

I want to be able to refer to systems on both my home and work networks by their hostnames rather than their fully-qualified domain names, so, ‘prey’ instead of ‘prey.thestaticvoid.com’ and ‘acad2′ instead of ‘acad2.es.gwu.edu’.

The Problem

I would typically set my home and work domains as the search setting in /etc/resolv.conf. Unfortunately, either NWAM or the Solaris DHCP client (I haven’t decided which) overwrites resolv.conf on every new connection. DHCP on Linux does the same thing, but I can configure it by editing dhclient.conf (or whatever is being used these days, it’s been a while. I think I just set my domains in the NetworkManager GUI and forget about it).

The Solaris DHCP client configuration is not nearly as flexible, and neither is NWAM which gives you the option of replacing resolv.conf with information supplied by the DHCP server, or provided by you, but not a mix of both. I do like having the nameservers set by the DHCP server, so supplying a manual configuration is not an option.

What I Tried

The first thing I tried was setting the LOCALDOMAIN environmental variable in /etc/profile. From the resolv.conf man page:

You can override the search keyword of the system
resolv.conf file on a per-process basis by setting the
environment variable LOCALDOMAIN to a space-separated list
of search domains.

I thought, great, a way to manage domain search settings without worrying about what’s doing what to resolv.conf. It didn’t work as advertised:

% LOCALDOMAIN=thestaticvoid.com ping prey
ping: unknown host prey
% s touch /etc/resolv.conf
% LOCALDOMAIN=thestaticvoid.com ping prey
prey is alive
% LOCALDOMAIN=thestaticvoid.com ping prey
ping: unknown host prey

Next, I considered adding an NWAM Network Modifier to set my search string in resolv.conf after a new connection is established. This worked reasonably well, but didn’t handle the case when you switch from one network to another, for example, from wireless to wired. The only events in NWAM that can trigger a script when the network connection changes happens before DHCP messes up resolv.conf.

Finally, in the course of my testing, I discovered that the svc:/network/dns/client service was restarting with every network connection change. I looked into its manifest and saw that it was designed to wait for changes to resolv.conf:

So I could write another service which depends on dns/client and restarts whenever dns/client does and I would have the last word about what goes into my configuration file!

My Solution

I wrote a service, svc:/network/dns/resolv-conf, with the following manifest:

which calls the script, /lib/svc/method/dns-resolv-conf containing:

So now I can set my search options like:

% svccfg -s resolv-conf setprop 'options/search="thestaticvoid.com es.gwu.edu"'
% svcadm refresh resolv-conf
% svcadm enable resolv-conf
% cat /etc/resolv.conf
domain  iss.gwu.edu
search iss.gwu.edu thestaticvoid.com es.gwu.edu
nameserver  161.253.152.50
nameserver  128.164.141.12

Problem solved! Or at least worked-around in the least hacky way I can!

CrashPlan is, as far as I know, the only online backup solution that officially supports Solaris, and it’s not half-assed either. The software is delivered as a standard SVR4 package which installs to /opt/sfw/crashplan and includes an SMF manifest. Normally I’d never trust consumer-oriented proprietary software like this, but their Solaris support instills confidence in me. I can only hope that they continue to maintain it, despite the uncertainty surrounding Solaris’s future.

Like I said, installation was a breeze. Looking back at my shell history, it was as easy as:

# cd /tmp
# wget http://download.crashplan.com/installs/solaris/install/CrashPlan/CrashPlan_3.0_Solaris.tar.gz
# tar -xvzf CrashPlan_3.0_Solaris.tar.gz
# pkgadd -d . CrashPlan
# svccfg import /opt/sfw/crashplan/bin/crashplan.xml
# svcadm enable crashplan

From there the GUI can be launched as a regular user by running /opt/sfw/crashplan/bin/CrashPlanDesktop. The user interface is clean and simple. On the first run, it walks you through setting up an account. New users get a 30-day free trial to CrashPlan+, which includes unlimited online backups. I’m still on my trial, but as long as it continues to work for me, I expect I’ll purchase a subscription for $5/month.

First thing I did after registering was to go into the security settings and change the archive encryption key type to use a private password. This encrypts the key which encrypts my data with a separate password so even if someone hijacks my CrashPlan account, they will not be able to restore any of my files. The other advanced option, supplying your own private data key, I would argue is less secure since the key is stored in-the-clear on the local system and it cannot be changed without invalidating all of your backups. Security is very important to me, so I am happy to see that they give control over these settings to the user, though I wish the backup agent were open-source to enable more public scrutiny. At the very least, I’d like for CrashPlan to provide more details about their encryption methods similar to SpiderOak.

Next I directed the software to backup my storage array mounted at /nest to CrashPlan Central and off it went. I’m currently seeing speeds around 6 Mbps (750 KB/s) which is slightly disappointing on my fast connection, but not unacceptable. They claim that they do not cap or throttle connections, though from what I’ve read, speed is largely dependent on which of CrashPlan’s many datacenters you are provisioned to. They’ve been experiencing much higher volume than normal with last week’s release of CrashPlan 3, so I hope to see increased speed when that activity subsides.

I do like that the backup actually takes place in the background, so the GUI is only ever necessary for changing settings and performing restores. I tested a restore and saw much better speeds around 16 Mbps (2 MB/s), though still not even close to saturating my internet connection.

My backup should hopefully be done by the new year and then it’ll just be a matter of performing small nightly incrementals.

The strange thing is that it connected from home with the very same profile and everything worked fine. I immediately suspected something was wrong with the routing tables, like maybe some of the routes installed by vpnc-script were conflicting with the routes necessary to talk to the VPN concentrator. I endlessly compared the routing tables between work and home and my working Ubuntu workstation, removing routes, adding routes, and manually constructing the routing table until I was positive it could not be that.

Everything I pinged worked. I could ping the concentrator. I could ping the gateway. I could ping the tunnel device. I could ping the physical interface—or so I thought.

As I was preparing to write a message to the vpnc-devel mailing list requesting help, I did some pings to post the output in the email. I ran

$ ping <concentrator ip>
<concentrator ip> is alive

which looked good, but I wanted the full ping output, so I ran

$ ping -s <concentrator ip>
PING <concentrator ip>: 56 data bytes
^C
----<concentrator ip> PING Statistics----
4 packets transmitted, 1 packets received, 75% packet loss
round-trip (ms)  min/avg/max/stddev = 9223372036854776.000/0.000/0.000/-NaN

For some reason, only the first ping was getting through. The rest were getting hung up somewhere. The really strange thing was that I saw the same behavior on the local physical interface:

$ ifconfig bge0
bge0: flags=1004843 mtu 1500 index 3
        inet 161.253.143.151 netmask ffffff00 broadcast 161.253.143.255
$ ping -s 161.253.143.151
PING 161.253.143.151: 56 data bytes
^C
----161.253.143.151 PING Statistics----
5 packets transmitted, 1 packets received, 80% packet loss
round-trip (ms)  min/avg/max/stddev = 9223372036854776.000/0.000/0.000/-NaN

I have never seen a situation where you couldn’t even ping a local physical interface! I checked and double checked that IPFilter wasn’t running. Finally I started a packet capture of the physical interface to see what was happening to my pings:

# snoop -d bge0 icmp
Using device bge0 (promiscuous mode)
161.253.143.151 -> <concentrator ip> ICMP Destination unreachable (Bad protocol 50)
161.253.143.151 -> <concentrator ip> ICMP Destination unreachable (Bad protocol 50)
161.253.143.151 -> <concentrator ip> ICMP Destination unreachable (Bad protocol 50)
^C

That’s when by chance I saw messages being sent to the VPN concentrator saying “bad protocol 50.” IP protocol 50 represents “ESP”, commonly used for IPsec. Apparently Solaris eats these packets. Haven’t figured out why.

I remembered seeing something in the vpnc manpage about ESP packets:

I enabled force-natt mode, which encapsulates the ESP packet in a UDP packet, normally to get past NAT, and it started working! In retrospect, I should have been able to figure that out much easier. First, it pretty much says it on the vpnc homepage: “Solaris (7 works, 9 only with –natt-mode forced).” I didn’t even notice that. Second, I should have realized that I was behind a NAT at home and not at work, so they would be using a different NAT-traversal mode by default. Oh well, it was a good diagnostic exercise, hence the post to share the experience.

In other vpnc related news, I’ve ported Kazuyoshi’s patch to the open_tun and solaris_close_tun functions of OpenVPN to the tun_open and tun_close functions of vpnc. His sets up the tunnel interface a little bit differently and adds TAP support. It solves the random problems vpnc had with bringing up the tunnel interface such as:

# ifconfig tun0
tun0: flags=10010008d0<POINTOPOINT,RUNNING,NOARP,MULTICAST,IPv4,FIXEDMTU> mtu 1412 index 8
        inet 128.164.xxx.yy --> 128.164.xxx.yy netmask ffffffff
        ether f:ea:1:ff:ff:ff
# ifconfig tun0 up
ifconfig: setifflags: SIOCSLIFFLAGS: tun0: no such interface
# dmesg | grep tun0
Jul 23 14:56:05 swan ip: [ID 728316 kern.error] tun0: DL_BIND_REQ failed: DL_OUTSTATE

The changes are in the latest vpnc package available from my package repository.