This time around I chose to build a lights-out manager for my Sun Ultra 24 server (which this blog is hosted on). With a LOM I can control the system’s power and access its serial console so I will be able to perform OS updates remotely, among other things.

Since I already owned one and I didn’t want to spend a lot of money, I chose to develop the project on top of the Arduino platform. I like the size of the Arduino, and the availability of different shields to minimize soldering. It’s also able to be powered by USB, which is perfect because the Ultra 24 has an internal USB port that always supplies power.

To make things a little more challenging for myself (because the Arduino is pretty easy on its own), I chose to implement a hardware UART to communicate with the Ultra 24′s serial port. Specifically, I chose to use the Maxim MAX3110E SPI UART and RS-232 transceiver. Great little chip.

For communication with the outside world, I bought an Ethernet shield from Freetronics. It’s compatible with the official Arduino Ethernet shield, but includes a fix to allow the network module to work with other SPI devices (such as my UART) on the same bus. I started to implement the network UI using Telnet, but after realizing I would have to translate the serial console data from VT100 to NVT, I switched to Rlogin, which is like Telnet, but assumes like-to-like terminal types.

Lastly, for controling the system’s power, I figured out how to tap into the Ultra 24′s power LED and switch. Using the LED, I can check whether the system is on or off, and using the switch circuit and a transistor, I can power the system on and off. I managed to do this without affecting the operation of the front panel buttons/LEDs.

I’ll spare you all of the implementation details (if you’re interested, you can read my report). Suffice it to say, the thing works as well as I could have imagined. Here is a screenshot of me using the serial console on my workstation:

From my research, the serial and power motherboard headers are the same on most modern Intel systems, so this LOM should work on more than just an Ultra 24. If you want to build one of your own, my code is available on GitHub and the hardware schematic is in the report I linked to above.

I’ve been working on a project off and on for the past year which uses the Spring Framework extensively. I love Spring for how easy it makes web development, from wiring up various persistence and validation libraries, to dependency injection, and brainless security and model-view-controller functionality. However, as the project has grown, I’ve become more and more frustrated with one aspect of Spring and Java web development in general: performance and resource usage. It’s so bad, I’ve pretty much stopped working on it altogether. Between Eclipse and Tomcat, you’ve already spent over 2 GB of memory, and every time you make a source code change, Tomcat has to reload the application which takes up to 30 seconds on my system, if it doesn’t crash first. This doesn’t suit my development style of making and testing lots of small, incremental changes.

So rather than buy a whole new computer, I’ve started to look for a new lightweight web framework to convert the project to. I really like Erlang and have wanted to write something big in it for a while, so when I found the Nitrogen Web Framework, I thought this might be my opportunity to do so. Erlang is designed for performance and fault-tolerance and has a great standard library in OTP, including a distributed database, mnesia, which should eliminate my need for an object-relational mapper (it stores Erlang terms directly) and enable me to make my application highly available in the future without much fuss. Nitrogen has the added benefit of simplifying some of the fancy things I wanted to do with AJAX but found too difficult with Spring MVC.

The thing I don’t like about Nitrogen is that it is designed to deliver a complete, stand-alone application with a built-in web server of your choosing and a copy of the entire Erlang runtime. This seems to be The Erlang/OTP Way of doing things, but it seems very foreign to me. I already have Erlang installed system-wide and a web server, Yaws, that I have a lot of time invested in. I’d rather use Nitrogen as a library in my application under Yaws just like I was using Spring as a library in my application under Tomcat.

Procedures

I start my new project with Rebar:

$ mkdir test && cd test
$ wget https://bitbucket.org/basho/rebar/downloads/rebar && chmod +x rebar
$ ./rebar create-app appid=test
==> test (create-app)
Writing src/test.app.src
Writing src/test_app.erl
Writing src/test_sup.erl
$ mkdir static include templates  # These directories will be used later

Now I define my project’s dependencies in rebar.config in the same directory:

These dependencies are taken from Nitrogen’s rebar.config. Next I write a Makefile to simplify common tasks:

I expect I’ll be tweaking this Makefile some more in the future, but it demonstrates the absolute minimum to compile the application. When I run make, four things happen the first time:

1. BASEDIR is defined as the current directory in include/basedir.hrl. We’ll use this later.
2. All of the Nitrogen dependencies are pulled from Git to the deps directory.
3. All of the code is compiled.
4. The static content from Nitrogen (mostly Javascript files) is symlinked into our static content directory.

Next I prepare the code for running under Yaws. First I create the Nitrogen appmod in src/test_yaws.erl:

This is taken from Nitrogen repository. I also modify the init/0 function in src/test_sup.erl to start the nprocreg application, similar to how it is done in Nitrogen proper:

Lastly, I add a function to src/test_app.erl which can be used by Yaws to start the application:

One other thing that I do before loading the application up in Yaws is create a sample page, src/index.erl. This is downloaded from Nitrogen:

I make sure to include basedir.hrl (generated by the Makefile, remember?) and modify the template path to start with ?BASEDIR. Since where Yaws is running is out of our control, we must reference files by absolute pathnames. Speaking of templates, I downloaded mine from the Nitrogen repository. Obviously, it can be modified however you want or you could create one from scratch.

Before we continue, I recompile everything by typing make.

Now the fun begins: wiring it all up in Yaws. I use my package for OpenSolaris which puts the configuration file in /etc/yaws/yaws.conf. I add the following to it:

Obviously, your paths will probably be different. The point is to tell Yaws where all of the compiled code is, tell it to start your application (where the business logic will be contained), and tell it to use the Nitrogen appmod. Restart Yaws and it should all be working!

Now for some cool stuff. If you run the svc:/network/http:yaws service from my package, or you start Yaws like yaws --run_erl svc, you can run yaws --to_erl svc (easiest to do with root privileges) and get access to Yaws’s Erlang console. From here you can hot-reload code. For example, modify the title in index.erl and recompile by running make. In the Erlang console, you can run l(index). and it will pick up your changes. But there is something even cooler. From the Erlang console, type sync:go(). and now whenever you make a change to a loaded module’s source code, it will automatically be recompiled and loaded, almost instantly! It looks something like:

# yaws --to_erl svc
Attaching to /var//run/yaws/pipe/svc/erlang.pipe.1 (^D to exit)

1> sync:go().
Starting Sync (Automatic Code Reloader)
ok
2>
=INFO REPORT==== 17-Feb-2011::15:03:10 ===
/docs/test/src/index.erl:0: Recompiled. (Reason: Source modified.)

=INFO REPORT==== 17-Feb-2011::15:04:20 ===
/docs/test/src/index.erl:11: Error: syntax error before: body

=INFO REPORT==== 17-Feb-2011::15:04:26 ===
/docs/test/src/index.erl:0: Fixed!

2> sync:stop().

=INFO REPORT==== 17-Feb-2011::15:07:17 ===
    application: sync
    exited: stopped
    type: temporary
ok

One gotcha that may or may not apply to you, is that Yaws should have permission to write to your application’s ebin directory if you want to save the automatically compiled code. In my case, Yaws runs as a different user than I develop as, a practice that I would highly recommend. So I use a ZFS ACL to allow the web server user read and write access:

$ /usr/bin/chmod -R A+user:webservd:rw:f:allow /docs/test/ebin
$ /usr/bin/ls -dv /docs/test/ebin
drwxr-xr-x+  2 jlee     staff          8 Feb 17 15:04 /docs/test/ebin
     0:user:webservd:read_data/write_data:file_inherit:allow
     1:owner@::deny
     2:owner@:list_directory/read_data/add_file/write_data/add_subdirectory
         /append_data/write_xattr/execute/write_attributes/write_acl
         /write_owner:allow
     3:group@:add_file/write_data/add_subdirectory/append_data:deny
     4:group@:list_directory/read_data/execute:allow
     5:everyone@:add_file/write_data/add_subdirectory/append_data/write_xattr
         /write_attributes/write_acl/write_owner:deny
     6:everyone@:list_directory/read_data/read_xattr/execute/read_attributes
         /read_acl/synchronize:allow

ACLs are pretty scary to some people, but I love ‘em

Other Thoughts

You would not be able to run multiple Nitrogen projects on separate virtual hosts using this scheme. Nitrogen maps request paths to module names (for example, requesting “/admin/login” would load a module admin_login) and module names must be unique in Erlang. I think it would be possible to work around this using a Yaws rewrite module, though I haven’t tested it. I imagine if one virtual host maps “/admin/login” to “/foo/admin/login” and another maps it to “/bar/admin/login”, then Nitrogen would search for foo_admin_login and bar_admin_login, respectively, eliminating the conflicting namespace problem.

Now that I’ve gone through all the trouble of setting up Nitrogen the way I like, I should start converting my application over. Hopefully I’ll like it. It would be a shame to have done all this work for naught. I’m sure there will be posts to follow.

I toyed around with lighttpd and read the documentation for nginx. Both seem to do what I need them to do, but I was more interested in the equally fast and light Yaws. I’ve been really into Erlang lately (more on that in another post), so this was a great opportunity to see how a real Erlang application works.

Installation

Naturally, Yaws isn’t included in OpenSolaris yet, but Erlang is! So it was fairly easy to whip together a spec file and SMF manifest for the program. Then it was as easy as:

$ pkgtool build-only --download --interactive yaws.spec
$ pfexec pkg install yaws

I’ve set the configuration to go to /etc/yaws/yaws.conf and the service can be started with svcadm enable yaws. When I’m completely satisfied with the package, I’ll upload it to SourceJuicer.

Configuration

I run two virtual hosts: thestaticvoid.com and iriverter.thestaticvoid.com. Those two domains also have “.org” counterparts which I want to redirect to the “.com” domain. The yaws.conf syntax to handle this case is very simple:

Should be pretty self-explanatory, but the nice thing is the pick_first_virthost_on_nomatch directive combined with the localhost block so that if anyone gets to this site by any other address, they’ll be redirected to the canonical thestaticvoid.com. I did actually run into a bug with the redirection putting extra slashes in the URL, but squashed that bug pretty quickly with a bit of help from the Yaws mailing list. That whole problem is summarized in this thread.

PHP

Yaws handles PHP execution as a special case. All you have to do is add a couple lines to the configuration above:

Reload the server (svcadm refresh yaws) and PHP, well, won’t work just yet. This actually took me an hour or two to figure out. OpenSolaris’ PHP is compiled with the --enable-force-cgi-redirect option which means PHP will refuse to execute unless it was invoked by an Apache Action directive. Fortunately, you can disable this security measure by setting cgi.force_redirect = 0 in your /etc/php/5.2/php.ini.

Trac

Trac needs a little work to get running in Yaws. It’s requires an environmental variable, TRAC_ENV, set to tell it where to find the project database. The easiest way to do that is to copy /usr/share/trac/cgi-bin/trac.cgi to the document root, modify it to set the environmental variable, and enable CGI scripts in Yaws by setting allowed_scripts = cgi.

But I decided to set up an appmod, so that trac.cgi could be left where it was, unmodified. Appmods are Erlang modules which get run by Yaws whenever the configured URL is requested. Here’s the one I wrote for Trac:

All appmods must define the out/1 function which takes an arg record which contains information about the current request. At the end of the function, the Yaws API is used to execute the CGI script with the extra environmental variables. This is compiled (erlc -o /var/yaws/ebin -I/usr/lib trac.erl) and enabled in the Yaws configuration by adding appmods = </trac, trac> to a server section. Then whenever someone requests /trac/foo/bar, Trac runs properly!

I also set up URL rewriting so that instead of requesting something like http://i.tsv.c/trac/wiki, all you see is http://i.tsv.c/wiki. This involves another Erlang module, a rewrite module. It looks like:

This module runs as soon as a request comes into the server and allows you to modify many variables before the request is handled. This is a simple one which says: if the request is “/” or the requested file doesn’t exist, append the request to the Trac appmod, otherwise pass it through unaltered. It’s enabled in the Yaws server by adding arg_rewrite_mod = rewrite_trac to yaws.conf. The Trac appmod must also be modified to make sure SCRIPT_NAME is now / so the application generates links without containing /trac.

WordPress

WordPress works perfectly once PHP is enabled in Yaws. WordPress permalinks, however, do not. A little background: WordPress normally relies on Apache's mod_rewrite to execute index.php when it gets an request like /post/2009/07/27/suexec-on-opensolaris/. mod_rewrite sets up the environmental variables such that WordPress is able to detect how it was called and can process the page accordingly.

Without mod_rewrite, the best it can do is rely on requests like /index.php/post/2009/07/27/suexec-on-opensolaris/ and use the PATH_INFO variable, which is the text after index.php. I think that looks ugly, having index.php in every URL. You would think that simply rewriting the URL, just as was done with Trac, would solve the problem, but WordPress is too smart, and always sends you to a canonical URL which it thinks must include index.php.

After more experimentation than I care to explain, I discovered that if I set the REQUEST_URI variable to the original request (the one not including index.php), WordPress was happy. This was a tricky exercise in trying to set an environmental variable from the rewrite module. But, as we saw with the Trac example, environmental variables can be set from appmods. And I found that data can be passed from the rewrite module to the appmod through the arg record! Here's my solution:

Nevermind the extra WebSVN rewriting code. Notice I set the opaque component of the arg. Then in the appmod:

I pull the data from the arg record and pass it to the call_cgi/5 function. Also of note here is the special way to invoke the PHP CGI. The location of the php-cgi executable is pulled from the Yaws configuration and passed as the second argument to call_cgi/5 so Yaws knows what to do with the files. You can surely imagine this as a way to execute things other than PHP which do not have a #! at the top. Or emulating suEXEC with a custom wrapper

Overall, this probably seems like a lot of work to get things working that are trivial in other servers, but I'm finding that the appmods are really powerful, and the Yaws code itself is very easy to understand and modify. Plus you get the legendary performance and fault-tolerance of Erlang.