Absurdly Certain

HOWTO: deploy multithreading on Matlab with the Theron C++ Actors library

2011-09-02T16:05:00.048-07:00

Preamble: What follows is a HOWTO guide for getting Theron-2.09 multithreaded C++ mex files built on Matlab. For my general musings on Matlab concurrency and Theron, see the previous post.

Building multithreaded Matlab mex files with Theron

This was pretty straightforward on the Mac, somewhat tricky but not terrible on Linux. I don't have any working C++ compiler with Matlab on Windows thanks to absolutely abysmal support from Microsoft and Mathworks (essentially Mathworks only supports Microsoft and Microsoft's compilers are impossible to get installed correctly). But the notes below may help you get things working on Windows anyway and I added some Windows specific notes at the very bottom. Before trying to get Theron working, I encourage you to write a simple C++ mex program (maybe a single-threaded version of your planned algorithm) and get it built on Matlab first, to confirm that your system is correctly set up for mex.

I'm fairly confident most Mac users can get this working. On Linux I have only tested on 2010 and 2011 64-bit versions of Matlab, and there are reasons to fear that earlier versions may have more trouble working. Fair warning.

Building Theron on Mac or other static mex environments

For this discussion, the fundamental difference between Mac and Linux versions of Matlab is how they handle mex compilation. When you mex a C++ source file on the Mac (confirmed up to Matlab 7.12), the system generates statically linked output (a statically linked executable I think, but it could be a static library; it doesn't really matter). On Linux, mex generates a shared library instead, which complicates things significantly when your build depends on the boost_thread library (see below).

For now, let's assume you are on a Mac or some other Matlab system that generates static mex files. One way to confirm this is to run mex on something simple with the -v option; this will show the gcc or g++ commands that mex is using. As long as the linking step doesn't include the -shared flag, you're okay.

In this case, your first step is to follow the getting started instructions on the Theron site: download Theron and build it, linking against your choice of Boost threads library. If you're not sure whether you have Boost threads installed already, you can try the locate libboost_thread command, possibly after running sudo updatedb if your locate database is out of date. If you don't have Boost threads already, the simplest way to get it is to use Macports. Once you have Macports installed, it's as simple as: sudo port install boost. Currently this gives you a very up-to-date boost version 1.47.

Theron is currently a little Windows-centric, so you'll need to tweak the makefile a little to get it to build. Theron author Ashton Mason tells me the next release should make things smoother for Gnu (non-Windows) people. First, as detailed in the Theron guide, you need to edit the makefile to point to your Boost installation. If you got Boost from Macports, this will probably look like this:

BOOST_INCLUDE_PATH = /opt/local/include/
BOOST_LIB_PATH = /opt/local/lib/
BOOST_RELEASE_LIB = boost_thread-mt
BOOST_DEBUG_LIB = boost_thread-mt

If you want to be able to run make clean to start over with the build process, I recommend also changing the following lines:

RM = rm -f

...

clean:
  ${RM} ${BUILD}$/*.o
  ${RM} ${BUILD}$/*.a
  ${RM} ${BUILD}$/*.ilk
  ${RM} ${BIN}$/*.pdb
  ${RM} ${BIN}$/*.ilk
  ${RM} ${BIN}$/*.exe
  ${RM} ${LIB}$/*.a
  ${RM} ${LIB}$/*.lib

I think that should be it, so go ahead and build. If you are only planning to use Theron from Matlab you only need the library, so I would use make library mode=release.

If all goes well, your build should finish pretty fast and you should now have the libtheron.a library in your THERONPATH/Lib folder. Now boot up Matlab and try to build a simple mex file. A Theron test program might look something like:

// theron_test.cpp
#include "mex.h"
#define THERON_USE_BOOST_THREADS 1
#include "Theron/Framework.h"
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
  Theron::Framework theron;
}

This looks simple, but actually the call to initialize the Theron Framework involves a fair amount of background stuff getting setup, so it should give you an indication of whether things are working. To compile this from Matlab, you'll need a little more than a simple mex theron_test.cpp.

Do not put spaces in any of these mex options; no space after the "D". First you need to tell the compiler what Theron function calls look like. You do this by pointing it to the Theron header files with the -I/THERONPATH/Include/ directive.

Next you need to tell mex to pull in the theron and boost_thread libraries. The standard way to do this is with the -L and -l options to the linker, but I find it cleaner to just include the full paths to the library files after your C++ file name; this tells the linker to look through the object files in the libraries when resolving the calls to Theron in your code. I also like using the -v option so I can see exactly what mex is up to.

So your full build command should look something like:

mex -v -I/THERONPATH/Include/ theron_test.cpp /THERONPATH/Lib/libtheron.a /LIBPATH/libboost_thread.so

If you got Boost from Macports, then the libboost part above should be /opt/local/include/libboost_thread-mt.dylib. You could also use libboost_thread-my.a if you want; I don't think it matters much.

Did it build? Hope so... Now try running theron_test from the Matlab command line. You should see nothing; nothing is good. Now go ahead and code up your algorithm using the Theron tutorial as a guide and enjoy simple Matlab concurrency!

Building Theron on Linux or other shared library mex environments

The process is a bit more complicated for Linux users. The upside is, you probably don't have to worry about getting your own Boost implementation (although chances are you already have one...). The issue is that in Linux, Matlab mex files are built as shared libraries instead of statically linked. This causes two problems. The immediate problem is that Matlab needs you to build Theron with the -fPIC option turned on so that it can use the Theron library in a shared build. This is not a big problem; just add -fPIC to the CFLAGS in the Theron makefile.

The second problem is the deal breaker: Matlab uses libboost_thread internally and has its own copy hidden away in its internal distribution folders. Since your mex file got built as a shared library, it will try to pull the boost_thread library in at runtime, and it will find Matlab's version (1_40_0) instead of your external version (1_42_0 currently in apt-get). This causes a segfault.

So there are two approaches to solving this. The better one would be to find some way to tell your mex file not to use Matlab's boost_thread. I found a discussion of this on MatlabCentral, but did not get this working; it seems like it might require rewriting parts of Theron to use dlopen; not very nice. I still feel like there should be some other way to get Matlab to behave, but I didn't find it. (For completeness: it does not work to point Matlab's internal boost_thread to an updated one; duh why did I even try that?)

So there is option 2: throw in the towel and build Theron against Matlab's internal boost_thread. This works at least for 64-bit Matlab 2010 or 2011, which has boost_thread 1.40.0 internally. I don't know that it will work for earlier or 32-bit versions of Matlab, which I believe use earlier versions. (A very old version of Matlab might not have this issue at all, as I think it was around 2007 that MathWorks started multithreading Matlab internal code in the first place. If you're using a very old Matlab, check the internal distribution folders for libboost_thread to see if it's there. locate should be your friend.)

Building Theron against Matlab's internal boost_thread is not too hard. The only trick is that Matlab doesn't have the header files in the distribution (at least I didn't find them). So you'll have to download boost_1_40_0 (or whatever version Matlab has internally on your system) and point Theron to those headers. If you use the wrong headers, you'll get segfaults. Theron's makefile should look something like:

BOOST_INCLUDE_PATH = /Downloads/boost_1_40_0/
BOOST_LIB_PATH = /usr/local/MATLAB/R2011a/bin/glnxa64/
BOOST_RELEASE_LIB = boost_thread
BOOST_DEBUG_LIB = boost_thread

...

CFLAGS += -c -Wall -fPIC

(The libboost_thread in the Matlab folder is only the versioned filename, not the soname, so I would have thought you might need to symlink a soname but that didn't seem to be necessary.)

If this builds correctly, you should be good to build your mex file from Matlab; the command is the same as above for Mac, except you point it to the internal Matlab boost_thread:

mex -v -I/THERONPATH/Include/ theron_test.cpp /THERONPATH/Lib/libtheron.a /MATLABLIBPATH/libboost_thread.so.1.40

Conclusions

This was a little bit of a pain, mostly because of Matlab's questionable decision to use shared libraries internally without (apparently) providing for situations where users build their mex files against the same libraries. The process is especially annoying if you have Mac and Linux machines that both need to be able to build your mex. Some steps would be a little smoother if Theron were less Windows-centric; Theron has a new release coming that should improve on this.

I think the current solution is stable, although it's tricky to be locked into the Matlab internal boost_thread version. The benefit is clear. Using Theron to multithread a simple algorithm, I got about 4.5x speedup using 6 cores, and 6x speedup using 12 threads on 6 cores with hyperthreading. This was a simple algorithm that didn't really require an Actors model, but it was great that I could get this done quickly (once I figured out the Matlab setup tricks) and without having to deal with low-level threads and mutexes at all.

Building Theron on Windows

As noted, I have not gotten any Windows compiler to work with mex, so these are purely hypothetical guidelines. If you are building on Windows you will probably find it simpler to use the Windows thread library instead of Boost. So just leave out the THERON_USE_BOOST_THREADS define and it should revert to using Windows threads instead. This means you should be safe from the internal Matlab shared library conflict problem that Linux has. But you may still need the -fPIC option added to your CFLAGS depending on whether Windows mex is setup to create static or shared libraries (I don't know).

Simple(r) concurrent Matlab programming with Theron

2011-09-02T14:15:00.018-07:00

Preamble: What follows are my (long-winded) musings on the current (Sep. 2011) state of concurrent programming in Matlab. For a specific guide to using the Theron C++ library for Actors concurrency in Matlab, see the following post.

Concurrent Matlab

Matlab is a standard programming environment for scientific calculations. In our lab, we use it for a lot of analysis, especially prototyping new analyses. Eventually, some of the performance critical stuff gets ported over to Java, but we still have plenty of things we need to run that are Matlab-only and need to be fast.

As personal computers add more and more cores without improving clock speeds much, there is more and more need for parallelized (AKA concurrent, multithreaded) software. Matlab is not a very popular language for supercomputing, but high-performance clusters also of course demand parallelized software. Matlab has some support for multithreading internally, so depending on which Matlab internal functions you use, you may see significant multicore use on your machine. But there is currently no support for explicitly writing multithreaded programs in vanilla Matlab.

One solution is to start multiple instances of Matlab and use communication abstractions like sockets (also not supported in vanilla Matlab) or the file system to interact between the processes. This is an okay solution for coarsely (AKA embarrassingly) parallel algorithms. Several nicely abstracted solutions exist. Matlab's own Parallel Computing Toolbox with its parfor construct for example. Free solutions include the multicore package from MatlabCentral FileExchange, and MatlabMPI/pMatlab from MIT. Multicore and pMatlab both use the file system for interprocess communication. (They have somewhat different abstraction models that make them appropriate for somewhat different styles of multithreading.) One upside to using the file system for interprocess multithreading is that it is trivial to use a network file system to make your code not only multiprocess but actually distributed to multiple machines.

But what if you need more finely grained parallelism, for example you want to code algorithms that flexibly break a problem into hundreds of sub-tasks and maximally exploit available hardware concurrency? At present, this seems to require you to either give up on Matlab entirely, or else write lower-level code that links into Matlab. The two methods I'm familiar with for linking lower-level code into Matlab are to write Java, which Matlab communicates with fairly smoothly, or else write C/C++/Fortran and bring it in with Matlab's "mex" functionality.

Java core concurrency is established, but based around low level threads and locks. It's a minefield of synchronization issues. There are libraries available for more abstracted approaches, but I haven't worked with them. And if a contained segment of code is really performance critical, writing it in native C/C++ makes sense.

So what are the capabilities for writing concurrent C++ and linking it into Matlab? Unfortunately current C++ compilers do not have native concurrency support. But this is expected to change in the not-too-distant future. C++11, the new standard, was approved last month with a new thread library based on the threading in the popular Boost distribution. Unfortunately, the C++11 threading standard looks a lot like Java core concurrency, built around low-level threads and mutexes with no support for higher-level, safer, simpler concurrency abstractions. Luckily, there are libraries available offering concurrency abstractions for C++. The one I'd like to highlight today is Theron.

Theron

Theron is a concurrency library for C++ based on the Actors model, in which most of the low-level thread/mutex concurrency implementation details are hidden from the user. It is not a strict Actors model based tightly on the academic definition of Actors, but it is in that spirit. It offers an Actors abstraction based on a backing thread pool, so that creating new Actors is cheaper than actually creating new threads, allowing you to trivially have hundreds or thousands of Actors in one program. The pool of worker threads services the Actors efficiently, allowing high exploitation of available hardware concurrency. Actors are also defined to communicate only through message passing and not through shared memory, eliminating most of the booby traps that make low level concurrent programming such a pain. (There are ways to pass shared memory between Theron Actors if you really need to, e.g. if you have large input data structures you don't want to copy into messages. It's then up to the user to ensure that the Actors interact with the shared memory in well defined, safe ways. Nevertheless I have found that the Actors abstraction is very helpful for simplifying concurrent code even if some degree of shared memory is slipped in.)

Another nice feature of Theron is that it is designed to allow easy swapping in of different underlying thread models. Currently it has wrappers for native Win32 threading as well as Boost threads. The expectation is that as C++11 std::thread compilers become available, it will be easy to adapt Theron to use the standard threading. Theron also has a very nice website with clear documentation of the simple, powerful API, a clear tutorial, getting started guide, and good user support. Although Theron has historically been somewhat Windows-centric, I was able to download Theron, build it on my work Mac, and get a multithreaded program running in Matlab in about a day. Getting it working in Linux took longer, but I think I have it figured out now too; see next post for details.

Update: libcppa

I recently came across another C++ Actors project, libcppa. This is much less mature than Theron, and relies on a lot of C++11 features for its core functionality, but it looks interesting. It's hosted on GitHub, so could be easy to dig into despite not a lot of documentation. But Theron is clearly the choice for the time being.

Broken Mac OS 10.6 Java

2011-08-04T10:37:00.007-07:00

Apple has a pretty lousy history with Java support, but this still surprised me. If you are having trouble getting programs to compile with older Java versions (e.g. for backward compatibility), you might want to take a look in this directory: /System/Library/Frameworks/JavaVM.framework/Versions

Here's what I found on my OS 10.6 system:

lrwxrwxr-x   1 root  wheel    5 Jun 29 13:37 1.3 -> 1.3.1
drwxr-xr-x   3 root  wheel  102 Feb 11  2010 1.3.1
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.4 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.4.2 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.5 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.5.0 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.6 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.6.0 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   59 Jun 29 13:37 CurrentJDK -> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents

I kept trying to get Eclipse to compile with 1.5 and kept getting incompatible binaries. Thanks Apple! YMMV, but I found this post a good guide to getting a real Java 1.5 installation.

Mac OS 10.6.8 breaks OpenSSL dev compatibility

2011-07-28T14:03:00.010-07:00

Update (2011-07-29): Globus Toolkit devs have told me that this is a frequent problem with Mac OS updates. Thanks Apple. So they recommend running with the --without-openssl-header-check flag if you don't want to go to the trouble of actually installing compatible headers.

If you are trying to build anything that requires OpenSSL on Mac OS 10.6 and have updated to 10.6.8, you may run into the following complaint:

checking OpenSSL header version... 9080cf (OpenSSL 0.9.8l 5 Nov 2009)
checking OpenSSL library version... 90812f (OpenSSL 0.9.8r 8 Feb 2011)
checking whether OpenSSL's headers match the library... no
configure: error: Your OpenSSL headers do not match your library. Check config.log for details.
If you are sure your installation is consistent, you can disable the check by running "./configure --without-openssl-header-check".
Also see contrib/findssl.sh for help identifying header/library mismatches.

For some reason, when Apple upgraded to OpenSSL 0.9.8r to address security issues they failed to update the headers, so now there is a version mismatch (i.e. header version is 0.9.8l but library version is 0.9.8r).

If you want to track this issue in more detail, you can find a copy of the findssl.sh script, either on your system somewhere or just find it online. If you download it online, you will probably have to hack the script to find Mac's .dylib shared library files instead of the Linux standard .so files. Alternatively, just try these commands:

locate opensslv.h
locate libcrypto.dylib

That should show you where the different versions of headers and libs are respectively. Then you can look inside the opensslv.h file in /usr/include and confirm that it is 0.9.8l instead of 0.9.8r.

In any case, the solution seems to be:

Get OpenSSL 0.9.8r from http://www.openssl.org/source/
Untar, copy the include/openssl directory to somewhere like /usr/include/openssl-0.9.8r
Point your build to these new headers. I decided to just blast the incompatible headers, so I moved my old /usr/include/openssl to /usr/include/openssl-0.9.8l and then symlinked openssl to openssl-0.9.8r.
Now it builds!

Combining USDA SR23 with glycemic index data

2011-04-22T17:16:00.007-07:00

USDA provides a National Nutrient Database for Standard Reference currently in Version 23 (SR23), which has loads of nutritional data on a good selection of food products. On my wishlist would be cross-referencing with UPC bar codes, as well as some additional data that are currently missing, such as glycemic index and separate soluble vs. insoluble fiber values.

I did find one website that has done a pretty job with glycemic index already: nutritiondata.self.com. In addition to pulling in glycemic index data, they've also calculated glycemic load and then built a statistical model to try to infer/impute glycemic load for foods that don't have published glycemic index data. And they also have some other summary statistics that they've developed in-house such as "Fullness Factor" and an overall healthiness rating. Kudos to them, but I wanted to play with these data myself. Also I didn't see a fiber breakdown there.

One data source I found that has both glycemic load and soluble vs. insoluble fiber (at least for some entries) is the Diet History Questionnaire II (DHQ2). The DHQ is available to download as a single giant table. I plan to play with this.

Another option is to link the glycemic index data derived for the DHQ back to SR23. This appears to be possible thanks to a couple of additional tables. First, DHQ provides their glycemic index data in a table referenced against food codes from the USDA Continuing Survey of Food Intakes by Individuals (CSFII). These codes match the food codes used in the USDA Food and Nutrient Database for Dietary Studies (FNDDS). The FNDDS database includes a table that links selected FNDDS/CSFII codes back to one or more SR23 codes. (Since my main interest in glycemic index is to compare glycemic index to sugar composition, and FNDDS only has a value for total sugar not individual components, I'm not too interested in FNDDS by itself.)

I'll be exploring DHQ vs. SR23 (augmented with glycemic indices linked from FNDDS+DHQ) and will report back.

Globus Toolkit 5.0.3 on Cygwin 1.7

2011-02-20T11:46:00.004-08:00

Cygwin patches (see earlier post) did not make it into GT5.0.3. The patch requirement is therefore the same as for GT5.0.2 (see earlier post). These patches may make it into future distributions.

I took slightly more detailed notes during the 5.0.3 build process, so you can now find more details about which packages require patches here.

DIY KitchenAid attachments

2011-02-14T13:06:00.009-08:00

Update: (2011-03-14): So this does not quite fit perfectly into the Attachment Hub, but it is good enough that with a housing for the shaft I'm pretty sure it will work. I'll take more precise measurements and post any updates.

Update (2011-02-18): Arrived no problem from Mending Shed; Parcel Post was quite fast actually. Don't have the KitchenAid here with me right now so will not be able to test at all for a few weeks.

Our KitchenAid stand mixer is probably the fanciest piece of hardware we own other than our car. I've been thinking for a while that there really ought to be a DIY modding community around this appliance; it's a powerful, well built kitchen motor waiting to be exploited (plus planetary gearing if that is helpful). But I haven't found much so far. Here's one nice rotisserie mod I found, and that's about it.

KitchenAid makes a number of official attachments. Some of them attach to the bottom head where the planetary gearing is, while others attach to a port on the front (the "Attachment Hub") that accepts square shafted ("Power Shaft") attachments. I searched around a little and was able to find a part that I'm pretty sure will fit the attachment hub: KitchenAid 9709315, the "clutch shaft". Amazon sells it.

Note that this doesn't include the shaft housing that will allow you to lock it into place; figuring out a way to hack this together will be the first job before anyone can really use this thing for DIY projects. I'm wondering if putting something together in plastic through Ponoko or the like will get the job done. It probably requires some bearings too.

I ordered one from Mending Shed (slightly cheaper than Amazon and will ship Parcel Post), will update with progress.

Faster median calculation and generic rank selections in MatLab via nth_element

2010-11-19T14:52:00.007-08:00

Update (2011-09-03): I've added new functions to the release that edit the arrays in-place. This is not good Matlab style but gives significant speedups for the very performance conscious. It appears it may be useful for me to add a multithreaded version as well, so this may come in an upcoming release.

C++ std::nth_element is a standard specification for an efficient rank selection algorithm. This algorithm can be used to solve a number of rank selection problems efficiently.

One example is median finding. A naive median finding algorithm, as implemented in MatLab (at least up to R2010a) is to sort the entire list of values and then just take the value in the middle position. But sorting the entire list is overkill; it is more efficient to do a sort of partial sort just to the point where the desired value is in the correct position. The version of this typically implemented in nth_element is known as quickselect or Hoare's Selection Algorithm. See Wikipedia: Selection algorithm for more details.

I find that a MatLab fast_median function based on nth_element runs about twice as fast as the native MatLab median function based on sort. Theoretically, the average complexity of quickselect is O(n), while the best performance complexity of the sort based method is O(n log n).

Finding median values is central to robust statistics, for example calculating the Median Absolute Deviation.

If you are interested, my MatLab wrapping of nth_element and an example implementation of fast_median are available at MatLab Central File Exchange.

Startup GNU Screen with multiple regions (split screen)

2010-10-14T10:18:00.008-07:00

Since I've been doing a lot of work remotely lately I've been brushing up my GNU Screen skillz.

One thing I'd like is to be able to reattach a screen with a number of regions (AKA split screens) setup automatically. Screen is supposed to get this in the next release, but it's been a while.

In the meantime, the following kludge seems to work well enough: (sleep 0.1; screen -X split) & screen -r. Basically, this says: "reattach to my screen and at the same time in the background wait a fraction of a second and then send the split command to screen from a separate thread".

This was a common enough pattern that I wrote it into a little shell script:

#!/bin/sh

splits=$1
if [ -z $splits ] ; then
  splits=1
fi

sleep=$2
if [ -z $sleep ] ; then
  sleep=0.1
fi

sleep $sleep
i=0
while [ $i -lt $splits ] ; do
  screen -X split
  screen -X focus bottom
  screen -X select `expr $i + 1`
  i=`expr $i + 1`
done

which is used: splitscreen 4 & screen -r.

There's plenty of room for improvement. A more principled replacement for the call to sleep would be to monitor the socket files. In my Mac OS 10.5 these are under /tmp/uscreens/.... (I believe the standard location is /usr/tmp/screens/...) This is purely empirical, but it appears the user permission on the socket file changes from -x to +x (i.e. becomes executable) when the screen is attached. So instead of sleep we should be able to do a wait until that permission change occurs.

In principle I think this could even be used to save and restore region settings. For example, rebind the split key to not only split but also save a flag to filesystem indicating how many regions, then read this flag when reattaching. Or we could just wait for the next release...

Bluegiga WT12 Breakout Board V0.2

2010-10-07T20:35:00.004-07:00

I just got my first ever printed circuit boards back from BatchPCB. Pretty exciting! One of the boards is a Toyota audio system interface, but it needs to have a chip replaced and the pinout is different, so that board is not totally usable. The other board is this breakout board for the Bluegiga WT12 bluetooth module.

This breaks out all the pins so you have complete freedom, and it's still pretty small. I have another version I will probably release that breaks out only the UART pins and is compatible with the BlueSmirf pinout. That one is much smaller.

The WT12 has iWrap4 firmware that is a very simple interface to a variety of common Bluetooth profiles. It's quite affordable; please see my Flickr page and BatchPCB page for more information and ordering.

Chords: "Holiday" by the Kinks

2010-10-01T00:51:00.013-07:00

The chords posted around online are a bit wrong. And it's such a great, simple song! Fixed it up a little:

Artist-The Kinks
Song-Holiday
Album-Muswell Hillbillies
Writer-Ray Davies
Tabbed by S. McNally, Peter H. Li


Intro
A-F#-B7-E7


Verse 1
A     F#     B7
Holiday,
          E7     A     F#
Oh what a lovely day today,
B7        E7        A       F#
I'm oh so glad they sent me away,
   B7            E7   A    F#   Bm Eb E
To have a little holiday today, holiday.

Verse 2
A     F#
Holiday,
B7           E7              A        F#  
And I'm just standing on the end of a pier,
B7          E7      A        F#
Hoping and dreaming you were here,
B7          E7     A            (turnaround: A F E C A ???)
To share my little holiday.


Verse 3 (type II)
A              F#        B7         E7
Lookin' in the sky for a gap in the clouds,
A           F#                   B7           E7
Sometimes I think that sun ain't never coming out,
        A         A7/G              D          B7
But I'd rather be here than in that dirty old town,
  E                                             (E+)
I had to leave the city cos it really brought me down.


Verse 4
Oh holiday, oh what a lovely day today,
I think I'll get down on my little ol' knees and pray, thank you Lord,
Thank heaven for this holiday today, holiday.


Bridge
    D           B7       A
I'm leaving insecurity behind me,
    D             B7               A
The environmental pressures got me down,
C#              C#7          D       B7
I don't need no sedatives to pull me round,
E                                              (E+)
I don't need no sleeping pills to help me sleep sound.


Verse 5
Oh holiday,
Oh what a lovely day today,
I think I'll get down on my little ol' knees and pray,
That's what I'll do,
Thank heaven for this holiday.

Verse 6 (type II)
Lying on the beach with my back burned rare,
The salt gets in my blisters and the sand gets in my hair,
And the sea's an open sewer,
But I really couldn't care,
I'm breathing through my mouth so I don't have to sniff the air.

Verse 7
Oh holiday,
Oh what a lovely day today,
I'm so glad they sent me away,
To have a little holiday.

(I sure miss OLGA sometimes...)

Globus Toolkit 5.0.2 on Cygwin 1.7

2010-09-29T14:39:00.013-07:00

Updates (2011-02-20): Cygwin patches did not make it into GT5.0.3, but they may make it into a future release.

Updates (2010-12-08): Binghamton University TeraGrid has written up instructions for applying the patches and getting GT compiled. Also, it appears new versions of GT will be coming out over the next few months. I plan to check them out for Cygwin compatibility.

If you are interested in compiling Globus Toolkit 5.0.2 on Cygwin, I have some simple patches that seem to get the job done. Perhaps they will make it into the next release. For now: http://lists.globus.org/pipermail/gt-dev/2010-September/001885.html

Note that I built this on Cygwin 1.7; Cygwin 1.5 will probably not work. Also you must have the gcc4.x Cygwin packages (C and C++) installed. Getting GT to build on gcc3.x is probably not much more work, but going with gcc4 is easier than worrying about unrecognized compiler flags all over the place.

You also need to specify the flag: --with-buildopts="-static=1" when running the main configure script. Building GT on Cygwin with dynamic linking is certainly beyond me, and is a pretty big undertaking, so this tells it to use only static linking. I recommend --with-buildopts="-verbose -static=1" so you can also see what is going on.

(For the record, on dynamic linking: I tried adding some flags to configure.ac or configure.in to let autoconf know that we wanted to make DLLs and do dynamic linking. But trying to run a modern version of autoreconf generated configure scripts that don't work with the GT package. Even if the build worked, this probably wouldn't be a good idea as making DLLs properly requires going through all the source and making sure data exports are handled correctly. At least one package (globus_libtool) is DLL ready and already generates a DLL and RTL properly, so we know it all works theoretically once the individual libraries are DLL ready. There may be other subpackages that are DLL ready, but they will probably have to be handled one at a time. Unfortunately, GPT doesn't really offer a method to build the whole Globus Toolkit with some parts statically linked and some dynamic.)

If you get a build failure with an error like "fork()" or "vfork()" "resource temporarily unavailable", this means your machine ran out of processes while building (due to what I'm going to call a "process leak" bug). Basically just rerun make and it should get farther each time before crashing, until finally it gets all the way through. Or build GT in smaller chunks; if you are just trying to get proxy certificate running you only really need "make gsi-myproxy gsi-openssh".

You may also have to circumvent GPT's somewhat outdated Perl libraries. If you have a problem getting Archive::Tar to open some of the source packages, then you should install the Cygwin CPAN package and then use that to install an updated version of Archive::Tar. You may then need to hide GPT's versions under $GLOBUS_LOCATION/sbin/perl.

Feel free to ping me with any questions/thoughts.

Eclipse Windows 7 Plugin Problems

2010-08-18T00:13:00.002-07:00

Wow, Eclipse is really a piece of work. I wasted an hour trying to get it to behave reasonably with multiple plugins installed on Windows 7. Online forums were little help until I finally came across the suggestion to install Eclipse into a user directory, i.e. one that doesn't require administrator privileges. Finally fixed; what a waste of time!

Compiling GnuCAP in CygWin

2010-08-07T14:54:00.009-07:00

Update: GnuCAP is now available as an official CygWin package, as is some dialect of SPICE.

Very simple patch to get recent development snapshots of GnuCAP to compile in CygWin:

--- md.h.bak    2010-03-19 00:21:26.030900000 -0700
+++ md.h        2010-03-19 00:34:18.810900000 -0700
@@ -100,6 +100,12 @@
  #define MS_DLL
#endif
/*--------------------------------------------------------------------------*/
+#ifdef __CYGWIN__
+  #ifndef RTLD_LOCAL
+    #define RTLD_LOCAL 0
+  #endif
+#endif
+/*--------------------------------------------------------------------------*/
/* some convenient names */
typedef std::complex COMPLEX;
typedef std::pair DPAIR;

Mac-style wheel zoom, Exposé/Switcher on Windows 7

2010-05-13T07:53:00.033-07:00

Having a Mac at work has been a mixed blessing, mostly good. There are some great OS features that I've become addicted to. Happily, I have discovered ways to duplicate them on my Windows 7 home machine, and with better customizability even.

Wheel zoom

On Mac OS, holding command and using the mouse wheel zooms you in and out of your screen view. This is on the operating system level, so it works in any application. Windows 7 has a similar (i.e. copied) feature where holding the windows key and pushing plus/minus zooms. Obviously it's nicer to use the mouse wheel. Luckily, with AutoHotKey it's easy to set this up on Windows and even make it better.

After installing AutoHotKey, run the following script:

; Only one instance running at a time
#SingleInstance force


; Mac-like zooming
~LCtrl & WheelUp::
  sendinput #{NumpadAdd}
return

~LCtrl & WheelDown::
  sendinput #-
return

~LWin & WheelUp::
  sendinput #{NumpadAdd}
return

~LWin & WheelDown::
  sendinput #-
return

I derived this from AeroZoom by wandersick, which is an excellent set of shortcuts for presentations. I pulled out just the very simple zooming functionality and made it duplicate the Mac functionality more. Thanks to wandersick for putting his tool out with the source!

My mouse has side buttons that I usually use just for forward/back. I added the following so that now I can do the wheel zoom without the keyboard at all, just by holding the left side button and wheeling:

; Additional mouse-only zooming
~XButton1 & WheelUp::
  sendinput #{NumpadAdd}
return

~XButton1 & WheelDown::
  sendinput #-
return

Exposé/Switcher

Exposé is the Mac OS feature that makes it easy to find windows when you have a thousand different things open at once. There is a very similar application for Windows called Switcher. I like having Mac OS setup so that when you push the middle button (the wheel) it calls up Exposé. With AutoHotKey, of course, it was simple to set up identical functionality on Windows 7 via Switcher.

On my home mouse, I find it easier to push both side buttons simultaneously than to push the wheel down, so I set it up that way. I also had trouble getting this to work with Switcher's default activation button (windows-`), so I added an additional Switcher hotkey (Switcher is very nicely customizable too) and then just setup AutoHotKey to trigger that:

; Mac-like expose, hooking into switcher
~XButton1 & XButton2::
  send #s
return

Hope that helps some other people suffering from Mac lust. Enjoy!

Cloud patterns from space

2010-05-13T07:50:00.002-07:00

WiReD has a cool gallery of satellite cloud photos.

Mourning dove feeding

2010-01-20T16:16:00.009-08:00

Finally got around to posting a video of the mourning doves that lived in our tomato plant last summer. I was working from home at the time, and this was the window right next to my desk, so I have a lot of videos and photos, but this is one of the more interesting ones: feeding time!

Dapple

2009-12-03T16:53:00.041-08:00

On vacation in Paris last month we visited the Musee D'Orsay to see some of the classics of Impressionism. One of my current favorites (and one of the most expensive paintings ever sold) is Renoir's Bal du Moulin de la Galette:

What I particularly like about this painting is Renoir's careful depiction of sunlight dappling through trees. Look at the man in the foreground, facing away from us, with the curious round dabs of sunlight across his back. If you focus on this point and think about it, the appearance of the dapples seems a little strange, yet holistically it feels right.

In fact, what Renoir has captured here is a true optical curiosity. As discussed by Marcel Minnaert in Light and Color in the Outdoors, sunlight dappling through trees often produces these beautiful, perfectly round or elliptical spots of light:

As Minnaert points out: "the surprising thing is that all these images have the same shape, although it is obviously impossible that all the holes and slits in the foliage happen to have the same shape."

This happens because holes in foliage that are sufficiently small result in dapples whose shape is dominated by the image of the sun, i.e. a disc, rather than the shape of the hole. If the sun were a mere point source rather than having a significant diameter, we would not see this effect. (I believe that the shape of the spot of light produced is essentially just the image of the sun, scaled by the projection throw length and then convolved with the shape of the hole, neglecting diffraction effects...)

This effect is most dramatic when something distorts or obscures the image of the sun. A classic example is the change in the appearance of tree dapples during a solar eclipse: as the sun becomes obscured, its image becomes crescent shaped, as do its dapples through the trees:

Update: new eclipse photo, this time at annular stage:

Gifts for etymologists, pt. 1

2009-09-14T21:08:00.006-07:00

In the Styx

2009-08-19T06:41:00.008-07:00

In The Amazing Adventures of Kavalier and Clay, Michael Chabon uses the idiom "out in the sticks", meaning "in the boondocks", "in the middle of nowhere". According to EtymOnline, this idiom arose around 1905, simply from the meaning of sticks as in trees as in a rural place.

I've heard the phrase plenty of times, but this must be the first time I've seen it (or at least first time I've noted it) in print. Without really thinking about it, my mental image of this phrase had always been "out in the Styx". Which of course, now that I actually think about it, seems pretty unlikely. Definitely more fun my way though...

Marketing Coup

2009-08-13T21:37:00.008-07:00

Overdone sales mailer from The Economist: "Handle with care. Contents may inspire."

Roget's Thesaurus:

Inspire: to breathe in
Synonyms: inhale, gasp, suck

Work around Selenium IDE's iteration deficiency with Prototype JavaScript evals

2009-08-06T08:32:00.011-07:00

Selenium IDE, the FireFox testing plugin, is a nice tool. One of its limitations is the lack of straightforward methods for iterating or doing other control flow during tests. This is one reason to switch to writing tests in Selenium RC. However, it is sometimes possible to work around the limitations from within Selenium IDE by evaluating arbitrary JavaScript. In particular, Rails applications using the standard Prototype JavaScript library have available some nice ways to iterate over elements.

For example, I recently wanted to write a Selenium IDE test to clear the shopping cart for a web app. To do this, I needed to iterate over all the elements in the cart, set their quantity to 0, and then update the cart. To accomplish the iterative set quantity to 0, we can use the getEval method to run arbitrary JavaScript: selenium.browserbot.getUserWindow().document.body.select('input[id^="line_item_"][id$="_quantity"]').each(function(elem){elem.value = 0}).

Going through this piece by piece:

  // This gets us the user window.  
  // Older versions might use getCurrentWindow() instead.
  selenium.browserbot.getUserWindow()
  
  // Gets us down to the document body
  // Because this is a standard Rails app, 
  // body has already been extended with Prototype.
  // select() invokes Prototype's $$() CSS selector.
  document.body.select
  
  // These are CSS3 attribute selectors
  // Basically we want to match id="line_item_*_quantity".
  // May require FireFox >= 3
  input[id^="line_item_"][id$="_quantity"]
  
  // Prototype's Enumerable.each() does the iteration,
  // with an anonymous function setting the values.
  each(function(elem){elem.value = 0})

Getting Selenium to wait for Ajax requests

2009-08-05T15:18:00.012-07:00

Update: See here for one caveat in using Ajax.activeRequestCount.

Using Selenium IDE to test Ajax, you can usually use a waitFor condition reflecting on DOM elements to tell you when Ajax calls have returned. However, if you just want to wait until all Ajax requests have returned without relying on testing against page elements, there is this approach:

Command: waitForCondition
Target:  selenium.browserbot.getUserWindow().Ajax.activeRequestCount == 0
Value:   10000

This works for Prototype based Ajax pages, e.g. standard Rails. If you're using a different JavaScript library, you'll need to use a different call; see here for some possibilities.

(For earlier versions of Selenium, you might need to use getCurrentWindow() instead.)

Getting revision number in Tortoise Svn

2009-08-05T10:20:00.001-07:00

http://stackoverflow.com/questions/89741/can-i-see-the-currently-checked-out-revision-number-in-tortoise-svn

Going back to earlier revisions in Subversion

2009-08-05T09:50:00.011-07:00

There are basically two different scenarios where you will want to rewind (AKA roll back, revert) your code base to an earlier version.

You just want to have an earlier version of the code, but don't need to work-on / commit-against the earlier revision.
You actually want to roll your project back to an earlier version and then work-on / commit-against the project with the earlier revision as your starting point.

To accomplish #1, you simply update your code back to the earlier revision (yeah, it's an update operation even though you're going backwards). Alternatively, if you want to be very antiseptic, you could checkout the earlier revision.

#2 is considerably more complicated. There are hacky ways to handle it, but the "proper" method is described here: http://jacwright.com/blog/75/how-to-roll-back-changes-using-subversion/

next as a way to return from a block

2009-08-03T22:48:00.044-07:00

In Ruby, you can't return from a block with return; this will instead return from the enclosing method; if there is no enclosing method (i.e. you are at toplevel), this will throw an error.

However, the Ruby next statement takes an argument, and seems to work as a means to return from a block without breaking out of any enclosing context.

Demonstration:

def yielder
 ret = yield
 puts 'After yield'
 return ret
end

Now from irb:

>> VERSION
=> "1.8.6"

>> yielder{1}
After yield
=> 1

>> yielder{return 1} # This tries to return from toplevel...
LocalJumpError: unexpected return

>> yielder{break 1} # This breaks completely out of the call to yielder...
=> 1

>> yielder{next 1; puts 'In block'} # Aha! This breaks out of the block but stays in the method!
After yield
=> 1

As an example of how this can be useful:

  mixed_collection.sort do |x,y|
    next x.class.name <=> y.class.name if x.class != y.class
    x <=> y
  end

My main question at this point is, does this work in Ruby 1.9?

Update: Works in Ruby 1.9.1