Broken Mac OS 10.6 (and 10.7 and 10.8 and...) Java

Apple has a pretty lousy history with Java support, but this still surprised me. If you are having trouble getting programs to compile with older Java versions (e.g. for backward compatibility), you might want to take a look in this directory: /System/Library/Frameworks/JavaVM.framework/Versions

Here's what I found on my OS 10.6 system:

lrwxrwxr-x   1 root  wheel    5 Jun 29 13:37 1.3 -> 1.3.1
drwxr-xr-x   3 root  wheel  102 Feb 11  2010 1.3.1
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.4 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.4.2 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.5 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.5.0 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.6 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   10 Jun 29 13:37 1.6.0 -> CurrentJDK
lrwxrwxr-x   1 root  wheel   59 Jun 29 13:37 CurrentJDK -> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents

I kept trying to get Eclipse to compile with 1.5 and kept getting incompatible binaries. Thanks Apple! YMMV, but I found this post a good guide to getting a real Java 1.5 installation.

Mac OS 10.6.8 breaks OpenSSL dev compatibility

Update (2011-07-29): Globus Toolkit devs have told me that this is a frequent problem with Mac OS updates. Thanks Apple. So they recommend running with the --without-openssl-header-check flag if you don't want to go to the trouble of actually installing compatible headers.

If you are trying to build anything that requires OpenSSL on Mac OS 10.6 and have updated to 10.6.8, you may run into the following complaint:

checking OpenSSL header version... 9080cf (OpenSSL 0.9.8l 5 Nov 2009)
checking OpenSSL library version... 90812f (OpenSSL 0.9.8r 8 Feb 2011)
checking whether OpenSSL's headers match the library... no
configure: error: Your OpenSSL headers do not match your library. Check config.log for details.
If you are sure your installation is consistent, you can disable the check by running "./configure --without-openssl-header-check".
Also see contrib/findssl.sh for help identifying header/library mismatches.

For some reason, when Apple upgraded to OpenSSL 0.9.8r to address security issues they failed to update the headers, so now there is a version mismatch (i.e. header version is 0.9.8l but library version is 0.9.8r).

If you want to track this issue in more detail, you can find a copy of the findssl.sh script, either on your system somewhere or just find it online. If you download it online, you will probably have to hack the script to find Mac's .dylib shared library files instead of the Linux standard .so files. Alternatively, just try these commands:

locate opensslv.h
locate libcrypto.dylib

That should show you where the different versions of headers and libs are respectively. Then you can look inside the opensslv.h file in /usr/include and confirm that it is 0.9.8l instead of 0.9.8r.

In any case, the solution seems to be:

1. Get OpenSSL 0.9.8r from http://www.openssl.org/source/
2. Untar, copy the include/openssl directory to somewhere like /usr/include/openssl-0.9.8r
3. Point your build to these new headers. I decided to just blast the incompatible headers, so I moved my old /usr/include/openssl to /usr/include/openssl-0.9.8l and then symlinked openssl to openssl-0.9.8r.
4. Now it builds!

Combining USDA SR23 with glycemic index data

USDA provides a National Nutrient Database for Standard Reference currently in Version 23 (SR23), which has loads of nutritional data on a good selection of food products. On my wishlist would be cross-referencing with UPC bar codes, as well as some additional data that are currently missing, such as glycemic index and separate soluble vs. insoluble fiber values.

I did find one website that has done a pretty job with glycemic index already: nutritiondata.self.com. In addition to pulling in glycemic index data, they've also calculated glycemic load and then built a statistical model to try to infer/impute glycemic load for foods that don't have published glycemic index data. And they also have some other summary statistics that they've developed in-house such as "Fullness Factor" and an overall healthiness rating. Kudos to them, but I wanted to play with these data myself. Also I didn't see a fiber breakdown there.

One data source I found that has both glycemic load and soluble vs. insoluble fiber (at least for some entries) is the Diet History Questionnaire II (DHQ2). The DHQ is available to download as a single giant table. I plan to play with this.

Another option is to link the glycemic index data derived for the DHQ back to SR23. This appears to be possible thanks to a couple of additional tables. First, DHQ provides their glycemic index data in a table referenced against food codes from the USDA Continuing Survey of Food Intakes by Individuals (CSFII). These codes match the food codes used in the USDA Food and Nutrient Database for Dietary Studies (FNDDS). The FNDDS database includes a table that links selected FNDDS/CSFII codes back to one or more SR23 codes. (Since my main interest in glycemic index is to compare glycemic index to sugar composition, and FNDDS only has a value for total sugar not individual components, I'm not too interested in FNDDS by itself.)

I'll be exploring DHQ vs. SR23 (augmented with glycemic indices linked from FNDDS+DHQ) and will report back.

Globus Toolkit 5.0.3 on Cygwin 1.7

Cygwin patches (see earlier post) did not make it into GT5.0.3. The patch requirement is therefore the same as for GT5.0.2 (see earlier post). These patches may make it into future distributions.

I took slightly more detailed notes during the 5.0.3 build process, so you can now find more details about which packages require patches here.

DIY KitchenAid attachments

Update: (2011-03-14): So this does not quite fit perfectly into the Attachment Hub, but it is good enough that with a housing for the shaft I'm pretty sure it will work. I'll take more precise measurements and post any updates.

Update (2011-02-18): Arrived no problem from Mending Shed; Parcel Post was quite fast actually. Don't have the KitchenAid here with me right now so will not be able to test at all for a few weeks.

Our KitchenAid stand mixer is probably the fanciest piece of hardware we own other than our car. I've been thinking for a while that there really ought to be a DIY modding community around this appliance; it's a powerful, well built kitchen motor waiting to be exploited (plus planetary gearing if that is helpful). But I haven't found much so far. Here's one nice rotisserie mod I found, and that's about it.

KitchenAid makes a number of official attachments. Some of them attach to the bottom head where the planetary gearing is, while others attach to a port on the front (the "Attachment Hub") that accepts square shafted ("Power Shaft") attachments. I searched around a little and was able to find a part that I'm pretty sure will fit the attachment hub: KitchenAid 9709315, the "clutch shaft". Amazon sells it.

Note that this doesn't include the shaft housing that will allow you to lock it into place; figuring out a way to hack this together will be the first job before anyone can really use this thing for DIY projects. I'm wondering if putting something together in plastic through Ponoko or the like will get the job done. It probably requires some bearings too.

I ordered one from Mending Shed (slightly cheaper than Amazon and will ship Parcel Post), will update with progress.

Faster median calculation and generic rank selections in MatLab via nth_element

Update (2011-09-03): I've added new functions to the release that edit the arrays in-place. This is not good Matlab style but gives significant speedups for the very performance conscious. It appears it may be useful for me to add a multithreaded version as well, so this may come in an upcoming release.

C++ std::nth_element is a standard specification for an efficient rank selection algorithm. This algorithm can be used to solve a number of rank selection problems efficiently.

One example is median finding. A naive median finding algorithm, as implemented in MatLab (at least up to R2010a) is to sort the entire list of values and then just take the value in the middle position. But sorting the entire list is overkill; it is more efficient to do a sort of partial sort just to the point where the desired value is in the correct position. The version of this typically implemented in nth_element is known as quickselect or Hoare's Selection Algorithm. See Wikipedia: Selection algorithm for more details.

I find that a MatLab fast_median function based on nth_element runs about twice as fast as the native MatLab median function based on sort. Theoretically, the average complexity of quickselect is O(n), while the best performance complexity of the sort based method is O(n log n).

Finding median values is central to robust statistics, for example calculating the Median Absolute Deviation.

If you are interested, my MatLab wrapping of nth_element and an example implementation of fast_median are available at MatLab Central File Exchange.

Startup GNU Screen with multiple regions (split screen)

Since I've been doing a lot of work remotely lately I've been brushing up my GNU Screen skillz.

One thing I'd like is to be able to reattach a screen with a number of regions (AKA split screens) setup automatically. Screen is supposed to get this in the next release, but it's been a while.

In the meantime, the following kludge seems to work well enough: (sleep 0.1; screen -X split) & screen -r. Basically, this says: "reattach to my screen and at the same time in the background wait a fraction of a second and then send the split command to screen from a separate thread".

This was a common enough pattern that I wrote it into a little shell script:

#!/bin/sh

splits=$1
if [ -z $splits ] ; then
  splits=1
fi

sleep=$2
if [ -z $sleep ] ; then
  sleep=0.1
fi

sleep $sleep
i=0
while [ $i -lt $splits ] ; do
  screen -X split
  screen -X focus bottom
  screen -X select `expr $i + 1`
  i=`expr $i + 1`
done

which is used: splitscreen 4 & screen -r.

There's plenty of room for improvement. A more principled replacement for the call to sleep would be to monitor the socket files. In my Mac OS 10.5 these are under /tmp/uscreens/.... (I believe the standard location is /usr/tmp/screens/...) This is purely empirical, but it appears the user permission on the socket file changes from -x to +x (i.e. becomes executable) when the screen is attached. So instead of sleep we should be able to do a wait until that permission change occurs.

In principle I think this could even be used to save and restore region settings. For example, rebind the split key to not only split but also save a flag to filesystem indicating how many regions, then read this flag when reattaching. Or we could just wait for the next release...

Bluegiga WT12 Breakout Board V0.2

I just got my first ever printed circuit boards back from BatchPCB. Pretty exciting! One of the boards is a Toyota audio system interface, but it needs to have a chip replaced and the pinout is different, so that board is not totally usable. The other board is this breakout board for the Bluegiga WT12 bluetooth module.

This breaks out all the pins so you have complete freedom, and it's still pretty small. I have another version I will probably release that breaks out only the UART pins and is compatible with the BlueSmirf pinout. That one is much smaller.

The WT12 has iWrap4 firmware that is a very simple interface to a variety of common Bluetooth profiles. It's quite affordable; please see my Flickr page and BatchPCB page for more information and ordering.

Chords: "Holiday" by the Kinks

The chords posted around online are a bit wrong. And it's such a great, simple song! Fixed it up a little:

Artist-The Kinks
Song-Holiday
Album-Muswell Hillbillies
Writer-Ray Davies
Tabbed by S. McNally, Peter H. Li


Intro
A-F#-B7-E7


Verse 1
A     F#     B7
Holiday,
          E7     A     F#
Oh what a lovely day today,
B7        E7        A       F#
I'm oh so glad they sent me away,
   B7            E7   A    F#   Bm Eb E
To have a little holiday today, holiday.

Verse 2
A     F#
Holiday,
B7           E7              A        F#  
And I'm just standing on the end of a pier,
B7          E7      A        F#
Hoping and dreaming you were here,
B7          E7     A            (turnaround: A F E C A ???)
To share my little holiday.


Verse 3 (type II)
A              F#        B7         E7
Lookin' in the sky for a gap in the clouds,
A           F#                   B7           E7
Sometimes I think that sun ain't never coming out,
        A         A7/G              D          B7
But I'd rather be here than in that dirty old town,
  E                                             (E+)
I had to leave the city cos it really brought me down.


Verse 4
Oh holiday, oh what a lovely day today,
I think I'll get down on my little ol' knees and pray, thank you Lord,
Thank heaven for this holiday today, holiday.


Bridge
    D           B7       A
I'm leaving insecurity behind me,
    D             B7               A
The environmental pressures got me down,
C#              C#7          D       B7
I don't need no sedatives to pull me round,
E                                              (E+)
I don't need no sleeping pills to help me sleep sound.


Verse 5
Oh holiday,
Oh what a lovely day today,
I think I'll get down on my little ol' knees and pray,
That's what I'll do,
Thank heaven for this holiday.

Verse 6 (type II)
Lying on the beach with my back burned rare,
The salt gets in my blisters and the sand gets in my hair,
And the sea's an open sewer,
But I really couldn't care,
I'm breathing through my mouth so I don't have to sniff the air.

Verse 7
Oh holiday,
Oh what a lovely day today,
I'm so glad they sent me away,
To have a little holiday. 

(I sure miss OLGA sometimes...)

Globus Toolkit 5.0.2 on Cygwin 1.7

Updates (2011-02-20): Cygwin patches did not make it into GT5.0.3, but they may make it into a future release.

Updates (2010-12-08): Binghamton University TeraGrid has written up instructions for applying the patches and getting GT compiled. Also, it appears new versions of GT will be coming out over the next few months. I plan to check them out for Cygwin compatibility.

If you are interested in compiling Globus Toolkit 5.0.2 on Cygwin, I have some simple patches that seem to get the job done. Perhaps they will make it into the next release. For now: http://lists.globus.org/pipermail/gt-dev/2010-September/001885.html

Note that I built this on Cygwin 1.7; Cygwin 1.5 will probably not work. Also you must have the gcc4.x Cygwin packages (C and C++) installed. Getting GT to build on gcc3.x is probably not much more work, but going with gcc4 is easier than worrying about unrecognized compiler flags all over the place.

You also need to specify the flag: --with-buildopts="-static=1" when running the main configure script. Building GT on Cygwin with dynamic linking is certainly beyond me, and is a pretty big undertaking, so this tells it to use only static linking. I recommend --with-buildopts="-verbose -static=1" so you can also see what is going on.

(For the record, on dynamic linking: I tried adding some flags to configure.ac or configure.in to let autoconf know that we wanted to make DLLs and do dynamic linking. But trying to run a modern version of autoreconf generated configure scripts that don't work with the GT package. Even if the build worked, this probably wouldn't be a good idea as making DLLs properly requires going through all the source and making sure data exports are handled correctly. At least one package (globus_libtool) is DLL ready and already generates a DLL and RTL properly, so we know it all works theoretically once the individual libraries are DLL ready. There may be other subpackages that are DLL ready, but they will probably have to be handled one at a time. Unfortunately, GPT doesn't really offer a method to build the whole Globus Toolkit with some parts statically linked and some dynamic.)

If you get a build failure with an error like "fork()" or "vfork()" "resource temporarily unavailable", this means your machine ran out of processes while building (due to what I'm going to call a "process leak" bug). Basically just rerun make and it should get farther each time before crashing, until finally it gets all the way through. Or build GT in smaller chunks; if you are just trying to get proxy certificate running you only really need "make gsi-myproxy gsi-openssh".

You may also have to circumvent GPT's somewhat outdated Perl libraries. If you have a problem getting Archive::Tar to open some of the source packages, then you should install the Cygwin CPAN package and then use that to install an updated version of Archive::Tar. You may then need to hide GPT's versions under $GLOBUS_LOCATION/sbin/perl.

Feel free to ping me with any questions/thoughts.