A new kind of music video: Radiohead's "House of Cards"

Radiohead’s latest music video, “House of Cards”, was shot using LIDAR (Light Detection and Ranging) rather than cameras:

Learn more about how it was made at the Google Code Radiohead page. The coolest part is that Radiohead is making the data available to anyone who wants to play with it.

They provide some sample Processing code to view the data, but for whatever reason I couldn’t get it to work, so I wrote a simple little iPhone OpenGL ES application:

I’m unable to test it on a real iPhone since I don’t have a paid developer account, but presumably it should work.

View other people’s visualizations at the YouTube House of Cards group.

Amazon S3 PHP helpers

The Amazon documentation for using S3 with PHP refers to an elusive function called “setAuthorizationHeader”. It’s apparently supposed to magically set the correct value for the Authorization header on a Pear HTTP_Request object. As far as I could tell, it didn’t actually exist — but I wanted it, so I wrote it:

Source

<?php

require_once ‘Crypt/HMAC.php’;
require_once ‘HTTP/Request.php’;

define("S3URL", ‘http://s3.amazonaws.com’);
define("AWSACCESSKEYID", ‘XXXXXXXXXXXXXXXXXXXX’);
define("AWSSECRETKEYID", ‘XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX’);

$s3_hasher =& new Crypt_HMAC(AWSSECRETKEYID, "sha1");

function s3_sign($StringToSign)
{
    global $s3_hasher;
    return hex2b64($s3_hasher->hash($StringToSign));
}

function hex2b64($str)
{
    $raw = ;
    for ($i = 0; $i < strlen($str); $i += 2) {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

function setAuthorizationHeader($request)
{
    $headers = $request->_requestHeaders;
    
    $HTTP_Verb      = $request->_method;
    $Content_MD5    = $headers[‘content-md5’];
    $Content_Type   = $headers[‘content-type’];

    // Get the date, or set it if not already:
    if (!isset($headers[‘date’])) {
        $Date = gmdate("D, d M Y H:i:s T");
        $request->addHeader("date", $Date);
    }
    else {
        $Date = $headers[‘date’];
    }
    
    // Canonicalize the Amazon headers:
    $CanonicalizedAmzHeaders = ;
    $amz_headers = array();
    foreach ($headers as $key => $value) {
        if (substr($key, 0, 6) == ‘x-amz-‘) {
            if (isset($amz_headers[$key]))
                $amz_headers[$key] .= ‘,’ . $value;
            else
                $amz_headers[$key] = $value;
        }
    }
    ksort($amz_headers);
    foreach ($amz_headers as $key => $value)
        $CanonicalizedAmzHeaders .= $key . ‘:’ . $value . "\n";
    
    // Canonicalize the resource string
    $CanonicalizedResource    = ;
    $host = $request->_generateHostHeader();
    if ($host != ‘s3.amazonaws.com’) {
        $pos = strpos($host, ‘s3.amazonaws.com’);
        $CanonicalizedResource .= ‘/’ . ($pos === false) ? $host : substr($host, 0, $pos);
    }
    $CanonicalizedResource .= $request->_url->path;
    // TODO: sub-resources "?acl", "?location", "?logging", or "?torrent"

    // Build the string to sign:
    $StringToSign = $HTTP_Verb . "\n" .
                    $Content_MD5 . "\n" .
                    $Content_Type . "\n" .
                    $Date . "\n" .
                    $CanonicalizedAmzHeaders .
                    $CanonicalizedResource;
    
    $Signature = s3_sign($StringToSign);
    
    $Authorization = "AWS" . " " . AWSACCESSKEYID . ":" . $Signature;
    
    // Set the Authorization header:
    $request->addHeader("Authorization", $Authorization);
}

function s3AuthURL($Resource, $HTTP_Verb = ‘GET’, $seconds = 120)
{
    // Calculate expiration time:
    $Expires = time() + $seconds;

    // Build the string to sign:
    $StringToSign = $HTTP_Verb . "\n" .
                    "\n" .
                    "\n" .
                    $Expires . "\n" .
                    $Resource;

    $Signature = s3_sign($StringToSign);

    // Build the authorized URL:
    return  S3URL . $Resource .
            ‘?AWSAccessKeyId=’  . AWSACCESSKEYID .
            ‘&Expires=’         . $Expires .
            ‘&Signature=’       . urlencode($Signature);
}

?>

Note: this hasn’t been tested extensively, so use it at your own risk. Post a comment or contact me at if you find any bugs. Also, IANAPHPE.

Just replace the XX’s with your keys and it should work with this sample code.

There’s also a function for creating query string authorized URLs.

Using command line tools to detect the most frequent words in a file

Antonio Cangiano wrote a post about “[Using Python to detect the most frequent words in a file](http://antoniocangiano.com/2008/03/18/use-python-to-detect-the-most-frequent-words-in-a-file/)”. It’s a nice summary of how to do it in Python, but (nearly) the same thing can be accomplished by stringing together a few standard command line tools.

I’m no command line ninja, but I’d like to think I have basic command of most of the standard filters. Here’s my solution:

cat test.txt | tr -s ‘[:space:]’ ‘\n’ | tr ‘[:upper:]’ ‘[:lower:]’ | sort | uniq -c | sort -n | tail -10

I’ll explain it blow-by-blow:

cat test.txt

If you don’t know what this does you’ve got a lot to learn. “cat” simply reads files and prints them to standard output (concatenates), for use by subsequent filters.

tr -s ‘[:space:]’ ‘\n’

“tr” is a handy tool that simply translates matching characters from the first set to the corresponding character of the second set. The first instance turns all whitespace characters (spaces, tabs, newlines) into newlines (“\n”) so that each word is on a separate line (the -s option “squeezes” multiple runs of newlines into a single newline).

tr ‘[:upper:]’ ‘[:lower:]’

The second instance translates all uppercase characters into lowercase (note: the two “tr”s are separate for clarity, but they could be combined into a single one).

sort | uniq -c

“sort” and “uniq” do exactly as their names imply, but “uniq” only removes adjacent duplicates, so you often want to sort the input first. The “-c” option for “uniq” prepends each line with the number of occurrences.

sort -n

We sort the result of “uniq”, this time by numerical order (“-n”) to get the list of words in order of the number of occurrences.

tail -10

Finally, we get the 10 most frequently occurring words by using “tail” to take only the last 10 lines (since the “sort -n” puts the list in ascending order)

It’s not perfect, especially since punctuation is included in the words, but the “tr” commands can be tweaked as needed.

Chipmunk Physics engine running on the iPhone

A couple months ago I hacked together a demo of [Chipmunk Physics](http://wiki.slembcke.net/main/published/Chipmunk) engine running on the iPhone using the unofficial SDK. It shows the standard Chipmunk demos, but also it reads in accelerometer data using the method described on the [Medallia blog](http://blog.medallia.com/2007/08/iphone_accelerometer_source_co.html).

For the most part it was a fairly simple translation from standard OpenGL to the OpenGL ES the iPhone uses. If anyone is interested in the source code, let me know and I’ll see if I can find it… *edit: source is posted. see below*

[iPhone Chipmunk Physics demo application](http://tlrobinson.net/misc/iphone_chipmunk_physics/chipmunk_demos.tar.gz) – run from the command line (either using MobileTerminal.app or over ssh), only tested on iPhone software 1.0.2.

*Update: [the source](http://tlrobinson.net/projects/iphone/Chipmunk-iPhone.tar.gz) – this is just modified from the original Chipmunk Physics demo, so the original license applies. I can’t support this at all, so you’re on your own. I believe I used the hacked iPhone toolchain version 2 (or 3?), so you may have to edit the Makefile if you have a different version or installed it somewhere other than /usr/local/bin/.*

Overriding library functions in Mac OS X, the easy way: DYLD_INSERT_LIBRARIES

Back at MacHack 2003 Jonathan Rentzsch talked about how to override functions and inject code in Mac OS X using [several neat tricks](http://rentzsch.com/papers/overridingMacOSX). He also released a framework called [mach_star](http://rentzsch.com/mach_star/) which has two components: mach_override and mach_inject. These are great, but overkill for some simple cases.

A much easier way of doing library function overrides is using the DYLD_INSERT_LIBRARIES environment variable (analogous to LD_PRELOAD on Linux). The concept is simple: at load time the dynamic linker (dyld) will load any dynamic libraries specified in DYLD_INSERT_LIBRARIES before any libraries the executable wants loaded. By naming a function the same as one in a library function it will override any calls to the original.

The original function is also loaded, and can be retrieved using the dlsym(RTLD_NEXT, “function_name”); function. This allows a simple method of wrapping existing library functions.

Here’s a simple example which prints out the path of every file opened using the “fopen” function (lib_overrides.c):

#include <stdio.h>
#include <unistd.h>
#include <dlfcn.h>

// for caching the original fopen implementation
FILE * (*original_fopen) (const char *, const char *) = NULL;

// our fopen override implmentation
FILE * fopen(const char * filename, const char * mode)
{
    // if we haven’t already, retrieve the original fopen implementation
    if (!original_fopen)
        original_fopen = dlsym(RTLD_NEXT, "fopen");

    // do our own processing; in this case just print the parameters
    printf("== fopen: {%s,%s} ==\n", filename, mode);
    
    // call the original fopen with the same arugments
    FILE* f = original_fopen(filename, mode);
    
    // return the result
    return f;
}

And a simple test program (overrides_test.c):

#include <stdio.h>
#include <string.h>

int main(int argc, char const *argv[])
{
    char hello[] = "hello world";
    
    FILE *fp = fopen("hello.txt", "w");
   
    if (fp) {
        fwrite(hello, 1, strlen(hello), fp);
        fclose(fp);
    }

    return 0;
}

Compiled and tested:

tlrobinson$ gcc -Wall -o lib_overrides.dylib -dynamiclib lib_overrides.c
tlrobinson$ gcc -Wall -o overrides_test overrides_test.c
tlrobinson$ DYLD_FORCE_FLAT_NAMESPACE=1 DYLD_INSERT_LIBRARIES=lib_overrides.dylib overrides_test
== fopen: {hello.txt,w} ==
tlrobinson$

There are certainly scenarios far more interesting than this, though!

Presenting GCCalc: a horrible abuse of GCC

Following an [interesting discussion on Reddit](http://programming.reddit.com/info/62v70/comments) about [first class functions](http://en.wikipedia.org/wiki/First-class_function) in C, I was inspired to see what I could do with this new-found knowledge. The result is what I affectionately call “GCCalc”, for reasons that will become clear below.

GCCalc is a simple command line calculator, much like the common [bc](http://en.wikipedia.org/wiki/Bc_programming_language) calculator on many Unix systems. It’s implementation, however, is *very* different than most calculators. While bc is said to have “C-like syntax”, GCCalc’s syntax *is* C. Whatever you enter on the command line automatically gets compiled, loaded, and executed, and the result is returned (as a double) and printed to the screen.

You can either enter expressions like:

round(46.95886*sqrt(1+2/9.99*sin((21%5)*pow(2,8))))

or you can enter whole C statements (as long as they’re on one line, for now) like:

int i; for (i=0;i<10;i++) { printf("hello world!\n"); } printf("goodbye\n"); Unfortunately variables are scoped to the function that wraps them, so they don't persist across multiple entries. However, you can access the last result using the "last" variable (a double). [Here's the source file](http://tlrobinson.net/projects/gccalc/gccalc.c), and here's a syntax highlighted version: It's been tested on Mac OS X (Leopard) and Linux (Ubuntu Gutsy), with GCC 4. Compile with "gcc -o gccalc gccalc.c" on OS X, or "gcc -o gccalc gccalc.c -ldl" on Linux.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>
#include <unistd.h>

#ifdef __ELF__
#define GCC_FLAGS "-fPIC -shared"
#define EXTENSION "so"
#else
#define GCC_FLAGS "-dynamiclib"
#define EXTENSION "dylib"
#endif

#define HEADERS "#include <stdio.h>\n#include<math.h>"

typedef double(func_return_double)(double);

unsigned count = 0;
char *cwd;
char tmp_path[1024] = {‘\0’};

void *lib = NULL;

int main(int argc, char **argv)
{
    double result = 0.0;
    char input_buffer[1024], code_buffer[2048], function_name[32], command_buffer[1024];
    
    // get out current directory, which we’ll use for tmp files (dlopen seems to need absolute paths)
    cwd = getcwd(NULL, 0);
    
    while (1)
    {
        // for unique function and file names (needed for dlopen/dlsym to work correctly)
        count++;
        
        // read in the next line
        printf(">> ");
        fgets(input_buffer, sizeof(input_buffer), stdin);
    
        // format the function name
        sprintf(function_name, "f%d", count);
        
        // format the code string: if it doesn’t contain a semicolon, assume it is just an expression
        if (strchr(input_buffer, ‘;’))
            sprintf(code_buffer, "%s\ndouble %s(double last) { %s\nreturn 0; }", HEADERS, function_name, input_buffer);
        else
            sprintf(code_buffer, "%s\ndouble %s(double last) { return (%s); }", HEADERS, function_name, input_buffer);
            
        // format the filename string, delete the file if it exists
        sprintf(tmp_path, "%s/libtmp%d.%s", cwd, count, EXTENSION);
        unlink(tmp_path);
        
        // format the gcc command string
        sprintf(command_buffer, "gcc -Wall %s -x c – -o %s", GCC_FLAGS, tmp_path);
        
        // execute gcc command, write out the code
        FILE *fp = popen(command_buffer, "w");
        fwrite(code_buffer, 1, strlen(code_buffer), fp);
        fprintf(fp, "\n");
    
        // pclose waits for gcc to terminate (fclose/close do NOT thus compilation will sometimes not finish prior to the dlopen)
        pclose(fp);

        void *ptr = NULL;
        
        // open the just-compiled dynamic library
        if ((lib = dlopen(tmp_path, RTLD_NOW|RTLD_LOCAL)) == NULL) {
            puts(dlerror());
        }
        // get the function pointer
        else if ((ptr = dlsym(lib, function_name)) == NULL) {
            puts(dlerror());
        }
        
        // execute it
        if (ptr != NULL)
        {
            func_return_double *func = (func_return_double*)ptr;
            result = (*func)(result);
            // print the result
            printf("=> %.*lf\n", (result/((int)result)>1.0)?5:0, result);
        }

        // clean up: close the library, delete the temp file
        dlclose(lib);
        unlink(tmp_path);
    }

    return 0;
}

Thanks to jbert on Reddit for the initial code and inspiration.

If only I had known about this back when The Daily WTF has having their [OMG WTF](http://omg.thedailywtf.com/) crazy calculator programming contest…