robjsoftware.info

Yow!  Two postings in one night!

One reason for my reduced posting here is that I’ve realized this blog works a lot better if I have a personal hacking project. Hacking on something I can publicly talk about makes for much more interesting material. So I am glad that I have a surfeit of possibilities there, one or more of which will definitely be happening this year. The next three posts are devoted to these lovely options, any or all of which may or may not wind up happening, but which are nonetheless interesting in their own right.

Up today: the Holofunk.

A year or two ago I became aware of a British performer named Beardyman. I am an old school raver, and so I still follow the electronic / dance music scene. Beardyman won the best beatboxer in the UK prize two years in a row (a feat none else have accomplished), and these days he does one-man musical shows with his voice and a bevy of samplers and sequencers. The net result is that he is literally a one-man band. I find his work incredibly inspirational, as I like the electronic music scene and my voice is the only instrument I am much good with. So Beardyman is one inspiration.

Another inspiration of mine is my old college friend Leon Dewan. He and his cousin have created the Dewanatron, a unique device for making sounds never before conceived. I had the pleasure of joining them in a performance in Seattle recently (that would be me on the left at 3:15), and seeing them hacking their musical hardware was quite compelling. It’s fun to make weird sounds and bring order out of chaos out of order.

Now then, please integrate the above influences with the following facts:

1) There exists a piece of software named Ableton Live. (How existentially qualified of me!) This is a very powerful sequencing / synthesizing / sampling application used by many DJs and electronic music producers. Recently the Ableton people have done a lot of work to integrate Ableton Live with another audio product named Max, which is an exceptionally scriptable and flexible audio toolbox. The combined product is named Max for Live. There are existing C# libraries that allow Max, and hence Max for Live, to be controlled over a TCP connection.

2) 3D graphics cards are currently shipping; in fact, most mid-end to high-end PC

graphics cards can already drive a 3D display, and the hardware is about to become very widely available.

3) Microsoft this year, if their rumored timeframes hold true, is going to ship an attachment for the Xbox 360. This attachment is currently called Project Natal. It is essentially mass-market full-body video-based motion capture and 3D body tracking. The idea is that you just wave your arms and move your body and the thing watches you and can track you in realtime. (Actually, current rumors have it that the thing has about 100ms of lag, so you can’t expect instantaneous response out of it.) If it lives up to the hype — admittedly a big “if” — it really could be a step forward in interface technology.

So, let’s combine the artistic influences with the technical possibilities.

Let’s say that in late 2010 you could put together a reasonably fast PC with a 3D graphics card, an installation of Max for Live, and a PC-based Natal development kit. Let’s further say that Natal actually works, and that it is possible to — with enough experimentation and false starts — build an application that lets you directly manipulate 3D objects. (Hopefully Natal has at least some ability to detect finger position, or grabbing anything will be hard….)

If you could hook up a grab-and-drag Natal application to Max for Live, you could build a gesturally controlled 3D sound space. Add a microphone and whatever instruments you like, and make it really easy to script new gestures and new kinds of 3D sound-controlling objects.

It’d be a sonic holodeck. Punch the throbbing red sphere and start recording a new loop. Grab a sound and rip it in half, then push the two copies slightly out of tempo. Wave your arms around to change pitch, volume, modulation, or crossfade. I have no real idea what the detailed interface would be since I haven’t played with Natal for real yet.

But what I do know is that there is a real chance this could be an immensely entertaining thing for creative musicians to play with. I also know that I really want to play with it myself!

So that’s my first possible 2010 hacking project: the Holofunk. That’s what I’ll call it. If, of course, I ever finish it. If I do get somewhere with it, I will open source it, probably on Codeplex. And if anyone else is equally inspired by this, or otherwise has more ability to execute on it than I do (given my great day job and busy family life), by all means feel free! There is going to be some seriously fun art coming out of the techno scene in the next few years, and I can’t wait. I love the 21st century!