robjsoftware.info

It’s been a very interesting month despite the fact that I haven’t touched a line of Holofunk code!  I want to deeply thank everyone who’s expressed excitement about this project — it has been a real thrill.

First I have a favor: if you like Holofunk, please like Holofunk’s Facebook page — that is a great way to stay in touch with this project and with other links and interesting things I discover.

In this post I want to mention a variety of other synesthetic projects that people have brought to my attention, and I want to recap the places that have been kind enough to mention Holofunk.

First and foremost, let me say that, as with my first Holofunk post, I find all of these projects very thought-provoking and impressive, and I am linking them here out of appreciation and excitement.  Since I have many plans for Holofunk, I do find myself wanting to take various aspects of these projects and build them into Holofunk.  I sincerely hope that the artists and engineers who have produced this work are appreciative of this, rather than threatened or irritated by it.  There are obviously a lot of us creating new musical/visual art out there, and I hope that others are as inspired by my work as I am by theirs.

Holofunk is and will remain open source, under the very permissive Microsoft Public License, so if anyone who’s inspired me winds up wanting to make use of something I’ve done, it is entirely possible.  (Please let me know if you do, though, as I’ll be very interested and pleased!)

Synesthesia On Parade

One project Beardyman mentioned to me was Imogen Heap’s musical data gloves.  It took me a while to get around to looking them up, but when I eventually did I was gobsmacked:

Imogen Heap is of course a brilliant and well-known artist, and these gloves are her vision for where she wants to take her performance.  Her technical partner in this project is Tom Mitchell, a Bristol professor of music who was kind enough to reply when I wrote him a gushing email.

The system he’s developed with Imogen is best documented by this paper in the proceedings of the New Interfaces for Musical Expression 2011 conference.  And now I need to go off and download and read the complete proceedings, because it’s all right up Holofunk’s alley.

Tom and Imogen are using 5DT data gloves, which are $1,500 for a pair with a wireless connection, as well as a pair of AHRS position sensors (about $500 each).  So their hardware is out of my hobby-only price league.  I am interested in the Peregrine glove (only $150 per), but unfortunately it’s exclusively left-handed at present, though I wrote them and they said Holofunk was quite exciting and they would love to be involved, so there’s hope!  Anyway for now I will stick with Wiimotes as they are cheap and relatively ubiquitous.

Latency is a huge concern for Tom — the AHRS position sensors have a 512Hz update cycle, which is extremely impressive.  The Kinect will never come close to that, which again motivates sticking with some additional lower-latency controls.  Plenty of people I showed Holofunk to at Microsoft want me to build a Wiimote-less version, and I probably will experiment with that — including using the Kinect beam array as the microphone — but it honestly can’t compete with a direct mike and button/glove input as far as latency goes.  Darren (Beardyman) specifically mentioned how impressed he was that I’d gotten the latency right (or at least close to right) on Holofunk; evidently lots of programmers he talks to build things that are very latency-unaware, making them useless for performance.  So while a pure-Kinect version would be very interesting (and obviously quite marketable!), it’s not my priority.

I am hoping to make some waves inside Microsoft as far as getting better low-latency audio support in Windows… ASIO shouldn’t be necessary at all, Windows — and Windows Phone — should be able to do low-latency audio just as well as the iPhone can!  And for proof that the iPhone gets this right, here’s our friend Darren rocking the handheld looper:

The app there is evidently Everyday Looper, and dammit if it shouldn’t be possible to write that for Windows Phone 7, but I don’t think it can be done yet.  This will change, by Heaven.  In fact, writing this post got me to actually look the app up, and that turns up this stunningly cool video demonstrating how it works.  Plenty of inspiration here too:

Good God, that’s cool.

One other project Tom mentioned is the iPhone / iPad app, SingingFingers:

That’s synesthesia in its purest form:  sound becomes paint, and touching the paint lets the sound back out.  I totally want to build some similar interface for Holofunk.  Right now a Holofunk loop-circle is dropped wherever you let go of the Wiimote trigger while you’re recording it, but it would be immensely straightforward to instead draw a stroke along the path of your Wiimote-waving, and then animate that stroke with frequency-based colors.  It would also be fascinating to allow those strokes to be scratched back and forth, though I’m not yet sure that a freeform stroke is the most usable structure for scratching.

The fellows behind SingingFingers have various other projects, equally crazy and intriguing.

I am sure I will turn up a colossal quantity of other excellent projects as I move forward with Holofunk, and I will certainly blog the pants off of them because it’s dizzying how much work is being done here, now that every computer and phone you touch can crank dozens of realtime tracks through it.  Wonderful time to be an electronic musician, and the future is dazzling…

Holofunk Gets Press

I also very much appreciate the sites that have linked to Holofunk.

Bill Harris, an excellent sports/gaming blogger, was nice enough to mention Holofunk.

Microsoft’s Channel 9 site put together a good description of Holofunk.

The number one Kinect hacking site on the web, KinectHacks.net, asked me to write up a description of Holofunk, which they posted.  They get mad hits, so this is lovely.  An experimental music/art collective in Boston, CEMMI, already contacted me as a result of the kinecthacks post!

…And now that I am surfing kinecthacks.net, I find that I might be wrong about how possible it is to do Holofunk with just Kinect.  This guy seems to get a lot of pretty fast wiggle action going on here:

Getting effects like that into Holofunk is definitely on the agenda for early next year.

Still Taking It A Bit Easy

Now, all that wonderfulness having been well documented , I must confess that I am still on low hacking capacity, Holofunk-wise.  And here’s where this post veers into totally off-topic territory, so you’ve been warned!

I’m a gamer, you see, and Q4 of every year is the gamer’s weak spot.  I’ve been playing the heck out of Deus Ex: Human Revolution, a really excellent homage to a famous game from ten years ago.  I played that game then, and I’m totally digging this one now.

Then on November 11th, the unbelievably huge game Skyrim ships.  My friend Ray Lederer is one of the lead concept artists on the game (check out this video of him at work), and the game could take over a hundred hours to complete, so that’s a month and a half shot right there.

And THEN, soon after THAT, the expected-to-be-superb Batman: Arkham City game comes out.  I played the first Batman game from these guys to smithereens, and I am expecting to do likewise with this one.

So… yeah… the next few months have some stiff competition.  However, given how much excitement there is around Holofunk, I do plan to make these be the only games I play in 2011.  There are just not enough hours in the day to read, watch, listen to, or play every good book, movie, track, or game in the world, LET ALONE do any actual work of one’s own!  So one has to be picky, and the above are my picks.

But Once That’s Over With…

My only specific goal for Holofunk in 2011 is to rewrite the core audio pump in C++ to get away from the evil .NET GC pauses.

Then, in 2012, I plan to get seriously down to business again, feature-wise.

The number one feature is probably going to be areas — chopping up the sound space into six or so regions, and allowing entire areas to be muted or effected as a whole.  That will allow Holofunk to become useful for actual song creation, since you’ll be able to bridge into other portions of a song in a coherent way.

The second feature will probably be effects.  Panning, volume, filtering, etc. — adding that stuff will do a huge amount for making Holofunk more musically interesting.

Then will come visuals — SingingFingers meets Holofunk.  Should make the display radically more interesting and informative.

After that, probably scratching / loop-cutting.  I have no idea what the interface will be, but being able to chop up loops and resample them is part of every worthwhile looper out there (see Everyday Looper’s awesome video above), so Holofunk has got to have it.  Going to be challenging to do it with just a Wiimote, but it’s got to be possible, it’s GOT to be!

And then, most likely, video.  Stenciling out Kinect video and time-synchronizing it with the loops could be all kinds of wacky fun — I cited this in my last blog post as the “live Monkey Jazz” possibility.

All that together should hopefully take only until mid-2012 or so, at which point I want to start rehearsing with it in earnest and actually performing with it.  If I can’t get a slot at a TEDx conference, I’m just not trying hard enough.

Thanks as always for your interest, and stay in touch — 2012 will be an epic year!  I feel much more confident saying things like that now that I’ve actually gotten this project off the ground 🙂