robjsoftware.info

Well!  Certainly has been a while since my last post here.  In fact, I learned an important lesson: don’t go a whole year between blog posts.  Because if you do, you might overlook the fact that your domain expired, because your spam filter ate all the emails from your registrar, and your domain might get taken over by a domain squatter that forges content in your name!

That’s why I now heartily welcome you to robjsoftware.info — probably a better domain name altogether, come to think of it.

So: this post is all about my Holofunk project.  If you have no idea what that is, please check it out and then come on back and get ready for a big old geek fest.  Sorry, this is pretty tech-heavy, no videos or cool pictures in here 😦

What Just Happened

Despite the complete lack of activity here, things have in fact been happening in Holofunk land.  I posted an update on Facebook in the middle of the year, so let me just crib from that July 14th post:

Gggggghhhhhh I have been trying to resist saying anything, but I can’t. I am too excited SQUEEEEEEEEE

So, many of you know my Holofunk project. I’ve gone completely dark on it since last fall when I realized I had to rewrite it, as it only ran on one computer (driver problems elsewhere), and it was only in 2D.

With the help of various open source collaborators, I’ve beavered away since January. Last night I finally got the core systems working again, and the rewritten version made its first music!!!

So immediately after that summer vacation ate the rest of the summer, and then in September I implemented a gesture-controlled GUI with buttons you could reach out and grab via Kinect hand detection.  (I learned the meaning of the term “diegetic” in the process.  I love this project for things like that.)  The whole goal here was to build an audio setup interface in the app itself, to avoid having some kind of clunky external “configurator” app or something.

And then… things broke.  The latest version of Windows 10 somehow hosed the Universal Windows Platform (“UWP”) version of my Kinect+Unity app; it would freeze after less than a second of color video.  My little Intel mini-PC totally failed to handle the new Unity version of Holofunk; the audio was terrible.  And my old audio interface wouldn’t work with AudioGraph; it said the input device was always busy.

Basically, just when I was raring to go with my new GUI, I suddenly found myself with a bunch of busted hardware, and nothing working again.  It actually crushed my morale quite seriously.  So I took some time off in October and November to recharge.

What’s Happening Now

I am looking at where I am and realizing a few things.  Regarding Kinect:

• The Kinect is dead — Microsoft has stopped making it.  So any problems I run into trying to make a UWP Kinect app are unlikely to get any support.
• Therefore I should back away from UWP when it comes to Kinect.
  • (Note that I am not going to back away from Kinect itself.  Nothing else can do what it does, and it’s indispensable to the performance experience I’m creating.  In fact I just bought two more to make sure I have backups!)
• The PC desktop version of my app seems to run just fine with Kinect — none of the UWP problems exist.

Regarding audio:

• Holofunk has always had problems with scaling up audio — using C# for audio handling is fundamentally a bad fit.  Audio is the most latency-critical part of the system, and garbage collection jitter can be a disaster.
• The VST plugin standard is ubiquitous in the electronic music world, and Holofunk needs to have access to that ecosystem; but UWP apps can’t include random VST DLLs.
• I need an audio engine that is not written in managed code, and that can work on both UWP / HoloLens devices and on the PC, especially because it needs to support VST plugins on the PC.

Putting all these things together, I have realized that I really need to build two apps, or one app that can ship in two forms:

• A desktop Unity app that:
  • runs on Windows desktop
  • uses the Mono runtime
  • supports Kinect
  • supports external USB audio interfaces
  • has a C++ audio engine
  • can load and run VST plugins
  • can detect hand poses and handle UI interaction
  • can send body data and sound data over the network to the HoloLens app
• A HoloLens UWP Unity app that:
  • runs on UWP devices (especially HoloLens)
  • uses the CLR / .NET runtime
  • does not support Kinect, but does support mixed reality
  • has a C++ audio engine (if running standalone, with local audio processing)
  • cannot load and run VST plugins
  • can receive body data and sound data over the network from the Kinect/PC app

With these available, the performance setup will be:

• Performer on stage:
  • wearing a HoloLens running the UWP app
    • The UWP app will be rendering body/sound data sent by the Kinect PC, rendering the UI locally
  • holding a microphone which broadcasts to the Kinect PC
• Kinect PC on stage watching performer:
  • running the PC app
  • handling all audio processing (receiving all microphone inputs, feeding sound to the room)
  • handling all gesture processing (using Kinect hand pose detection)
  • providing video output which feeds a projector viewed by the audience
  • sending sound and body pose data over the network to the HoloLens

The result will be that the Kinect PC is really mostly driving the show, but the performer nonetheless gets to immersively inhabit the performance rendered by their HoloLens.

Or, if there is no performance going on, the whole app could just run with all audio processing on the HoloLens… the only limitation would be no VST support.

Or, if the HoloLens version isn’t even done yet, two people could have all kinds of fun with just the Kinect PC app.

So all the above is the plan for 2018:  manifest this massive multi-machine music modality.

But First

Observant readers will have noticed that both the apps posited above need a C++ audio engine.  And after looking around I’ve concluded that there is no such audio engine that suits my requirements, which are:

• MINIMAL LATENCY
• supports both UWP apps and desktop apps
• focused on in-memory buffering
• VST support

So I have decided to write my own.  That’s right: the next key piece of Holofunk’s development is implementing a C++ sound engine.  From my investigation to date it looks like the right sound API to use is Windows WASAPI.  (AudioGraph is managed only, and the ASIO standard is not supported for UWP apps.)

So between now and the end of 2017 I am going to put together the smallest, lightest, most minimal sound engine I possibly can, and hook up the PC build of Holofunk to it.  If  I get that working, I will add VST support.  Also, this whole sub-project will be open sourced on GitHub, because there’s really nothing Holofunk-specific about it.

If anyone out there knows WASAPI and/or wants to get involved in an open source Windows-ecosystem C++ sound engine with VST support, let me know — I need beta testers and code reviewers for this piece!  (And if anyone knows of an open source sound engine that meets these requirements already, let me know as well; I’d be happy to save some work!)

Once this is done and re-integrated with some more UI, then I’ll have a 3D Unity Kinect version of Holofunk that is basically at the functionality level of Holofunk circa 2015, only with Unity and WASAPI powering it.

At that point I’ll probably start working on networking with HoloLens, because that’s the future, baby.

Bits From My Git

If you’ve been reading this blog for a long time, you have the patience of a saint, and you would be right to wonder whether after all these years (years?!?!) there is really anything actually going on here.  I mean, how many times can I rewrite this freaking project?  And how many technologies are going to crumble out from under me in the process?  And when am I going to have some more damned demos and fun jams like in the olden days???

I feel your pain.  But it’s still gonna be a while yet.  I do believe that once I get the bare-bones audio working, I will start jamming around with people, and I sure would love for that to happen in January 2018 or so.  Or February 2018 at the latest.  Not really too far off.

But in the meantime, I can at least throw you this bit of evidence that I really am doing something, honest.  Here is my git commit log from this year, since creating my new repository for the Unity version of my app.  The Kinect Unity plugin I’m using is excellent, but it’s not open source, so I’m no longer using a public repository.  (For those, if any, still following along without being coders:  these messages are my notes to myself whenever I make a change I want to keep in my main source code.)

So here’s the result of git log –pretty=format:”%ad %s” –date=simple run against my private repository.  Prepare for a glimpse far too deep into my late night machinations and ululations.  Yes, I use too many exclamation points when I am even slightly excited.  And… keep the faith, as I now go off to hack C++ sound APIs deep into the long winter nights….

• 2017-12-02 UWP version no bueno con Kinect sur Win10 1709. But PC version… looking very delightful! Time to go back to the future.
• 2017-11-24 Pick up just a couple other bits
• 2017-11-24 Update to Unity 2017.2.f2 (or thereabouts) and latest (and probably final) K2-Asset. AND, with Tascam 2×2 and non-pre-release Kinect, all foundation techs work on my main home PC again!!!
• 2017-09-30 Welp, audio graph finally gets set up on NUC… and audio is hideous, unrecognizable, distorted, broken. F M L.
• 2017-09-30 Debug simple issues in audio setup UI,. Good: shows actual inputs & outputs, gets through CreateAudioGraphAsync! Bad: hits weird COM exception immediately after, blows up, leaves machine in weird state with lots of WinRT exceptions.
• 2017-09-28 Woot! The entire GUI flow actually works end to end!
• 2017-09-25 Woohoo!!! Stateful text field input buttons working together with GUI state machine and hand state capture functioning properly!!!!! NOYYYCE
• 2017-09-25 Text fields and OK buttons sort of kind of work properly now. But only the *first* OK button is OK. The second one, not so much. DEBUGGING TIME but first DINNER.
• 2017-09-25 Work in progress.
• 2017-09-24 Handedness detection works and carries state properly through tutorial. BOO && YA.
• 2017-09-24 Getting GUI state machine dialed in (instantiated rather than static; tracks which hand the user’s working with in tutorial). Latest code not tested yet, hit sleep timeout.
• 2017-09-22 Color and skeletal data working OK on NUC in release x64!!! Looks like unblocked?! ONWARDS ONCE AGAIN and maybe OK perf on Surfaces….
• 2017-09-22 Sigh… try disabling color map, user map, and background removal… still no joy on main PC. Let’s try on NUC.
• 2017-09-20 Latest local steps towards actual audio device selection.
• 2017-09-12 Start ACTUALLY IMPLEMENTING AUDIO INPUT/OUTPUT DEVICE SELECTION.
• 2017-09-10 Fix regression with hand events getting fired repeatedly; make sure OK button actually works! (and it does! And so does latest Windows insider release, with Kinect & Unity! PHEW.)
• 2017-09-10 Make the OK button do something!
• 2017-09-05 Actually got the CanvasTextButton working properly, sorting out all types of bugs along the way. Moral: when multiple hands can touch, having “isTouched” be per-update state, set idempotently by possibly multiple hands, is a good pattern. If it doesn’t scale, we’ll optimize it later.
• 2017-09-04 Theoretically got things wired up to support OK button capturing input from hand state machine. Tomorrow, will verify.
• 2017-09-04 Darn problem where adding source files to the UWP project puts them in the UWP generated sources directory and then they never get checked in :-O
• 2017-09-03 Don’t forget the scene changes
• 2017-09-03 Moving towards a real-ish event architecture. Already managed to unify “Loopie touching” and “Canvas UI element touching” so that’s a good win already 😀
• 2017-09-03 Semitransparent bones, OK-button background highlight.
• 2017-08-31 DIEGETIC OK BUTTON REACTS!!!!! In a terribly hacky way, but still IT WORKS! GraphicRayCaster does the Right Thing.
• 2017-08-30 Skeletal joint position averaging, helps a lot. Still very smooth. Working towards GUI position interception.
• 2017-08-28 Woot! New tutorial steps worked first time! Closed hand image works! Diegetic UI positioning works! Not sure what happened to damn GUIController component, but easy enough to sort out 🙂
• 2017-08-28 Add closed hand image and tutorial step for it. Start moving towards interactive OK button.
• 2017-08-06 Upgrade to Unity 2017.1, only it works! THANK YOU RUMEN!
• 2017-08-05 Guarded transitions! Tutorial interactions! Interaction states! Things are coming along nicely 🙂
• 2017-08-05 Fading and GUI state transitioning works!
• 2017-08-05 Welp, Unity 2017.1 borks UWP initialization of Kinect. Looks potentially like an async deadlock involving Wait loop in KinectManager? Anyway, UWP-mocking design means I’M NOT BLOCKED and can work on tutorial!!!
• 2017-07-16 Fix deletion bug — made Destroy() separate from Delete() but forgot to update the call site 😛
• 2017-07-15 YESSSSSS MUTING UNMUTING DELETING ALL WORK AGAIN. soooooooo simple and easy…… ahhhhhhhhhhh
• 2017-07-15 Good ol’ “while (samplesRemaining > 0) …” loop is back and working as great as ever 😀 Thanks, past me!
• 2017-07-14 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
• 2017-07-13 Reviving the diagnostics to sort out the audio timing issues. WILL SORT IT ALL, this is all familiar stuff from five years ago 😀
• 2017-07-12 O M F G — I JUST HEARD SOMETHING!!!!!!!!!!!!!!!!!!!!
• 2017-07-10 …oops, forgot to save before pushing
• 2017-07-10 Getting closer… now running into some confusion about the exact time when shutting the recorded stream. BIG progress though!
• 2017-07-10 Oh yeah, HolofunkAudioTrack.BeatPositionNow is a thing and UWP-audio tracks handle time properly. NOW FOR ACTUAL PLAYBACK
• 2017-07-09 HUGE step forward! AudioGraph initialization working AND not totally breaking Unity & Kinect on UWP!!!!!
• 2017-07-07 Wow. Actual audio code in Holofunk2! Lots of TODOs still, and too tired to finish tonight, but it’s mostly there now 😀
• 2017-07-05 Fix a few issues in UWP version — need to sort out what is up with ThreadContract & Unity thread…
• 2017-07-03 OH GOODNESS now tracks handle FinishRecording properly. IT IS ALL SOOOO LOVELY.
• 2017-07-03 DA BEAT MEASURE IS BACK IN DA HOOOOOUSE
• 2017-07-03 Good progress towards having multiple beat-measure doodads per loop. Now just needs debugging.
• 2017-07-01 Progress towards BeatMeasures for loopies! Got the scene graph right… now just need to write the code to set all the colors properly.
• 2017-07-01 Toddle along towards Tracks keeping their duration.
• 2017-06-30 First taste of thread contracts. Don’t draw bones to untracked joints. Don’t add Unity C# files in UWP solution :-\
• 2017-06-30 The clock actually DOES something now!
• 2017-06-30 OK fine, the clock is a singleton already. Emulate Unity in this regard.
• 2017-06-30 Got multiple colors in and working, and a proto-clock, and tucked the bones away nicely where they belong. Now to have groovy boppin’ shapes!
• 2017-06-28 Semi-transparent loopies for the big win!
• 2017-06-28 WOW! Got translucent bone skeleton working with all my own rotated 3D axis/angle cylinders and camera-aligned positions and EVERYTHING!!!!!
• 2017-06-25 Forgot to save good old .csproj 😛
• 2017-06-25 Half-baked state… working towards having the right upcall interfaces and locking stories. Old code got rid of inter-thread queue, it seems, instead opting for direct locking! Woddya know. Will try do to likewise, with much use of ThreadContract.
• 2017-06-25 OUTSTANDING. State machine seems solid, ready for sound!!!
• 2017-06-19 Rebuild UWP solution to take it on the road to Idaho!
• 2017-06-17 HELLS YAAAAA — the INTERACTION ARMED/RECORD/POINTING/MUTE/UNMUTE STATE MACHINE LIVES AGAIN BAYBEEEEEE 2017-06-17 YEEEEEEHAAAAAW mute and unmute WORKS!!!!!
• 2017-06-16 WOOOOO I can TOUCH LOOPIES AGAIN!!! Figured out how to set material colors, got loopie touch updating working, used the tip about setting the scene update ordering… AWWWW YISSSSS
• 2017-06-14 3D objects being created!!!!! 2017-06-10 Just like that, mute/unmute revived with proper icons and interaction. Ahhhh. FINALLY.
• 2017-06-09 WOOOO got state machine working and revived game object stack and got sprites instantiating and BASIC HAND STATE MACHINE IS BACK!!!!!!
• 2017-06-04 Added hand-pose smoothing back in. Instant, dramatic improvement, just as expected! Better structure than last version, too.
• 2017-06-03 Got per-hand state machines back! But MAN, hand pose is noisy noisy NOISY. Did I really average it out before? Have to do that again it looks like… and also have to sort out how the heck the Unknown transitions are working, because that’s FLAKY. But… tomorrow!
• 2017-06-03 WOOT got multiple players and properly factored components and WEEEEE!!!!!
• 2017-06-03 Starting to work out the right Unity structure for a player and their components.
• 2017-05-25 OH YEAH! Sprites working for both hands, no rotation, correct Z order!!!!! WOOOOOO
• 2017-05-23 Moving to BackgroundRemoval + JointOverlay.
• 2017-05-18 Move Core and StateMachine down into Unity-land. Import Skinner and play around with it — scale and translation issues, but definitely cool anyway.
• 2017-05-18 Bring in StateMachine and AudioGraph libraries, or at least the vestiges thereof. No idea why project now has all kinds of NuGet complaints though…
• 2017-05-17 Background removal + avatar = AWESOMESAUCE!!!!!!!!!
• 2017-05-16 Move to Unity 5.6.1f1 and it indeed fixes the crash on quit/alt-tab! Thanks again Rumen for the heads up on that one!
• 2017-05-13 ACTUALLY GET CORE UWP LIBRARY WORKING (modulo build griping about version mismatch issues) WITH UNITY, KINECT, & UWP. ***finally***. NO GOING BACK NOW DANGIT
• 2017-05-13 Unbreak things *again* — upgrade to Kinect/Unity v2.13, and confirm UWP app actually works. Now to see if scene changes saved properly….
• 2017-05-09 OK, after fixing local Unity installation (5.6 upgrade, but not latest Kinect/Unity assets), UWP Kinect Unity app once again ACTUALLY RUNS. Now not going to MESS with it for SOME TIME.
• 2017-05-08 Whoops, can’t import non-UWP assemblies into UWP app. So, converted Core and need to convert StateMachine to UWP style. But let’s move this over on git first.
• 2017-05-03 Get Core and StateMachine back in there and compiling again. Exclude binaries and remove packages.config that was pulling in dear departed SharpDX.
• 2017-05-03 Try again to see what is some kind of reasonable Unity configuration to check in Hopefully visible metadata and text format for assets will help.
• 2017-04-18 .gitignore file from https://github.com/github/gitignore/blob/master/ Unity.gitignore