This JanPlan I worked as a lead developer for the Integer X, a group working on creating the Interactive Tracking Floor. While the project went from its conception to the final stage through many iterations - and in the end did not even use the floor in the way planned originally - it was a lot of fun and a great learning experience. This post will outline what was my role in the project.
The idea of creating an installation that will use a walkthrough space and try to engage its actors in a short interaction timespan was created very organically by the group. One of the seeding ideas that lead to the final concept was my pitch to create a heat map of a building, based on the motion of people around a Colby building. As the ideas iterated, the concept merged with a concept for music generator based on a path of an actors walk and the ITF was born. In its original design I created, the ITF was supposed to look as follows:
In this plan, the person would leave behind a trail of color and movement through various intervals would trigger various dub step samples. However, the idea had to be ultimately at first minimized (we would mount the projector over a limited space) and then ultimately scrapped just days before final presentation (as there was no way to safely mount the projector pointing on the ground from high altitude). In the end, we ended up going through with the floor tracking, but presenting the video feed on the white wall over the walkway to Pugh we were monitoring.
Software design and implementation
At first we considered using Isadora as our main source of visual presentation and visual analysis. Soon it turned out that the visual tracking we can obtain using macros prepared for Isadora would be erratic, as Isadora does not keep a track on what is being currently in its blob buffer. At this point I took it upon myself to propose and design a better pipeline that our product could use. Being acquainted with both the Quartz 2D/3D platform that is integrated into OS X, as well as with OpenCV through the prior exposure to Computer Vision, I proposed moving away from Isadora to Quartz (for effective and high quality graphic rendering which would be necessary for aesthetically pleasing tracks) and OpenCV. The idea was supported after I proposed an improved pipeline that would allow us to go as far as to use multiple computers in the process and differentiate between people accurately. (as the first concepts included having two separate computers for tracking and visual/audio compositions)
At this point Brendan and Devon began acquainting themselves with OpenCV (due to their prior exposure to C), while Steven started learning about possible effects of Quartz compositions. I started investigating networking and data-passing capabilities of Quartz that would let us interface multiple computers together. The research was successful, but we ultimately we ended up downscaling to one Mac mini for the project. As it was proven to be very difficult to use networking between Quartz and Isadora, Steven and me came up with a MIDI interface between these two programs that we would use to pass data between them.
Since at this point Brendan and Devon gave up on creating a functional blob tracking (as their software was relatively slow), I decided to take on the task myself. Based off the original Kineme's CVTools, I used their skeletal patch structure to implement various blob tracking algorithms (especially using labeling, and LK). However, we found that the tracking tools severely leaked memory, what ultimately led to a couple of hours hunting for memory leaks. Going forward, I continued to perfect the motion and blob tracking, until a very reasonable system being able to process about 15-20 fps was written. With a couple of days left, the plugin was ready, but at this point the group has decided that due to the significant integration that has already occurred between blob tracking as implemented in Isadora and Csound it will not be time feasible to use Quartz/OpenCV blob tracking (as it would require severe rewrite to Alex's music playing software).
(on image above: OpenCV-based motion tracking with example rectangles overlaid; you can see that velocities decay on my head produces multiple blobs, but this can be alleviated by using minimal blob size for display; the fourth, unmarked person was relatively stationary for about 10 seconds, and as such is not picked up as a blob)
As such I took upon myself to recreate our visual projection from Steve's circles-and-squares to more elaborate, particle system based Quartz composition which was used throughout the presentation and was easy to integrate with current settings. The system turned out to be very visually attractive:
During the presentation, for a short period of time, as a tongue-in-cheek joke, I added nyan cats as sprites overlying my particle systems and replaced Alex's music with a long loop of the nyan cat video. Needless to say, the audience was very amused and definitely appreciated the joke. Unfortunately, this cannot be said about the person that was currently staffing the Pulver's front desk. :-)
Throughout the process I created a number of interesting patches for Quartz Composer that wrapped OpenCV calls. What I am most proud of is the fact these can be further used in teaching Computer Vision, as with Quartz they provide an extremely easy - and fast - prototyping pipeline that can be used to create effective and powerful OpenCV presentations and simple applications. I plan on continuing "porting" OpenCV to Quartz in hopes that it can be used effectively as a teaching tool. I keep on adding and extending patches successively and the source code can be downloaded from my GitHub account.
My participation in this JanPlan class was a very last minute decision, and my arrival to the class was delayed by a couple of days. Even so, I managed to quickly find a team with which I had a great time cooperating and creating (in the words of one of the passers-by) 'it's such a cool idea!' art. It was very frustrating when my OpenCV tracker didn't make it to the final cut, and when the system would randomly crash due to the overload of data being sent to Csound. It was however a great team exercise that I have enjoyed very much.