Alright,

The Metal Petals are ready to roll. Check em out at the New Interfaces for Musical Expression (NIME) show this Thursday, May 6th at Tonic in NYC. I’ll be rippin’ it up… literally!

http://www.tonicnyc.com/
http://stage.itp.nyu.edu/nime/tonic.htm

As a final test of the feasability of building and performing on the Metal Petals, I made a small scale model of the instrument from heavy paper foam board, and set the camera upon it.

The camera now is first keying out much of the background to make the image simpler, and then tracking 4 points on the petals. These points directly correspond to pitch, playback speed, position randomness (where in the course of the music a particular grain of sound is playing), and the length of the grain in milliseconds.

As spec’ed out in the beginning, the piece starts as noise. As I started to cut and bending the petals, the music became somewhat recognizable as the various elements shifted closer to “normal”. There is unpredictable interplay between the elements since shifting one petal sometimes pushes another petal to a new position. I feel this makes the piece more interesting from an improvisatory outlook.

In the end, the petals were totally cut up and flattened, and the song was just recognizable enough. All in all it was an almost perfect culmination of what I’d had in mind months ago. Now i need to transpose it up to the scale of a full human body.

This project has been carefully planned out and the creation process has been set up in discrete stages with clear goals since the very beginning, thanks to Gideon’s organizational skills, and my own interest in organizing myself and maintaining the purity of the idea regardless of technology, aesthetics, or any other tangential considerations. All in all, I feel like I’ve achieved most of my goals, whether or not the actual public performance goes well.

I’m now on the steps to creating a composition for the Metal Petals ‘instrument’. The composition itself will be an MSP patch sequencing and playing back processed samples. Although controlling a ready-made soft-synth and sequencer (like Reason and Digital Performer) would make the composition simpler to do, it would lack the sort of control and abstract sonic texture that I feel is necessary to make this piece compelling.

Granular Synthesis is an audio processing technique that uses individual “grains” of audio to construct a meaningful sound or texture. This has obvious relevance to Metal Petals, which is in essence a performance based on solving a puzzle constructed of musical pieces (e.g. grains).

After trying to write my own granular synthesis algorithms from scratch, I was giving Luis Alvarez, and the sewing machine I gave him, a ride to Manhattan when he told me he had been studying granular synthesis in the “Frameworks” class. As they started playing with “the granular synthesis patch made by someone in Japan, he had thought of its relevance to the Metal Petals and my ideas of reconstruction of music.

This was a lucky coincidence since my own granular synthesis patches were outputting extremely choppy sounds at best and none of the ITP gurus, some of whom helped create Max/MSP/Jitter, would even feign to respond to my request for advice. (There seems to be a surprising amount of vanity in the nether-regions where music and video intermingle).

The mother of all granular synthesis patches that Luis sent me is made by Nobuyasu Sakodna and is quite difficult to find online despite its popularity among those in the know. His website is http://www.bekkoame.ne.jp/~nsakonda/ and the patch is aptly titled “MSP Granular Synthesis patch”.

My intention is to become familiar with the sounds I can get out of this patch, change it around a bit so that it suits my needs more exactly, and then hook it up to the video tracking patch and come up with an idea of how the movement in the video image will evolve the music and reassemble the grains over time. All the while, the design has to be built to accommodate an unavoidably high level of error in the tracking algorithm.

Something weird happened to me while designing my instrument. Double click on the movie to play.

As you can see, I have a new camera – the Fire-i Unibrain. The cycling74.com FAQ, written by Jeremy Bernstein, who I also consulted, recommends firewire cameras for live video because USB and DV cameras both compress the video image which must then be decompressed by the computer. This compression/decompression causes significant latency in the image and makes these unsuitable for live video and music.

The Itchy Scratch movie was done using Jean-Marc Pelletier’s cv.jit package. Although the pixels appear to have “flocking” behavior, this is really just a by-product of the blob tracking and the way I’m playing with the “bugs” in it all, if you get my pun.

My website was partially deleted this week – I don’t know how, but there should be backups somewhere in NYU – I’m in the process of finding out where. So here’s my update via email:

This week, I spent more time getting the cv.jit externals to do what I want them too. There are so many objects and so many options with this package that I really have to pick a couple of the cv objects and stick with them. Having spent a good deal of time playing with these now, I see a bunch of hurdles:

-designing the actual instrument with computer vision in mind is absolutely necessary. In order to hone in on the part of an image that I’m interested in without getting too involved in computer vision, it is much easier to do so with careful lighting and coloring of objects. I think rather than being a setback, this is actually a positive thing for the performance. Careful lighting and coloring can be done in a way so as to enhance the entertainment value of the performance.

-I started talking to a guy with bad teeth and dirty hands on the subway – he was relatively eloquent and well-informed, but it ends up he collects scrap metal for a living. He helped me eliminate choices of metals for my piece, and gave me a tip of a scrap yard near my house in Brooklyn that probably has thin gauge aluminum like the type I’m looking for.

-I haven’t yet started worrying about the music composing part of the piece, right now I’m imagining music divided up into samples which are arranged in a standard sequencer. The cv.jit patches will map blobs in the video feed to features of these samples, such as where they fall in a measure, playback speed, effects, and possibly pitch (although this sounds like it would make things difficult).

-I also haven’t started using live video – up until now, I’ve been using saved movies so I could control my parameters and get to know the cv.jit capabilities. I have a USB camera, but it doesn’t work on Mac. I have to either buy a new camera, or switch to PC.

A great collection of computer vision objects, cv.jit, for Jitter is available from Jean-Marc Pelletier, a Canadian grad student at the Institute of Advanced Media Arts and Sciences in Japan. I’ve been playing with them in order to get the basic principles of object/blob and shape tracking working. They’re here: http://www.iamas.ac.jp/~jovan02/cv/

Here’s an email from him:

Thanks for the feedback, Amos!

Jean-Marc

My tactic so far is to chroma-key out all but one color – the color of the petals – then feed this image into an edge-detection/object segmentation patch to divide the flower into its individual petals. From here, the object tracking patch follows each petal through its contortions. Although it’s currently far from working, the idea is simple enough that I’m relatively confident that this will work in the end.

Another set of motion tracking objects is available as part of the Tap Tools at http://www.sp-intermedia.com/taptoolsmax/ . I haven’t yet been able to get these working on my machine, but rumor has it they’re a very useful although it seems they’re better known for audio than video. The video objects output a level of movement within regions, not actual tracking like that done by jit.findbounds.

Also available is Cyclops, http://www.cycling74.com/products/cyclops.html, a motion tracking object by Eric Singer. It costs $100, and divides the video frame into a grid which is then analyzed for greyscale, threshold, color, and motion. This sounds similar to the Tap Tools, and I hear that this functionality is mostly incorporated into Jitter itself now.

A shipment of QProx capacitive sensors arrived this week, but rather than get all teched out right away, I would like to wait until the motion tracking idea fails before playing with these.

In related news, I’ve adapted my original concept to fit another project I’m doing in another course taught by Frank Migliorelli. This project is a kids museum interactive installation. Check the link to “Interactive Design for Kids” from my website . This project uses the metaphor of an ecosystem to allow kids to experience collaborative music composition firsthand.

Metal Petals

Metal Petals is a musical puzzle.

Overview
The performer creates a musical composition by twisting, cutting, and gnarling flower petals made out of metal. Shaping the metal petals shapes the sound.

Sound
The sound begins as chaotic incomprehensible noise. The performer tames the noise by shaping the metal. The performer’s job is to shape the composition into something the audience can enjoy.

Materials
The metal petals are made of light thin metal sheeting, such as aluminum. The performer shapes the metal petals with tin snips and a hammer.

The Process
As the performer snips and shapes the metal petals, the sounds of working the metal are incorporated into the musical composition. The sound at any given moment is a stylized reflection of the shape of the “flower”. The performer reacts and adapts to changes in the sound in an improvisatory way.