The Metal Petals are ready to roll. Check em out at the New Interfaces for Musical Expression (NIME) show this Thursday, May 6th at Tonic in NYC. I’ll be rippin’ it up… literally!

http://www.tonicnyc.com/
http://stage.itp.nyu.edu/nime/tonic.htm

The camera now is first keying out much of the background to make the image simpler, and then tracking 4 points on the petals. These points directly correspond to pitch, playback speed, position randomness (where in the course of the music a particular grain of sound is playing), and the length of the grain in milliseconds.

As spec’ed out in the beginning, the piece starts as noise. As I started to cut and bending the petals, the music became somewhat recognizable as the various elements shifted closer to “normal”. There is unpredictable interplay between the elements since shifting one petal sometimes pushes another petal to a new position. I feel this makes the piece more interesting from an improvisatory outlook.

In the end, the petals were totally cut up and flattened, and the song was just recognizable enough. All in all it was an almost perfect culmination of what I’d had in mind months ago. Now i need to transpose it up to the scale of a full human body.

This project has been carefully planned out and the creation process has been set up in discrete stages with clear goals since the very beginning, thanks to Gideon’s organizational skills, and my own interest in organizing myself and maintaining the purity of the idea regardless of technology, aesthetics, or any other tangential considerations. All in all, I feel like I’ve achieved most of my goals, whether or not the actual public performance goes well.

Granular Synthesis is an audio processing technique that uses individual “grains” of audio to construct a meaningful sound or texture. This has obvious relevance to Metal Petals, which is in essence a performance based on solving a puzzle constructed of musical pieces (e.g. grains).

After trying to write my own granular synthesis algorithms from scratch, I was giving Luis Alvarez, and the sewing machine I gave him, a ride to Manhattan when he told me he had been studying granular synthesis in the “Frameworks” class. As they started playing with “the granular synthesis patch made by someone in Japan, he had thought of its relevance to the Metal Petals and my ideas of reconstruction of music.

This was a lucky coincidence since my own granular synthesis patches were outputting extremely choppy sounds at best and none of the ITP gurus, some of whom helped create Max/MSP/Jitter, would even feign to respond to my request for advice. (There seems to be a surprising amount of vanity in the nether-regions where music and video intermingle).

The mother of all granular synthesis patches that Luis sent me is made by Nobuyasu Sakodna and is quite difficult to find online despite its popularity among those in the know. His website is http://www.bekkoame.ne.jp/~nsakonda/ and the patch is aptly titled “MSP Granular Synthesis patch”.

My intention is to become familiar with the sounds I can get out of this patch, change it around a bit so that it suits my needs more exactly, and then hook it up to the video tracking patch and come up with an idea of how the movement in the video image will evolve the music and reassemble the grains over time. All the while, the design has to be built to accommodate an unavoidably high level of error in the tracking algorithm.

As you can see, I have a new camera – the Fire-i Unibrain. The cycling74.com FAQ, written by Jeremy Bernstein, who I also consulted, recommends firewire cameras for live video because USB and DV cameras both compress the video image which must then be decompressed by the computer. This compression/decompression causes significant latency in the image and makes these unsuitable for live video and music.

The Itchy Scratch movie was done using Jean-Marc Pelletier’s cv.jit package. Although the pixels appear to have “flocking” behavior, this is really just a by-product of the blob tracking and the way I’m playing with the “bugs” in it all, if you get my pun.

This week, I spent more time getting the cv.jit externals to do what I want them too. There are so many objects and so many options with this package that I really have to pick a couple of the cv objects and stick with them. Having spent a good deal of time playing with these now, I see a bunch of hurdles:

-designing the actual instrument with computer vision in mind is absolutely necessary. In order to hone in on the part of an image that I’m interested in without getting too involved in computer vision, it is much easier to do so with careful lighting and coloring of objects. I think rather than being a setback, this is actually a positive thing for the performance. Careful lighting and coloring can be done in a way so as to enhance the entertainment value of the performance.

-I started talking to a guy with bad teeth and dirty hands on the subway – he was relatively eloquent and well-informed, but it ends up he collects scrap metal for a living. He helped me eliminate choices of metals for my piece, and gave me a tip of a scrap yard near my house in Brooklyn that probably has thin gauge aluminum like the type I’m looking for.

-I haven’t yet started worrying about the music composing part of the piece, right now I’m imagining music divided up into samples which are arranged in a standard sequencer. The cv.jit patches will map blobs in the video feed to features of these samples, such as where they fall in a measure, playback speed, effects, and possibly pitch (although this sounds like it would make things difficult).

-I also haven’t started using live video – up until now, I’ve been using saved movies so I could control my parameters and get to know the cv.jit capabilities. I have a USB camera, but it doesn’t work on Mac. I have to either buy a new camera, or switch to PC.

Here’s an email from him:

Thanks for the feedback, Amos!

Jean-Marc

My tactic so far is to chroma-key out all but one color – the color of the petals – then feed this image into an edge-detection/object segmentation patch to divide the flower into its individual petals. From here, the object tracking patch follows each petal through its contortions. Although it’s currently far from working, the idea is simple enough that I’m relatively confident that this will work in the end.

Another set of motion tracking objects is available as part of the Tap Tools at http://www.sp-intermedia.com/taptoolsmax/ . I haven’t yet been able to get these working on my machine, but rumor has it they’re a very useful although it seems they’re better known for audio than video. The video objects output a level of movement within regions, not actual tracking like that done by jit.findbounds.

Also available is Cyclops, http://www.cycling74.com/products/cyclops.html, a motion tracking object by Eric Singer. It costs $100, and divides the video frame into a grid which is then analyzed for greyscale, threshold, color, and motion. This sounds similar to the Tap Tools, and I hear that this functionality is mostly incorporated into Jitter itself now.

A shipment of QProx capacitive sensors arrived this week, but rather than get all teched out right away, I would like to wait until the motion tracking idea fails before playing with these.

In related news, I’ve adapted my original concept to fit another project I’m doing in another course taught by Frank Migliorelli. This project is a kids museum interactive installation. Check the link to “Interactive Design for Kids” from my website . This project uses the metaphor of an ecosystem to allow kids to experience collaborative music composition firsthand.

Metal Petals is a musical puzzle.

Overview
The performer creates a musical composition by twisting, cutting, and gnarling flower petals made out of metal. Shaping the metal petals shapes the sound.

Sound
The sound begins as chaotic incomprehensible noise. The performer tames the noise by shaping the metal. The performer’s job is to shape the composition into something the audience can enjoy.

Materials
The metal petals are made of light thin metal sheeting, such as aluminum. The performer shapes the metal petals with tin snips and a hammer.

The Process
As the performer snips and shapes the metal petals, the sounds of working the metal are incorporated into the musical composition. The sound at any given moment is a stylized reflection of the shape of the “flower”. The performer reacts and adapts to changes in the sound in an improvisatory way.

The performer wields a large heavy mallet and stake. By using the mallet to shape the sheet of metal, the timbre of the motor knocks and their speed is affected, which in turn alters the sound generated by the Max patch.

Heavy Metal Flower
Pieces of bendable metal sheet in a tulip-like shape about 4 ft. tall. The piece starts as incomprehensible noise. The performer stands in the middle of the flower twisting, bending, and trimming the metal with shears until it becomes apparent that there is a recognizable tune encoded within the metal. By bending in a step-by-step manner, the performer slowly exposes the tune until finally it is all very clear and obvious and sweet sounding.

Throughout the semester, we will conceptualize, design, build, and perform a new musical instrument. I expect many issues will arise. Below I have summarized my thoughts on the subject having just a cursory understanding of the topic. The assigned readings were a good primer that evoked my thoughts on the basics of modern music and technology.

The article, “The Limitations of Mapping as a Structural Descriptive in Electronic Insturments” by Joel Chadabe addresses the traditional view of electronic instrument design as a means of designing a mapping between human input and instrument output. He concludes that viewing electronic instruments as purely mapping devices is outdated due to a continually blurring division between direct and indirect mapping.

This seems obvious. Having limited knowledge of history, I wonder why the question of mapping was ever considered to be of such magnitude. Modern electronic music technology allows for a direct musical reaction to a human gesture, for example, while still triggering other semi-autonomous musical reactions which are not so direct. In other words, that there will be a reaction to a human gesture can (and maybe should) be directly mapped, but very often, the details of just what this reaction will be cannot be predicted if the system is so designed.

In “Principles for Designing Computer Music Controllers”, Perry Cook points out some hard-learned guides to successful instrument design. The general gist of his article boils down to a few simple points: limit what the instrument is capable of, and design the instrument for a particular piece, not as a purely self-congratulatory feat. He argues that extra “options” on an instrument are extremely confusing and non-productive for the performer and the composer, since both are often overwhelmed with the possibilities of what the instrument “could” do, and never settle on any particular use for the instrument.

This is somewhat counter-intuitive in that the age of digital equipment is often flaunted for having “opened up the realm of the possible” to include just about everything. What Cook is saying, is that “everything is possible” is an utterly useless belief in real-life applications where one must dumb-down the instrument to make it useful to the performer and composer.

The issue of the instrument being designed for a particular musical piece is interesting. In my own experience composing music, I often tried to understand the meaning of composition with modern computer equipment. Working on one idea of sound for hours, you come to realize that what you are composing is the final product, the performance itself. The mass-reproducible sound file that sits on your hard drive is both composition and performance. Like that Jorge Luis Borges story, “On Exactitude in Science”, where the king’s mapmakers create a map so exact, it ends up being exactly the same size as the kingdom, the use of digital sound sculpting tools results in such a detailed documentation of the composition and the composer’s intentions, that the end composition is actually the sound of the composition itself being played.

Cook, by recommending that an instrument be built for a particular composition is further equating the instrument and the composition. It is interested to ponder whether all three – instrument, composition, and performance – are not antiquated terms which now represent one and the same thing. Is this just a phase, or have we totally moved past the classical terminology of music?

A related issue is the idea of reproducing what was possible before modern digital equipment arrived on the scene. Almost 100% of music on the radio and on TV is mostly artificial, meaning that the sounds are generated digitally and the rare recordings of actual instruments and performers are augmented and “touched up” in the editing studio afterwards. Yet the vast majority of this music replicates the sound and feel of “authentic” instruments and real performances such as those which existed before the digital era. Very rarely do you find a recording (except in electronica) in the which the sound is obviously digital and inauthentic.

Is the replication of “authenticity” an unavoidable preference of the human species (much like some argue linearity in films is a result of a human predisposition to straight stories). It seems relatively clear that the “postmodern reality” of art is one that is based upon sampling and simulation, but are these inherently related to “authenticity”?. Likewise, music and film both have time-based elements – what does this mean for the form of these arts? Are they dependent upon human limitations in our understanding of time? if so, have we already reached the limits of this man-made concept?