Freeze Frame: Audio, Aesthetics, Sampling, and Contemporary Multimedia

Ken Jordan and Paul D. Miller a.k.a. DJ Spooky that Subliminal Kid

pdf (24 Kb)

Paul D. Miller's Preamble:

In an era of intensely networked systems, when you create, it's not just how you create, but the context of the activity that makes the product. Let's think of this as a hypothetical situation become real, and then turn the idea inside out and apply it to music - operating systems, editing environments, graphical user interfaces - these are the keywords in this kind of compositional strategy. During most of the spring of 2002 I was working on an album called "Optometry." I thought of it as a record that focused on "the science of sound - as applied to vision." Think of it as a kind of "synaesthesia" project navigating the bandwidth operating between analog and digital realms. "Optometry" was constructed out of a series of audio metaphors about how people could think of jazz as text, of jazz as a precedent for sampling - of jazz as a kind of template for improvisation with memory in the age of the infinite archive. In sum, the album was a play on context versus content in a digital milieu using sampling as a "virtual band" of the hand. Flip the situation into the here and now of a world where file swapping and peer-2-peer bootlegs are the norms of how music flows on the web, and "Optometry" becomes a conceptual art project about how the "hypertextual imagination" holds us all together. Seamless, invisible, hyper-utilitarian... those are some of the words that describe the composition process of "Optometry." What's new here? In 1939 John Cage made a simple statement about a composition made of invisible networks that was called "Imaginary Landscape." The piece was written for phonographs with fixed and variable frequencies (consider that there was no magnetic tape at that time), and radios tuned to random stations. The idea for Cage was that the music was an invisible network based on "chance operations." As Cage would later say in his famous 1957 essay "Experimental Music," "Any sounds may occur in any combination and in any continuity." The sounds of one fixed environment for him were meant to be taken out of context and made to float - think of it as audio free association, and you get the first formalist ideas of the origins of DJ culture. But what does this have to do with jazz?

In 1964 Ralph Ellison gave a discussion before the Library of Congress about writing jazz criticism. In it he discussed Henry James's fascination with Americaness - think of it as an echo of the Cage notion, and flip the code into a different cipher - you arrive at Henry James' critique of Americanness as "a complex fate." The Ellison lecture was called "Hidden Name/Complex Fate" and Ellison takes us on a journey through elements "absent from American life." In this text Ellison would flip the mix and build a template for a new kind of literature - that's the echo of "Imaginary Landscape" that intrigues me. He said: "So long before I thought of writing, I was playing by weather, by speech rhythms, by Negro voices and their different timbres and idioms, by husky male voices and by the high shrill singing voices of certain Negro women, by music by tight places and wide spaces in which the eyes could wander..." Again, the invocation of an imaginary landscape made of the hyper-real experiences of living in a world made of fragments. That's what "Optometry" inherits. Think of it as a dialectical triangulation between the idea of being made from files of expression put through places that are not spaces, but code. Gesture is the generative syntax, but once the sounds leave the body, they're files. And that's the beginning...

1.

When computers communicate over a network, they do so through sound. Before information can be sent over wires run between computers, it must first be translated into tones. The composer Luke Dubois, of Columbia University's electronic music department, has described the static you hear when a modem connects as a hyper-accelerated Morse Code, a billion dots and dashes sung each second, too fast for the human ear to discern. This has been true since the dawn of networked computing. When the first two nodes of the Internet, at UCLA and Stanford, were brought online in 1969, Charlie Kline at UCLA famously initiated the connection by typing "login." After keying the letter "l" he received the appropriate echo back along the phone line from Stamford. The same with the letter "o." But when he hit "g" the system crashed; the audible reply from Stamford never reached its destination.

In 1972, Ray Tomlinson modified a program meant for ARPANET, the precursor to the Internet, that would let people send each other data as small "letters." He chose the @ sign for addresses for a simple reason: the punctuation keys on his Model 33 Teletype made it easy to type; it was a convenient way to lend a geographic metaphor to an otherwise abstract place made up of data and people's interaction with the nodes that hold the data together. In one fell swoop, Tomlinson signaled that data could be both a place and a linguistic placeholder for digital information as a complete environment. By using the @ symbol, he restated what modernist artists and composers had been pointing out for over a century: when information becomes total media in the Wagnerian and the Nietzschian sense in, we arrive at the "Gesamkunstwerk" or "the total artwork." The Situationists referred to this as a "psycho-geography." Antonin Artaud wrote an essay about it called "Theater and It's Shadow;" for him it was based on the interaction of different forms of alchemy. When Artaud coined the term "virtual reality" in his 1938 essay "The Alchemical Theater," he anticipated a realm where signs, symbols, letters, and ciphers were all placeholders in the rapidly changing landscape of a society that faced the surging tides of industrial culture's mad race to become an information culture. It was a phrase to describe a mind trying to make sense of the data road kill on the side of the information highway being built in the minds of artists whose dreams punctuated an immense run on sentence typed across the face of the planet as technology carried the codes out of their minds and into the world. In the 20th century, one symbol - "@" - ushered in a new world linked by the intent of people to communicate. This is a world of infinitely reflecting fragments, vibrating, manifesting a hum, making music.

The connection between sound and networked computing is more than the product of technical convenience. It can be traced to the first visionary articulation of the digital age. In his seminal essay from 1945, "As We May Think," Roosevelt's science advisor, Vannevar Bush, proposed the creation of a device he called the memex, which provided the inspiration for what later became the networked personal computer. Bush's memex system had the ability to synthesize speech from text, and, conversely, to automatically create text records from spoken commands. He wrote enthusiastically of the Voder, which was introduced at the 1939 World's Fair as "the machine that talks." "A girl stroked its keys and it emitted recognizable speech," Bush wrote. "No human vocal cords entered in the procedure at any point; the keys simply combined some electrically produced vibrations and passed these on to a loud-speaker." Bush also discussed another Bell Labs invention, the Vocoder, an early attempt at a voice recognition system. Central to his vision of the memex was the notion that sound would circulate through the system, available for easy retrieval and manipulation.

Today that ease of access and malleability is transforming the way musicians conceive of and make music. It is now simple to convert sound into digital streams, so it can flow anywhere across the computer network, to be manipulated by a continually growing array of software. Real time collaborations between musicians across the Net are becoming common. Online collaborations that are not real time are commonplace. The combination of databases (for storage), software (for manipulation), and networks (for interactivity between databases, software, and musicians) is challenging many long held notions of what music making can or should be. Established boundaries are blurring.

This blurring comes from a basic premise behind computing: that all information can be translated from its original form into binary code, and then re-articulated in a new form in a different medium. Texts can be stored in a database as ones and zeros, and later output as images or sounds. Ted Nelson, the man who coined the terms "hypertext" and "hypermedia" in the mid-1960s, was among the first to appreciate the full range of opportunities that networked computers make possible. In 1974, he proposed the playful idea of "teledildonics," a computer system that would convert audio information into tactile sensations. Why should music only enter the body through the ear? Why not through the skin, or through the eye?

Artists have been using computer networks for collaboration at least since 1979, when I.P. Sharp Associates made their timesharing system available to an artist's project called "Interplay." Organizer Bill Bartlett contacted artists in cities around the world where IPSA offices were located, and invited them to participate in an online conference - essentially a "live chat" - on the subject of networking. At the time this technology was rare and expensive; artists had no access to it. "Interplay" is often referred to as the first live, network-based, collaborative art project.

Around the same time, the innovative use of satellites by artists such as Nam June Paik, Joseph Beuys, Douglas Davis, Kit Galloway, and Sherrie Rabinowitz were connecting performers across great distances in collaborative, interactive pieces. A dancer in New York would improvise to music played in Paris, while video of the two would be edited into a single performance for broadcast in, say, Berlin.

Although these pioneering telematic works did not make use of networked computing - bandwidth and processor speeds were not yet great enough to allow for it - they set precedents for the real time network-based interaction between artists that became possible in the 1990s, as the technology improved and costs came down.

Online collaboration today takes many forms. Using Web-based music technologies, artists are working together to create new music. There are online studios that connect artists across great distances, and Web-based jams between musicians who have never laid eyes on one another. At the same time, even more popular are "collaborations" between artists who are not even aware that a "collaboration" is taking place. Referred to as "remixes" or "bootlegs," digital files of a wide range of recorded material are being cut up and manipulated into entirely new works of art - blending distinct and unlikely source materials into singular creations. Of course, this kind of unsolicited collaboration challenges some long-held notions of intellectual property, and an artist's unique affiliation with his or her own output. But at the same time, it brings back the idea of a shared folk culture, where creative expression is the property of the community at large, and can be shared for everyone's benefit. Digital technology may be a route that reconnects us to aspects of our tribal roots.

As new as these techniques are, however, they retain a continuity with pre-digital compositional approaches. The network simply allows musicians to perform together online, replicating the experience they have always had when jamming in the same room. At the same time, the mixing of distinct aural elements certainly does not require digital technology; analog sound mixing dates at least to John Cage's 1939 performance of Imaginary Landscapes, which featured a mix of turntables and radios. From this perspective, computer networks simply contribute to long standing tendencies in composition that preceded the digital era.

However, some composers are exploring a wholly original, uncharted musical terrain, one that is unthinkable without networked computers. In these works, the sound experience is created through the real time participation of the listener in the making of the performance itself. These online sound art pieces rely on the interactive engagement of the listener, who helps to shape the specifics of the performance through her choices and actions, which are communicated to the music making software over the wired network. In this way, the traditional distinction between "artist" and "audience" begins to melt away, as the "listener" also becomes a "performer."

2. Composing With Software

When the software conditions the experience, it conditions the music. One thing that many people notice when they start making music online is that the Web is a powerful vortex; it doesn't let you go. There is no single way to end a session; rather, there are many ways. There are bootlegs of everything that has ever made it onto the Net.

Multiplicity is an unwavering factor in the online experience. Look at sites like Afternapster.com. You will find hundreds of peer-to-peer networks, each of which is the private preserve of a file sharing community. These can be seen as the operational mode of a culture of distributed networks, held together by a common thread: each represents a particular taste as distributed through the system.

As Artaud said (in an incredible pre-cognition of the digital era's constant stream of information guiding any creative act): "All true alchemists know that the alchemical symbol is a mirage as the theater is a mirage... [It's] the expression of an identity existing between the world in which the characters, objects, images, and in a general way all that constitutes the virtual reality of the theater develops..." In a way, collaborative music making on the Net requires an interaction of people and software that turns almost all normal contact between musicians into a mediated experiment with the hypothetical. Is there a human on the other side of the screen? The sounds can only give you a hint. The software is a window onto a realm governed only by the uncertainty of that fact. The connections displace physicality in a way that leaves you a victim of context. This is the experience of tele-composing. It makes the creative act become a cog in the abstract machines of the software that mediate it.

Using online studio software, such as Rocket, Pro Tools, or Reason, allows you to mix equally with either musicians or found sounds. Through the software interface, there is a certain equivalency. Collaboration can take place in real time between people, or between the remnants of creativity that people leave behind - the Net is full of such suggestive debris. In this context, the only limitation comes from the bottleneck that bandwidth places on file exchanges. The quicker the speed, the richer the environment.

Another effect of software is to dematerialize the musical instrument. It does this by distributing the qualities of an instrument across the various peripherals that control the sounds that the software generates. Algorithm displaces rhythm and becomes the environment in which to create. MAX/MSP is an open ended software environment that lets you create templates for virtual instruments - it allows you to make an aggregate of whatever sounds you run through its parameters. Almost all process oriented software behaves like this. Editing environments such as Pro Tools or Digital Performer function as dissecting tables of sound; they allow the musician to compose material from multiple layers of tracks and files, and to then condition the total output. It's like building music out of Lego blocks.

That is, either Lego blocks or samples. Online, everything is a sample. Every audio element becomes a potential fragment for manipulation and recontextualization. Sampling follows the logic of the abstract machinery of a culture where there are no bodies - just simulations of bodies. The fragment speaks for the whole; the whole is only a single track drifting through a vast database. The basic structure of "assemblage," the method of collage, holds sway here. Think of this terrain as object-oriented programming with beats. Take the file, edit it: import/export/MIDI/SMTP.

Time code synchronizes the fragments, and makes it work wherever you are... FTP controls the data exchange as a basic source of file exchange... Lee Perry popularized the term "versioning" by using a series of vocal tracks that were taken out of context and de-familiarized through sound effects programming. This can be done either as a live process or improvised on a virtual "mixing board." Software that allows real time online jamming is appearing from every corner of the globe. But is your online collaborator a person or a bot? Or a combination of the two?

On the Web, collaborators can come in all guises. The White Stripes have no bass player. Steve McDonald, the bass player for Red Kross, felt that the White Stripes tunes could use more bottom. So each week he adds a bass guitar part to one or two White Stripes songs, and makes them available as "bootleg" MP3s. Jack White, the White Stripes' front man, has apparently given these remixes his blessing.

3. Interacting With Intelligent Networks

Once, every sound had a distinct source. A door slammed shut, a horn was blown, a guitar string was strummed. Audio came from a discrete event, it was tied to a discernable action.

Networked music challenges this notion by displacing sound from its origin, moving audio freely from one location to another, giving it a presence in and of itself. John Cage brought this quality into modern music with his 1939 piece, Imaginary Landscape. A performance that combined turntables and radio broadcasts, this work introduced networked interactivity into music making.. Cage mixed into his performance various transmissions that came over the airwaves, and with them created an entirely new composition. Sound separated from its source in this manner becomes a "free floating signifier," to borrow a phrase from Roland Barthes. The musical elements are liberated from a specific time and place, allowing them to be recontextualized in the final composition.

Robert Rauschenberg pursued something similar in the mid-1960s with his interactive, sound-emitting sculpture, Oracle. Rauschenberg's collaborator on the project, Bell Labs engineer Billy Kluver, described Oracle as "a sound environment made up of five AM radios, where the sounds from each radio emanates from one of the five sculptures. The viewer can play the sculpture as an orchestra from the controls on one of the pieces, by varying the volume and the rate of scanning through the frequency band. But they can not stop the scanning at any given station. The impression was that of walking down the Lower East Side on a summer evening and hearing the radios from open windows of the apartment buildings."

By the early 1970s, as the technology became more accessible, more artists began to explore the potential of networked media - both audio and video - to create unique forms of interactive expression. These artworks grew from the notion that meaning would emerge from media as it circulates freely within a network - and that meaning can be enhanced through strategic interventions by the artist or audience. Douglas Davis' 1971 performance, Electronic Hokkadim, produced at the Corcoran Gallery, was based on the interactions between telephone callers and broadcast television. Nam June Paik pursued what he referred to as a "cybernated art," based on the transmission of information through video and audio networks. Paik's 1973 television broadcast, Global Groove, stands as a landmark event in this trajectory. Fragments of performances by artists of various traditions - Western and Eastern, popular and elitist, traditional and modern - were strung together in a frenetic, continuous flow across the screen. Paik himself "performed" the broadcast as a live mix, choosing his streams as a DJ does today, manipulating images through a video synthesizer, using rhythm as the underlying principle of composition.

Enabling and manipulating the continuous flow of information was a principal concern behind the design of the networked personal computer. But before the mid-1980s, bandwidth constraints and limited processing power made the use of these tools prohibitively expensive for artists. However, it was long apparent to the pioneers of networked media - such as Davis, Paik, and Roy Ascott - that their artistic explorations with satellites and local wired networks would lead to computer-based work, once the technology had caught up to their vision.

Among the earliest musicians to dedicate themselves to the potential of networked computing were The Hub, perhaps the world's first "computer network band," which was founded at Mills College in 1985. One of the members describes their method as follows: "Six individual composer/performers connect separate computer-controlled music synthesizers into a network. Individual composers design pieces for the network, in most cases just specifying the nature of the data which is to be exchanged between players in the piece, but leaving implementation details to the individual players, and leaving the actual sequence of music to the emergent behavior of the network. Each player writes a computer program which make musical decisions in keeping with the character of the piece, in response to messages from the other computers in the network and control actions of the player himself. The result is a kind of enhanced improvisation, wherein players and computers share the responsibility for the music's evolution, with no one able to determine the exact outcome, but everyone having influence in setting the direction. The Javanese think of their gamelan orchestras as being one musical instrument with many parts; this is probably also a good way to think of The Hub ensemble, with all its many computers and synthesizers interconnected to form one complex musical instrument. In essence, each piece is a reconfiguration of this network into a new instrument."

Implicit in this approach is the idea that, within the network, a kind of intelligence is in circulation. David Wessel, at the University of California at Berkeley, has been working with his colleagues along these lines since the late 1980s, bringing together the fields of computer music and neural networks. Could an instrument become intelligent, and adapt to in an automated manner to a musician's playing style? Could it learn the preferences of a particular musician, and modify itself in response to what it learns? Using the MAX programming environment, Wessel began to experiment with musicians in a network context. "We have obtained reliable recognition of complex guitar strumming gestures and limited numbers of spatial gestures," he wrote. "With such procedures and much more research, we might conceivably move towards adaptive, personalizable instruments.... one will have to decide when to standardize or fix the instrument and let the musician learn the appropriate gesture and when to let the instrument adapt to the specialized approach of a player. How to rig the training harnesses on ourselves as players and on our instruments as expressively responsive musical tools will be a question of scientific, aesthetic, and social concern." Once meaningful information is circulating within a computer network, the opportunity emerges for a relevant interaction. As Wessel suggests, networked computer tools will lead musicians into making choices about aspects of their performance that had previously never had to be asked, such as: how "smart" do I want my instrument to be?

The notion that music can emerge from an intelligent, interactive environment has drawn some composers to compositional forms that would be inconceivable without telecommunications technology. One example is Atau Tanaka's 1998 installation, Global String. The work consists of a physical string, 15 meters long, that stretches from a floor diagonally to the ceiling of a room. At the ceiling, the string is connected to the Internet. "It is a musical instrument wherein the network is the resonating body of the instrument through the use of a real-time sound-synthesis server," writes Tanaka. "The concept is to create a musical string (like the string of a guitar or violin) that spans the world. Its resonance circles the globe, allowing musical communication and collaboration among the people at each connected site."

Ping, a site-specific sound installation by Chris Chafe and Greg Niemeyer, takes a similar approach. Ping has been described as "a sonic adaptation of a network tool commonly used for timing data transmission over the Internet. As installed in the outdoor atrium of SFMOMA," for the millennial exhibition 010101, "Ping functions as a sonar-like detector whose echoes sound out the paths traversed by data flowing on the Internet. At any given moment, several sites are concurrently active, and the tones that are heard in Ping make audible the time lag that occurs while moving information from one site to another between networked computers." In effect, Ping makes music out of the data flow of the Net - the constant motion of digitized fragments in real time is given an aesthetic form.

The composer and theorist Randall Packer has explored this line of telematic composition in a number of pioneering collaborative installations. For Mori, an "Internet based earthwork" first mounted in 1999 by Packer with Ken Goldberg, Wojciech Matusik, and Gregory Kuhn, the trembling movements of California's Hayward Fault are picked up by a seismograph, converted into digital signals, and sent over the Internet to the installation. This data stream triggers a series of low frequency sounds that vibrate through the installation, viscerally connecting the visitor to the moment-by-moment fluctuations of the earth's actual movement.

In what he has referred to as "artistic research projects," Packer has further explored the possibilities of interactive, telematic musical works. One such installation, Telemusic, was staged by Packer and his collaborators Steve Bradley and John P. Young at the Sonic Circuits VIII International Festival of Electronic Music and Arts in St. Paul, Minnesota, in November, 2000. Telemusic brought together live performers, audio processing of their performances, and real time participation from the public through a Web site, www.telemusic.org. As the performers read from a script, their delivery was effected by audio processing triggered by the mouse clicks of visitors to the Web site. The final mix in the room was then streamed to the Web site, so a visitor could hear the final musical composition that she had contributed to by clicking a mouse.

In order to create this direct form of interactivity, Packer's team had to develop an interface between impulses captured over the Internet and a server hosting MAX software. This circular experience, in which listener is also a participant in the making of a musical work, is indicative of the direction where the Internet is suggesting that music should go - as the distinction between "artist" and "audience" begins to slip away, and we find ourselves dipping into the data flow, listening to the music that it makes, and that we make with it.

www.djspooky.com

www.21cmagazine.com