The Theater of Pattern Formation


The Theater of Pattern Formation is nonstationary; a work in progress, largely irreversible. What follows is the view from 9 May 2003, when talking to funders ...

0. Sound samples.

I. Introduction

While the theories and concepts that describe unpredictable and random processes in nature are well understood, those appropriate to capturing nature’s structures and patterns do not exist in any where near a similar state of development.
This is the research frontier addressed by the burgeoning field of complex systems. This project is dedicated to engaging the public in an exploration, even a celebration, of this frontier—discovering patterns in natural, social, and engineered systems. How do we detect novel patterns in the world around us? How do we extend our vocabularies, beyond a current palette of what is expressible, to describe new structures and regularities that we have discovered? These questions go to the heart of the scientific and engineering enterprises; they bear on cultural evolution and aesthetic appreciation. Fortunately, the last several decades have produced wholly new approaches to the questions of the emergence of pattern.

The project will bring the most recent concepts of complex systems to the public via a new class of all-digital theaters. To do this, we propose a strongly educational approach that is oriented toward fundamental scientific research on perceiving patterns in sensory-immersive environments and that, via a companion ITR proposal, will develop the necessary acoustic and image computing and display technologies.


One of the essential goals of the project is to bring the exciting, contemporary ideas of nonlinear mathematics and physics to the public. The program material under development will give audience members access to results in the fields of structural complexity, deterministic chaos, and pattern formation. This will be done by contrasting natural imagery and sound and the simulation of natural phenomena in a way that plays on our innate abilities to perceive novel patterns in space and in time. Its scientific research component addresses key issues of the aural and visual psychophysics of presenting dynamic patterns that range across the complexity spectrum, in immersive environments. The project will also develop, using recent theories of structural complexity, ways to quantitatively measure structure and pattern in simulated model systems to automatically explore and discover engaging patterns and behaviors. These will be used to intelligently design program content and to demonstrate, during live performances, how pattern formation and discovery can occur.

II. The Theater of Pattern Formation

The central task of the present proposal is a production that we call the Theater of Pattern Formation. Roughly speaking, there are two components to the production. The first concerns exploring dynamical pattern formation in acoustic feedback systems. The second addresses, in a complementary way, spatiotemporal pattern formation in image feedback systems. Individually, these systems have received a fair amount of scientific, mathematical, and aesthetic investigation. One of the key novel aspects of the Theater of Pattern Formation is the attempt to cross couple these individual modalities (acoustic and visual) and to present the result as a live, ever-unfolding program in an immersive environment. The overall architecture of the system is shown in the figure.

We will develop educational programs for exploring the patterns generated by large-scale, complex, nonlinear systems. The results will be brought to the public using sensory-immersive environments, with a particular focus on the relatively new technology of all-digital dome theaters. There are two sets of tools developed by the companion ITR proposal that we will use to produce the programs. These consist of (i) real-time sound and image synthesis algorithms optimized for clustered multiprocessors that can be used for on-line investigation, as well as interactive theater performances, and (ii) a small-scale, lost-cost multi-channel acoustic and -image presentation system that uses off-the-shelf components for research and development of wide-screen program material. The new technology applications, that will arise in the ITR project and in the program production funded by the present proposal, will be complemented by visual and aural psychophysics studies designed to understand directly how audiences perceive and discover new spatial and temporal patterns that are presented via sensory-immersive environments. This will form an important part, that we believe is new and novel, of our evaluation of the program and performance content.


We explain the basic historical and educational motivations behind the development of the Theater of Pattern Formation program material in the following sections.

II.A Dynamical Pattern Formation in Acoustic Feedback Systems

Throughout the 20th century many musicians looked to technological breakthroughs as a means to expand both aesthetic and acoustic resources of musical expression. The resulting interactions between musicians, composers, engineers, and scientists produced a rich and diverse body of scientific research that contributed to the fields of acoustics, psychoacoustics, electrical engineering, bioacoustics, and scientific sonification, for example. At a deeper intellectual level the discourse between scientists and musicians has often been profound. In recent years this dialogue has only intensified due to breakthroughs in nonlinear mathematics and complex systems being applied to what can only be described as the beginnings of a scientifically grounded theory of sound and music. In recent years composer David Dunn ( has been exploring the sonification of the global behavior of hyper-chaotic analog circuits modeled in the digital domain. The emergent complexity of these systems results from the dynamical attributes of coupled chaotic attractors interacting in a high-dimensional phase space. The control of various circuit parameters determines a range of instabilities and structural couplings between distinct chaotic circuits, allowing different self-organizing behaviors to emerge. In many ways this work extends from and reinforces Dunn's previous work in bioacoustics and biomusicology and his interest in understanding pattern formation in natural sound systems [24-26]. Comparisons and interactions between these natural and artificial systems should shed light on how similar dynamical properties might be operating at their generative levels [27-29].

II.B Spatiotemporal Pattern Formation in Image Feedback Systems

Recent breakthroughs in the mathematics of spatial-pattern forming systems have revealed the intricate mechanisms responsible for the emergence of order in large-scale systems. This progress derives from research in the physics of phase transitions and critical phenomena [12,13], center manifold (pattern formation) theory [10,11,30], and nonlinear dynamics [7-9] over the last three or four decades. The result is that the patterns and structures which emerge in many natural phenomena can be analyzed and explored in great detail. Though the mathematics can be used to guide investigations into realms to which it does not directly apply, the sheer complexity and scale of most pattern-forming systems require the use of very powerful simulation engines. One component of research on pattern formation has been an exploration of how iterative, nonlinear image processing systems produce patterns. A simple example of this is found in video feedback systems in which a standard video camera, connected to a video display monitor, is pointed at the monitor screen, closing a hybrid electronic-optical feedback loop. With patience it becomes relatively easy to interactively explore a vast arena of pattern formation by varying the available controls (brightness, contrast, hue, focus, f/stop, camera angle, and so on). The result is a dazzling array of spatiotemporal patterns that spontaneously emerge from the feedback system [31-34]. Surprisingly, very many of these patterns correspond to those exhibited by physical, chemical, and biological systems. An important complementary aspect of pattern formation research has been a more direct attempt to say what patterns are and how we detect them [35-37]. One practical result of this work is a new set of techniques for designing nonlinear filters for image sequences which detect various kinds of coherent objects that are spontaneously generated in spatially extended systems [38,39]. These methods will form a key part of how the patterns formed in the image-processing domain are fedback and control aspects of the dynamics in the acoustic domain.

II.C Integrating Acoustic and Visual Patterns: Psychophysics of Sensory-Immersive Environments

The scientific need to process and comprehend large data sets has converged with increasingly powerful and available digital media technologies [40,41]. This has driven the expansion of the fields of scientific visualization and sonification at a time when it has also become evident that purely mathematical analysis is not the only way for comprehending complex data [42,43]. This is particularly true when attempting to translate mathematical and scientific breakthroughs to a general audience. The techniques of computer-based visualization and sonification often optimize the highly evolved capacity of our perceptual systems to recognize patterns and comprehend relationships. While preliminary research suggests that different sensory modalities are better at perceiving certain types of information [44], further research is necessary to identify what data features are most apprehensible to particular sense modalities [45,46]. Multi-modal interactions, such as those that occur between simultaneous visual and auditory displays, are even more mysterious (and potentially useful) [47,48]. While the challenge of cross-modal interaction has been a major issue of the contemporary electronic and media arts, it has received very little scientific investigation. The question of how visual events can change the perception of a sound and vice versa remains an open and salient research frontier [49,50].

III. Program Development


The Theater of Pattern Formation will allow the public to explore these frontier questions by bridging sound and image. The result is a large-scale feedback system in which pattern information flows between both acoustic and visual modalities.


III.A Audio Input to Pattern Formation Engine


While there has been a substantial amount of research in the application of nonlinear dynamics to musical synthesis and timbral modeling [20], there have been virtually no substantial attempts at applying these new principles of  “self-organization” to multiple structural aspects of media performance or to integrated multi-modal data displays. Attempts to integrate sound and image within popular media forums have been limited to intuitive juxtapositions that are often arbitrary or culturally driven. The technical issue to be addressed pertains to the need for appropriate and robust translation software to allow for the control of visual pattern formation by an audio source that is driven by fundamental and related mathematical principles. A further technical challenge is to allow for flexibility in the selection of appropriate audio parameters that can be mapped in real-time as parallel or interactive data streams to the visual display and its correlated control parameters [51]. MIDI (Musical Instrument Digital Interface) is one of the few control protocols available that is sufficiently well developed and reliable for this kind of mapping. MIDI will form the basis of our development strategy.


III.B Pattern Input to Acoustic Feedback System


The inherent integrative capacity of our aural perception is increasingly being appreciated as well suited for the display of high-dimensional data and transient data that can be missed by visual displays [44]. Data environments with many changing variables or where complex information must be monitored over time are particularly appropriate for sonification display. The unique attributes of such sonification displays emerge from the particular perceptual advantages of the human auditory system: the ability to process multiple auditory sources and streams, rapid detection, affective response, and high sensitivity to temporal characteristics and aural gestalt formation [52-55]. Despite the advantages that these unique perceptual characteristics might provide, specific research that addresses the optimal means of mapping data to specific parameters of sound remains in its infancy. While it might appear that the technical problems posed for the control of sound by visual data are a mere reversal of those stated in the previous section, the problem is much more complex and hindered by the lack of appropriate research as previously discussed. What constitutes and determines visual pattern recognition is drastically different from what determines auditory pattern recognition, particularly with regard to its temporal aspects [56-58]. Once again the MIDI control protocol is one of the only generalized computer protocols from which research into real-time interaction between temporal changes in image and audio data might proceed. Our approach is to create software to perform the real-time recognition and mapping of visual objects into the control parameters of complex acoustic behaviors.


IV. Program Dissemination and Evaluation


In addition to its research focus, the Theater of Pattern Formation will provide, to both lay audiences and the scientific community generally, an educational entry into the world of pattern formation, complexity, and chaos, one that celebrates recent major innovations in our understanding of the origins of order and disorder in nature and in mathematics. While the project will consist of a variety of mathematically driven visualizations-sonifications, these will go beyond mere didactic expositions of scientific principles. Our agenda is to create an integrated perceptual environment that focuses on leading-edge scientific results and builds an experiential understanding of fundamental principles of natural pattern formation and their mathematical descriptions.


The ultimate goal is to produce live performances of the Theater of Pattern Formation in dome theaters. To the best of our knowledge, these performances will be a first for such large-screen theaters, both in terms of aesthetic and scientific content and as live interactive events in such venues. We will also produce fixed program material on single DVDs for classroom use and multichannel DVDs for distribution to all-digital dome theaters. When appropriate these visual and auditory materials will be placed in context with verbal descriptions and other educational resources that explain and further elaborate the underlying scientific concepts.


A model for this kind of educational experience is Turbulent Landscapes, The Forces that Shape Our World [15]. (See Turbulent Landscapes---A Dialogue and Turbulent Landscapes: The Forces that Shape Our World.) One of the PIs was principal scientific adviser to this NSF-sponsored exhibition. This show allowed visitors to the San Francisco Exploratorium to interact with 30 different kinds of pattern formation and deterministic chaos in a hands-on manner and through an audio tour guide. Each of the 30 exhibits was designed and built by an artist, with the overall content selected to reflect one or another recent concept or result in nonlinear mathematics and physics. The show was one of the Exploratorium's most popular. In 1996 it was estimated to have brought in 300,000 visitors during its six months at the Exploratorium. The show has been on constant national and international tour since then. It is currently showing at the British Museum. Many valuable lessons were learned in developing this project. Aside from the technical issues of how to display and engage the public in complex systems, the most important was simply the exhibition's overwhelming popularity, which demonstrated the public's strong desire to learn about large-scale complex systems and the patterns that organize them.


To develop successful program material, we will consider a number of related issues, including sound localization, visual perception of novel objects and structures in space and in time, a subject's tendency to perceive patterns when there are none and to miss them when there, and so on.


Another important aspect of the Theater of Pattern Formation will be, as program developers, our own discovery of novel and interesting patterns within the vast search spaces of pattern forming systems. Similarly, there is the question of whether or not viewers will detect the patterns and structures when presented in an immersive environment. The question common to both program development and audience experience is a fundamental one: What is a pattern and can we quantitatively measure the degree of structure in temporal sequences of images and sounds? To address these questions we will adapt recent work on quantitative measures of structural complexity and randomness [35-37]). We will use the measures developed there to aid in our automatically exploring and discovering engaging patterns and behaviors. These measures will also be used to monitor audience reaction to the programs we develop. Again, we believe these efforts to determine the impact of the program material will introduce novel approaches to evaluating educational impact.

Bibliography: Coming.

Sound samples.


Performances & presentations:

    The Theater of Pattern Formation

        1 December 2003, CUNY Graduate Center

    The Making Of `The Theater of Pattern Formation'

        1 December 2003, Working Session, CUNY Graduate Center

    Apollo by Day, Dionysus by Night, Theme Camp

        Burning Man, August 2004

    CEAIT Festival, Center for Experiments in Art, Information and Technology

        California Institute of the Arts

        Roy and Edna Disney/CalArts Theater, Los Angeles, 28 January 2006

    Arts Technology Center, University of New Mexico

    ToPF In Sensory Immersive Environments

    Chaos All The Way Down:

        YouTube video snippet

        “Chaos All The Way Down”, Leonardo (2009) in press. Download PDF.

        Preshow to Collapse (sudden falling down)

        Premiered Mondavi Center Studio Theatre, 25-28 October & 1-4 November 2007

James P. Crutchfield & David D. Dunn

Art & Science Laboratory