audiovisual interactivity

Andreas Weixler 2001/02/03

 computer music and interactive media arts
midi, audio and video with object oriented programming in Max/msp/Nato/GEM/Jitter

deutsch hier
in general locations of the Workshop
set up (net work)

about my art work
Max/msp and object oriented programming

content in detail I) combination of data streams of audio, midi and video within Max/msp/Nato/GEM/Jitter

II) algorithmic composition, computer music and interactivity

a) synchronizing of traditional score writing and computer timing
b) generating structures in Max
c) midi controlling
III) interactive communication in media arts

IV) audio visual Interactivity

possibilities of interaction of sound and vision:
a) sound -> sound
b) sound -> image
c) image -> sound
d) image -> image
e) mixed media

short curriculum vitae
locations of the workshop

Bergische Universität Wuppertal, workshop audiovisual interactivity and Max/Msp/Jitter
Lehrstuhl für Visuelle Kommunikation (institut for visual communication), Prof. Hans Günter Schmitz, organized by Wolfram Zwanziger, 12th-14th July 2004
Dongeui University, Pusan, South-Corea, From Minimal to Audiovisual, lecture by Andreas Weixler and Se-Lien Chuang for Pusan Electronic Music Association, November the 13th of 2003
TNCA Applied Music Department, Tainan National College of the Arts, Taiwan, 21. + 22. April 2003
Institut für Neue Musik, Staatliche Hochschule für Musik Freiburg, Germany, 7.-9. Februar 2003
Weinberg Castle, education centre of Upper Austria, lecture on 7th of May 2002, Event of the Bruckner-Conservatory
Institute of Electronic Music, University of Arts in Graz, Austria, lecture and workshop on 5th and 10th of December 2001, with Winfried Ritsch and Johannes Zmölnig, in German
National Taiwan University of Arts,  Taipeh 13th. of November 2001, with Se-Lien Chuang, in Chinese and English.
Hochschule für Grafik und Buchkunst Leipzig, Germany, 16th - 17th of July 2001, in German
Nagoya City University - School of Design and Architecture, Japan, 19th January 2001, with  Se-Lien Chuang and Mikako Mizuno in Japanese and English.


The schedule of the workshop is very flexible from a lecture of 2-4 hours up to a practical workshop over several days
It is considered to be useful for beginners aswell as for advanced students of composition and media arts.
set up
I will bring a Macintosh Powerbook and Software like Max/msp and Nato/GEM and some other equipment.
equipment at the location of the workshop:
  stereo audio equipment (2 speakers and a stereo amplifier)
video- and data projector
DVD video player PAL/NTSC
midi keyboards (at least one)
Macintosh computer (for student training)
Max/msp/Jitter (can be installed temporarily on several machines)

extras (optional)
  ethernet-hub for local AppleTalk Network
overhead projector
Nato modular
USB- or DV cameras

about my art work

as a composer I am not only using software in a way they obviously offer. I am always intrested in new combinations and maybe alternative use of software, which means using software differently from their first intention (like using a vacuum cleaner as a sound source).
I am using many different software for creating computer music and media art. Also in teaching I like to encourage my students to go their own way in using software.

The desire of creating my own technical tools was fullfilled by using object oriented languages like Max for ISPW on NeXT computer and nowadays Max/msp on Macintosh.
Of course in my artwork I also use standard software for hard disk recording (SoundDesigner, ProTools a.o.), writing CDs, creating Midi-Sequences and score printouts (like Cubase) and so on and last but not least I use pencil and paper...

In computer music I was creating pieces in different fields.
All my compositions somehow belong together as a work in progress, so you can see a comparison in using the available technology as well as in the structure of the composition itself. Some pieces quote each other, and sometimes I use the recordings of an instumental piece of mine to create electronic sound in the studio for the next one.
Now I would like to show you some examples of these catagories:

1) midi controlled devices
 - sampler & synthesizer
2) algorithmic composition
 - score
 - structure of composition
3) sound processing
 - studio production
 - multi channeling
 - realtime
4) interactivity
 -interactive communication
 -audio reacts to audio
5) audio visual
 -  music video/audio-visual composition
 - interactivity
  -sound processes image
  - image processes sound
  -image processes image

although these categories seem to be clear, very often a piece of computer music is based in more than one catagory.

for example
Der letzte Wassermann (by Andreas Weixler)
includes many of these categories: first it is a live instrument with tape, second it is multi channeling (4 channels), third it is granular synthesis as sound processing of the recording of bowed piano, marimba and a string trio.

or  Trinity (computer animation by Se-Lien Chuang, computer music by Andreas Weixler)
first it is an audio visual and a music video. The visual is a computer animation, the music is sound processing of comtemporary instrumental sounds.

Max generally

Max is a programming language, specially designed for media artist and musicians. It is object oriented, this means its programming is graphical. You have many objects, which are representing programming routines. By connecting these objects you design the flow of data and its processings. This is called a patch and finally is your own program written in Max.
MIDI is a digital language for electronic musical instruments like synthesizer, sampler, computer and some special devices.
MSP is a collection of Max objects for audio processing.
NATO/GEM are very new objects for quciktime video and OpenGL graphics
JITTER is a set of video, matrix, and 3D graphics objects for the Max graphical programming environment
Max/msp offers all kind of interaction of midi, audio and video objects. For example the audio can be controlled and affected by MIDI data as well as by video or audio itself.
Many artists made their patches as libaries to re-use it in other programming or share it with others through the internet.

more information on

content in detail I) combination of data streams of audio, midi and video within Max/msp with Nato/GEM or with Jitter

II) algorithmic composition, computer music and interactivity

a) synchronizing of traditional score writing and computer timing
b) generating structures in Max
c) midi controlling
III) interactive communication in media arts

IV) audio visual Interactivity

possibilities of interaction of sound and vision:
a) sound -> sound
b) sound -> image
c) image -> sound
d) image -> image
e) mixed media

I) combination of data streams of audio, midi and video within Max/msp with Nato/GEM or with Jitter
What I consider special about the graphical programming environment is that you can combine all kinds of data and "see" how the data flows. All data of midi, audio and video can function as input and also as output.

The Max object orientated programming language as I want to introduce to you consist of

a) Max, the basic programm.
    It has mathematical funktions and was designed for midi applications.
b) msp, the object library for audio processing.
c) Nato, the object library for video processing based on Quicktime.
d) GEM, Graphics Environment for Multimedia. It `s objects can create graphics based on OpenGL.
e) Jitter, quicktime video and OpenGL animation
functions of Max/msp/Nato/GEM

1) algorithmic

as in all programming languages you can do complex mathematical calculations
the midi standard (MIDI = musical instruments digital inteface) enables the system to communicate with electronic musical instruments and can be also used for controlling several external devices.
3) DSP
the objects of msp is opening the world of DSP (Digital Sound Processing) to Max, e.g. sampling, FFT, analysis, resynthesis and others
the LCD object in Max allows simple graphic skills made out of circles, rectangles and lines and use those interactively for input and output.
A portation of GEM by Shigeyuki Hirai (Kyoto) from PD for Linux/IBM computer to the Macintosh-platform opens the possibilities of OpenGL.
There are also several third party externals for displaying graphics and 3D, for example max3d by Shuichi Chino.
the Nato modules give full access to all Quicktime video functions. Again it can provide input by cameras and output on a screen and/or projector.
Also for visuals some other third party objects exist, which have the advantage to be free of charge like o- object by Ichi, mov by Shuichi Chino, videoin by Erik Singer
very well documentated object about quicktime video and OpenGL animation,
organisation and manipulation of matrices, and much more..

all these modules can be combined and mapped to each other and depend on each other algorithmically.
I call this the script, which is considered to be part of the artistic work. To make libraries and re-use them makes programming more comfortable by the time and creates your own personal way. For this it is necessary to think about a good architecture and naming.

example WAON


Waon live at Experimental Intermedia, New York, USA

interactive audio visual performance for two instruments by Andreas Weixler.

description: based on the Japanese spirit of harmony the computer senses harmony between the players and acts accordingly in generating audio and visual.

Studio: Sound Design Lab/Nagoya City University, School of Design and Architecture, Japan
premiere: 15.12.2000 Experimental Intermedia, New York, USA
selected for ISEA 02, Symposium of Electronic Art, in Nagoya, Japan

technique: Max/msp (fiddle~ objekt for analysis of pitch and dynamic, generating algortihmic structures, LCD object for displaying graphics)

II) algorithmic composition, computer music and interactivity

a) synchronizing of traditional score writing and computer timing
b) generating structures in Max
c) midi controlling

a) synchronizing of traditional score writing and computer timing

displaying timing as rhythm
computers "think" in time of milli seconds, but musicians like to play with traditional score printing, which means they divide a pulse of time into proportions, like quavers, triplets etc.
If you like to write down a computer generated composition into tradtional score printing, you need a comparable timing.

It is very easy if you take tempo of 60 (means 60 beats per minute). In other words this means one stroke per second.
To divide it is now easy for computer timing as well as for the score:

a crotchet will be divded as follows:

the logical division into semiquavers:

division into a quintelet:
division into triplets of semiquavers
(numbers are approximated)

If you overlay them all you get a resolution of one second as follows:

At the beginning and at the end it is less dense as in the middle part, and also you can see it is symmetrical.

! So what is this for ? This means an absolute timing of a computer calculation is transformed into a proportion of time. The musiscian, who play this algorithmically generated part, does not have to play absolute time for example of  2 seconds and 34 milli seconds (which appears to be impossible for a human), but the musical correct proportion of 2 crotchets and the third of a triplet of semiquavers, which is the next closest proportion.
Of courses this means there is some approximation in the relations shown above. By the choice of crotchets, semiquavers, quintlets and triplets of semiquavers as the possible divisions of a time base of 60 beats per minute, I choose relations which make sense to both, the computer calculated accurate time and the musical rhythm a musicians can play accordingly.

In other cases one has to be forced into the timing of the other, either the musician into absolute timing of the computer by playing with a click track or the computer has to be forced into the system of a proportion of time, i.e. rhythm, via score following for adjusting the time basis of his list of notes accordingly to the human interpretation of the score.

b) generating structures in Max

in der Zeit des Blühens (A.Weixler 2000)
algorithmic composition for traditional Chinese ensemble

instruments: dongxiao, sheng, pipa, guhzeng, daruan, yanzin, erhu and percussion
duration: 7 minutes
ensemble: China Found Music Workshop - Taipeh/Taiwan
premiere: 05.04.2000 Konzerthaus Wien, Hörgänge

this score shows the variaton of the density of the rhythms very clearly .
The structures of timing are generated by a Max patch, the data is exported as a midi file and transformed into a score by importing it into Cubase. The score then has been re-interpretated by hand writting to make sure a musical appearance of the data conversion.

the Max patch "appear" for generating the structures of the rhythms and for instrumentation

c) midi controlling

Soundbarrier, live instruments will be acoustically displayed over multichannel loudspeaker system.
Max-patch: LAP


HyperControl is a Max patch for controlling the software HyperPrism for sound processing.
In that way the sound processing can be controlled algorithmically.

Jade for orchestra on tape ( A.Weixler 1996)
description: computer music, algorithmic controlled sound processing of a contemporary orchestra
Dauer: 5:54
Hardware: PowerMac 7200/90, Macintosh Quadra 650 div Peripheriegeräte
Software: ProTools 16Track, TDM Plugins: HTD, BRN, GRM, Hyperprism 1.5.2, GRM Tools 1.55, SoundDesigner 2.8, Alchemy 3.0, HyperControl, Max 3.0, Soundhack 0.8
Studios: University of East Anglia , Norwich, BEAST Birmingham Electro Acoustic Sound Theatre, England

UA: Kunsthalle Stockerau 31.8.96
weitere Aufführungen:
WUK Wien 3-13.10.96 ProjektTheater Wien-New-York,
15.10.96 Theatro Austrian Sound Check/Steirischer Herbst ,
14.u.15.11.97 Projekttheater Eva Brenner-Minoriten Graz ,
26.6.1997 European Electroacoustic Music Festival in Brüssel ,
11.2.98 Electroacoustic Music Concert Montreal/Canada ,
15. 01.98 Konzert "Musik des 20. Jahrhunderts" Tiroler Landeskonservatoriums,
10.11.98 Sumida Triphony Hall-Tokio/Japan
13.03.1999 Lange Nacht der Neuen Klänge, Konzerthaus Wien
11.05.99 Lüneburg 21. internatinale Studienwoche zeitgenössischer Musik
15.12.2000 Experimental Intermedia, New York

III)  interactive communication in media arts

the cat and the mouse live in Lueneburg, Germany

the cat and the mouse (1998)       ca.12´00
interactive realtime computer music for electric guitar, piano, interactive communication and ISPW audio processing

concept and prepared piano: Se-Lien CHUANG
ISPW- Max: Wolfgang MUSIL (Lüneburg), Thomas MUSIL (Graz)
electric guitar & Opcode- Max: Andreas WEIXLER

premiere: at the 21st international week of contemporary Music, Lüneburg, Germany 12.05.99
other premieres: ESC Labor/Graz at the V: NM Festivals 18. 05.1999
duration: circa 12´00

 - - - - - - - in C  (1998)
realtime processing for electric guitar, piano, interactive communication and ISPW audio processing

Se-Lien Chuang, composition and piano
Andreas Weixler, Opcode Max programming and electric guitar
Wolfgang Musil, ISPW-Max programming
Studios: Opcode-Max programming at Atelier Avant,
ISPW-Max programming Institute for Elektroacoustic Vienna

premiere: at the Alte Schmiede/Kunstverein Wien, Festivals Elektronischer Frühling ´98 Westöstlicher Divan:  Taiwan in Wien - Wien in Taiwan 29/04/1998

The improvised melody in this piece is basically displayed with 7 church-modes.
The cat and the mouse is computer aided composition in which freezing, modulation, filtering and granular synthesis are used to complete the sound processing.
The reason for the cat and the mouse to be challenging and gripping is because it's a interactive real time game between guitar, piano and the computer.
When the musicians are playing, the computer records the sounds in real time, processes the recording sounds, and stores it up. Then the musicians release the stored sounds through the computer by improvising on instruments. Involvement, intertwining and interlocking flow into this performing stage.

Communication in varied layers

1. The improvisation is an transparent communication between the musicians (guitar/piano)

2. The stream of information in the main computer by sound processing (ISPW)

3. Concealed communication between the musicians, the computer and among surroundings (Laptops)
The Laptops send off the digital operation commands from the interpreters on the main computer (ISPW) and carry out the exchange of the information amongst them.

premiere. Festivals Elektronischer Frühling, Vienna 1998, 11.05.1999, 21.International Studyweek for contemporary Music Lüneburg/Germany
18. 05.1999, ESC im labor, Graz/Austria, in series of V: NM Festivals 1999

IV)  audio visual Interactivity

possibilities of interactivity beween sound and visuals:
a) Sound -> Sound, b) Sound -> Image, c) Image -> Sound, d) Image -> Image

a) Sound -> Sound

a1) the dynamic of an instrument controlls a sampler

O.zon (Andreas Weixler & Se-Lien Chuang 1997)

description: interactive composition for piano solo in multi channeling.

premieres: 2.8.97 14. Künstlerbegegnung St. Lambrecht;
3.11.97 Institut für Elektronische Musik,Graz, 12.05.99 Lüneburg 21. international festival of contemporary music

Hardware: PowerMac 7200/90, Ensoniq EPS 16+, Tx802
Software: Opcode-MAX
Se-Lien Chuang, pian; Andreas Weixler, computer system

a2) interactive audio & realtime processing

a crush on You live at Experimental Intermedia, New York, USA

a crush on You
interactive improvisation for the Chinese dulcimer Yan-zin and realtime audio processing

concept and Max/msp Programming: Andreas Weixler
Yan-Zin: Se-Lien Chuang
Studio: SUSS Sheffield University Sound Studio und Atelier Avant1999,
Sound Lab Nagoya City University 2000

premiere 25.09.1999 Weingut Gasser, Styria, Austria
other premiers: Festival Elektrokomplex 12.12.1999 at Rhiz Wien,
Stadtapotheke Trofaiach (Austria) 22.09.2000,
Experimental Intermedia New York 15.12.2000,
Sound Lab Nagoya City University, Japan  19.01.2001

b) Sound -> Image

 interactive audio visual

The Story of  H e i k e live in Nagoya, Japan

The Story of  H e i k e (Japanese traditional narrative story)
interactive audiovisual performance

the biwa player tells the traditional Japanese story of Heike.
The computer cotnrolled choice of picture interactively accompanies the play of the biwa and the singing voice of the narrator.

        Kagusho Kitagawa,  biwa
        Mikako Mizuno, art concept
        Se-Lien Chuang,  visuals & realtime arrangement
        Andreas Weixler, interactive programming

Premiere:  Nagoya City University - School Design and Architecture, Japan,
Friday 16th of February 2001, Sound Lab

technique: Max/msp (fiddle~ object for analysis of pitch and dynamic, generating algorithmic structures), and X-Pose visual sampler.

living cave live in Laval, France
© Chuang Se-Lien

living cave
interactive audio-visual installation by Se-Lien Chuang, Max/msp programming by Andreas Weixler

description: The room reacts to the audience in algorithmic choice of pictures and their transformation and processes realtime audio within an electroacoustic composition.

Studio: Sound Design Lab/Nagoya City University, School of Design and Architecture, Japan
premiere: 19.01.2001 Nagoya, Sound Design Lab/Nagoya City University, Japan
further premieres 16.-2o th of May 2001 at  Laval Virtual and VRIC (Virtual International International Conference), France.

technique: Max/msp (fiddle~ object for analysis of pitch and dynamic, generating algortihmic structures), and X-Pose visual sampler.


 the movement and speed of a video by Se-Lien Chuang is controlled by pitches. Soundstudio of Nagoya City University, School of Design and Architecture.

c) Image -> Sound
by video object tracking one can follow the movement of one or more objects in a video in the picture of a live camera.
For these colours will be defined as well as regions in the picture. These will then create a data flow of  predefined parameters.

Two examples, one with the Nato object 242.pupille and one with the software BigEye by STEIM in Amsterdam.

Ouroboros at Durham Art Gallery, Canada

Ouroboros (2000)
Video on the Genom project by Gertrude Moser-Wagner, interactive sound modulation by Andreas Weixler

For the realization in sound regarding the original concept of Gertrude Moser Wagner´s "Ouroboros" ( I used techniques of computer music:
I first used granular synthesis of the spoken word "Ouroboros," and then added the distorted resynthesis of a pitch tracking signal, which I calculated corresponding to the two
channels of the stereo granular process - alltogether as a realtime process. This meets the idea of the genome project to reorganize original (gen-, audio-) material.

In a version of audio visual interactivity the movment of the ROL6 worm is followed by video object tracking of an radioactive marker. The parameters generated by this are used for controlling the sound process.
produced at Atelier Avant Aweixler
Hardware: PowerMacintosh 7200, Software: Max/msp 3.5.9, Nato.

premiere  08. 09 2000 Durham Art Gallery, Canada
other presentations: Kentler International Drawing Space, New York. 7.10. 2000

Japanese origami paper is controlling the sound. Experiments with Big Eye object tracking. The generated midi data will be processed for sound and visuals.

d) Image -> Image
experiments: object tracking with Big Eye and Nato controls Quicktime videos, i.e. by the Mov object or by the Nato object

e) mixed media
in media arts of course this categories may be mixed.

e.g. living cave (Chuang-Weixler 2001), uses audio input and generates an ambience of visuals and surrounding sound out of this.

Very often Max/msp is used in combination of other software and hardware.
e.g. X-pose visual sampler, or BigEye object tracking, Hyperprism  sound processing and others.

living cave in Laval Virtual, Frankreich

short curriculum vitae

Andreas Weixler  

*19.2.1963 Graz (Stmk/A)

1983 University of Music and Technical University Graz: electrical engineering, sound engineering, from 1986 Graz University of Music and Dramatic Arts: composition (Andrzej Dobrowolski, Younghi Pagh- Paan, Beat Furrer), diploma 1995, Mag. art. (M.A. in arts) 1997;
2000-2001 special researcher in computer music and audiovisual media arts at Nagoya City University, School of Design and Architecture, Nagoya, Japan within the Japan Foundation Fellowship Programm.

performances in Europe,  Asia, North and South America. Numerous musical projects and groups in the field of jazz related music, music for contemporary theatre and  dance, contemporary composition and computer music, video and interactive media arts.

recent performances: Laval Virtual 01 (France), Experimental Intermedia 00 (NYC, USA), Artport 01 (Nagoya, Japan), ISEA International Symposium on Electronic Art (Nagoya, Japan)

1992 - 1999 Graz University of Arts, Institute for Electronic Music: teaching contract,
since 1997  at the Bruckner-conservatory  Linz: teacher for music and media technology

1992 to 1999 editor of a cycle of publications "Beiträge zur Elektronischen Musik" (contributions to  electronic music),
1992-1996, member of  the board of  the composers union "die andere saite", since 1996 member of the board of  the Austrian Society of Electroacoustic Music.

Se-Lien Chuang 

*13.8.1965 Chiayi City (Taiwan), since 1991 residence in Graz (Austria/EU);

1990 Bachelor of Arts, Tamkang University/Taiwan: German studies;
1996 diploma, University of Music and Dramatic Arts Graz: piano education-teaching license (Walter Groppenberger); since 1996 University of Music and Dramatic Arts Graz: composition (Beat Furrer), University of Music and Dramatic Arts Vienna: Electroacoustic Music (Tamas Ungvary, Wolfgang Musil); 1999 diploma with distinction, Bruckner-Conservatory Linz/Austria: Music and Media Technology  (Adelhard  Roidinger, Karlheinz Essl);
1999 scholarship of Werkstadt Graz for Silicon Graphics studio training (computer animation), Austria
2000-01 special research in computer music & audiovisual arts at Nagoya City University, Japan

video and media artist, pianist for classical and contemporay interactive music, composer for contemporary instrumental music,  computer music;
composer in residence 1996: SAMT (Studio for Advanced Music and Media Technology) Linz/A, UEA  (University of East Anglia) and BEAST (Birmingham Electro Acoustic Sound
Theatre) England, 1999:  SUSS (Sheffield University Sound Studio) England;

concerts & presentations: d.a.s. (die andere saite) 1997-2000, 1997-2000 and numerous musical projects and groups 1996-2000 in Austria, Germany, Belgium, Netherland, England, France, Hungary,USA, Russia, Brazil, China and Japan;

recent performances: ISEA 02 (International Symposium on Electronic Art, Nagoya, Japan), SICMF 02 (Seoul International Computer Music Festival), Laval Virtual 01 (France), Experimental Intermedia 00 (NYC, USA), Artport 01 (Nagoya, Japan)

created 10.11.2k1, update 19.09.2004 Brussels
back to Atelier Avant main page