Rise of AI Conference 2019 · Berlin

Symphonia Universalis (#SU): a journey into sound

Marc Engenhart · Head at Engenhart - Bureau for design

At Rise of AI 2019 in Berlin, Marc Engenhart, head of design studio Engenhart, presents Symphonia Universalis (#SU), a man-machine art installation his six-to-eight-person team has spent nearly a year building for the Beethoven anniversary in Bonn in 2020. He traces the idea to visiting Beethoven's birthplace and reading Beethoven's character as visionary, humanist and boundary-expanding, which he treats as a template for an artificial persona. To build it, his team digitized Beethoven's full catalogue (Opus 1 through 138 plus roughly 90 unlabeled works) into MIDI data, engineered features describing his persona from letters and research, consulted a criminal profiler to draft a psychological profile translated into algorithms, modeled mood and crisis states such as his breakup, and fed all of it into a recurrent neural network alongside live human input he calls a "heartbeat." The result, installed as a black cube people can visit, touch and feel vibrate, continuously composes new music from public contributions worldwide, intended to run without updates or a defined end date.

Key takeaways

  • Engenhart's team digitized Beethoven's full catalogue, Opus 1 through 138 plus about 90 unlabeled works, into MIDI data to train the system.
  • A criminal profiler helped draft a psychological profile of Beethoven that Engenhart translated into algorithmic features.
  • The system combines a trained Beethoven 'persona' with a live human 'heartbeat' input, since Engenhart says intuition cannot yet be modeled by machines.
  • The installation, a black cube first set up in Bonn in 2020, is designed to run with no planned end date and no software updates.
  • Engenhart chose Beethoven specifically for the 2020 anniversary of his 250th birthday rather than to clone or improve on his music.

Detailed notes

Origins: Beethoven's birthplace and the idea for #SU

Engenhart traces the project to a visit to Beethoven's family home in Bonn, where a sculpture and an architecturally distinct ceiling led him to think about consciousness and to see Beethoven, described in his research as a revolutionary, humanist, forward thinker, visionary and cosmopolitan nature lover, as resembling an artificial intelligence system in his own time. He decided to build a model combining coding, design and public participation for science, contemporary art, and the future of music.

Building a Beethoven persona from data

Engenhart says his first approach, to write an algorithm that composes like Beethoven, was a mistake, since similar AI-music projects already exist. Instead his team digitized Beethoven's entire written catalogue, Opus 1 through Opus 138 plus around 90 unlabeled compositions, into MIDI data, cleaning the handwritten source material (Partitur) to preserve musical expression and tempo accurately. On top of this MIDI dataset, they engineered features for a recurrent neural network representing Beethoven's persona, drawn from letters and research; consulted a criminal profiler to write a psychological profile that Engenhart translated into algorithms; and modeled situational and emotional states, including weather and an existential crisis modeled on Beethoven's breakup.

System design: persona, heartbeat and evolution

Engenhart describes the resulting system as combining a machine-learning "persona" layer trained on Beethoven's data and character with a human "heartbeat": live MIDI input from human intuition, since he says intuition itself cannot yet be modeled by machines. A public perception layer lets visitors sense the installation directly, and an "evolution" layer, the recurrent neural network, continuously collects intuitive musical ideas contributed by musicians worldwide via a website and recomposes them into new music.

The installation: a black cube in Bonn

The physical installation is a black cube, first set up in a museum in Bonn in 2020, combining visual, audio and touch (vibration) presentation. Engenhart plans to invite contributors from 150 nationalities to submit musical ideas online, which the system uses to compose live music continuously, with no planned end date or software updates; if the system ever produces only white noise, he treats that as a valid outcome of the research rather than a failure.

Q&A highlights

Asked why he chose Beethoven rather than another composer, Engenhart says Beethoven fit the 2020 anniversary (250 years since his birth) and Bonn's setting, and that the goal was never to clone Beethoven but to use him as a well-documented, recognizable teacher for the model. Asked whether digital, algorithmic composition suits classical music, he says the point is not to improve on Beethoven but to recreate the conditions for a musical revolution, and that the installation includes a touch interface precisely because pure listening is not the only intended experience. Asked about future engineering, he says the system also uses motion tracking, so that a crowded room of visitors can make it behave as "anxious," which the team will study by comparing recorded situations, such as many visitors present alongside intuition input arriving live from abroad.

Quotes

I want to create a model for science, for contemporary art, for the public to perceive, to get their hands on and understand, and for the future of music.

Marc Engenhart

The SU is a live composing intelligent machine.

Marc Engenhart

Every day, every second, worldwide, people can contribute to new music, that's the SU.

Marc Engenhart

Questions answered in this talk

What is Symphonia Universalis (#SU)?

According to Marc Engenhart at Rise of AI 2019, #SU is a man-machine art installation, a recurrent neural network trained on Beethoven's full musical catalogue and persona, combined with live human input, that continuously composes new music from contributions submitted by the public worldwide, first installed in Bonn in 2020.

Why did Marc Engenhart base his AI music project on Beethoven specifically?

Marc Engenhart said at Rise of AI 2019 that he chose Beethoven because the installation launched for the 250th anniversary of Beethoven's birth in Bonn, and because Beethoven's well-documented life and letters provided rich data, not because the goal was to clone or improve his music.

Will the #SU system receive software updates after launch?

According to Marc Engenhart at Rise of AI 2019, #SU is designed to run indefinitely with no planned end date and no updates, since it is a research and art installation meant to observe what happens over time, not a product meant to keep improving.

Mentioned

Symphonia Universalis (#SU), recurrent neural network, MIDI data, human-machine interface, feature engineering, Beethoven persona modeling

Keep exploring

More from Rise of AI Conference 2019

More on AGI & Future of Intelligence

Transcript

Auto-generated transcript, not edited. · 3,219 words

Show the full transcript

okay we Justin we have 36 seconds left so I will introduce you to sue good afternoon ladies and gentlemen so as I can see we're a few but it's not the point today I want to introduce you to the zoo the zoo is called symphonia universities who are just two little letters but I think it's interesting to have like a word or a description expression which stands for what we have done what I have done my name is mark in heart I'm human I'm real but in my early days I experienced code as a second language I'm German so my mother tongue is German I speak a little English but code was also like a and I say a language I started to talk or cold early when you engineer something when you design something your designer so I think every person in this room is somehow as well designer because we humans we create things artist is a less I do petition and the opposite of the designer but I'm also a musician futurist

and thinker but it's human that's a big list I know but it's just human I want to thank Fabien and Veronica Veronica Fabian beside for inviting me here I think it's a beautiful conference I'm here the third time so thank you for that um let's start how it all started now you say what is the zoo yes I'm funny Rizal is and you heard a little tune in there at the beginning it was Beethoven yeah recognizer it's just a big brand as a brand it's a brilliant musician it what Beethoven but something was different as he heard it some tones were different so we're here the rise over a conference and my early days as you can see me here three years my mother's house or my father's or my upon just a parents house getting in touch with the first interface I have experience creating music was my guitar I still have it I still play it and it's a very of course analog interface there's nothing to do with any engineering or scientific

computing our creating tones it was just my first interface but it has had a very very intensive impression on my life as designer and so on when we skip some years from the first picture to it's like three years before I once visited the city of Bonn in Germany because I was interested in the city of course and secondly I was interested in their less I human called Beethoven because who is and still is our brilliant creator of music and I went in the house of Beethoven as parents and family and I saw a chamber like this and there was just this sculpture and I've just read the designs I was written it was the birthplace of Beethoven the place where this human still started to get in touch with our analog world and I just as I'm design of course I'm a visual guy or see things like precisely and in detail I just saw that this guy is looking with the sculpture of course the artist it made the sculpture put it there

there's nothing in his room just him just like is her element and he's looking to ceiling and why I'm showing you this it was the moment where I thought hmm okay visionary maybe just this visionary maybe just one more thing maybe you can recognize that these walls which are authentic historic or just very very neutral the balls you can see here sorry here but the ceiling is different it's a different model in the environment it's made up there's in the architecture it's different and I of course walk through this earth searching for places which are different this is a place at the Museum in Germany where you can go in in somehow an environment stand in and see you in different levels different shapes different atmosphere is different whatever and the combination of both brought me to something which I said hmm maybe it's consciousness it thought maybe it's like Beethoven and then of course visited in born a very special place as well it's the place where Beethoven was just like baptised and become

a person of our world for introduced own my introduction is also like this element this is a sculpture you can also see like in the area of the Beethoven house and what I experienced is a visual interpretation of a model of Beethoven which somehow has special hair and the say like this and of course you can say yeah a special hair maybe you it was his head at that time but for me it was a model of visualization of his conscious and when I do the design and adapt this sculpture to a model which is just like more precise it's a black box imagine this is a Beethoven avatar like this imagine it's a box which is man-machine interface creating evolutionary new music that's the zoo when Beethoven was described at his time and in our times he was a revolutionize his time given he was someone who expanded the boundaries of music his time given he was humanist and forward thinker his time given he was a visionary and intellectual maverick reaching out creating

new music cosmopolitan the nature lover and a radical role model I found this in Beethoven research and as you can see we're here the Rise of AI conference it's clear maybe he's something like an artificial intelligence system so I thought okay that's just how it is I have to start and create a model like that because I can compare different fields which are coding designing presentation participation some sort of the right time given because we are in the middle of their digital revolution and I would thought about I want to create a model for science for contemporary art because I'm artists for the public to perceive or to get their hands on and understand and for the future of music and as you can see not that easy so we're here at the conference and I give you a short inside the first time ever how we build that I'm not building that alone I have a team for six to six to eight people working on this almost a year now and now we

start to get better okay the first mistake I did or made was I wrote down how to build an algorithm that composes music like Beethoven I said okay maybe you know there are a lot of different companies on this world also doing this creating AI music recomposing music to find new music or maybe composing music to a visual image or whatever and I said this will be my goal how to build an algorithm that behaves like Beethoven and regionals in the future okay let's start the first step is of course is everybody doers find data to the data mining precise your selection and make somehow sure that the data is perfect what is the data of Beethoven that's it everything hit every composition he ever created his data in the world we have in a digital when you see that compress this is Opus one to opus 138 in addition 90 additional compositions which were not labeled like opus one or label focus 138 this composition you can see here in typographic compressed of course

are written compositions but we have today the ability to digitalize that into MIDI data MIDI data you can play by any sequence so you have at home and just play Beethoven it's quite good the quality is not so good so we have to deal with a lot of like it the pop in German is called party tour while you see the handwritten data and then you just like transform it to the digital world and you have to make sure that is perfect because of course like the expressions tombola tempe musical expressions i'll need to be perfect so to have embossed on they say best quality on data this is just a view on how it looks like up is one nine ten and so on and this is how it looks like when you see it on a step sequencer interface this is like a music sequencer in the middle you can see like the numbers this is me t cold well justjust configures old gym musical elements and the second step was extended feature

engineering for the recurrent neural network we were setting up to compose music to have some expressions on that it's very important I say I need to have a ratio of Beethoven because I wanted to create a model like Beethoven ratio is for me my MIDI data set MIDI music the persona which is characteristics of Beethoven we try to translate out of books and research and scientific elements like letters from him writing to his partner or family or whatever and we'd estimate algorithms out of like characteristics of Beethoven next is animal I just called because I know criminal law jeez I don't know the words in English it's like a profiler a man from from the year the police department I called him and said hey can you do a profile on Beethoven in the certain or a certain situation write it down and I will translate it to an algorithm you said yes of course but what is algorithm I said yeah no worries I need just the text from you and I will do

the rest another feature is living in bioethics for this model and see if you have a nature situation imagine Beethoven composing raining some raining thunderstorm what it's different so we also create like algorithms which describe the atmosphere and one very important element is an existential crisis beethoven got dumped so when we do a digital version of that I'll try we don't want to clone Beethoven or do the same just setting up an environment which is almost the same like Beethoven his time given X times of crisis we have the human part on this model which is impulses for creativity because they all know intuition is not possible by human by machine it's not possible at that time now we don't have intuition by machine as prediction but humans can do the intuition we have in this model a heartbeat which is life MIDI input from human intuition we have a public perception because it's a human machine interface it's an art installation where you women can go and have a perception sensing what is doing

what is the zoo and we have evolution which is is a recurrent neural network collecting intuitive ideas by musicians all over the world I will give you an overview right next and the system recomposes every musical idea with a deep learned si identity he has from his first level to have an overview machine learning part human swamp out put on features together create an artificial intelligence system as a recurrent neural network and that's it let's assume in combination we manage or it's possible to have an research and experiment how human machine intuition could work within the fields of creating sound and music bring all together on the left side every computer worldwide will get an interface by a website to just play in some notes to the zoo the zoo will learn from it actively well know how Beethoven would have composed certain elements out of this and now of course do new music the zoo is a live composing intelligent machine and because it's human-machine study we have on the right side because it's

set up in a sec a scene I can see a black cube and people can visit so we have a visual presentation we have an audio presentation and a touch because su translates what it's compose what the composition into there's like vibration the first initiation of the su will be set up where I invite 100 50 nationalities around the world to have our variety on human music ideas well being 2020 them how we will will look like how we will sound like will ask maybe that's how it will look like we build the black cube put it in a museum that's it it's just a black box maybe you know this expression from our world and the artificial it can vibrate but it will not it's not the speaker it's just a sculpture people can go in which with we are to see something like this this is the artificial world where you have compositions at certain date and time people sent in like intuition elements you can go in you can go out and

not a point of view it would sound like the piano paid often used for composition because we can do the new composition by a MIDI and play it live at a certain time the system composes go back two years to have this composition at that certain point of time and when it's at the end when the system decides to become a white noise sound nothing to hear for humans it's like these because the zoo is also like a research element a scientific installation you can experience that by yourself 2020 in Bonn where Beethoven was born because we installed it there and one day we'll push the button to initiate the 150 culture music ideas and then every day every second worldwide people can contribute to new music that's the suit thank you very much for your attention I know I know it was a lot but I have 20 minutes we have still time for questions we already music excuse me for the music yep that's right correct so why not other composers is it

difficult once your system is already trained for Beethoven music is it easier to train it for other composers the Susan machine human machine intelligent installation it's an art installation I had to decide to use Beethoven as a teacher because we will set it off in 2021 it's the birth I say the University 250 years of Beethoven we will be his age 215 and that's why we selected precisely Beethoven himself calls could train with Marla I don't know Britney Spears whatever it's not but in this the zoo is a symphony or neuroscientist from Beethoven we're now not clone Beethoven it's just because I have to make the decision ok let's do it like this because it fits it's an art intern installation perf to the environment of bond people will recognize it better as a public installation to have talks on the eye and so on also like an impulse to the public because they can touch and Beethoven is a good brand so I'd fit well we have a data set which is very good

for music as well welcome but as a provocative question is it really good to use classical music on digital devices so to say because I don't hear the difference very good but I know that people would never listen to Beethoven digitally yeah maybe I can yes it's good that you asked that you described of course the art installation was focused on Beethoven and if we say okay let's pick something out I don't know if you know the national you need any rock band when we do the same with that kind of music people will of course reacted to do it differently and classical music is precise but India on the other side also very complex because of a lot of instruments bring it and so I don't know what what's easier to find let's see what happens that's most important because the human intuition combined with a train system and in one model there's a main goal to check what happens in about five years because one more thing it will be set up with

no time limitation we will install it in 2020 and it's of course I gonna say I'm just living no eternally but how they say and it was mind just like one line and the concept was it has to be eternal and there will be no updates ok yeah yes that's a good point because you know every product you do an applied AI just has a function to have something or to build something better to have a better prediction on whatever it is or to make a like a better product like this this zoom is more research art installation because you can see what happens it's not about making great your Beethoven it's not about that it's about creating an environment which could be almost the same as at that time beta1 was given to have something like a musical revolution the zoo tries to find an unscientific or artificial habitat where something can happen which is music evolution if it's white noise and if the zoo decides to compose in frequencies you never be able

to hear that is human it's alright because we can see it in data you can see we cannot just like perceive or it's not like here Bolden but as we have also like a vibration interface because the sculpture in the museum or at a certain pop place is also like touchable because Beethoven was dumped people will also have an sense of how it will sound like I think we don't need to mic no like ask question that's great thank you you mean for them for the future engineering or for the perception at the Museum or just the system yeah we have also like I said it's for me it's the persona at our level where we destined eight out of out of patrons history and research like very important takes a touch points where the music changed somehow because of public reaction know when we run where we read a letter he describes itself like a piano concert in Bonn to a friend we label the certain sentences describing important change chains in the system

and this we translate into algorithms and build it all together to make they say environment to decide as well as well because we have life from life trekking or motion tracking sensors where you say okay the sewers all of course as a composition unit you can say it's a human machine interface but it's it's composing music and when there are takes like 200 people standing like this in the museum and want to touch it it will be anxious yes and it's not Beethoven which will be anxious or who it's assumed saying oh my god is a little complicated right now I wanna I don't want to compose it's because it's it's important and as we archive all these situations like record we can by research of course having some certain points say what happens at that certain time 200 people in the room intuition from Brazil via Internet with a salsa element like Popham

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.