“What Would Music Look Like?” An AI-Driven Leap From Malta to Berlin

When sound becomes sight, music finds a new language. Maltese composer and producer André Tabone has turned this idea into reality with an AI-driven project that lets algorithms ‘imagine’ what music might look like. His work, developed during his studies in Berlin, blurs the line between performance, engineering, and visual art – transforming every note into motion.

During a performance in Berlin, André Tabone, a Maltese electronic composer and producer, presented a piece in which a pianist and a cellist played while AI-generated visuals based on the music being played accompanied them.

This performance was part of Tabone’s final-year project for his Bachelor of Arts in Electronic Music Production and Performance at Catalyst Institute for Creative Arts and Technology in Berlin. He previously completed a Master’s in Engineering at the University of Malta in 2020. In 2021, he returned to Berlin to work on a project combining music and AI visualisation, supported financially through the Malta Arts Scholarship Scheme.

‘I was intrigued by the physical forms that music can be imagined as beyond sound. I was curious about how it works and what the result would be, without the pressure of having a refined performance as the outcome,’ Tabone tells THINK.

Tabone’s project began as a final-year assignment. It evolved into a system that uses audio analysis and artificial intelligence to generate images based on a musical piece’s timbre, pitch, and dynamics.

‘I trained a model to turn 10-second snippets of sound into a word, which acted as the prompt for the AI image generator. Then I pushed that further,’ Tabone says. ‘I stopped it a bit earlier, where it was still an abstract representation called the token and switched out the image generator with a video generator.’

From Tokens to Texture

Tabone’s process involves using an audio model to convert sound into a representation, which is then used to guide a video model. Signal processing is applied to ensure the visuals reflect characteristics of the music.

‘If the sound has a high pitch, I would inject the word glass, or if it’s a warm timbre, I can inject the word warmth – it’s very much like prompting,’ Tabone says.

For one sequence, he staged the whole thing in a sparse, gallery-like prompt: “A person in a minimalistic art exhibit where the painting is contained.”

‘This is where I inserted what the AI thinks of the music – leaking outside of the canvas. I emphasised the loudness of the music by how much it leaks,’ he adds.

As the music becomes louder, the visuals become more complex, with thicker lines and more objects. When the music is quieter, the visuals become simpler. The result is an abstract visualisation that responds to the music.

Two Tracks, One Question

Tabone used two unreleased pieces, Postcard and Prologue, as the basis for his research. Postcard includes a recording of children in a schoolyard outside his Berlin apartment, made when he was missing Malta.

‘It starts with a feeling or an experience. Postcard was my experience of feeling super homesick. I recorded children playing outside my window and turned the result into somewhat of a snapshot – a sonic postcard back home,’ Tabone says. To keep faith with the feeling, he stripped the production of heavy electronics, allowing the reverberated piano to shoulder the narrative.

Prologue is more experimental, with changing pitch, rhythm, and ambience, and AI-generated visuals that also change throughout the piece.

Live, he presented a pre-rendered visual set synced to performance – a practical necessity over romantic improvisation. ‘It takes hours to generate these videos, so each song had a corresponding pre-made video projected behind me while I played,’ he says.

Piano First, Then Electricity

Beyond the code is a musician who learned to tell a story with hammers and strings. Tabone is semi-classically trained, with the piano as his centre of gravity and electronics as the atmosphere around it.

‘My main instrument is the piano. I’ve often been told my music sounds like film scores. There’s a lot of storytelling, and I use foley, recordings from outside, to give context.’

The modern conundrum, whether digital tools take soul out of music or widen its horizon, is not a debate for Tabone.

​’They definitely expand the possibilities. I’ve developed different piano configurations with digital tools. If the piano is the centre, I want you to feel inside it: key-noise, room creaks, all of it. If it’s in a thicker mix, the tone has to be sharper so it cuts through. These tools let you choose the perfect piano for the piece,’ he says.

Coda

What would music look like? In Tabone’s hands, not like a literal translation, not a piano turning into a CGI piano. It seems like a leakage – the colour and form escaping a frame, proportionate to the loudness of a feeling. It looks like a silent room as a projector warms. It seems like Malta and Berlin are speaking across a thin wire of nostalgia called Postcard. And it sounds like a composer who is happy to let images approximate while resonating piano strings sing the truth.