Sound Art Prompted by AI Inspired Lutherie

I have been tinkering with generative art (especially music) since around 2008. Back then, if you wanted to implement a neural network to create art-like works, you had to hand code your perceptrons using exotic Java libraries.

While the past 10 years or so have seen tremendous simplifications and advances for coding machine learning systems in general (TensorFlow) and abstractions for music in particular (NSynth/Magenta for instance), in my personal compositorial practice I’ve moved away from such systems in the direction of more simple algorithmic models I sometimes use to pre-produce material. I did not, however, dig any deeper into the area of autonomous musical "minds".

AI as Extension of My Own Imagination

Forward to 2022 and the hype around image generators and large language models. Even though for me as someone who has been fishing in this pond a long time ago the developments aren’t surprising, it has sparked new interest in me. But not, as you might have guessed, to offload my artistic processes to an artificial mind. Much rather what I am experiencing these days I would classify as intellectual extensions of, and "significant other" to my own imagination. This is manifesting itself especially in fringe concerns that inform my musical and sound artist practice, but I have no special training in. The following is an account of one such interaction with two artificial intelligences.

Prompting the Artist

While I’ve been exploring the problem space of image generators for resonAIte, I stumbled upon an interesting use case: Using AI to prompt me (!) to explore sonic territories.

The process (which isn’t completed yet) is the following:

Ask ChatGPT to come up with concepts for fantastical/non-existent/„impossible a la M.C.Escher“ musical instruments
Ask StableDiffusion to produce visualizations of those
Sonify them

Work In Progress

Without much further explanation I would like to put forth my interaction with ChatGPT and StableDiffusion. As you will see, currently I've stopped at producing the instrument visualizations, but will certainly explore the territory of sonifying them soon.

ChatGPT

Q: Can you conceive a few concepts for non-existent, fantastical musical instruments? I would like to pass them as prompts into an image generator like Midjourney or StableDiffusion.

Certainly! Here are a few concepts for fantastical musical instruments:

Harmonic Crystal: The Harmonic Crystal is a translucent, diamond-shaped instrument that emits ethereal sounds when touched or struck. Each facet of the crystal produces a different tone, and skilled musicians can manipulate the crystal's vibrations to create hauntingly beautiful melodies.

Aerochimes: Aerochimes are a set of wind-powered instruments resembling delicate, metallic flowers. Suspended in the air, they sway gently with the breeze, creating soothing harmonies. Each flower-like structure has thin, shimmering petals that act as resonators, amplifying the musical notes produced by the wind passing through them.

Luminostrum: The Luminostrum is an instrument made up of several glowing orbs of varying sizes and colors, suspended from a sturdy frame. When the orbs are struck or rubbed, they emit radiant sounds that correspond to their respective hues. Musicians can create dazzling, multi-sensory performances by combining rhythmic patterns and light choreography.

Aquaphonium: The Aquaphonium is a water-based instrument that harnesses the power of flowing liquid to produce melodies. It consists of a series of interconnected glass tubes, each filled with water at different levels. By manipulating valves and blowing air through the tubes, musicians control the flow and pitch of the water, generating mesmerizing, cascading sounds.

Pyrophone Organ: The Pyrophone Organ is a grand, multi-tiered instrument that produces music using controlled flames. Each pipe of the organ has a small flame burning at its base, and when air is blown through the pipes, the flames modulate to create different pitches. The result is a visually stunning display of fire and sound, where skilled performers orchestrate fiery compositions.

Q: Great! Can you name some more?

Certainly! Here are a few more fantastical musical instrument concepts for your image generator:

Celestial Harp: The Celestial Harp is a celestial-themed instrument with strings made of shimmering stardust. Its frame is adorned with intricate constellations, and when the strings are plucked or strummed, they produce celestial melodies that resonate with the cosmos itself.

Thunder Drum: The Thunder Drum is a large, cylindrical drum embedded with powerful lightning generators. When struck, it creates thunderous percussive beats accompanied by lightning flashes. Skilled musicians can manipulate the intensity and rhythm of the thunder and lightning to create electrifying performances.

Echo Flute: The Echo Flute is a slender, transparent flute-like instrument that harnesses the power of echoes. When played, the flute emits a hauntingly pure sound that reverberates through space, bouncing off invisible surfaces and creating a rich, immersive auditory experience.

Harmonic Glove: The Harmonic Glove is a wearable instrument worn on the hand that translates gestures and movements into musical notes. Equipped with sensors and intricate patterns, it allows musicians to "play" the air by waving their hand, enabling them to create intricate melodies and harmonies through expressive movements.

Chime Blossoms: Chime Blossoms are a collection of flower-shaped instruments suspended from branches, each producing unique chime-like sounds. As gentle breezes pass through the enchanted garden, the blossoms sway and create a mesmerizing symphony of delicate tones, resembling a natural wind chime orchestra.

Quantum Resonator: The Quantum Resonator is a mesmerizing instrument that utilizes quantum entanglement to generate ethereal sounds. Made of intricate, intertwined crystal structures, it harnesses the mysterious properties of quantum mechanics to produce otherworldly, resonating tones that seem to transcend space and time.

Q: Now, let's modify the challenge to produce "impossible" instruments a la M.C. Escher. Can you try that?

Certainly! Here are some concepts for "impossible" musical instruments inspired by the intricate and mind-bending works of M.C. Escher:

Infinity Harp: The Infinity Harp appears as a never-ending loop of strings that twist and intertwine, defying conventional geometry. Despite its seemingly impossible structure, when plucked or strummed, it produces an infinite range of harmonies that seem to resonate from multiple dimensions simultaneously.

Möbius Organ: The Möbius Organ is a perplexing instrument shaped like a twisted Möbius strip, where the keyboard and pipes form an unbroken, looping surface. When played, the organ emits complex melodies that continually shift and transform, as if the music is flowing along an endless, enigmatic pathway.

Penrose Percussion: The Penrose Percussion is a set of drum-like instruments with irregular shapes and patterns inspired by Penrose tiling. The drums' surfaces are covered in intricate, interlocking motifs that appear to tessellate forever. When struck, they produce rhythmic beats and resonances that defy conventional musical rhythms.

Paradoxophone: The Paradoxophone is a surreal instrument composed of interconnected brass and woodwind elements that twist and intersect in impossible ways. Its pipes merge and separate, creating an intricate network that seemingly violates the laws of physics. When played, the Paradoxophone produces disorienting melodies that challenge the listener's perception of space and sound.

Recursive Bells: The Recursive Bells are a set of nested bell-like structures, each containing smaller replicas of itself, creating an infinite regression of chimes. When struck, the bells resonate with harmonic overtones that cascade infinitely, producing an enchanting, cascading soundscape that defies traditional musical composition.

Fractal Strings: The Fractal Strings are a set of interwoven, self-replicating strings that branch and bifurcate, creating intricate fractal patterns. When plucked or bowed, these extraordinary strings produce a mesmerizing cascade of harmonics, with each iteration of the fractal generating new tonal complexities.

These "impossible" musical instruments inspired by M.C. Escher's art challenge our perception of form and function, creating a fascinating fusion of music and visual illusion. Enjoy exploring the impossible and surreal through your image generator!

StableDiffusion

Note: I used PromptGen to create some variations of the descriptions above which are more likely to produce interesting visualizations.

Harmonic Crystal

a translucent, diamond-shaped musical instrument, ornate glass-wires, high-tech, lasers and cyberpunk, in the style of pixar animation, trending on Artstation, 8K, octane render

Aerochimes

a set of wind-powered instruments resembling delicate, metallic flowers, with a metallic orb, in the foreground, a fantasy magical forest, a magical fantasy fantasy, trending on artstation and featured on pixiv and deviantart

Luminostrum

an instrument made up of several glowing orbs of varying sizes and colors, suspended from a sturdy frame with a high wall made out of thousands of pieces of glass in the center, in the style of studio ghibli. dark and moody, moody, atmospheric, epic, cinematic, very very very very very beautiful.

Aquaphonium

a water-based instrument consisting of a series of interconnected glass tubes, each filled with water at different levels, in the style of peter mohrbacher and dan mumford, featured on artstation, photorealistic, octane render, cinematic, elegant, intricate, 8k

Chime Blossoms

a collection of flower-shaped instruments suspended from branches, each producing unique chime-like sounds, resembling a natural wind chime orchestra, a high contrast ultradetailed photorealistic painting by jan van eyck, audubon, rene magritte, agnes pelton, max ernst, walton ford, andreas achenbach, ernst haeckel, hard lighting, masterpiece

Quantum Resonator (1-3)

a mesmerizing instrument made of intricate, intertwined crystal structures, that utilizes quantum entanglement to generate ethereal sounds. ilya kuvshinov, zhelong xu, oscar Ukono, concept art, digital concept, smooth, sharp focus, highly detailed. octane. zbrush. artstation trending. ray tracing. 8k. intricate. nikon d850

a mesmerizing instrument made of intricate, intertwined crystal structures, that utilizes quantum entanglement to generate ethereal sounds. ilya kuvshinov, greg rutkowski, zabrocki, karlkka, jayison devadas, trending on artstation

a mesmerizing instrument that utilizes quantum entanglement to generate ethereal sounds, made of intricate, intertwined crystal structures, highly detailed, digital art, hyperrealism, octane rendering, smooth, sharp focus, illustration, trending on art station, artgerm and greg rutkowski and bo chen and jin xiaodi

Celestial Harp

a celestial-themed instrument with strings made of shimmering stardust. Its frame is adorned with intricate constellations and components, and its intricate and highly detailed isometric design is a truly magical & haunting work of art.

Thunder Drum

a large, cylindrical drum embedded with powerful lightning generators. When struck, it creates thunderous percussive beats accompanied by lightning flashes. The drum has a powerful aura, creating a tornado of force. Hyperdetailed digital painting, trending on artstation, cgsociety, concept art, magic realism

Echo Flute

a slender, transparent flute-like instrument that harnesses the power of echoes and dispels vibrations, an ultrafine detailed illustration by james jean, final fantasy, intricate linework, bright colors, behance contest winner, vanitas, angular, altermodern, unreal engine 5 highly rendered, global illumination, radiant light, detailed and intricate environment

Harmonic Glove

a wearable instrument worn on the hand that translates gestures and movements into music instruments, a futuristic music instrument, in the style of zaha hadid, futuristic, futuristic, futuristic, futuristic, neon colors, bright colors, bright colours, trending on artstation

"Impossible Instruments"

Infinity Harp

a harp consisting of a never-ending loop of strings that twist and intertwine, defying conventional geometry, dark fantasy, intricate, elegant, highly detailed, digital painting, artstation, concept art, matte, sharp focus, illustration, art by josan gonzalez

Möbius Organ

a perplexing instrument shaped like a twisted Möbius strip, where the keyboard and pipes form an unbroken, looping surface, and a microscopic photo of the inside of the keyboard and pipe, in a surreal, cosmic, cinematic, ultra realistic, lens flare, 8k, hyper detailed, photo realistic

Penrose Percussion

a set of drum-like instruments with irregular shapes and patterns inspired by Penrose tiling, high detail, photorealistic, intricate abstract, detailed textures, octane render, unreal engine, 8k

Paradoxophone

a surreal instrument composed of interconnected brass and woodwind elements that twist and intersect in impossible ways. Its pipes merge and separate, creating an intricate network that seemingly violates the laws of physics

Recursive Bells (1-2)

a set of nested bell-like structures, each containing smaller replicas of itself, creating an infinite regression of chimes

a set of nested bell-like structures, each containing smaller replicas of itself, creating an infinite regression of chimes in the background, and a variety of variations of styles to create a sense of ordered chaos, from chimes to the background, in the style of balthus, high detail, ultra realistic, 4k, trending on artstation, photorealistic

Fractal Strings

a set of interwoven, self-replicating strings that branch and bifurcate, creating intricate fractal patterns around the strings, highly detailed portrait, scifi, digital painting, artstation, concept art, smooth, sharp foccus ilustration, artstation hq