Design Concept: VR Synesthesia Simulator
Synesthesia is a phenomenon where some people experience their senses intermixing with each other. They might see sounds, or hear sights, or even physically feel words. It’s a mental condition apparently attributed to many great artists, from Vincent Van Gogh to Billie Eilish to allegedly even Beyonce.
I have always wanted to experience synesthesia myself, but I always assumed it would be impossible. Certain pieces of art can give me a fleeting sensation of it—Van Gogh’s paintings and Kanye’s 808s & Heartbreak come to mind. Visuals at a concert, when done well, can give a feeling of seeing the music. But that’s a sort of inferred collective synesthesia, where I can sense the impact of it on the work, but I don’t really get to experience it as someone with the condition would.
It occurred to me recently that augmented reality headsets like Apple’s Vision Pro, along with generative AI, could provide the technology to do a reasonable real-time synesthesia simulator. I’m adding it to the list of projects I’d undertake if I had unlimited resources, but for now it’s just a thought experiment.
The Vision Pro, along with generative AI, would fail to reproduce all iterations of the condition, but it could hypothetically reproduce the following types of synesthesia:
- Grapheme Color Synesthesia, who see colours or shapes next to specific words or characters.
- Number Form, who see particular forms or shapes for some numbers.
- Chromesthesia, who sees sounds as colours or shapes.
- Ticker Tape, who see strings of words scroll underneath people as they talk.
Here are some art/tech projects which explore other aspects of synesthesia. Google’s Play a Kandinsky is beautiful, but is again an interpretation of an artist’s view.
The game Audiosurf matches audio to visuals in a synesthetic way. A seemingly defunct smartphone VR app seemingly tried to accomplish the same concept, though it glitches on my phone and hasn’t been updated in 6 years. There are some fascinating video visualizations of synesthesia, like this YouTube video of a violinist who sees notes as coloured shapes.
My version would be real time, and importantly, individualized. You would choose the type of synesthesia you wanted to recreate, or even the specific artist whose synesthesia you wanted to experience, and everything you saw and heard would be filtered to recreate the look and sound of that type.