Sonified neural data: how it changes audience engagement

Sonified Neural Data: How It Changes Audience Engagement

A range of 80 to 97 percent. That is the decline in insect populations represented by a prominent ecological sonification project, compressed into an auditory signal that audiences could follow as a changing pattern rather than confront as a dense statistical statement. The number itself is not the point. The mechanism is.

Sound is unusually good at carrying change over time. A listener may not be able to read a raster plot of neural spikes or interpret a calcium-imaging heatmap, but they can often notice that a rhythm has thickened, a tone has risen, or a pattern has become unstable. That does not mean the audience automatically understands the biology. It means sonification can make a change perceptible before the listener has learned how to read the original representation.

This distinction matters. Neural data sonification is no longer confined to a few experimental performances or research demonstrations. The Data Sonification Archive tracks hundreds of projects, with public engagement developing alongside scientific research and accessibility work. The practical question is no longer simply whether neural activity can be converted into sound. It is how the conversion changes what people notice, remember, and believe they have understood.

For researchers, science communicators, and exhibition designers, the central problem is calibration. A sonification can preserve a meaningful signal and still fail as communication. It can sound compelling while hiding the feature it was meant to reveal. It can invite an audience into a dataset, or turn that dataset into an opaque piece of electronic music.

From EEG Pioneers to Modern BCMIs: The Evolution of Neural Sound

The first documented EEG sonification dates to the 1930s. By 1965, Alvin Lucier’s Music for Solo Performer had turned amplified brainwave signals into triggers for percussion. These early works established an important premise: electrical activity associated with the brain could become an audible event rather than remain an invisible trace on an instrument panel.

The early systems were also limited by their dimensionality. One channel was often connected to one sound source or one control parameter. A theta rhythm might influence pitch; an alpha rhythm might alter volume. The result could be technically faithful while remaining perceptually thin. The audience heard a changing signal, but had few tools for distinguishing one neural condition from another.

Modern brain-computer music interfaces have expanded the mapping space. Instead of treating the EEG as a single stream, they can separate activity into frequency bands and distribute those bands across several musical parameters. Melody may represent one dimension, rhythmic density another, while harmonic tension, timbre, or spatial position carry additional information.

The change is not just an increase in technical sophistication. It changes the listener’s task. A single tone that rises and falls can communicate fluctuation. A coordinated set of changes can suggest relationships between fluctuations. Yet the second experience is not automatically intelligible. More channels create the possibility of richer communication, but also create more opportunities for interference.

A 2023 user-engagement study involving 72 subjects found a strong relationship between aesthetic appeal and perceived usability in sonifications using melodic, rhythmic, and chordal contexts. Participants did not merely report that they liked the more musical outputs. They also tended to regard them as more usable for engaging with the underlying information. That is a useful finding, but it should not be translated into a claim that a listener can decode a complex neural signal without instruction. Perceived usability is not the same as unaided scientific comprehension.

A single EEG channel mapped to pitch produces a changing tone. A multi-parameter mapping can produce relationships—but only if the listener is given a way to learn what those relationships mean.

The Mapping Architecture

A practical neural sonification begins by deciding which features deserve to be heard. The most common mistake is to treat every available variable as equally important. Neural datasets are full of signals that are valuable for analysis but unsuitable for direct public presentation.

This table is an architecture, not a universal code. Pitch does not inherently mean “higher neural activity,” and dissonance does not inherently mean “dysfunction.” Those meanings are assigned by the designer. The audience must be told what has been assigned, and the assignment must remain stable long enough for a listener to form an expectation.

That stability is where musical structure becomes useful. Repeated mappings give the ear something to compare. If a particular neural feature always makes a rhythm denser, the listener can begin to hear departures from a baseline. If the same feature sometimes changes pitch, sometimes timbre, and sometimes spatial position, the sound may remain expressive but become difficult to interpret.

The information ceiling of a sonification is therefore not determined by the number of variables in the dataset. It is determined by the number of relationships an audience can follow at once. One-to-one mapping may discard too much. One-to-many mapping may overwhelm. The design problem sits between those two failures.

The Psychology of Auditory Data: Why Musical Mapping Drives Engagement

“Engagement” is an imprecise word until it is connected to a behaviour or a measure. In sonification research, it may refer to reported enjoyment, perceived usability, focused attention, interaction time, recall, or a willingness to explore the system further. These are related, but they are not interchangeable.

The 2023 D2M, or Data-to-Music, study used a User Engagement Scale adapted from human-computer interaction research. Its participants evaluated sonifications with different levels of musical mapping complexity. Three dimensions are particularly relevant to public-facing neural data:

1. Aesthetic appeal: whether the output sounds intentional and structured rather than arbitrary.

2. Perceived usability: whether the listener believes the sound helps them detect or explore a change in the data.

3. Focused attention: whether the sound holds attention for the duration of the experience.

The relationship between aesthetic appeal and perceived usability is important because musical organisation gives the listener a predictive framework. A recurring pulse establishes a temporal expectation. A melodic contour creates direction. A harmonic change marks a transition. When the data alters one of these patterns, the deviation becomes perceptible.

That is not the same as saying that music makes data self-explanatory. It makes certain differences easier to notice. The listener still needs context to connect the difference to a neural process.

This is why raw neural data and sonified audio should not be treated as competing versions of the same thing. A raw EEG trace preserves the original measurement pathway and can support exact inspection. Its visual form also makes some features explicit to a trained reader, such as amplitude, timing, and channel relationships. Sonified audio is better suited to temporal comparison, shared listening, and situations in which visual attention is already occupied.

The most effective public installation often uses more than one of these representations. The sound can draw attention to a transition while the screen shows which signal or region produced it. The visual layer can provide anatomical orientation; the audio layer can make a changing temporal pattern easier to follow. In that arrangement, sonification is not a replacement for the data. It is an additional route into it.

Why Raw-to-Audio Conversion Often Fails

A direct conversion, in which every voltage sample becomes an audio sample, may be informationally complete in a narrow technical sense. It does not necessarily communicate anything to an untrained listener. The result can sound like noise because the signal has been preserved without being given a perceptual hierarchy.

A public audience usually needs a baseline and a limited number of meaningful deviations. The baseline may be a resting or reference condition. The deviation might be an increase in rhythmic density, a shift in register, or a movement across the stereo field. The listener does not need to hear every recorded fluctuation. They need to hear the fluctuations that matter to the explanation.

Musical mapping helps because it creates a grammar of comparison:

• A recurring pulse can establish a reference state.
• A change in pulse density can indicate a change in event rate.
• A constrained melodic range can make a shift more noticeable.
• Harmonic tension can mark a relationship between signals, provided the audience knows that tension is the chosen metaphor.
• Silence or reduction can be more informative than continuous complexity when the system needs to mark a transition.

The calibration should be tested with people who have not seen the dataset before. Ask what they noticed, what they think changed, and what they believe the change represented. If listeners identify the most dramatic musical feature rather than the intended neural feature, the design is communicating the wrong thing.

Lessons from Fish and Chips: Balancing Spatial Resolution and Artistic Impact

The Fish and Chips installation at Ars Electronica in 2001 remains a useful case study because it made the tension between spectacle and resolution visible. The project used low-resolution signals from in-vitro zebrafish neural cells to drive a robotic drawing arm. Audiences could watch marks appear on paper as a living neural preparation influenced the system.

The installation’s strength was immediate legibility at the level of action. The audience could see that something was producing something else. The biological signal was not presented as a static graph; it was connected to a visible, unfolding process. That relationship created an entry point even for visitors without a background in neurobiology.

The weakness was the loss that occurred in translation. A spatially distributed neural network was condensed into a relatively limited audio output. The audience could encounter the temporal behaviour of the signal, but not necessarily its topology: which cells or regions were related, where activity originated, or how propagation moved through the network.

This is one of the defining problems of neural sonification. Biological neural systems are distributed in space, while sound unfolds in time. A single stream can preserve sequence and rhythm, but it cannot preserve every spatial relationship at the same time. Something has to be selected, collapsed, or re-encoded.

The BrainWaves project, presented at NIME in 2006, approached the problem through spatial auditory patterns and collaborative tactile controllers. Rather than forcing a distributed dataset into one undifferentiated stream, it treated distribution as part of the listening experience. Multiple players, controls, and spatial positions could represent different aspects of propagation.

The lesson is not that one project replaced the other. They answered different communication needs.

1. Use a single channel when temporal dynamics are the main story. A seizure-like spike, a sleep-stage transition, or a stimulus-evoked response may be easier to follow as one carefully shaped stream.

2. Use spatial or multi-channel sound when topology matters. Functional connectivity, propagation waves, and regional activation patterns require some way of distinguishing sources and relationships.

3. Use a hybrid design when both dimensions matter. Pitch and rhythm can carry temporal behaviour, while panning, speaker placement, or movement can carry spatial information.

4. Keep the number of simultaneous cues limited. Spatialisation does not solve overload by itself. It can make several streams distinguishable, but only if the audience has time to learn their positions and roles.

The artistic impact of Fish and Chips came partly from its visible apparatus and its sense of biological agency. Its scientific communication value depended on what the audience could connect that spectacle to. The more visually or sonically dramatic the installation becomes, the more deliberate the explanatory layer must be. Otherwise the living neural system becomes a source of atmosphere rather than an object of understanding.

A sonification can preserve a neural rhythm while discarding neural geography. The design must decide which loss the audience can afford.

Beyond Visualization: Higher-Order Sonification of 3D Brain Structures

Higher-order sonification addresses a different limitation: how to represent structure that cannot be reduced to a single line or a flat image. A 2025 study published in Scientific Reports introduced a method for translating three-dimensional structural MRI data into sound using statistical descriptors associated with cosmological analysis, including approaches developed for describing spatial density distributions.

The comparison with cosmology is not decorative. Both brain imaging and cosmic-background analysis involve patterns distributed through space and visible at different scales. Brain tissue has local intensity variation, regional clustering, and larger-scale asymmetries. A framework designed to describe spatial fluctuation can therefore be adapted to describe the organisation of voxel values.

The resulting approach attempts to retain scale-dependent features that a simple projection would flatten. Instead of turning an entire volume into one average value, it can encode how density or intensity is distributed across local and global scales. The audio is not a literal acoustic model of the brain. It is a translation of statistical properties into an auditory form.

That distinction is useful in public communication. Visitors do not need to believe that a low tone is “the sound of grey matter” in any direct physical sense. They need to understand that the tone, texture, or progression is linked to a defined property of the image. The translation becomes credible when the rules are visible and consistent.

What Higher-Order Sonification Can Preserve

The benefit is not that audio suddenly becomes a complete substitute for a volume rendering. It is that the listener gains access to relationships that may be difficult to perceive in a crowded visual display. Sound can also operate in parallel with an image. A screen might show the anatomy while the audio emphasises the distribution of intensity or the difference between two volumes.

That parallel modality has a practical consequence: each channel must do a different job. If the image and sound both encode the same variable with equally rapid changes, the audience may experience redundancy without gaining clarity. If the image establishes location and the sound marks temporal or statistical change, the two modes can reinforce one another.

The cost is interpretive. Higher-order mappings are usually less immediately musical than a simple pitch-to-value conversion. Their tonal relationships may not follow familiar harmonic conventions, and the audience may need a short explanation before the sound becomes meaningful. A brief decoder is not an admission of failure. It is part of the instrument.

Interactive Science Communication: Scaling Neural Data for Public Exhibits

A confirmed public-engagement example using Raspberry Pi and Sonic Pi concerns DNA sequence sonification, not neural data. It demonstrates that low-cost hardware and open software can support an interactive scientific sound installation. It does not, by itself, prove that neural sonification is viable at scale. Neural datasets bring additional problems: acquisition, preprocessing, privacy and consent in human studies, biological variability, and the need to explain what a particular signal represents.

The distinction matters because “low-cost” and “scalable” are not synonyms. A small computer may be sufficient to generate sound, but a public neural installation also needs a stable data pipeline, clear interaction logic, maintenance, and an explanatory framework that survives repeated encounters with visitors.

The most common failures are design failures rather than hardware failures.

Context Without a Lecture

An audience needs to know what it is hearing before it can interpret a change. The opening explanation can be short, but it must identify the source, the represented variable, and the direction of the mapping. “This sound changes with the activity recorded from the preparation” is more useful than “this is the brain in music.”

A context frame should answer three questions:

• What biological system produced the data?
• Which feature has been mapped to sound?
• What should the listener pay attention to?

The frame should not promise more than the sonification can deliver. If the installation represents firing rate rather than the location of individual neurons, it should say so. If the sound is based on a statistical summary, the visitor should not be encouraged to hear it as a literal recording of microscopic events.

Establishing a Baseline

Without a reference, a visitor may hear difference without knowing that it is difference. A baseline can be a resting condition, an averaged state, or a deliberately simplified reference pattern. It gives the ear a point of comparison.

The baseline does not need to be long, and it does not need to be scientifically exhaustive. Its job is to establish the mapping. Once visitors recognise the reference pulse, register, texture, or spatial position, later changes can become perceptible as departures.

This is also where the installation can avoid a common mistake: beginning with the most complex signal. Complexity may impress for a moment, but it gives the listener no stable surface against which to measure change.

Interaction and Agency

Interactive exhibits often hold attention because they give visitors a reason to listen again. A slider controlling stimulus intensity, a button that changes the selected neural feature, or a choice between two recording conditions can turn the audience from passive recipient into investigator.

Agency should remain connected to interpretation. If a visitor can manipulate a parameter without understanding what it controls, the interaction becomes a synthesiser disguised as a scientific instrument. The best controls have an explicit relationship to the data: choose a frequency band, move through a recording, compare two regions, or adjust the balance between temporal and spatial information.

The DNA sequence examples built with Raspberry Pi and Sonic Pi are relevant here as models of accessible interaction, not as evidence about neural data specifically. They show how an inexpensive system can make a biological encoding tangible. Neural exhibits can borrow that openness while retaining stricter explanations about signal processing and biological meaning.

Avoiding Signal Overload

Raw multi-channel neural data rarely makes a good public soundtrack. Filtering and quantisation are not necessarily distortions; they are editorial decisions that determine which relationships survive in the listening experience.

A practical exhibit might:

1. Choose one primary variable and one supporting variable.

2. Establish a reference state before introducing change.

3. Keep the mapping stable across the visitor’s session.

4. Offer a visual or printed decoder for the sound.

5. Let visitors compare two conditions rather than asking them to interpret one complex stream in isolation.

6. Record interaction duration and visitor responses where appropriate, while avoiding the assumption that longer listening automatically means better understanding.

The final point is important. A visitor may stay because the sound is attractive, because the interface is confusing, or because the installation creates a strong atmosphere. Engagement data needs to be paired with questions about interpretation and recall.

The Decoder Is Part of the Work

A printed card, projected legend, or short spoken introduction can translate a mapping into an invitation:

• higher pitch corresponds to greater power in a selected frequency band;
• denser rhythm corresponds to a higher event rate;
• a change in harmonic tension corresponds to a selected relationship between channels;
• movement from left to right corresponds to a change in spatial source or region.

The wording should be exact enough to prevent false conclusions and simple enough to support listening. “You are hearing 50,000 neurons responding to a light pulse” is not a useful frame when the data does not support that neuron count or stimulus description. A more responsible explanation would identify the preparation and the measured feature without inventing a level of resolution the system did not record.

The bottleneck in public neural sonification is not signal fidelity alone. It is design fidelity: whether the mapping gives the audience a pattern to learn instead of another layer of noise.

What Neural Data Sonification Can Change

The strongest case for neural data sonification is not that sound is more truthful than a graph. It is that sound changes the conditions of attention. A visual display can be scanned, compared, and revisited. Sound unfolds and disappears, but it can make temporal relationships immediate and allow several people to attend to the same event together.

That change in attention can support science communication in several ways:

• It can make variation audible to people who would not know how to read a neural trace.
• It can give a group a shared object of attention in a gallery, classroom, or laboratory event.
• It can supplement visual displays for visitors with different sensory access needs.
• It can make relationships between signals perceptible as rhythm, harmony, or movement.
• It can turn a static explanation into an exploratory encounter.

Each benefit has a corresponding risk. Accessibility is not guaranteed by adding sound; a poorly designed audio layer can exclude listeners or create sensory overload. Musical appeal can increase curiosity while encouraging visitors to mistake artistic coherence for biological certainty. Interactivity can increase agency while allowing arbitrary manipulation to masquerade as experimentation.

The most defensible approach is therefore neither raw fidelity nor unrestricted artistic interpretation. It is a declared translation. The designer explains what has been selected, what has been discarded, and what the audience can reasonably infer.

Neural data sonification changes audience engagement when it gives listeners a stable mapping, a meaningful comparison, and enough context to connect an audible pattern to a biological question. Musical structure can make neural activity easier to approach. Spatial audio can recover some of what a single channel loses. Higher-order methods can carry statistical features of three-dimensional structures that are difficult to preserve in a flat display.

But none of these techniques removes the need for explanation. The most persuasive sonification is not the one that sounds most like music or most like a machine. It is the one in which the audience can hear a difference, learn what that difference represents, and understand the limits of the translation.

The architecture is part of the message. The explanation is part of the instrument.