Why do we need to know about the brain?

(1,4). Music and Emotion

Many of us remember the very first moment when music took hold of our body and soul. Suddenly, the music hits you with overwhelming heat. You are transported to a different place. You even forget who you are.

I have long wondered how we can experience such intense emotions by listening to structured abstract sounds: not a human voice, not a story. Perhaps something primordial in music, something that has been with us even before we had words to describe it.

But what is it, really?

(1,2). Music and Brain

Today, we believe that our soul resides in the brain (or perhaps the entire nervous system, or even all the environment with which we interact). Let’s say that neuroscientists believe that many parts of our soul live in the brain.

Music is, after all, an auditory signal, in the same way that everything is, after all, made of atoms. The modern neuroscience gives a good explanation of how sounds are processed in the auditory cortex—how the spectral and temporal energy fluctuations are decomposed and encoded, and how these are further processed in the higher-order auditory cortices. Still, it remains unclear in many parts. For example, how does the brain segregate auditory objects? What determines the foreground and background?

(2,4). Brain and Emotion

And how much do we know about emotion? What is emotion? Why are we still debating over the optimal data type of emotion (binary vs. float) after 100 years? What is wrong with this field?

Psychology, as a science, bears a fundamental challenge in attempting to measure something that cannot be measured directly. Even for perception, we cannot directly measure one’s percept in their consciousness (qualia): we don’t know whether the redness my friend sees is the same redness I see.

The only hope is that my friend’s redness is at least reliable throughout their life and comparable across objective objects. Psychophysicists have been working to reconstruct perception based on countless simple comparisons (AB or ABA). In a way, this is a tedious process of learning a massive kernel matrix (billions by billions) through each of pairwise comparisons.

But how can we learn such a kernel for emotions? Do we always find music A is more enjoyable than music B? We experience redness in the same way whether we are tired or energetic (or so we believe). But the emotional experiences are strictly tied to the external stimulus. One day, I prefer music A over music B. On another day, I prefer B over A, and I don’t know why. This lower reliability (or the stochastic nature of affective experience as compared to perceptual experience) poses an even greater challenge for affective science to become a science.

(2). Brain

And how much can we really know about the brain by measuring its activity? Can we understand how the Space Invaders works by recording all electrical activity of Atari [.HTML]?

Moreover, we are so far from “recording perfect activities” of the brain, even the invasive electrophysiology in mice [.HTML]. What can these squish scalp EEG and BOLD-fMRI can tell us about the actual brain?

Of course, it’s the best we have now. But is it really do anything more than just giving us false hope that these noisy signals might tell us the ancient secret? Have we been just analysing the EEG electrode noise and the MRI artefacts and fantasising about them? 🤡

(3,1). Machine and Music

Can a machine understand music better than a human? At least, it has non-decaying memory and rapid computation speed.

SUNO’s overfitted models can replicate real-world music to the point that it can be ruled copyright infringement by a German regional court [.HTML]. But that does not mean that the models understand what music is.

(3,2). Machine and Brain

Some network structures (e.g., CNNs) are inspired by how we think the sensory cortex works. But many network architectures and most implementations have nothing to do with the human brain.

Can we understand the human brain better through machine representations? What does it mean that we can predict human brain activity using machine embeddings of external stimuli? How can we move from “mindless” prediction to “insightful” understanding and explanation?

(3,4). Machine and Emotion

How well can a machine predict or decode emotions? Only as well as the quality of the training set. And because emotion is stochastic, I suspect that the internal consistency of a human rater would already be quite weak. The agreement across raters would be substantially lower than for other types of datasets.

(1,2,3,4) Music, Brain, Machine, and Emotion

My goal is to bring all of these together and make a contribution to the Computational Neuroscience of Musical Emotion. To have a science, we need reliability, consistency, and validity. How can we improve on these? 🤨