The human brain is the next top­ic of dis­cus­sion. The brain is much like a com­put­er, but it is a com­put­er that works by very dif­fer­ent prin­ci­ples from say, the com­put­er that runs your cell phone. That com­put­er, which was designed by peo­ple, has very sep­a­rate hard­ware and soft­ware stacks. They’re designed to be as inde­pen­dent as pos­si­ble, so that the phone can run any software.

Image of a microchip being shown as "vs" a human brain

The human brain works by the oppo­site prin­ci­ple. The hard­ware and the soft­ware are inti­mate­ly inter­linked. Now, there are about, we esti­mate based on ani­mal stud­ies, five hun­dred dis­tinct brain mod­ules in the human brain, and these are wired togeth­er in a com­pli­cat­ed, inter­con­nect­ed network.

Graph showing the various major brain regions and a complicated web of individual connections to and from each of them to the various others.

And on a short time scale, the net­work is fair­ly fixed, and your fleet­ing thoughts all arise by inter­ac­tions of infor­ma­tion through this net­work. Each indi­vid­ual brain area is itself fair­ly com­pli­cat­ed. If we look at the visu­al cor­tex of an ani­mal, we find that there is a map of visu­al space in the visu­al cor­tex, and on top of that there are say ten or twen­ty oth­er visually-relevant dimen­sions rep­re­sent­ed in that same brain area. So if we have five hun­dred brain areas each rep­re­sent­ing twen­ty dimen­sions, we have ten thou­sand dimen­sions rep­re­sent­ed in the brain.

Now, peo­ple have known for over a thou­sand years that brain func­tion was prob­a­bly some­what local­ized. But until recent­ly we had no way to actu­al­ly mea­sure the brain in liv­ing humans. So most of our con­cep­tions about brain func­tion and brain anato­my were based most­ly on superstition.

But about twen­ty years ago there was a new method devel­oped for mea­sur­ing meta­bol­ic activ­i­ty in the brain that is asso­ci­at­ed with neur­al activ­i­ty, and that’s called MRI or func­tion­al MRI. An MRI mea­sures brain activ­i­ty in small vol­u­met­ric units about the size of a pea called vox­els. And you can mea­sure these meta­bol­ic units all over the brain, and you can use this there­fore to map brain activ­i­ty, as shown here.

A small video area showing various movie clips, with a flattened brain map below showing different areas activating in response.

Still from Jack Gallant, Human Brain Activity Elicited By Natural Movies With Sound”; dur­ing this sec­tion, Gallant plays a 15-second por­tion from ~2:00

This is the brain of one human sub­ject watch­ing a movie. We inflate the brain and we flat­ten it out so you can see the entire cor­ti­cal sur­face, and we’re paint­ing brain activ­i­ty on the cor­ti­cal sur­face as this sub­ject watch­es the movie. And you can see that these pat­terns are dynam­ic and com­pli­cat­ed and con­stant­ly shifting.

Flattened brain map, with various smudges of color representing areas of activation.

Now, if we do a con­trolled exper­i­ment, for exam­ple we have peo­ple lis­ten to nar­ra­tive speech, we can actu­al­ly pull fea­tures out of the speech and we can look to see where those fea­tures are rep­re­sent­ed in the brain. So for exam­ple, green on this map shows where seman­tic infor­ma­tion about speech, the mean­ing of speech, is rep­re­sent­ed in the brain. And you can see that the mean­ing of lan­guage is rep­re­sent­ed in wide swaths of the brain. 

We can drill down into the seman­tic mod­el and col­or each vox­el accord­ing to its seman­tic selec­tiv­i­ty. So the col­ors on these maps indi­cate dif­fer­ent kinds of seman­tic con­cepts. Animals, vehi­cles, tools, any­thing you can think of. Social inter­ac­tions, they’re rep­re­sent­ed in lan­guage, and you can see that these maps are fair­ly rich and complicated.

Several flattened brain map, with the same coloring as previous, showing similarities and differences across individuals.

We can do this map­ping in indi­vid­ual sub­jects, so we can recov­er these seman­tic maps in each indi­vid­ual sub­ject. You can see that the motifs are some­what com­pa­ra­ble across sub­jects, but there are also sub­stan­tial indi­vid­ual differences. 

And if we col­lect data from a group of say, sev­en or ten sub­jects, we can use mod­ern machine learn­ing and gen­er­a­tive mod­el­ing meth­ods to recov­er an atlas of seman­tics selec­tiv­i­ty. And this atlas shows that there are about two hun­dred dis­tinct brain areas that’re involved in rep­re­sent­ing the mean­ing of lan­guage. And these areas are shared across all humans, but their exact posi­tion and size dif­fers in dif­fer­ent individuals.

Now, to show you how pow­er­ful this atlas approach is, we can do brain decod­ing. And in brain decod­ing, we take our mod­el that we’ve devel­oped of the brain (and this can be a mod­el for any­thing, vision or lan­guage) and we reverse it. And instead of going from the stim­u­lus to the brain activ­i­ty, we go from the brain activ­i­ty back to the stim­u­lus. And we actu­al­ly decode the infor­ma­tion rep­re­sent­ed in a giv­en brain area. So we can build one brain decoder that rep­re­sents or recov­ers the infor­ma­tion in each brain area. 

So, this is a decoder built for [the] pri­ma­ry visu­al cor­tex. On the upper left is the movie that sub­jects saw. On the upper right is the decod­ed result. Primary visu­al cor­tex rep­re­sents the struc­tur­al ele­ments in movies, like the local edges and tex­tures. And you can see that this decoder does a fair­ly good job of recov­er­ing the struc­tur­al infor­ma­tion, even though we’re decod­ing one brain area.

This is a decoder that is recov­er­ing infor­ma­tion from high­er level-visual areas that rep­re­sent the seman­tic infor­ma­tion in movies and scenes, the objects and actions in the movies. And you can see that this decoder also works quite well. It recov­ers talk­ing, and woman, and man; cov­ers all the seman­tic infor­ma­tion in the movies.

So, at this point it’s still ear­ly days for non­in­va­sive mea­sure­ment of human brain activ­i­ty. We’re much like the sit­u­a­tion was in pho­tog­ra­phy in the ear­ly 18th cen­tu­ry. The pic­tures aren’t very good yet. They’re some­what dim. But as long as we keep invest­ing in neu­ro­science, we going to devel­op bet­ter meth­ods for mea­sur­ing the brain. And when­ev­er we can mea­sure the brain bet­ter, we can build a bet­ter brain decoder.

So, bet­ter brain decoders are in our future. These will be used to decode inter­nal speech. They’ll be portable. They’ll be used for brain-aided CAD cam, dream decod­ing, all kinds of things. And these bring up inter­est­ing, I think, neur­al eth­i­cal issues hav­ing to do with pri­va­cy and use of infor­ma­tion. You know, how are you going to pro­tect your brain infor­ma­tion, who has rights to use it, what will it be used for, and how do we con­trol that?

Thank you very much for your time.

Further Reference

The Gallant Lab at UC Berkeley

Help Support Open Transcripts

If you found this useful or interesting, please consider supporting the project monthly at Patreon or once via Cash App, or even just sharing the link. Thanks.