The human brain is the next top­ic of dis­cus­sion. The brain is much like a com­put­er, but it is a com­put­er that works by very dif­fer­ent prin­ci­ples from say, the com­put­er that runs your cell phone. That com­put­er, which was designed by peo­ple, has very sep­a­rate hard­ware and soft­ware stacks. They’re designed to be as inde­pen­dent as pos­si­ble, so that the phone can run any software.

The human brain works by the oppo­site prin­ci­ple. The hard­ware and the soft­ware are inti­mate­ly inter­linked. Now, there are about, we esti­mate based on ani­mal stud­ies, five hun­dred dis­tinct brain mod­ules in the human brain, and these are wired togeth­er in a com­pli­cat­ed, inter­con­nect­ed network.

And on a short time scale, the net­work is fair­ly fixed, and your fleet­ing thoughts all arise by inter­ac­tions of infor­ma­tion through this net­work. Each indi­vid­ual brain area is itself fair­ly com­pli­cat­ed. If we look at the visu­al cor­tex of an ani­mal, we find that there is a map of visu­al space in the visu­al cor­tex, and on top of that there are say ten or twen­ty oth­er visually-relevant dimen­sions rep­re­sent­ed in that same brain area. So if we have five hun­dred brain areas each rep­re­sent­ing twen­ty dimen­sions, we have ten thou­sand dimen­sions rep­re­sent­ed in the brain.

Now, peo­ple have known for over a thou­sand years that brain func­tion was prob­a­bly some­what local­ized. But until recent­ly we had no way to actu­al­ly mea­sure the brain in liv­ing humans. So most of our con­cep­tions about brain func­tion and brain anato­my were based most­ly on superstition.

But about twen­ty years ago there was a new method devel­oped for mea­sur­ing meta­bol­ic activ­i­ty in the brain that is asso­ci­at­ed with neur­al activ­i­ty, and that’s called MRI or func­tion­al MRI. An MRI mea­sures brain activ­i­ty in small vol­u­met­ric units about the size of a pea called vox­els. And you can mea­sure these meta­bol­ic units all over the brain, and you can use this there­fore to map brain activ­i­ty, as shown here.

Still from Jack Gallant, ​“Human Brain Activity Elicited By Natural Movies With Sound”; dur­ing this sec­tion, Gallant plays a 15-second por­tion from ~2:00

This is the brain of one human sub­ject watch­ing a movie. We inflate the brain and we flat­ten it out so you can see the entire cor­ti­cal sur­face, and we’re paint­ing brain activ­i­ty on the cor­ti­cal sur­face as this sub­ject watch­es the movie. And you can see that these pat­terns are dynam­ic and com­pli­cat­ed and con­stant­ly shifting.

Now, if we do a con­trolled exper­i­ment, for exam­ple we have peo­ple lis­ten to nar­ra­tive speech, we can actu­al­ly pull fea­tures out of the speech and we can look to see where those fea­tures are rep­re­sent­ed in the brain. So for exam­ple, green on this map shows where seman­tic infor­ma­tion about speech, the mean­ing of speech, is rep­re­sent­ed in the brain. And you can see that the mean­ing of lan­guage is rep­re­sent­ed in wide swaths of the brain. 

We can drill down into the seman­tic mod­el and col­or each vox­el accord­ing to its seman­tic selec­tiv­i­ty. So the col­ors on these maps indi­cate dif­fer­ent kinds of seman­tic con­cepts. Animals, vehi­cles, tools, any­thing you can think of. Social inter­ac­tions, they’re rep­re­sent­ed in lan­guage, and you can see that these maps are fair­ly rich and complicated.

We can do this map­ping in indi­vid­ual sub­jects, so we can recov­er these seman­tic maps in each indi­vid­ual sub­ject. You can see that the motifs are some­what com­pa­ra­ble across sub­jects, but there are also sub­stan­tial indi­vid­ual differences. 

And if we col­lect data from a group of say, sev­en or ten sub­jects, we can use mod­ern machine learn­ing and gen­er­a­tive mod­el­ing meth­ods to recov­er an atlas of seman­tics selec­tiv­i­ty. And this atlas shows that there are about two hun­dred dis­tinct brain areas that’re involved in rep­re­sent­ing the mean­ing of lan­guage. And these areas are shared across all humans, but their exact posi­tion and size dif­fers in dif­fer­ent individuals.

Now, to show you how pow­er­ful this atlas approach is, we can do brain decod­ing. And in brain decod­ing, we take our mod­el that we’ve devel­oped of the brain (and this can be a mod­el for any­thing, vision or lan­guage) and we reverse it. And instead of going from the stim­u­lus to the brain activ­i­ty, we go from the brain activ­i­ty back to the stim­u­lus. And we actu­al­ly decode the infor­ma­tion rep­re­sent­ed in a giv­en brain area. So we can build one brain decoder that rep­re­sents or recov­ers the infor­ma­tion in each brain area. 

So, this is a decoder built for [the] pri­ma­ry visu­al cor­tex. On the upper left is the movie that sub­jects saw. On the upper right is the decod­ed result. Primary visu­al cor­tex rep­re­sents the struc­tur­al ele­ments in movies, like the local edges and tex­tures. And you can see that this decoder does a fair­ly good job of recov­er­ing the struc­tur­al infor­ma­tion, even though we’re decod­ing one brain area.

This is a decoder that is recov­er­ing infor­ma­tion from high­er level-visual areas that rep­re­sent the seman­tic infor­ma­tion in movies and scenes, the objects and actions in the movies. And you can see that this decoder also works quite well. It recov­ers talk­ing, and woman, and man; cov­ers all the seman­tic infor­ma­tion in the movies.

So, at this point it’s still ear­ly days for non­in­va­sive mea­sure­ment of human brain activ­i­ty. We’re much like the sit­u­a­tion was in pho­tog­ra­phy in the ear­ly 18^th cen­tu­ry. The pic­tures aren’t very good yet. They’re some­what dim. But as long as we keep invest­ing in neu­ro­science, we going to devel­op bet­ter meth­ods for mea­sur­ing the brain. And when­ev­er we can mea­sure the brain bet­ter, we can build a bet­ter brain decoder.

So, bet­ter brain decoders are in our future. These will be used to decode inter­nal speech. They’ll be portable. They’ll be used for brain-aided CAD cam, dream decod­ing, all kinds of things. And these bring up inter­est­ing, I think, neur­al eth­i­cal issues hav­ing to do with pri­va­cy and use of infor­ma­tion. You know, how are you going to pro­tect your brain infor­ma­tion, who has rights to use it, what will it be used for, and how do we con­trol that?

Thank you very much for your time.