Joey Eschrich: Thank you Ed and thank you to the first pan­el for set­ting a high bar and kick­ing us off and mak­ing me ner­vous. So I’m here to talk to you about a real­ly bad Frankenstein adap­ta­tion that I love, Splice. Has any­body seen this movie? Show of hands. Ah, there’s like six or sev­en Spliceheads in here. Very exciting.

Okay. Splice is a sci­ence fiction/horror hybrid. It was released in 2009. And the film fol­lows the efforts of a mar­ried cou­ple of genet­ic engi­neers played by Adrien Brody and the very tal­ent­ed Sarah Polley, who work for a big phar­ma­ceu­ti­cal com­pa­ny, and their job is to cre­ate genet­ic hybrid crea­tures for med­ical appli­ca­tions. They start off cre­at­ing these kind of worm-like beings. But they’re not sat­is­fied with that, and so they decide to splice human DNA into the mix. And they’re they’re hop­ing in a kind of Victor Franenstein‑y, hand-wavy way to like, rev­o­lu­tion­ize the human race, right. Like they want to cre­ate an organ­ism that will pro­duce genet­ic mate­r­i­al that could cure can­cer, that could cure Parkinson’s, that would you know, in some again very hand-wavy way just solve all the prob­lems that we have.

And you know, they end up cre­at­ing some­thing sen­tient and it’s kind of cute in a creepy squid-like way. And so they decide to raise it in secret, of course. Because as Nancy said, some­thing hor­ri­ble has to hap­pen right off the bat or else you don’t have a sto­ry. So Splice is a mod­ern day Frankenstein sto­ry. And for those of you who are sort of sci­ence fic­tion and hor­ror heads, it’s crossed with the grue­some bio­hor­ror of clas­sic sci­ence fic­tion movies Alien and The Fly.

It’s also frothy and over­wrought. It’s a lit­tle nuts. It goes total­ly off the rails near the end. And that messi­ness is pre­cise­ly why I love it so much. I think it in and bad movies like it—bad but kin­da smart movies like it—tell us a lot about the moment we live in. And in this case I think about the sense of dis­trust and para­noia we have about biotech­nol­o­gy and these oth­er Frankensteinian tech­nolo­gies like AI and geo­engi­neer­ing and things like that in this moment, as we’ve start­ed to talk about already, of great pos­si­bil­i­ty and per­haps great per­il as well.

So in adapt­ing Frankenstein to this con­tem­po­rary moment of actu­al human/pig hybrids, for those of you who have been read­ing your sci­ence and tech news week, with design­er babies—as Nancy talked about—on the hori­zon, the film­mak­ers behind Splice make impor­tant deci­sions about which ele­ments of Shelley’s nov­el to car­ry through and which to trans­form or leave out. You know, just like any adapters of a myth or well-worn sto­ry, they want to tai­lor it to their own social and in this case tech­no­log­i­cal moment.

And my basic premise is these deci­sions are real­ly mean­ing­ful. And in this case they shape the themes and eth­i­cal mes­sages of the film, and they shape the ways that it departs from its source mate­r­i­al. And so today I want to talk about one real­ly impor­tant depar­ture that Splice makes from Shelley’s nov­el as a way to kind of set up this pan­el. My pan­el is about unin­tend­ed consequences.

So with­out fur­ther ado, here’s a brief clip. And this is hap­pen­ing when the crea­ture, which is devel­op­ing at a vast­ly accel­er­at­ed rate, is very young.

[clip was exclud­ed from recording]

Rush out and see it, seriously. 

So, names are real­ly impor­tant. And giv­ing some­thing a name, whether it’s a human or a child or a pet or like, your car—right, an inan­i­mate object—it lets us imbue it with a per­son­al­i­ty. To see it as an inde­pen­dent being, with goals, and emo­tions, deserv­ing of our atten­tion and affec­tion. It’s no sur­prise that so many of our friend­ly tech­nol­o­gy con­glom­er­ates these days are cre­at­ing vir­tu­al assis­tants that have names and per­son­al­i­ties. They’re encour­ag­ing us to build emo­tion­al con­nec­tions with their brands and to kind of imbue those brands with all kinds of asso­ci­a­tions about desires and sens­es of humor and things like that.

In Frankenstein, Shelley very inten­tion­al­ly has Victor nev­er give his cre­ation a name. And I think this is real­ly, indeed, quite inten­tion­al. It’s awk­ward, I think, as a nov­el­ist to have a major char­ac­ter with no name. And it makes the writ­ing hard­er. When refer­ring to the crea­ture, Shelley has Victor use a bunch of dif­fer­ent sub­sti­tutes for a name. He calls the crea­ture a wretch, a demon, a mon­ster, and many oth­er ter­ri­ble, insult­ing things.

Shelley goes to all this trou­ble, I think, because the lack of a name sym­bol­izes in a real­ly pow­er­ful way Victor’s rejec­tion of the crea­ture. He aban­dons it right after he brings it to life. He makes no attempt to care for it, to teach it, to help it accli­mate to the world. In the nov­el the crea­ture becomes vio­lent and venge­ful, pre­cise­ly because he’s reject­ed, first by Victor then by oth­er peo­ple, large­ly because he’s so large, so ugly. He’s scary-looking, right. His lack of a name brings home the idea that he’s barred and shunned from human soci­ety, and the pain of that exclu­sion is what turns him bad—he’s not born bad.

Which brings us to Splice. And on the oth­er hand, in this movie—you can start to see it here—Dren is social­ized, edu­cat­ed, loved. Later in the film the sci­en­tists hide with her in a barn, where they cre­ate a sort of grotesque, Lynchian par­o­dy of a tra­di­tion­al 50s sub­ur­ban nuclear fam­i­ly. This movie has a kind of dark comedic under­side to it and it real­ly comes out in this pas­tiche of nuclear fam­i­ly life.

And these aren’t per­fect par­ents by a long­shot. But they do try, and they care for Dren. They screw up a lot but they try. And you can real­ly see of course, in this clip Sarah Polley’s char­ac­ter start­ing to real­ly build a bond with this crea­ture. And this is a real­ly piv­otal scene, because you can see in the con­flict between the two sci­en­tists that this is the start of the crea­ture tran­si­tion­ing from being a spec­i­men to being a daugh­ter. That name ​“spec­i­men” real­ly becomes this stick­ing point between the two of them.

But of course this ends in vio­lent may­hem. This movie ends hor­ri­bly, just like Frankenstein, with death and with a real­ly shock­ing, bru­tal sex­u­al assault, actu­al­ly. Sarah Polley’s char­ac­ter ends up alone and despon­dent, just like Victor at the end of the nov­el. So we end up in the same place.

And so to go back to the nov­el, the les­son I drew from it is that Victor’s sin— This is one read­ing, any­way. That Victor’s sin was­n’t in being too ambi­tious, not nec­es­sar­i­ly in play­ing God. It was in fail­ing to care for the being he cre­at­ed, fail­ing to take respon­si­bil­i­ty and to pro­vide the crea­ture what it need­ed to thrive, to reach its poten­tial, to be a pos­i­tive devel­op­ment for soci­ety instead of a disaster.

Splice on the oth­er hand has this just very dif­fer­ent eth­i­cal pro­gram. It has a very dif­fer­ent les­son for us. It says that some lines should­n’t be crossed. Some tech­nolo­gies are too dan­ger­ous to med­dle with. It’s pos­si­ble for sci­en­tists, these sort of well-meaning sci­en­tists who we kin­da like and you know, we like the actors, they can fall vic­tim to hubris. They can shoot too high. And even though they try their best, again the exper­i­ment ends in blood and sor­row. These peo­ple, these char­ac­ters, do some­thing tru­ly ground­break­ing and they fail to pre­dict and under­stand the con­se­quences of their actions. They avoid Victor’s mis­take. They stick around and hold the crea­ture close. But the unin­tend­ed con­se­quences of their actions are still catastrophic.

And as we’ve already start­ed to talk about, we’re in a moment when these Frankensteinian tech­nolo­gies seem to be becom­ing more and more real­i­ty. AI, genet­ic engi­neer­ing, robot­ics, geo­engi­neer­ing, promise to make us health­i­er and more effi­cient, and even help to com­bat the exis­ten­tial threat of cli­mate change.

But Splice warns us that if we try to do these rad­i­cal­ly ambi­tious things right, and we make an earnest effort to do them right, might unleash ter­ri­ble unin­tend­ed con­se­quences any­way. We might wipe out the econ­o­my. We might give rise to the robot upris­ing that every­body likes to ref­er­ence in their Future Tense pieces. We might wreck our envi­ron­ment even faster. And for Splice it’s just not about how respon­si­bly we do sci­ence or whether we stick around and care and love. It’s about the idea that some inno­va­tions are just a bridge too far.

And so to help me con­tin­ue to explore this theme of unin­tend­ed con­se­quences, I would like to wel­come our three expert pan­elists to the stage. First Sam Arbesman is the Scientist in Residence at Lux Capital, and the author of the book Overcomplicated: Technology at the Limits of Comprehension. Susan Tyler Hitchcock is the Senior Editor of books for the National Geographic Society and the author of the book Frankenstein: A Cultural History, which has just been immense­ly help­ful to me in under­stand­ing and untan­gling all of this. And Cara LaPointe is an engi­neer who has worked with autonomous sys­tems for both sci­ence and defense appli­ca­tions and devel­op­ment, field­ing oper­a­tions, and pol­i­cy devel­op­ment. Thank you so much for being here with me.

Joey Eschrich: So I’m sort of inter­est­ed, whether you’re new to Frankenstein, rel­a­tive­ly new like Patric, or whether you are kind of like some­one who’s lived and breathed Frankenstein your whole life, what got you inter­est­ed in the first place? Susan, you have this entire very ency­clo­pe­dic and help­ful book about the Frankenstein phe­nom­e­non. Sam, your work with inven­tors and tech­nol­o­gy star­tups seems to me to be evoca­tive of some of the themes of the sto­ry, these cre­ators at the cusp of some­thing new. And Cara, I’m guess­ing that there’s some con­nec­tion between your work with autonomous sys­tems and the autonomous sys­tems that we see in the nov­el in the 19^th cen­tu­ry. So I’m inter­est­ed to hear from each of you kind of what res­onates with you to start us off.

Susan Tyler Hitchcock: So, my fas­ci­na­tion with Frankenstein goes back to my graduate—well no, real­ly my edu­ca­tion, my fas­ci­na­tion with the lit­er­a­ture of the Romantics, the British Romantics. They rep­re­sent a time of cul­ture wars as inter­est­ing as the 60s, when I start­ed my fas­ci­na­tion with these char­ac­ters and their literature. 

And also today. I mean, there were a lot of amaz­ing things hap­pen­ing in their day, and I began with an inter­est in Percy Bysshe Shelley. I ulti­mate­ly taught human­i­ties to engi­neer­ing school stu­dents, and I had the great oppor­tu­ni­ty Halloween day of teach­ing a class on Frankenstein. And for that class, I brought—I actu­al­ly wore—a Halloween mask. Green, ugly, plas­tic Frankenstein mask. And we start­ed talk­ing about the dif­fer­ence between the nov­el and the cur­rent cul­tur­al inter­pre­ta­tion. And that’s what start­ed me. From that point on I start­ed col­lect­ing Frankensteiniana. And I have hun­dreds of objects. And then I wrote a book.

Eschrich: We should’ve done this at your house. Sam, how about you?

Hitchcock: I have them hid­den away.

Samuel Arbesman: So I guess my inter­est in the themes of Frankenstein, the themes of I guess soci­etal impli­ca­tions of tech­nol­o­gy more gen­er­al­ly, began I guess through influ­ences from my grand­fa­ther. My grand­fa­ther, he’s 99. He’s actu­al­ly been read­ing sci­ence fic­tion since essen­tial­ly the mod­ern dawn of the genre. He read Dune when it was seri­al­ized, before it was actu­al­ly a book. He gave me my first copy of the Foundation tril­o­gy. And a lot of the sci­ence fic­tion that I’ve been espe­cial­ly drawn to is the ones that kind of real­ly try to under­stand a lot of the soci­etal impli­ca­tions of the gad­gets, as opposed to just the gad­gets of the future themselves. 

And in my role at Lux— It’s a VC firm that does early-stage invest­ments and kind of— I guess any­thing that’s at the fron­tier of sci­ence and tech­nol­o­gy. And so one of my roles there is involved in try­ing to con­nect groups of peo­ple that are not tra­di­tion­al­ly con­nect­ed to the world of ven­ture and star­tups. And so relat­ed to that, when a lot of tech­nol­o­gists and peo­ple in the world of Silicon Valley are build­ing things, there’s often this kind of tech­noutopi­an sense of like, you build some­thing, it’s this unal­loyed good, it must be wonderful. 

But of course there’s often a lot peo­ple who are think­ing about the social, reg­u­la­to­ry, eth­i­cal, legal impli­ca­tions of all these dif­fer­ent tech­nolo­gies. But they’re often in the world of acad­e­mia, and they’re often not talk­ing to the peo­ple who are build­ing these things in the start­up world. And so one of things I’ve actu­al­ly been doing is try­ing to con­nect these two dif­fer­ent worlds togeth­er to make real­ly sure that both par­ties are as engaged as possible.

And actu­al­ly, even going back to the sci­ence fic­tion part. Since sci­ence fic­tion more holis­ti­cal­ly looks at a lot of the impli­ca­tions of these kinds of things as opposed to just say­ing oh, the future is the fol­low­ing three gad­gets and what they’re going to be, sci­ence fic­tion is real­ly good at say­ing, ​“Okay, here is a sce­nario. Let’s actu­al­ly play it out,” I’ve actu­al­ly been work­ing to try to get sci­ence fic­tion writ­ers involved in talk­ing to the world star­tups and real­ly try­ing to make them think about these kinds of things. I don’t think I’ve actu­al­ly got­ten peo­ple involved in like, explic­it­ly Frankensteinian sto­ries involved, but yes—

Eschrich: Everybody who gets mon­ey from you guys has to watch Splice before [inaudi­ble].

Cara LaPointe: Well it’s inter­est­ing that Sam talks about this kind of holis­tic approach. So, I’m an auton­o­my sys­tems engi­neer, but I’ve worked in devel­op­ment sys­tems, using sys­tems, the pol­i­cy impli­ca­tions. So I kind of come at autonomous sys­tems from a lot of dif­fer­ent angles. So what’s real­ly inter­est­ing to me about the Frankenstein sto­ry is it real­ly seems to delve into the idea of the ethics of cre­ation. Should we or should we not cre­ate the technology?

But I think it was brought up in the first pan­el, when it comes to auton­o­my, when it comes to arti­fi­cial intel­li­gence, this tech­nol­o­gy is being devel­oped. So I think it’s real­ly more pro­duc­tive to think about okay, what is the ethics of how, where, when, why, you’re going to use these types of tech­nolo­gies. Because I think some­one said the genie’s out of the bot­tle, right. These things are being devel­oped. So that’s what to me is very inter­est­ing, is kind of mov­ing from that con­ver­sa­tion about cre­at­ing, to how are these tech­nolo­gies actu­al­ly used. 

The thing about autonomous sys­tems is you start to move into a world where we’ve use machines for a long time to do things, right. But now we’re start­ing to get to a place where machines can move into the cog­ni­tive space in terms of the decision-making space. There’s a real­ly inter­est­ing con­struct that we use in the defense world some­times called the OODA Loop—Observe, Orient, Decide, and Act. It’s just kind of a way to describe doing any­thing. So observ­ing the world, you’re sens­ing the things around you. Orienting is kind of under­stand­ing what you’re sens­ing. And then decid­ing what you want to do to achieve what­ev­er your pur­pose is. And then you act.

We’ve use machines for a long time to do the sens­ing. We have all kinds of cam­eras and oth­er types of sen­sors. And we’ve used machines to act for us for a long time. But what’s real­ly inter­est­ing with tech­nol­o­gy today is we’re on this cusp where these cog­ni­tive functions—machines can move into this cog­ni­tive space. So fig­ur­ing out kind of where and when and how we want machines to move into the cog­ni­tive space, that’s what I think is real­ly inter­est­ing. And I think even from very ear­ly on, Frankenstein was bring­ing up those ideas of when you bring some­thing into that cog­ni­tive space. So that’s why I think it’s pret­ty fascinating.

Eschrich: So, Susan I was hop­ing you could ground is in how peo­ple at Mary Shelley’s his­tor­i­cal moment are think­ing about unin­tend­ed con­se­quences. As Ed said the word ​“sci­en­tist” isn’t even in use yet. But are there oth­er ways peo­ple are think­ing and talk­ing about the ethics of cre­ation and respon­si­bil­i­ty. And how is Shelley kind of build­ing on the con­text that she’s in to kind of cre­ate this theme in Frankenstein and devel­op it.

Hitchcock: Yeah, well there’s an inter­est­ing inter­sec­tion between her lega­cy from her par­ents and the sci­ence going on. Her father I find a real­ly impor­tant influ­ence on the nov­el because William Godwin was real­ly— I think of him as being the father of our mod­ern day lib­er­al con­cept that peo­ple aren’t evil, that bad actions come because peo­ple have been influ­enced by hatred, by anger, by neg­a­tive out­side influ­ences. That is, that evil is made not born. And I think that that real­ly car­ries through. It’s as if Mary Shelley want­ed to ani­mate that phi­los­o­phy of her father’s.

But at the same time, there are these fas­ci­nat­ing exper­i­ments going on at the time. Galvani, the whole idea of the spark of life, what is the spark of life, in these amaz­ing exper­i­ments. Not only with frogs, which is sort of the famous one, but even with corpses. Introducing elec­tri­cal stim­uli to bod­ies and mak­ing them move, mak­ing the eyes of a corpse open up, mak­ing it sit up, that sort of thing. Those things were being done at the time, and they were kind of like sideshow events that the pub­lic would go to.

So there was a lot of sci­ence hap­pen­ing that opened up a lot of ques­tions of should we real­ly be doing this, and that is a lot of the inspi­ra­tion behind Frankenstein as well. You don’t real­ly see that hap­pen­ing in the nov­el, but it’s so inter­est­ing that instant­ly the retelling of the sto­ries bring elec­tric­i­ty in as the ​“spark of life.” 

Eschrich: So, that point about social con­text and those sort of social con­struc­tion­ist beliefs of William Godwin is real­ly appro­pri­ate, I think, and also some­thing that her moth­er Mary Wollstonecraft was very adamant about. She wrote a lot about wom­en’s edu­ca­tion and the idea that the way that women were edu­cat­ed social­ized them to be sub­mis­sive and sort of…she called them ​“intel­lec­tu­al­ly mal­formed” and things like that. This idea that they were kind of vio­lent­ly social­ized away from being intel­lec­tu­als and cit­i­zens and full mem­bers of society. 

Both Sam and Cara, I think you both have some inter­ac­tion, Sam through your book and through Lux, and Cara through year your engi­neer­ing work, with sys­tems that learn and adapt. The sys­tems that work in a social con­text and have to solve prob­lems in com­plex ways. So, is this sort of social con­struc­tion­ist think­ing, this idea that the social con­text for the oper­a­tion of these tech­nolo­gies actu­al­ly affects the way they work and what they become… How do we kind of react to that in this moment?

Samuel Arbesman: One of the clear exam­ples this kind of thing is the whole like, arti­fi­cial intel­li­gence, machine learning—especially with deep learn­ing, how this is like, we’re hav­ing a moment of deep learn­ing right now. And with these sys­tems and even though the algo­rithms of how they learn are well under­stood, often­times the result­ing sys­tem based on once you kind of pour a whole bunch of data into it, the result­ing thing might actu­al­ly be very pow­er­ful, it might be very pre­dic­tive. You can iden­ti­fy objects and images, or help cars dri­ve by them­selves, or do cool things with voice recog­ni­tion. How they actu­al­ly work, kind of the under­ly­ing com­po­nents and the actu­al thread with­in the net­works, are not always entire­ly under­stood. And often­times because of that, there’s moments when you’re actu­al­ly just like, the cre­ators are sur­prised by their behavior.

So we were talk­ing about this ear­li­er of like, there’s the Microsoft chat bot Tay, I guess a lit­tle more than a year ago, when it was designed to be a teenage girl, end­ed up being a white suprema­cist. It was because there was this mis­match between the data that they thought the sys­tem was going to get and what it actu­al­ly did get. And there was this like…the social­iza­tion in this case was wrong. And you can actu­al­ly see this also in sit­u­a­tions with IBM Watson, where the engi­neers who were involved in Watson want­ed the sys­tem to bet­ter under­stand slang, just kind of every­day lan­guage. And so in order to teach it that, they kind of poured in Urban Dictionary. And then it end­ed up just curs­ing out its cre­ators. And that was also not intended.

And so I think there’s a lot of these kinds of things of rec­og­niz­ing that the envi­ron­ment that you expose a sys­tem to, and the way it kind of assim­i­lates that, is going to affect its behav­ior. And some­times you only dis­cov­er that when you actu­al­ly inter­act with it. And so I think that’s kind this iter­a­tive process of— As opposed to in Frankenstein, where it’s like you build the thing, build the crea­ture, hope­ful­ly it’s per­fect. Oh no, it sucks. I kind of give up and run away. 

I think in tech­nol­o­gy ide­al­ly there’s this iter­a­tive process of under­stand­ing. You build some­thing. You learn from it. You actu­al­ly kind of find out that there’s a mis­match between how you thought It going to work and how it actu­al­ly does work, embod­ied by glitch­es and fail­ures and bugs. And then you debug it and make it bet­ter. So rather than kind of just view­ing it as we ful­ly under­stand it or we nev­er under­stand it, there’s this con­stant learn­ing process and social­iza­tion kind of real­ly mak­ing sure you have the right envi­ron­ment to make sure it gets as close as pos­si­ble to the thing you actu­al­ly want it to be. 

Hitchcock: There’s a lack of knowl­edge, though, of what that the forces that you’re putting on to— Whether it’s the crea­ture or the sys­tems. You know, maybe we don’t have the capa­bil­i­ty of ful­ly under­stand­ing, or ful­ly know­ing. Like pour­ing the Urban Dictionary in. They did­n’t know what influ­ences they were mak­ing on the thing.

Arbesman: And actu­al­ly relat­ed to that, there’s this idea from physics. It’s this term of when look­ing at a com­plex tech­no­log­i­cal sys­tem or just com­plex sys­tems in gen­er­al, of robust yet frag­ile. So the idea that when you build a sys­tem, it’s often extreme­ly robust to all the dif­fer­ent even­tu­al­i­ties that you’ve planned in, but it can be incred­i­bly frag­ile to pret­ty much any­thing you did­n’t think about. And so there’s all these dif­fer­ent excep­tions and edge cas­es that you’ve built in and you’re real­ly proud of han­dling them, and sud­den­ly there’s some tiny lit­tle thing that just makes the entire thing cas­cade and fall apart. And so yeah, you have to be very wary of rec­og­niz­ing the lim­its to how you actu­al­ly designed it. 

LaPointe: I think it’s real­ly inter­est­ing to think about the sys­tem. We’re using the word sys­tem to talk about a machine that’s being cre­ative. When I think of ​“sys­tem,” I actu­al­ly think of the inter­ac­tion between machines and peo­ple. Time and time again in his­to­ry, tech­nol­o­gy comes in, inno­v­a­tive emerg­ing tech­nolo­gies come in and actu­al­ly change the fab­ric of our lives. And I think that the whole Industrial Revolution, right. I live thir­ty miles out­side of DC but I can dri­ve in every day. I mean, that would be unheard of cen­turies ago. 

But then think of the per­son­al com­put­er, think of the Internet. You actu­al­ly live your life dif­fer­ent­ly because of these tech­nolo­gies. And so we’re on the cusp of the same kind of social change, when it comes to autonomous sys­tems. Autonomy is going to change the fab­ric of our lives. I don’t know what it’s going to look like. But I can tell you it is going to change the fab­ric of our lives over the com­ing decades. So it’s inter­est­ing when you’re talk­ing about a sys­tem to under­stand that it’s not kind of there’s one way, it’s not how we’re just teach­ing a machine and teach­ing a sys­tem. You’ve got to under­stand how kind of we col­lec­tive­ly as a sys­tem evolve. And so I think that’s just an inter­est­ing way to kind of think about fram­ing it as you move for­ward talk­ing about these types of technologies.

Hitchcock: What do you mean when you say auton­o­my is going to be shap­ing our future? What is auton­o­my, that you’re talk­ing about?

LaPointe: So, auton­o­my— You know what, there is no com­mon def­i­n­i­tion of auton­o­my. Many days of my life have been spent in the debate about what auton­o­my and autonomous means. So you know, at some point you just moved beyond. But auton­o­my is when when you start to get machines to move into the cog­ni­tive space. Machines can start mak­ing deci­sions about how they’re going to act. 

So the exam­ple I love to use, because a lot of peo­ple have had them, seen them, the Roomba vac­u­ums, right? I got a Roomba—I love it. But it’s fun­ny because when you think of a tra­di­tion­al vac­u­um, you have a tra­di­tion­al vac­u­um, you turn it on and what’s it doing? Its job is to suck up the dirt, right. And you move and decide where it’s going to go. Okay well, a Roomba, what’s its job? Its job is to suck up dirt and clean the floor. But it now decides how the pat­tern it’s going to fol­low around your room, or around you what­ev­er the set space is, to clean that. So auton­o­my is, as you’re start­ing to look at machines start­ing to get into the deci­sion space…

And I think one of the things that we real­ly need to address and fig­ure out as these machines come in— And it’s much more than just a tech­ni­cal chal­lenge, it’s all these oth­er things we’re talk­ing about— …is how do we trust these sys­tems? You trust some­body when you can rely on it to be pre­dictable, right. And we have this kind of intu­itive trust of oth­er peo­ple, and we know that they’re not going to be per­fect all the time. We have kind of this under­stand­ing, and your under­stand­ing of what a tod­dler’s going to do is dif­fer­ent than what a teenager’s going to do, is dif­fer­ent than what’s an adult going to do. So you have kind of this intu­itive knowledge.

So as we’re devel­op­ing these autonomous sys­tems that can act in dif­fer­ent ways, it’s real­ly impor­tant for us to also spend a lot of time devel­op­ing and under­stand­ing what this trust frame­work is, for sys­tems. So as Sam was say­ing, when you have an autonomous sys­tem, when I turn that Roomba on I don’t know the path it’s going to take around the room. I don’t know if it goes straight or goes left or does a lit­tle cir­cle. I have three kids and a dog, so it does a lot of the lit­tle cir­cles where it finds those dirt patch­es, right. I don’t know just look­ing at it instan­ta­neous­ly if it’s doing the right thing. I have to kind of wait to see as it’s done its job if it did the right thing. So it’s decid­ing or fig­ur­ing out how you real­ly trust sys­tems and test and eval­u­ate sys­tems is going to be fun­da­men­tal­ly dif­fer­ent with autonomous sys­tems, and this to me is one of the real chal­lenges that we are facing—we are fac­ing as a society.

So think about auton­o­my in self-driving cars. A lot of peo­ple like to talk about self-driving cars. And this is a tech­nol­o­gy that is devel­op­ing apace. Well, what are the chal­lenges? The chal­lenges are how do you inte­grate these into the human, exist­ing sys­tem we already have? How do you trust the self-driving cars? I mean, if there’s ever one acci­dent does that mean you don’t trust all self-driving cars? I know a lot of dri­vers who’ve had an accent, and they still are trust­ed to dri­ve around, right. But you know, we don’t have that kind of same lev­el of intu­itive under­stand­ing of what is pre­dictable and reliable.

Arbesman: And to relate to that, with­in machine learn­ing— I was going back men­tion­ing how these sys­tems are mak­ing these some­what eso­teric deci­sions where they work, but we’re not always entire­ly sure why they’re mak­ing these things and that makes it dif­fi­cult to trust them. And so there’s been this move­ment of try­ing to cre­ate more explain­able AI, actu­al­ly kind of gain­ing a win­dow into the decision-making process of these systems. 

And so relat­ed to the self-driving cars, it’s one thing when you— We have a pret­ty decent sense of like…intuitive sense of mind of like, when I meet some­one at an inter­sec­tion how they’re going to kind of inter­act with me in my car ver­sus their car. They’re not entire­ly ratio­nal but I kind of have a sense. But if it’s a self-driving car, I’m not real­ly entire­ly sure the kind of decision-making process that’s going on. And so if we can cre­ate cer­tain types of win­dows into under­stand­ing the decision-making process that’s real­ly important. 

And I think back in terms of the his­to­ry of tech­nol­o­gy, and so the first com­put­er my fam­i­ly had was the Commodore VIC-20. And I guess William Shatner called it the won­der com­put­er of the 1980s. He was the pitch­man for it. I was too young to pro­gram at the time, but one of the ways you would get pro­grams is you had these things called type-ins. You would actu­al­ly just get a mag­a­zine and there would be code there and you would just type the actu­al code in. 

And so even though I did­n’t know how to pro­gram, I could see this clear rela­tion­ship between the text and what the com­put­er was doing. And now we have these real­ly pow­er­ful tech­nolo­gies, but I no longer have that con­nec­tion. There’s a cer­tain dis­tance between them, and I think we need to find ways of cre­at­ing sort of a gate­way into kind of peek­ing under the hood. And I’m not entire­ly sure what those things would be. It could be a sym­bol, maybe just like a progress bar—although I guess those are only ten­u­ous­ly con­nect­ed to real­i­ty. But we need more of those kinds of things in order to cre­ate that sort of trust.

Eschrich: Yeah, it seems to me that the rul­ing aes­thet­ic is mag­ic, right. To say oh, you know Netflix, it works accord­ing to mag­ic. The iPhone, so much of what hap­pens is under the hood and it’s sort of for your pro­tec­tion. You don’t need to wor­ry about it. But I think we’re real­iz­ing espe­cial­ly with some­thing like cyber­se­cu­ri­ty, which is a big unin­tend­ed con­se­quences prob­lem, right. We offload every­thing onto the Internet to become more effi­cient, and sud­den­ly every­thing seems at risk and inse­cure, in a way. We’re real­iz­ing we might need to know a lit­tle bit more about how this stuff actu­al­ly works. Maybe mag­ic isn’t good enough all the time.

Arbesman: And one of the few times you actu­al­ly learn about how some­thing works is when it goes wrong. Like some­times the only way to learn about a sys­tem is through the fail­ure. And you’re like, oh! It’s like I don’t know, the chat bot Tay is becom­ing racist. Now we actu­al­ly real­ize it was kind of assim­i­lat­ing data in ways we did­n’t expect. And yeah, these kind of bugs are actu­al­ly teach­ing us something. 

Hitchcock: Which brings us back to Frankenstein.

Arbesman: Yes.

Eschrich: Thank you. Thank you so much.

Hitchcock: Because Victor was so fas­ci­nat­ed and excit­ed and proud and delight­ed with what he was doing. And then when he saw what he had done, it’s like…checking out. Horrible. End of his fas­ci­na­tion and delight. And begin­ning of his downfall.

Eschrich: I want­ed to say, and I’m going to kind of prompt you, Sam. That Frankenstein’s very…haughty about his real­ly— You know, I think you can read it psy­cho­log­i­cal­ly as a defense mech­a­nism. But he’s so haughty lat­er about the crea­ture. He’s very dis­dain­ful of it, he sort of dis­tances him­self from it. All the unin­tend­ed con­se­quences it caus­es, he sort of works real­ly hard to con­vince his lis­ten­ers and the read­er that he’s not respon­si­ble for that. As if not think­ing ahead some­how absolves him.

But Sam, in your book Overcomplicated, you talk a bit about this con­cept of humil­i­ty, which dates all the way back to the Medieval Period. And I feel like the con­ver­sa­tion we’ve been hav­ing reminds me of that con­cept. Talking about how to live with this com­plex­i­ty in a way that’s not scorn­ful, but that’s also not kind of mys­ti­fied and helpless.

Arbesman: Yeah, so when I was writ­ing about humil­i­ty in the face of tech­nol­o­gy I was kind of con­tract­ing it with two extremes which we often tend towards when we think about or when we’re kind of con­front­ed with tech­nol­o­gy we don’t ful­ly under­stand. So, one is the fear in the face of the unknown, and we’re like oh my god self-driving cars are going to kill us all, the robots are going to rise up. 

And the oth­er extreme is kind of like the mag­ic of Netflix or the beau­ti­ful mind of Google. This almost like reli­gious rev­er­en­tial sense of awe. Like these things are beau­ti­ful, they must be per­fect… And of course, they’re not per­fect. They’re built by imper­fect beings. Humans.

And these two extremes, the down­side of both of these is they end up cut­ting off ques­tion­ing. When we’re so fear­ful that we can’t real­ly process the sys­tems that we’re deal­ing with, we don’t actu­al­ly try to under­stand them. And the same thing, if we think the sys­tem is per­fect and won­der­ful and wor­thy of our awe, we also don’t query. And so humil­i­ty, I’ve kind of used that as like the sort of halfway point, which actu­al­ly is pro­duc­tive. It actu­al­ly ends up allow­ing us to try to query our sys­tem, but rec­og­nize there are going to be limits. 

And so going back to the Medieval thing, I kind of bring this idea from my Maimonides, from the 12^th cen­tu­ry philosopher/physician/rabbi. And in one of his books, The Guide for the Perplexed, he wrote about sort of like, there are clear lim­its to what we can under­stand and that’s fine. He had made his peace with it. And I think in lat­er cen­turies there was sort of a sci­en­tif­ic tri­umphal­ism that if we apply our minds the world around us, we’ll under­stand everything. 

And in many ways we’ve actu­al­ly been extreme­ly suc­cess­ful. Which is great. But I think we are rec­og­niz­ing that there are cer­tain lim­its, there are cer­tain things we’re not going to be able to under­stand. And I think we need to import that into the tech­no­log­i­cal realm and rec­og­nize that even the sys­tems we our­selves have built, there are cer­tain cas­es where— And it’s one thing to say okay, ​“I don’t under­stand the iPhone in my pock­et.” But if no one under­stands the iPhone com­plete­ly, includ­ing the peo­ple who cre­at­ed it and work with it on a dai­ly basis, that’s an inter­est­ing sort of thing. 

And I think this humil­i­ty is pow­er­ful in the sense that it’s bet­ter to work that and rec­og­nize our lim­its from the out­set so that we can sort of build upon them and con­stant­ly try to increase our under­stand­ing, but rec­og­nize that we might not ever ful­ly under­stand a sys­tem. As opposed to think­ing we are going to ful­ly under­stand it, and then be blind­sided by all of these unin­tend­ed consequences.

Eschrich: So Susan, I’m going query you on this first. Because I feel like the oth­er two are going to have stuff to say too, but I want to get the Frankenstein angle on it. So what do we do? Like, should we… How do we— What does Frankenstein tell us about how we pre­pare for unin­tend­ed con­se­quences since they’re inevitable, clear­ly. Like, we’re sort of inno­vat­ing and dis­cov­er­ing very quick­ly, things are chang­ing quick­ly. Should we ask sci­en­tists and engi­neers to reg­u­late them­selves? Should we cre­ate rigid laws? Do researchers need more flex­i­ble norms that they agree— You know, what does Frankenstein— You know, what does this mod­ern myth we’re con­stant­ly using to frame these debates kind of have to say about them?

Hitchcock: What does the—oh, gosh. 

Eschrich: I have in my mind some­thing that you said.

Hitchcock: You do? Maybe you should say it, because— 

Eschrich: You said some­thing about— Well, I want to prompt you. So, you said when were talk­ing in advance and I was pick­ing your brains about this, about how secre­tive Victor is. About how he removes him­self from his colleagues.

Hitchcock: Well, it’s true. Yes, indeed. Victor is rep­re­sen­ta­tive of a sci­en­tist who works in secret, all by him­self, does not share. And as a mat­ter of fact even the James Whale film, it’s the same thing. I mean, Victor goes up into a tow­er and he locks the door, and his beloved Elizabeth has to knock on the door to ever see him. I mean, it is per­pet­u­at­ed in the retelling of Frankenstein, this whole idea of a sci­ence that is solo and not shared.

And you know, thanks for the prompt because maybe that’s a good idea, that the we share the sci­ence, that we talk about it. And I think shar­ing it not only with oth­er sci­en­tists but also with philoso­phers, psy­chol­o­gists, human­ists. You know, peo­ple who think of— And bioethi­cists. People who think about these ques­tions from dif­fer­ent van­tage points, and talk about them as the sci­ence is being devel­oped. That is about what human beings could do, I think. That’s about the best we could do.

LaPointe: I think this idea of shar­ing is real­ly crit­i­cal. So, from the kind of developer/operator perspective—and I come from kind of a Navy back­ground, mil­i­tary back­ground, it’s real­ly impor­tant that you get the peo­ple who are devel­op­ing sys­tems talk­ing to the peo­ple who are using sys­tems, right. We get into trou­ble when peo­ple have an idea of ​“Oh, this is what some­body would want,” and they go off and devel­op it kind of in iso­la­tion, right. Maybe not secret, but in iso­la­tion, just…there’s a lot of kind of stovepipes and large orga­ni­za­tions. And it’s real­ly impor­tant to cre­ate these robust feed­back loops. 

And we have this say­ing in the Navy that sailors can break any sys­tem, so you always when you build some­thing you want to make it sailor-proof, right. But it’s real­ly a fab­u­lous thing to take a new idea, take a new design, take a pro­to­type, and give it to sailors. Because there’s noth­ing like 19 and 20 year-olds to com­plete­ly take apart what you just gave them, tell you why all the rea­sons you thought it was going to be great are com­plete­ly stu­pid and use­less, and tell you the thou­sand oth­er things you can do with a system. 

So I think this kind of idea of shar­ing in the devel­op­ment— So, shar­ing in terms of talk­ing to peo­ple who are oper­a­tors, talk­ing to peo­ple who are the infra­struc­ture devel­op­ers, right. You think about, kind of going back to the self-driving cars, think about how we inter­act with the dri­ving infra­struc­ture. When you come to a stop light, what are you doing? You are visu­al­ly look­ing at a stop light that will tell you to stop or go. Do you think that is the best way to inter­act with a com­put­er? That’s real­ly real­ly hard. It’s real­ly real­ly hard for a com­put­er to look and visu­al­ly see a different-colored light and take from that the instruc­tions of whether to stop or go. 

So you have to kind of include the peo­ple who are devel­op­ing the infra­struc­ture, and include the pol­i­cy­mak­ers, include the ethi­cists. I mean, you have to bring—back to this holis­tic idea—you have to bring every­body in as you’re devel­op­ing tech­nol­o­gy, to make sure you’re devel­op­ing it in a way that works, in a way that’s use­ful, in a way that’s going to actu­al­ly be the right way to go with tech­nol­o­gy. And I think that’s a real­ly good exam­ple from Frankeinstein, is that because he’s kind of solo and design­ing some­thing that to him is bril­liant, and maybe if he had stopped and talked to any­body about it they would’ve said, ​“Hey maybe that’s not the most bril­liant idea in the world.”

Arbesman: Yeah, and so relat­ed to this, in the open source soft­ware move­ment there’s this max­im of with enough eye­balls all bugs are shal­low. The idea that if enough peo­ple are work­ing on some­thing then all bugs are going to be I guess root­ed out and dis­cov­ered. Which is not entire­ly true. There are bugs that can actu­al­ly be quite seri­ous and last for like a decade or more.

But by and large you want more peo­ple to actu­al­ly be look­ing at the tech­nol­o­gy— And also going back to the robust yet frag­ile idea, that you want to make it as robust as pos­si­ble, and to do that you need as many peo­ple involved to deal with all the dif­fer­ent even­tu­al­i­ties. But you also just need the dif­fer­ent kinds of…like, dif­fer­ent peo­ple from dif­fer­ent modes of think­ing, to real­ly try to under­stand. Not just to make the sys­tem as robust as pos­si­ble but real­ly as well thought-out as pos­si­ble. And I think that’s a real­ly impor­tant thing.

LaPointe: Kind of crowd­sourc­ing your devel­op­ment. If you think about what’s going on with self-driving cars, one of the most impor­tant things that’s hap­pen­ing today that’s going to feed into that is actu­al­ly just the autonomous fea­tures in oth­er cars that are being deployed, and all this information-gathering, that because there are so many peo­ple out there and so many cars out there, and if there are auton­o­my algo­rithms in some of these oth­er cars. 

And lit­tle things—there’s lots of cars today that help you park, they help you do all these oth­er things. They help you stay in the lane, right. And so those can all have unin­tend­ed con­se­quences. But as you learn from that, and the more wide­ly you’re test­ing and test­ing this kind of incre­men­tal approach— You know, I like to say ​“rev­o­lu­tion through evolution,” right. You build a lit­tle, test a lit­tle, learn a lot. And I think that’s a real­ly good way to try to pre­vent unin­tend­ed consequences.

So instead of just talk­ing about man­ag­ing unin­tend­ed con­se­quences when they hap­pen, try to bring as many peo­ple as you can in from dif­fer­ent fields and try to think through what could be pos­si­ble con­se­quences, and try to mit­i­gate them along the way.

Arbesman: And relat­ed to the process of sci­ence more broad­ly, in sci­ence peo­ple have been recent­ly talk­ing a lot about the repro­ducibil­i­ty cri­sis and the fact that there’s cer­tain sci­en­tif­ic research that can’t be repro­ducible. And I think that real­ly speaks to the impor­tance of open­ing sci­ence up and actu­al­ly mak­ing sure we can share data, and actu­al­ly real­ly see­ing the entire process, and putting your com­put­er code online to allow peo­ple to repro­duce all these dif­fer­ent things, and allow peo­ple to actu­al­ly par­take in the won­der­ful messi­ness that is sci­ence as opposed to kind of just try­ing to sweep it under the rug. And I think that’s real­ly impor­tant, to real­ly make sure that every­one is involved in that kind of thing.

Eschrich: So we have time for one more quick ques­tion. I actu­al­ly want to address it to you, Susan, at least first. And hope­ful­ly we’ll get a quick answer so we can go to ques­tions and answers from every­body else.

Arbesman: I’m lis­ten­ing to you all talk about diver­si­fy­ing this con­ver­sa­tion and engag­ing non-specialists, it strikes me that one irony there is that Frankenstein itself, this poi­so­nous fram­ing of Frankenstein as this like ​“don’t inno­vate too far; dis­as­trous out­comes might hap­pen; we might trans­gress the bounds of accept­able human ambi­tion.” That this is actu­al­ly a road­block to hav­ing a con­struc­tive con­ver­sa­tion in a way, right. All of these themes that we’re talk­ing today about today of unin­tend­ed con­se­quences and play­ing God are in fact dif­fi­cult for peo­ple to grap­ple with in big groups. I won­der if you have any thoughts about that, Susan. Other ways to think of the nov­el, maybe, or recode it for people.

Hitchcock: Well, yeah. You know, I think that cul­ture has done the nov­el dis­ser­vice. Because I actu­al­ly think that the nov­el does­n’t end— The nov­el does not end with every­body dead. Nor does Splice, by the way. 

Eschrich: There’s a cou­ple peo­ple alive at the end of Splice. [crosstalk] And of course the com­pa­ny is mas­sive­ly popular.

Hitchcock: Oh, there’s also a preg­nant woman at the end.

Eschrich: That is true.

Hitchcock: Uh huh, that’s what I’m think­ing about. So, Frankenstein ends with the mon­ster, the creature—whatever we want to call him, good or bad—going off into the dis­tance and poten­tial­ly liv­ing for­ev­er. And also, Victor Frankenstein is you know, yes indeed he is say­ing, ​“I should­n’t have done that. And you, Walton,” who is our nar­ra­tor who’s been going off to the North Pole, indeed lis­tens to him, still wants to go to the North Pole, but his crew says, ​“No no, we want to go home. We’re too cold—”

Eschrich: They’re wor­ried they’re going to die.

Hitchcock: Yeah.

Eschrich: Yeah.

Hitchcock: I know. But there are still these fig­ures in the nov­el, both the crea­ture and Walton to some extent, who are still quest­ing. [crosstalk] Still questing.

Eschrich: They have moral agency, to some extent.

Hitchcock: Yeah. And I don’t know why I got onto that from your ques­tion, but—

Eschrich: You refuse to see the nov­el as pure­ly black at the end, I think.

Hitchcock: Yeah. Oh, I know. I was going to say cul­ture had done it a dis­ser­vice because I think cul­ture has sim­pli­fied the sto­ry to say sci­ence is bad, push­ing the lim­its are bad. This is a bad guy, and he should­n’t have done it. And I don’t think that it is that sim­ple, frankly. In the nov­el or today, for that matter.

Joey Eschrich: Alright. Well, I am going to ask if any­body out here has ques­tions for any of our panelists.

Audience 1: Thank you very much for a great dis­cus­sion. I’m curi­ous about what seg­ment of soci­ety real­ly wants the self-driving cars. And one of the con­cerns is that there’ll be a lethar­gy that will come upon the rid­er, per­haps, or the one who’s in the car and such and not real­ly ready to— 

Let’s say your Roomba. If your Roomba could­n’t get through— It would stall because it could­n’t get into it, you’d have to inter­act to reset it or some­thing. So I’m just won­der­ing, in a self dri­ving car if you’re not going to be able to have to do any­thing, then you’re not going to be maybe aware of what’s real­ly going around. So, Musk is the one who start­ed the whole idea, and yet is it going to tar­get just a cer­tain seg­ment of soci­ety as opposed to you know, every­one has to be in a self-driving car?

Cara LaPointe: Well, I’m not going to speak to I think in terms of who’s dri­ving. I think that’s a lot of peo­ple who’ve been dri­ving the self-driving cars. But your idea of when you have peo­ple that were for­mer­ly dri­ving and who are now the pas­sen­gers, I think this is actu­al­ly a real­ly impor­tant issue with autonomous sys­tems is one of the most dan­ger­ous parts with any autonomous sys­tem is the hand­off. The hand­off of con­trol between a machine and between peo­ple. And it does­n’t mat­ter whether you’re talk­ing about cars or oth­er sys­tems. We can be talk­ing about a plane on autopi­lot going to pilot on a plane. It’s a per­fect exam­ple. And it’s that not hav­ing that full sit­u­a­tion­al aware­ness, so when you have this hand­off that’s a real­ly dan­ger­ous time for any system. 

So I think this is one of the chal­lenges and that’s why when I define the sys­tem, I don’t think you can just define the machine, right. You have to define the sys­tem of, how is the sys­tem going to work togeth­er between a machine and a per­son and how they’re going to work together.

Audience 1: We as humans don’t have that capac­i­ty of putting off [inaudi­ble]. We’re not going to [inaudi­ble] ask the machine to fig­ure out. That’s the con­se­quence of that. So I don’t think we humans are wired at that lev­el to under­stand how they all fuse togeth­er and what con­se­quence results from it.

LaPointe: Well I think cog­ni­tive load is a real­ly big issue for engi­neers as well. Just think about we live in an age of so much infor­ma­tion, right. How much infor­ma­tion can a per­son process? And frankly you have data. There’s tons of data. You have so many cen­ters, you can bring in so much data—how do you kind of take that data and get the knowl­edge out of it and turn it into infor­ma­tion. And I think real­ly part of the art of some of this, is how you take so much data and turn it into infor­ma­tion and deliv­er it to the human part of a sys­tem? Or even the machine part of a sys­tem. The right infor­ma­tion at the right time to make the over­all sys­tem successful.

Samuel Arbesman: And relat­ed to that, there’s the com­put­er sci­ence’s Danny Hillis. He’s argued that we were liv­ing in the Enlightenment, and we kind of applied our brain to under­stand the world around us. And we move from the Enlightenment to the Entanglement, this era of like, every­thing hope­less­ly inter­con­nect­ed, we’re no longer ful­ly going to under­stand it. And I think to a cer­tain degree we’ve actu­al­ly been in that world already for some time. It’s not just that self-driving cars are going to her­ald this new era. We’re already there, and I think the ques­tion is how to just actu­al­ly be con­scious of it and try our best to make sure we’re get­ting the rel­e­vant infor­ma­tion and con­stant­ly iter­a­tive­ly try­ing to under­stand our sys­tems as best we can. 

And I think that goes back to it in terms of think­ing about how we approach what under­stand­ing means for these sys­tems. It’s not a bina­ry sit­u­a­tion. It’s not either com­plete under­stand­ing or total igno­rance and mys­tery. There’s a spec­trum of you can under­stand cer­tain com­po­nents, you can under­stand the lay of the land with­out under­stand­ing all of the details. And I think our goal when we design these tech­nolo­gies is to make sure that we have the abil­i­ty to kind of move along that spec­trum towards greater under­stand­ing, even if we nev­er get all the way there. I think that’s in many cas­es fine.

Eschrich: I’d like us to move on to our next question. 

Audience 2: I want to dig in a lit­tle on the dia­logue that we may all agree it would be a good idea to involve more peo­ple in at the start of con­ceiv­ing of these tech­nolo­gies. And ide­al­ly, I think we might agree that some pub­lic moral­i­ty would be a good ele­ment to include. But say hypo­thet­i­cal­ly we lived in a soci­ety where prac­ti­cal­ly we’re not real­ly good at hav­ing con­ver­sa­tions among the pub­lic that are thorny and espe­cial­ly that include tech­ni­cal details. I mean, just say that that hap­pened to be the case. 

And I just want to clar­i­fy, is the val­ue of broad pub­lic con­sen­sus on input, or is the val­ue more on hav­ing a diver­si­ty of rep­re­sen­ta­tive thought process? And if the val­ue’s on some­thing like open­ness and trans­paren­cy, that might have a dif­fer­ent infra­struc­ture of feed­back where­as if it’s on some­thing about diver­si­ty of thought, you might think of a sort of coun­cil where you have a philoso­pher and a human­ist and what­ev­er. So I think often­times we end up say­ing some­thing like, ​“We should have a broad con­ver­sa­tion about this and that’s how we’ll move for­ward,” but sort of dig­ging in on what that might actu­al­ly look like and how to get the best val­ue in our cur­rent society.

Eschrich: Thank you for that ques­tion. I’m just going to ask that we keep our respons­es quick just so we can take one more real­ly quick ques­tion before we wrap up.

Susan Tyler Hitchcock: They’re not mutu­al­ly exclusive. 

Eschrich: Oh look at that. A quick response. Either of you want to add anything?

Arbesman: So one thing I would say is…this is maybe a lit­tle bit to the side of it. But peo­ple have actu­al­ly looked at when you kind of bring in lots of dif­fer­ent, like diver­si­ty of opin­ions when it comes inno­va­tion, often­times the more diverse the opin­ions, I think the low­er the aver­age val­ue of the out­put but the high­er variance. 

So the idea is like, for the most part when you bring lots of peo­ple togeth­er who might speak lots of dif­fer­ent lan­guages and jar­gons, it often fails. But when it does suc­ceed it suc­ceeds in a spec­tac­u­lar fash­ion in a way it would­n’t have oth­er­wise. And so I think we should aim towards that but rec­og­nize that some­times these con­ver­sa­tions involve a lot of peo­ple talk­ing past each oth­er and so we need to do our best to make sure that does­n’t happen.

LaPointe: But I think specif­i­cal­ly to mak­ing sure you bring diverse voic­es from dif­fer­ent seg­ments of soci­ety and dif­fer­ent back­grounds into the con­ver­sa­tions is real­ly impor­tant. I always like to tell peo­ple auton­o­my, autonomous sys­tems, it’s not a tech­ni­cal prob­lem. It’s not like I can put a bunch of engi­neers in a room for a cou­ple of months and they could solve it. There are all these oth­er aspects to it. So you need to make sure you bring all the oth­er peo­ple. You bring the lawyers, you bring the ethi­cists, you bring every­body else. You know, the users, all the dif­fer­ent peo­ple. So I think you just have to be very thought­ful when­ev­er you are look­ing at devel­op­ing a tech­nol­o­gy to bring all those voic­es in at an ear­ly stage.

Eschrich: Okay. One more very quick question.

Tad Daley: Yeah, thanks. I’m Tad Daley. It’s for you, Cara. In the last pan­el, Nancy Kress I thought made a very com­plex, sophis­ti­cat­ed argu­ment about genet­ic engi­neer­ing. It has great ben­e­fits, also enor­mous risks. I think Nancy said gene edit­ing, some aspects of that is ille­gal. But then Nancy said but of course you can go offshore.

So I want to ask you to address those same things, Cara, about autonomous sys­tems. I think you’ve made clear that they have both risks as well as great ben­e­fits. Do you think it ought to be reg­u­lat­ed at all, and if so who should do the reg­u­lat­ing giv­en that if Country A does some reg­u­la­tion, in our glob­al­ized world it’s the eas­i­est thing in the world to go to Country B.

LaPointe: I think it’s a great ques­tion and some­thing that we inter­nal­ly talk a lot about. I think the thing about auton­o­my to under­stand is that every… Autonomy is ulti­mate­ly soft­ware, right. It is soft­ware that you’re putting into hard­ware sys­tems that helps move into this cog­ni­tive decision-making space. Now, every piece of auton­o­my that you devel­op, this soft­ware you devel­op, it’s dual-use.

So that was my ear­li­er point in terms of I don’t think it’s real­ly use­ful to talk about should you reg­u­late devel­op­ment, because auton­o­my is being devel­oped for a lot of dif­fer­ent things. So what you real­ly need to think about is okay, this tech­nol­o­gy is being devel­oped so how, where, when should the tech­nol­o­gy be used? I think those are the use­ful con­ver­sa­tions to have in terms of how it’s reg­u­lat­ed, etc. You know, where is auton­o­my allowed to be used, where it’s not allowed to be used. But the idea that you could some­how reg­u­late the devel­op­ment of auton­o­my I just don’t think is fea­si­ble or realistic.

Eschrich: Okay. I with a heavy heart have to say that we’re out of time. We will all be around dur­ing dur­ing the hap­py hour after­wards, so we’d love to keep talk­ing to you and answer­ing your ques­tions and hear­ing what you have to say. And thank you to all of you for being up here with me and for shar­ing your thoughts with us.

And to wrap up I’d like to intro­duce our next pre­sen­ter Jacob Brogan, who is an edi­to­r­i­al fel­low here at New America. And Jacob also writes bril­liant­ly about tech­nol­o­gy and cul­ture for Slate mag­a­zine. And he’s here to talk to you about a fan­tas­tic Frankenstein adaptation.