When Darius asked me to speak I had to think a lit­tle bit about what I would say to peo­ple who make soft­ware agents, which I think is real­ly real­ly cool. And to me, in think­ing about it, I think what is a bot to me? A bot is fun­da­men­tal­ly a piece of soft­ware that involves per­son­al­i­ty. And I’ve had a long-running inter­est in build­ing phys­i­cal robots that have per­son­al­i­ty of vary­ing degrees. So I pro­posed to give a talk about that, and I’m going to go from here.

So here’s me. I did research in col­lege in a robot­ics group ded­i­cat­ed to build­ing machines that inter­act­ed with peo­ple in socia­ble ways. I’ll give a lit­tle overview of some of the things I worked on in that lab, some of the oth­er projects the lab did, under­stand­ing what per­son­al­i­ty is, and then at the end a lit­tle bit about how machines can per­haps gen­er­ate it, and what soft­ware robots can take from that.

So, these are the two main projects I worked on in this lab. On the top left is my grad stu­dent Jeff Lieberman, who’s wear­ing a shirt which has mark­ers that let you track his body and then we cre­at­ed soft­ware that would then train you, using feed­back on how to move. The idea there was either for ther­a­peu­tic cas­es or per­haps to help some­one with their ten­nis game. 

At the bot­tom, far more rel­e­vant here, is Autom, a robot I worked on. It’s not a very tall robot, it’s about a tor­so. It sits on your desk. When you walk in, it looks at you. Its eyes track you. It speaks to you with a gen­er­at­ed voice. This start­ed before me, but I worked on it in 2006, 2007, so this was before Siri was a thing. And it had a touch­screen, which is an impor­tant part of some­thing I’ll describe lat­er. But you inter­act with it by as its speak­ing to you respond­ing via touch­ing the screen on its chest level.

This group was start­ed by Cynthia Breazeal. (A quick shout out to Professor Breazeal.) She built what is cred­it­ed as being one of the first socially-interacting robots. That’s her with Cog as a PhD stu­dent in Rodney Brooks’ lab.

Here’s one of three robots I’ll give as exam­ples for things that were worked on here. I think this is a real­ly inter­est­ing robot. This is a robot called Leonardo. Leonardo was built in col­lab­o­ra­tion between this robot­ics group and a prop shop. So, they pro­fes­sion­al­ly build crea­tures. You can prob­a­bly tell this was not the sole work of a bunch of MIT stu­dents. And Leonardo exists in a room that can tell where peo­ple are. It’s instru­ment­ed with cam­eras. And that let peo­ple do research using Leonardo to build a sys­tem where the robot can appear to rec­og­nize objects. 

So, one of the exper­i­ments they did was you would have a shell game. You’d have an object and would hide it. And the grad stu­dent would ask Leonardo the robot to iden­ti­fy where the object was. And it could under­stand the human lan­guage and then point to where the object was. And they devel­oped this to a point where the grad stu­dent could then leave the room and put on a hood, and come back as a vil­lain (so some­one Leonardo did­n’t rec­og­nize, because his face was cov­ered), and then change where the object was. And then Matt would leave and come back again and come back as him­self, and then ask Leonardo, ​“Oh, where’s the object? Is it under where I left it?” and Leonardo, this robot, was capa­ble of express­ing, non-verbally, using only facial expres­sions that it knew some­thing that you did­n’t know. Which is pret­ty advanced. This is con­sid­ered like, four year-old cog­ni­tive devel­op­ment, that I know some­thing that you don’t know, and that we have dif­fer­ing impres­sions of what things are like. This is real­ly cool.

This was a dif­fer­ent direc­tion. This is Ryan Wistort’s work. Ryan built these robots that I real­ly liked. This is a lit­tle bit like—if any­one’s famil­iar with Keepon, a cute lit­tle robot. I’m includ­ing this because Leonardo is at one end of a spec­trum of the extent to which you can build some­thing with extreme com­plex­i­ty and very detailed fea­tures. These robots, Tofu, are basi­cal­ly the oth­er end of it. They’re ani­ma­tron­ic pup­pets, and these were an explo­ration in using tech­niques bor­rowed from ani­ma­tion. So basi­cal­ly they squash up and down, and then they stretch, and all of the emo­tion­al expres­sion of these robots come from their eyes and that abil­i­ty to squash and stretch. So I think what I learned from these is that you can have on the one hand this very advanced sort of inter­ac­tion, but you can also have quite a rich inter­ac­tion with some­thing that’s far simpler.

And then last­ly this was the flag­ship robot of this research lab. This is a robot called Nexi. Nexi has, you can tell, a very advanced face. This is quite up there for what research robots do. It has a jaw that moves. The eyes move left, right, up, and down. It can track your eyes. It can blink. And it has actu­al robot­ic eye­brows, with mul­ti­ple degrees of free­dom that they can move it. So, it’s quite able of express­ing any facial expres­sion you could want it to do. You can have it do that and it can appear to hold a real con­ver­sa­tion. It’s also on a mobile plat­form and has hands. So it was called the mobile, dex­trous, social robot. It can do all of these things. 

And that was real­ly neat because this opened up this avenue to do greater test­ing. This is research that Jin Joo Lee did, which is real­ly real­ly fun­ny to me. I think this has to be one of the more uncom­fort­able exper­i­ments you could do. Because Nexi has hands, they want­ed to find out what does it take to make some­one feel more trust, or less trust, in a robot. So this starts to get into build­ing a per­son­al­i­ty. Jin Joo Lee worked with a team of psy­chol­o­gy stu­dents at Northeastern University, and they iden­ti­fied a cou­ple of behav­iors that make you trust peo­ple less, or make peo­ple gen­er­al­ly— They thought they want­ed to test this. Does this in fact inspire lack of trust? And those are things like if some­one maybe touch­es your face, or leans back, or cross­es their arms, or touch­es your hand.

Those are all things that are iden­ti­fied as inspir­ing lack of trust. So they had Nexi, the robot, do this in these psy­chol­o­gy exper­i­ments. I’m mere­ly an arm­chair psy­chol­o­gist, but to me the guy looks pret­ty uncom­fort­able there. And this was seen as being a real­ly cool thing to do with a robot in a research con­text, because you can have a grad stu­dent or some­one you’ve hired—an actor, maybe—come and try and do this exper­i­ment, but that per­son­’s going to have oth­er things asso­ci­at­ed with them and maybe they’re in a bad mood today and just no one trusts them, you know, it’s a Tuesday. Who knows? But the robot can do the same thing every time.

And this is fas­ci­nat­ing to me. And of course the result was peo­ple did feel less trust­wor­thy toward the robot. That’s a mind-blowing sen­tence to me, but on anoth­er lev­el is intu­itive and makes per­fect sense.

So this rais­es two ques­tions for me. One is how do you under­stand affect? And then how do you under­stand what the per­son you’re inter­act­ing with is feel­ing, also? So in think­ing about that part of it, Nexi of course had a voice, and I think one of the ear­li­er speak­ers ref­er­enced what tone of voice you use. We’ve dis­cussed also gen­der in robot­ics. And so I want­ed to raise for this audi­ence a relat­ed, very inter­est­ing study, not from the same lab but from a dif­fer­ent one, where these peo­ple were build­ing basi­cal­ly cars with a soft­ware agent built in.

Drivers who inter­act­ed with voic­es that matched their own emo­tion­al state had less than half as many acci­dents on aver­age as dri­vers who inter­act­ed with mis­matched voices!
Matching In-car Voice with Driver State: Impact on Attitude and Driving Performance

In short, they want­ed to add a voice to the car, that you would inter­act with, that would tell you help­ful, use­ful things. And in doing this, they want­ed to know, can this voice influ­ence peo­ple to be hap­pi­er. Which sounds hor­ri­ble, right? But, can you have this voice some­how be an excit­ed voice and have that make peo­ple feel more excit­ed? How does that inter­ac­tion work? And they found that peo­ple pre­ferred to inter­act with a robot whose voice matched their own affect. So if you were depressed, you’d feel bet­ter talk­ing to a depressed robot. And if you were just a hap­py per­son, you would pre­fer talk­ing to a hap­py robot. So then affect detec­tion became real­ly impor­tant and interesting.

So that brings me back to Autom. This is the robot that I worked on. And you can see here, this was a lat­er mod­el. Cory end­ed up leav­ing the lab and start­ing to sell them as a prod­uct. It has the same touch­screen in the front, and you can see the UI that was built for Autom. You can see it has three but­tons, and again this is still a touch­screen, so Autom speaks and the per­son responds. And I thought this was extreme­ly clever. The solu­tion found here for fig­ur­ing out how some­one feels is there are three respons­es. The ques­tion says, ​“Hello. I see it’s late, so I’ll help you get through this quick­ly and start mak­ing progress toward your goals.” Autom’s is basi­cal­ly a real­ly advanced food diary. So you inter­act with it, tell it what you ate, and then it would help you maybe lose weight or whatever.

And try­ing to deter­mine what sort of mood a per­son­’s in is done right here in this inter­face. These three answers are exact­ly equiv­a­lent. They all just get you to the next screen. The first one is like, ​“Hi.” The sec­ond one is ​“Good to see you.” And the last one just says ​“Okay.” And in see­ing peo­ple use this—it’s fascinating—people would select the one that matched their own inter­nal voice. They would­n’t even reg­is­ter the oth­er pos­si­bil­i­ties. So, I think this was some­thing to learn in the course of build­ing robots that I think tremen­dous­ly affects my sense of how peo­ple per­ceive per­son­al­i­ty in robots.

So, at this point I became sort of obsessed with this idea of how do you put per­son­al­i­ty into a thing? The idea that peo­ple would feel less trust in a robot… It’s a pile of bolts and com­put­er, right? Why would you feel one way or anoth­er? So I start­ed to try and say, okay so what is per­son­al­i­ty? And then when we say, you know…some peo­ple we would describe as hav­ing a lot of per­son­al­i­ty. So, what is that? What’s going on there, when some­one has a lot of personality?

That led me (I promise this isn’t direct­ly, of course) to Tom Waits, who’s a musi­cian. So, describ­ing this a lit­tle more for me for how it relates to this, it’s like Tom Waits is a musi­cian. A vari­ety of char­ac­ters come out in his work. And you can tell from this pic­ture he’s very expres­sive. And I actu­al­ly, in the way that very exu­ber­ant peo­ple in their ear­ly twen­ties will do, I would go around this robot­ics lab and just sort of try to shake peo­ple at the shoul­ders and say, ​“Why can’t we build a robot with as much per­son­al­i­ty as Tom Waits?” And my old­er friends the grad stu­dents were like, ​“Can we start with some­thing a lit­tle easier?”

But nev­er­the­less, I went on explor­ing. So what makes Tom Waits so personality-filled? And I thin it’s a cou­ple things. He’s very expres­sive. So, he does things that are very emo­tion­al and that inspire emo­tion in oth­ers. It’s gen­er­a­tive. So these things come from with­in him some­how, per­haps in response to an envi­ron­ment, but they’re def­i­nite­ly of Tom Waits. And then they’re unex­pect­ed. Like, if tomor­row one of us were to wake up and say, ​“I’m going to do Tom Waits today,” you could­n’t, right? You just could­n’t even take that one. Something that he’s doing is not predictable.

From there I want to share a lit­tle video. It’s short, but hope­ful­ly also amus­ing. This is my favorite robot, and the best robot I’ve ever found that incor­po­rates per­son­al­i­ty in the way I think I mean that has those three qual­i­ties. This is the Little Yellow Drummer Bot, made by a guy named Frits Lyneborg. He’s lets​maker​o​bots​.com, and I’ll just go ahead and show you. It’s a musi­cal robot.

So you can see in this robot­’s behav­ior, it’s doing a lot of those things. It’s very expres­sive, right? Like, it’s very dif­fi­cult to see that robot—and you know, I heard some peo­ple laugh­ing [inaudi­ble]. But it gives you a sense of joy, I think. It gives me one, any­way. It’s gen­er­a­tive, so it’s doing some­thing new every time. It’s actu­al­ly play­ing and then lis­ten­ing; it records itself. And then again plays music against its own record­ing of the sur­face it’s play­ing music on. And then it’s unex­pect­ed, right? It’s respond­ing to its envi­ron­ment. It’s based on some rules, but you don’t real­ly know what’s going to hap­pen every time.

And so for me, those are the answers and those are the things I would think about that I would leave you with here at Bot Summit in build­ing your cool soft­ware that I want to inter­act with, and find out how they work. 

Thank you very much.