When Darius asked me to speak I had to think a lit­tle bit about what I would say to peo­ple who make soft­ware agents, which I think is real­ly real­ly cool. And to me, in think­ing about it, I think what is a bot to me? A bot is fun­da­men­tal­ly a piece of soft­ware that involves per­son­al­i­ty. And I’ve had a long-running inter­est in build­ing phys­i­cal robots that have per­son­al­i­ty of vary­ing degrees. So I pro­posed to give a talk about that, and I’m going to go from here.

So here’s me. I did research in col­lege in a robot­ics group ded­i­cat­ed to build­ing machines that inter­act­ed with peo­ple in socia­ble ways. I’ll give a lit­tle overview of some of the things I worked on in that lab, some of the oth­er projects the lab did, under­stand­ing what per­son­al­i­ty is, and then at the end a lit­tle bit about how machines can per­haps gen­er­ate it, and what soft­ware robots can take from that.

So, these are the two main projects I worked on in this lab. On the top left is my grad stu­dent Jeff Lieberman, who’s wear­ing a shirt which has mark­ers that let you track his body and then we cre­at­ed soft­ware that would then train you, using feed­back on how to move. The idea there was either for ther­a­peu­tic cas­es or per­haps to help some­one with their ten­nis game. 

At the bot­tom, far more rel­e­vant here, is Autom, a robot I worked on. It’s not a very tall robot, it’s about a tor­so. It sits on your desk. When you walk in, it looks at you. Its eyes track you. It speaks to you with a gen­er­at­ed voice. This start­ed before me, but I worked on it in 2006, 2007, so this was before Siri was a thing. And it had a touch­screen, which is an impor­tant part of some­thing I’ll describe lat­er. But you inter­act with it by as its speak­ing to you respond­ing via touch­ing the screen on its chest lev­el.

This group was start­ed by Cynthia Breazeal. (A quick shout out to Professor Breazeal.) She built what is cred­it­ed as being one of the first socially-interacting robots. That’s her with Cog as a PhD stu­dent in Rodney Brooks’ lab.

Here’s one of three robots I’ll give as exam­ples for things that were worked on here. I think this is a real­ly inter­est­ing robot. This is a robot called Leonardo. Leonardo was built in col­lab­o­ra­tion between this robot­ics group and a prop shop. So, they pro­fes­sion­al­ly build crea­tures. You can prob­a­bly tell this was not the sole work of a bunch of MIT stu­dents. And Leonardo exists in a room that can tell where peo­ple are. It’s instru­ment­ed with cam­eras. And that let peo­ple do research using Leonardo to build a sys­tem where the robot can appear to rec­og­nize objects. 

So, one of the exper­i­ments they did was you would have a shell game. You’d have an object and would hide it. And the grad stu­dent would ask Leonardo the robot to iden­ti­fy where the object was. And it could under­stand the human lan­guage and then point to where the object was. And they devel­oped this to a point where the grad stu­dent could then leave the room and put on a hood, and come back as a vil­lain (so some­one Leonardo didn’t rec­og­nize, because his face was cov­ered), and then change where the object was. And then Matt would leave and come back again and come back as him­self, and then ask Leonardo, Oh, where’s the object? Is it under where I left it?” and Leonardo, this robot, was capa­ble of express­ing, non-verbally, using only facial expres­sions that it knew some­thing that you didn’t know. Which is pret­ty advanced. This is con­sid­ered like, four year-old cog­ni­tive devel­op­ment, that I know some­thing that you don’t know, and that we have dif­fer­ing impres­sions of what things are like. This is real­ly cool.

This was a dif­fer­ent direc­tion. This is Ryan Wistort’s work. Ryan built these robots that I real­ly liked. This is a lit­tle bit like—if anyone’s famil­iar with Keepon, a cute lit­tle robot. I’m includ­ing this because Leonardo is at one end of a spec­trum of the extent to which you can build some­thing with extreme com­plex­i­ty and very detailed fea­tures. These robots, Tofu, are basi­cal­ly the oth­er end of it. They’re ani­ma­tron­ic pup­pets, and these were an explo­ration in using tech­niques bor­rowed from ani­ma­tion. So basi­cal­ly they squash up and down, and then they stretch, and all of the emo­tion­al expres­sion of these robots come from their eyes and that abil­i­ty to squash and stretch. So I think what I learned from these is that you can have on the one hand this very advanced sort of inter­ac­tion, but you can also have quite a rich inter­ac­tion with some­thing that’s far sim­pler.

And then last­ly this was the flag­ship robot of this research lab. This is a robot called Nexi. Nexi has, you can tell, a very advanced face. This is quite up there for what research robots do. It has a jaw that moves. The eyes move left, right, up, and down. It can track your eyes. It can blink. And it has actu­al robot­ic eye­brows, with mul­ti­ple degrees of free­dom that they can move it. So, it’s quite able of express­ing any facial expres­sion you could want it to do. You can have it do that and it can appear to hold a real con­ver­sa­tion. It’s also on a mobile plat­form and has hands. So it was called the mobile, dex­trous, social robot. It can do all of these things. 

And that was real­ly neat because this opened up this avenue to do greater test­ing. This is research that Jin Joo Lee did, which is real­ly real­ly fun­ny to me. I think this has to be one of the more uncom­fort­able exper­i­ments you could do. Because Nexi has hands, they want­ed to find out what does it take to make some­one feel more trust, or less trust, in a robot. So this starts to get into build­ing a per­son­al­i­ty. Jin Joo Lee worked with a team of psy­chol­o­gy stu­dents at Northeastern University, and they iden­ti­fied a cou­ple of behav­iors that make you trust peo­ple less, or make peo­ple gen­er­al­ly— They thought they want­ed to test this. Does this in fact inspire lack of trust? And those are things like if some­one maybe touch­es your face, or leans back, or cross­es their arms, or touch­es your hand.

Those are all things that are iden­ti­fied as inspir­ing lack of trust. So they had Nexi, the robot, do this in these psy­chol­o­gy exper­i­ments. I’m mere­ly an arm­chair psy­chol­o­gist, but to me the guy looks pret­ty uncom­fort­able there. And this was seen as being a real­ly cool thing to do with a robot in a research con­text, because you can have a grad stu­dent or some­one you’ve hired—an actor, maybe—come and try and do this exper­i­ment, but that person’s going to have oth­er things asso­ci­at­ed with them and maybe they’re in a bad mood today and just no one trusts them, you know, it’s a Tuesday. Who knows? But the robot can do the same thing every time.

And this is fas­ci­nat­ing to me. And of course the result was peo­ple did feel less trust­wor­thy toward the robot. That’s a mind-blowing sen­tence to me, but on anoth­er lev­el is intu­itive and makes per­fect sense.

So this rais­es two ques­tions for me. One is how do you under­stand affect? And then how do you under­stand what the per­son you’re inter­act­ing with is feel­ing, also? So in think­ing about that part of it, Nexi of course had a voice, and I think one of the ear­li­er speak­ers ref­er­enced what tone of voice you use. We’ve dis­cussed also gen­der in robot­ics. And so I want­ed to raise for this audi­ence a relat­ed, very inter­est­ing study, not from the same lab but from a dif­fer­ent one, where these peo­ple were build­ing basi­cal­ly cars with a soft­ware agent built in.

Drivers who inter­act­ed with voic­es that matched their own emo­tion­al state had less than half as many acci­dents on aver­age as dri­vers who inter­act­ed with mis­matched voic­es!
Matching In-car Voice with Driver State: Impact on Attitude and Driving Performance

In short, they want­ed to add a voice to the car, that you would inter­act with, that would tell you help­ful, use­ful things. And in doing this, they want­ed to know, can this voice influ­ence peo­ple to be hap­pi­er. Which sounds hor­ri­ble, right? But, can you have this voice some­how be an excit­ed voice and have that make peo­ple feel more excit­ed? How does that inter­ac­tion work? And they found that peo­ple pre­ferred to inter­act with a robot whose voice matched their own affect. So if you were depressed, you’d feel bet­ter talk­ing to a depressed robot. And if you were just a hap­py per­son, you would pre­fer talk­ing to a hap­py robot. So then affect detec­tion became real­ly impor­tant and inter­est­ing.

So that brings me back to Autom. This is the robot that I worked on. And you can see here, this was a lat­er mod­el. Cory end­ed up leav­ing the lab and start­ing to sell them as a prod­uct. It has the same touch­screen in the front, and you can see the UI that was built for Autom. You can see it has three but­tons, and again this is still a touch­screen, so Autom speaks and the per­son responds. And I thought this was extreme­ly clever. The solu­tion found here for fig­ur­ing out how some­one feels is there are three respons­es. The ques­tion says, Hello. I see it’s late, so I’ll help you get through this quick­ly and start mak­ing progress toward your goals.” Autom’s is basi­cal­ly a real­ly advanced food diary. So you inter­act with it, tell it what you ate, and then it would help you maybe lose weight or what­ev­er.

And try­ing to deter­mine what sort of mood a person’s in is done right here in this inter­face. These three answers are exact­ly equiv­a­lent. They all just get you to the next screen. The first one is like, Hi.” The sec­ond one is Good to see you.” And the last one just says Okay.” And in see­ing peo­ple use this—it’s fascinating—people would select the one that matched their own inter­nal voice. They wouldn’t even reg­is­ter the oth­er pos­si­bil­i­ties. So, I think this was some­thing to learn in the course of build­ing robots that I think tremen­dous­ly affects my sense of how peo­ple per­ceive per­son­al­i­ty in robots.

So, at this point I became sort of obsessed with this idea of how do you put per­son­al­i­ty into a thing? The idea that peo­ple would feel less trust in a robot… It’s a pile of bolts and com­put­er, right? Why would you feel one way or anoth­er? So I start­ed to try and say, okay so what is per­son­al­i­ty? And then when we say, you know…some peo­ple we would describe as hav­ing a lot of per­son­al­i­ty. So, what is that? What’s going on there, when some­one has a lot of per­son­al­i­ty?

That led me (I promise this isn’t direct­ly, of course) to Tom Waits, who’s a musi­cian. So, describ­ing this a lit­tle more for me for how it relates to this, it’s like Tom Waits is a musi­cian. A vari­ety of char­ac­ters come out in his work. And you can tell from this pic­ture he’s very expres­sive. And I actu­al­ly, in the way that very exu­ber­ant peo­ple in their ear­ly twen­ties will do, I would go around this robot­ics lab and just sort of try to shake peo­ple at the shoul­ders and say, Why can’t we build a robot with as much per­son­al­i­ty as Tom Waits?” And my old­er friends the grad stu­dents were like, Can we start with some­thing a lit­tle eas­i­er?”

But nev­er­the­less, I went on explor­ing. So what makes Tom Waits so personality-filled? And I thin it’s a cou­ple things. He’s very expres­sive. So, he does things that are very emo­tion­al and that inspire emo­tion in oth­ers. It’s gen­er­a­tive. So these things come from with­in him some­how, per­haps in response to an envi­ron­ment, but they’re def­i­nite­ly of Tom Waits. And then they’re unex­pect­ed. Like, if tomor­row one of us were to wake up and say, I’m going to do Tom Waits today,” you couldn’t, right? You just couldn’t even take that one. Something that he’s doing is not pre­dictable.


From there I want to share a lit­tle video. It’s short, but hope­ful­ly also amus­ing. This is my favorite robot, and the best robot I’ve ever found that incor­po­rates per­son­al­i­ty in the way I think I mean that has those three qual­i­ties. This is the Little Yellow Drummer Bot, made by a guy named Frits Lyneborg. He’s lets​maker​o​bots​.com, and I’ll just go ahead and show you. It’s a musi­cal robot.

So you can see in this robot’s behav­ior, it’s doing a lot of those things. It’s very expres­sive, right? Like, it’s very dif­fi­cult to see that robot—and you know, I heard some peo­ple laugh­ing [inaudi­ble]. But it gives you a sense of joy, I think. It gives me one, any­way. It’s gen­er­a­tive, so it’s doing some­thing new every time. It’s actu­al­ly play­ing and then lis­ten­ing; it records itself. And then again plays music against its own record­ing of the sur­face it’s play­ing music on. And then it’s unex­pect­ed, right? It’s respond­ing to its envi­ron­ment. It’s based on some rules, but you don’t real­ly know what’s going to hap­pen every time.

And so for me, those are the answers and those are the things I would think about that I would leave you with here at Bot Summit in build­ing your cool soft­ware that I want to inter­act with, and find out how they work. 

Thank you very much.

Further Reference

Darius Kazemi's home page for Bot Summit 2016.

Help Support Open Transcripts

If you found this useful or interesting, please consider supporting the project monthly at Patreon or once via Square Cash, or even just sharing the link. Thanks.