[This presentation was posted in four pieces. Section markers appear at the breaks as there were sometimes brief gaps between them.]
Fox Harrell: Firstly, thank you Kenny and the organizers for inviting me and having me here. And one of things I’ll talk about is that social activism and using technology for empowerment, it’s partly outreach and the activism within the world, but also becoming learners, doers, creators, builders of our own technologies. And also understanding the ways that the technologies themselves build oppressive structures. And so we’ll see a bit of that.
So here’s what I do. I direct the Imagination, Computation, and Expression Lab at MIT. And we research and develop artificial intelligence and cognitive science-based computing systems for creative expression, cultural analysis and social change.
So today’s talk has two different parts to it. So I’ll talk about computational identity technologies (That’s self imagination—how we imagine ourselves through social networks, online virtual worlds, through games.), and the project I’m running, which is National Science Foundation supported, called the Advanced Identity Representation Project, or the AIR Project. And then an example ICE Lab project for social change that worked with the Truth and Reconciliation Commission of Liberia.
So this is my motivation, to start off with. In the real world we can creatively represent ourselves in dynamic ways. So, we can vary our gesture, our discourse, our posture, our fashion, life stories, the way we tell our stories. And all of this is with an astounding sensitivity to social context.
Now, computer technologies like computer games, social networking, and virtual worlds are much more primitive than what we do in the real world. And so Kenny introduced me as the way of the future. Actually I think the ways that we’ve learned to negotiate the world—so the black experience, the ways we’ve had to adapt tragedies for survival, all of that I think is much more advanced than what we have here in these technologies. Because these technologies require us to represent ourselves through computational data structures, through algorithms. So they have much less nuance than we have in the real world.
So it raises a few questions for us. So that’s, how can we serve the human need for self-expression and identity construction using the computer? How can we represent ourselves in dynamic ways or even think about our power relationships with other people in the world, social issues, and oppression?
Let me give you an example of what the problem is. Do people know these games already? This is World of Warcraft here on the left, and this is Elder Scrolls IV: Oblivion on the right. So, in Elder Scrolls IV, let’s consider how it might stigmatize users. So, the ostensibly “African” characters are called the Redguard are described in the essentialist stereotype of a black athlete. So you read the manual or the instructions, they’re the most naturally talented warriors in Tamriel. The Redguards are also physically blessed with hardy constitutions, quickness of foot. So what happens in gameplay terms is it gives you bonuses eventually to your running and jumping ability. [audience laugther]
We can go a bit further, so let’s look under the hood of the game. What this is is a chart of all the default statistics for a character. You’ll notice things that are a bit interesting here. Like if you’re an orc and you happen to be female— One thing I should say is that most of these games say racial divisions for fictitious races like if you’re an orc or a troll or an elf. This one actually has Norwegian peoples, the Redguard which are black people, the Bretons which are the French. So it saves those kind of racial changes that are normally fantastic races and applies them to what are ostensibly real races.
So if you happen to be a female orc you’ll see here by default you’re ten points more intelligent than your male counterpart. If you’re a human Breton, which is the French group that I mentioned, you’re twenty points more intelligent by default than your Norwegian or black counterparts. So the choice of race and gender within this game results in ability-based stereotypes.
I was interviewed for a site, Boing Boing, which is a blog, and criticized a number of different games for the limited abilities to represent ourselves, and an interesting thing happened, which was that that was reblogged again by a gaming site. The original Boing Boing article was called something like “Professor Fox Harrell and His Chimerical Avatar” [“Chimerical Avatars and Other Identity Experiments from Prof. Fox Harrell”], how these avatars that change based on what you do, your emotions, and the way that you interact within the world.
When it was reblogged on this site called Kotaku, they changed the title to “Making Avatars That Aren’t White Dudes is Hard.” They also changed the goal of what I was trying to accomplish, since they said, “Fox Harrell wants to create avatars that look, well, like he does.” Maybe they meant maybe my social category, not just a kind of pure narcissism. Anyway, it spawned a series of incendiary comments about it.
So I published another article through that same blog and suggested ten different ways that we can improve our avatar representation.
And maybe it had an impact, because Elder Scrolls V: Skyrim—some people might know that—this is the new default Redguard character. If you play the game, this is what the character looks like now. And the interesting thing, though, is that my critique—the article had a big picture of my face along with an avatar. So it was a bit interesting. Maybe they responded to the article. But the thing is, regardless of how the character looks it still is going to be twenty points less intelligent than your French counterpart.
…number of ways that they actually could have improved it. Say, tie a life story into the way that you construct your characters or experience would be a very simple kind change.
But we can go a lot deeper than just looking at default stats. Let’s look at the underlying identity elements that are broken down into data structures. This is from a game called Neverwinter Nights. And if you look under the hood, actually, you have data structures for race, even blood color by race—something almost like the one-drop rule. And so changing race in this game actually doesn’t change the appearance of your character. But many items have racially-based bonuses.
Gender is interesting if you look underneath the hood. Because when you play you just see the character, but in the data structure there are actually five different genders. You might think that it’s somewhat more expansive than say a binary gender representation. Anyway, if you happen to be male, both, other, or none—which are four of the gender representations—you have a default male body type. So in fact despite these five different [gender] types under the hood, you have just two binary genders within it.
So the point of all this is that these elements are built into the very data structure of the games. So just taking up a character in a virtual world and saying, “I’m going to look like somebody different than I am,” it doesn’t do that much, actually, in order to think about ourselves through a different lens or think about ourselves as a new identity or to empower ourselves. In fact, a certain type of oppression or discrimination is built into the underlying structure of the code. So it means it behooves us to build new technologies or think about becoming creators or does or learners and builders of technology.
In social networking there are problems, too. So for example this is an old Facebook page, but opting in or opting out to an identity is just say, joining a group. And so you see in Facebook, if you want to be [somewhat?] Native American, opt into the Native American group, with its stereotypical so-called noble savage metaphors here. It’s quite a simplistic model of group membership.
I won’t go through all of these, but there are a number different problems that we have, whether it’s computer games, social networking, virtual worlds, any of our online accounts, any of these digital representations. And it’s the fact that our identities are reduced to statistics. Social categories are reduced to just graphical models and skins, so just nothing but appearance. Character change isn’t driven by anything like emotion or actually meeting people or what you do, but rather combat, spatial exploration, and acquiring objects.
In social networking, we have a simplistic model that joining a community is just click a button and you’re a member of the community. You don’t actually have to live, work, breathe with the community.
Virtual worlds, we have— So what about states like transitioning or becoming? I was actually on a panel once with a transgender digital media artist. That was someone who was actually in a state of transition. I said what about becoming? Well that’s something we all deal with as you become an expert, for example, from being a novice. We’re always in states of transition, but usually you’re always just something fixed in digital technology. Limited cultural diversity. So there are a lot of limitations that we have that in current technologies.
But you might ask what’s even the big problem just because well, it’s just a game, right? It’s just a virtual world. Well, there’s research—some of my own empirical research, Jerry Bailenson at Stanford is another person—that shows that changes to the virtual representations impact issues like interpersonal confidence, body image, students’ perception of themselves as learners and does. So our real-world interaction with people can be changed significantly by the way we interact through our virtual identities. And if you even have an account for your cell phone, you have a virtual identity in some kind of way. It’s not just these pyrotechnic examples in the video games I mention.
So, addressing these problems I think will make more diverse user groups, provide better customizability, make for more salient and powerful experiences, invent new forms of art, entertainment, and identity. So in short, we can do a lot better than the current state.
So you can say that we’ve at least been trying to take some modest steps towards doing so in the Imagination, Computation, and Expression Lab. And so with the support of the National Science Foundation I initiated a five-year project that’s called the Advanced Identity Representation Project. That’s developing a toolkit that works across platforms—social networking, avatars, profiles, characters—that enables rich self-expression; that can change dynamically; address issues like social stigma, bias, prejudice; and is based on cognitive science models of how we categorize in the world.
So instead of just the naïve intuition of a game designer who might not know anything about the experience of diverse groups of people, we’re looking at how we actually cognitively categorize in the world. And it’s much different than people assume. We don’t actually just have categories [where] we just try to force people into small boxes. What people tend to do is to have a prototype, models of family resemblances. It’s much more flexible than the way that people imagine in most of these infrastructures.
And we can learn a lot, even going back to 1903 from say WEB Du Bois and double-consciousness. So as I mentioned, before the kind of rich ways that we navigate the world give us a lot of power and insight in how we can develop more powerful technologies. So the dual awareness of people from marginalized groups of their self-perception and social stigma that tends to be attributed to the group.
Everyday self-presentation. So the fact that we adapt to perform identities like gestures and discourse, register for different social situations.
And even recently in ’99, identity torque. That’s the psychologically painful experience of a person’s self-conception differing from broader stigmatizing perceptions reinforced by classification infrastructures.
So I mean it’s just essentially the same thing that Du Bois said back in the day, but just the fact that these infrastructure we have, like the game for constructing our characters, or online accounts, actually result in psychologically painful experiences where you have some disjunction between who you are in the real world and then the way that you have to represent yourself through those technologies.
Audience 1: Do identity torque again. Just explain a little more.
Harrell: Sure. So, it’s essentially what Du Bois was talking about, but the idea of torque is the idea of twisting. It’s a twisting of our biographies, of our life stories, against the technologies that we have to engage. So when you have to go and use— It’s that sense I have say creating the character in Skyrim, and oh, it looks like me now, but I have to be a little bit less intelligent if I want to look like me. So maybe I should look like the French character or Roman character. So my biography being twisted up against that infrastructure.
And the same thing when you create an online profile on Facebook and you have to kowtow to commercial interests, for example. Or you have to represent yourself in a group just by opting into the group. So it’s that sense that you have much richer self and story, and when you go to use these technologies then there’s a kind of psychologically painful experience that’s detrimental to your health and happiness.
So, the sort of things that our toolkit can allow are say changing user’s self-representation for different social groups. Basic things that people do—code-switching. So you have different experiences. Again, the things that we do in the real world can give us a lot of insights for new technologies.
For better or for worse, right? So, stereotyping or passing. Are people passing on Facebook? That doesn’t just mean racial passing, but say what if we’re— I think Kenny and I were talking just earlier about a movement towards say punk rock music or skateboarding within the black community. What if people just want to seem a bit more like they’re in that scene than they actually are? There’s a kind of passing there. So can we identify this kind of passing that we have within social networking?
What about swapping between multiple identities the way that we use different entities in different situations?
What about aspirational identities? Not just a fictitious identity or something negative like passing, but what about what you see yourself as in the future? We model our aspirational identities.
And one of the things that I’ve noticed is that in social networking or computer games, at some level there’s some level of abstraction or some similarity between the way we represent ourselves in these different technologies. And so what I did here was just draw a graph that shows a Facebook profile and a character in a game called Dragon Age. And so in Facebook you have your profile, you have these links like “friends with” and your groups of friends. You have different pages, different things that you like there. And in Dragon Age you have different skills and classes like you might be a rogue, or you might use a dagger. So there’s some level at which you can abstract and start to compare these identities to process them in the same way. You can start to say is this identity like another identity? And so that’s where you can start to use computing to think about these kind of issues, like Du Bois and others have thought about.
…go much into all of the detail, but I’ll tell you about some of the kind of things our system allows. So let’s say discovering social categories. You might look at everybody in your social network and then find who are the people that tend to like the same thing? The people that like certain films, what are the kind of movies that they tend to like also. Or what are the kind of activities, places that they go. And so you can find social categories that are built into the network, that people haven’t explicitly gone in there and labeled.
What about finding people that are like you, not just by going in and saying “I want to find everyone who’s a woman who lives in Cambridge,” but to say what about the things that you do or the things that maybe you like?
You can predict belonging in categories and say, “based on what I know about these other social categories, I think that this person would fit into this other category.” And some of these aren’t just utilitarian tools. They’re also just tools to think with, to think about issues. Like how can you think about belonging from a computational perspective? Because most computer scientists don’t deal with issues like belonging as a member of a group.
Seeing identities in terms of one another. So that’s again this kind of passing, or say even posing. Is a person trying to be a member of my group? And so you could see their identity through the lens of a group that you find yourself in.
I’ll show you just a little bit of what this looks like. I’ll scroll through it. You can do things, say see the system with a few different games like say a game, StarCraft, Civilization here, and then from there say “what do other people in this person’s social network who like these games also tend to like?” And so this is an MIT student’s social network. And for his network for some reason, these gamers also tend to like some series of these classical music composers. That’s something that you wouldn’t have known that ahead of time, before using the system.
You can go in and relabel the categories, too. This is a category called “hipsters,” and maybe [inaudible] independent films or other sort of things. Then you find things in other categories that the person might just tend to like. Or things in people’s particular local community, even. So, sports fan.
So the interesting thing, too, is it’s not just trying to categorize all sports fans, it’s very local for that person’s network. So when you talked about punk rock music before, one person’s punk rock music might be pop punk rock, or Green Day, or some kind of band. Someone else might be underground. Someone else might be strictly black punk rock. It could be international. So it’s very particular for your local community. So what the system can do is find okay, who like sports in his own network and then what other things they like. They tend to like the Cold Stone Creamery. But also they tend to like some specific restaurant that just happens to be in Watsonville, California. So it’s not just anything, it’s something very particular to their own network.
And so now we can go back to those things things I talked about before, identity torque, double-consciousness, and passing, things we know about from the real world, and say— We actually implemented those in a computational way. So when I talk about passing, we could say— The mechanism you could use to think about passing on a computers is altering some of your representation to more closely resemble a member of another category. So you have all of these kind of characteristics coded in the data structure, and to think about it in ways in social networking to anonymously explore identities that are different than your own, whether for good or for ill. So it’s just a way we can say that these phenomena exist but in technical terms that are also in dialogue with what we do in everyday life. The same with double-consciousness.
Multiple visual representations. We built systems, for example a game where depending on what you do—if you’re walking through the suburbs in your character, then if you act aggressively or if you pray or if you punch someone, the character changes in a totally different way. If you use your cell phone or take out your wallet, you start to become more commerce-oriented, like a tycoon. Like a Monopoly man, you have stock charts bursting out of your head, or money bags start to appear. So your character’s just constantly changing. It’s not just getting new weapons, you’re changing in more poetic ways.
What I want to do now is just segue to another project which is a quite different but related to identity discussion, because it helps us to tell stories from the point of view of different entities. And this project is called the Living Liberia Fabric. We’ve implemented a number of different projects. This one was initiated [by] a colleague in international affairs, but it’s quite timely now, especially since Ellen Johnson Sirleaf was just awarded the Nobel Peace Prize.
Many of you might know that the West African nation of Liberia was never formally colonized. It was politically established in 1847 by freeborn African Americans, Africans freed from captured slave ships, all of whom were settled in an area that would become Liberia by the American Colonization Society. So this was a collusion of interests. It was slave owners, the US government… So I’m not saying that it’s a kind of benevolent experience. In fact, the conflict between the local populations with a series of a profiteers who who aided them initiated civil wars from 1989 until 2003. About 250,000 people were killed. That’s one third of the population that was displaced.
And so one of the things that Ellen Johnson Sirleaf did was to appoint a Truth and Reconciliation Commission modeled on the more famous one in Post-Apartheid South Africa. And one of the things that the TRC said is that memorialization is a necessary part of furthering the peace process. And so that’s—
And so I had my students meet with the Truth and Reconciliation Commissioner from Liberia, peace museum experts, diaspora Liberians who were in Georgia, also survivors of civil war—a gentlemen whose own son was abducted to become a child soldier, tragically—and went around and collected a number of stories and published books about this. So we really engaged with the community, spent a long time just to try to understand that conflict. So seeing ourselves as stakeholders, too. Not like we’re going to use technology for outreach but saying how do we relate to this through our own histories, our own biographies.
And so what the system does is it uses users’ actions to reveal multimedia content, video, photographs, texts, and an AI system I wrote called GRIOT that simultaneously coordinates themes, narrative structure, and media assets, by finding analogies between them.
And so what we did was find a series of information through our experiences talking to people, we encoded it here at three different levels—a visual level, at the level of the kind of frame that it interacts with, and then stakeholders groups, different kinds of themes, activists themes. And so we entered all this information in a form that could be manipulated computationally. You’ll have to read this. It’s fascinating, the format that our system could use. And then behind that, it structures a narrative differently each time you interact with it.
So what I’ll do is I’ll show you just a little bit of the way that the system works. And so the text is a bit a bit hard to read on the projector. Here it says “a silent moment for loss.” So what you do, before I start it, is you have a series of different figures here. When you start it, you just hear a bit of ocean sound. If you click on one of the figures, what that does is pick a stakeholder group, like woman or child. But that stakeholder group could mean woman survivor, it could be woman combatant. So you’re not choosing everything, you’re choosing one aspect of it.
That will pick a series of clips that will appear within the fabric. And then from those clips, you can click on just one of them. Then the next clip will be similar to the one you just picked. So if you click on something about a woman that deals with activism, then the one that comes after that will deal with activism. Then the next one might be a child that deals with activism. So it’s a way that we’re using AI not just to generate a story but rather to improvisationally make sure that there’s thematic links between it. So I’ll just play a little bit of it so you can see what what it looks like.
[The demonstration is viewable in the original video from ~2:30–5:25. There is also a demonstration clip on YouTube, showing a different instance of the project.]
So here you see all the other related stakeholder groups, and also kind of lost people that might be related to that group.
I should also say it’s a bit artificial showing it here. It says something like this is what the technology can do. Because the idea of the system was to say not what can we do for Liberia but to say how can we root computation within the culture of Liberia. So that’s when you can ground computing practices within diverse cultural practices. And so it’s a diverse perspective. We’re not just saying how can we go there and culturally plunder for the sake of technology, and so it’s a big difference here.
And these videos were collected by a colleague actually in Liberia, driving around in a truck from documentary film, from archival footage. So there’s a close personal connection to a lot footage, too.
So I’ll wrap up here before moving on to Malia. And so I’ll just say that the conclusion is just that computing can be used for subjective expression and social change. It’s not just a kind of objective medium. And so in the kind of work we do, we think about imagination. Imagination as a kind of artful thought, interactive narrative, poetry, gaming, as a space for self-expression and self-imagination. Computational identity systems, imagination as categorization, and more robust, more powerful forms of categorization that learns from life experience, learns from the experience of the marginalized or the survivors, and the dignity of people who struggle for social change. And our projects are just a few modest steps towards those ends. Thank you.