Introducer: Welcome every­body for the next talk, When Algorithms Fail in Our Personal Lives.” It is a one hour talk. Our love­ly speak­er with us today is Caroline Sinders. She’s a user researcher for IBM. She’s also an artist, a researcher, a video game design­er. She’s from the States, and I should also men­tion that she is a mem­ber of the NYC Resistor hack­er space. And I see some fans over here. Cool.

We already learned in a bunch of talks over the course of Congress what algo­rithms do when they fail. Yesterday we learned about how algo­rithms can dis­crim­i­nate, or not dis­crim­i­nate in the hir­ing process, and Caroline is going to tell us a lit­tle more about when it’s bet­ter not to use algo­rithms because there are some things that algo­rithms just can’t do, that humans can do.

So please give it up for Caroline and enjoy the talk. Thank you very much.

Caroline Sinders: Hi, every­one. I’m Caroline Sinders. I should prob­a­bly first spec­i­fy that I am speak­ing here of my own accord and not on behalf of IBM. So just FYI. And that this is actu­al­ly a pre­sen­ta­tion also on a very strange and spe­cif­ic art project I did back in late November. So, when algo­rithms fail in our per­son­al lives. This is prob­a­bly the best way to describe me because I live on the Internet.

I’ve spent the last two years study­ing online fan­doms, com­mu­ni­ties, Internet cul­ture, and online harass­ment. And this is what I do for fun out­side of work. I think a lot about lan­guage and con­ver­sa­tion as iden­ti­fiers, and I spend a lot of time read­ing the way in which con­ver­sa­tions unfold on dif­fer­ent sub­red­dits on Reddit, 4chan, 8chan, Wikipedia, the way Wikipedia is used as a con­ver­sa­tion­al tool not just to upload infor­ma­tion, and obvi­ous­ly Facebook, Twitter, LinkedIn, and Instagram.

And the one thing I’ve sort of learned from all this is that each of these dif­fer­ent plat­forms have a very dif­fer­ent iden­ti­ty and they have a very dif­fer­ent way in which con­ver­sa­tion sort of has evolved lin­guis­ti­cal­ly to that plat­form. The way we talk on Reddit is a lot dif­fer­ent than the way we talk on Twitter. And I think that that is due to the infra­struc­tur­al design of the plat­form itself, as well as the ways in which the plat­form iden­ti­fies itself to users, so like a code of con­duct.

Two years ago, actu­al­ly like a year and a half ago, I real­ly start­ed focus­ing on online harass­ment. I specif­i­cal­ly focused on Gamergate. As a video game design­er, I saw Gamergate kind of affect­ing the com­mu­ni­ty around me. I’m not dri­ven to study harass­ment by why it hap­pens, but rather how. How does harass­ment unfold on dif­fer­ent sorts of plat­forms, and how do plat­forms allow for dif­fer­ent kinds of com­mu­ni­ca­tion? And real­ly how open is the gen­er­al user on a plat­form? How con­nect­ed are they to the pri­va­cy poli­cies? And how aware are they of how exposed they are, and how per­me­able their data and infor­ma­tion is?

So TLDR, I explore com­plex emo­tions and emo­tion­al reac­tions with­in sys­tems. And I’m going to briefly cov­er some of the anti-harassment research I’ve done.

So some­times I write things, the Internet does­n’t like them. Last April, the Internet sent a SWAT team to my mom’s house. If you don’t know what swat­ting is, it’s a very pop­u­lar online harass­ment tac­tic that man­i­fests itself IRL.” Oftentimes a fake vio­lent phone call is placed to a local police depart­ment. That vio­lent phone call trig­gers the mil­i­ta­rized police to be deployed, and this is what hap­pened to my mom.

Sometimes pan­els I’m on get can­celled because maybe they’re kind of con­tro­ver­sial. I sub­mit­ted a design pan­el to SXSW and it was can­celled due to harass­ment and threats of vio­lence.

So I guess my work seems kind of con­tentious. I think that it’s pret­ty straight­for­ward. It’s gen­er­al­ly design. I don’t know why any­one would have a real­ly mas­sive opin­ion around it. But the thing I sort of want to talk about, actu­al­ly, and what I care about explor­ing, is how do sys­tems affect behav­ior? I said ear­li­er I’m here not as an IBM rep­re­sen­ta­tive, but I spend most of my day job work­ing on Watson, work­ing in arti­fi­cial intel­li­gence as a user researcher around con­ver­sa­tion­al ana­lyt­ics for chat robots. That’s kind of a mouth­ful.

What I mean by that is I spend a lot of time work­ing with soft­ware that allows users to set up chat robots. So I think a lot about the ways in which I’m design­ing soft­ware to help peo­ple design con­ver­sa­tions that robots have with peo­ple. So when I say I believe that sys­tems affect behav­ior, I live that every day, and I think about the ways in which the struc­ture of an inter­face actu­al­ly will lead peo­ple to con­verse and what that would look like.

In the past two years of doing I guess broad heuris­tic ethno­graph­ic research, what I’ve come to real­ize is users have a myr­i­ad of dif­fer­ent prob­lems that can be solved in sim­i­lar ways, but yield rad­i­cal­ly dif­fer­ent results. Meaning that we can sort of start approach­ing ways to solve prob­lems of harass­ment through either new kinds of algo­rithms or a real­ly flex­i­ble UI on top of an algo­rithm, ie. what if we gave peo­ple more robust pri­va­cy set­tings and allowed users to start to artic­u­late the ways in which they’re reach­able and how their data, real­ly their con­ver­sa­tions, are read?

And I think this because algo­rithms real­ly aren’t that smart, and lan­guage with­in an algo­rithm is decon­tex­tu­al­ized into data. We as users pro­vide con­text to lan­guage. Language is what we make it. But in a sys­tem it’s not sim­ply just bits of data. So I was dri­ven by this thought: How could I make a flex­i­ble sys­tem to solve a vari­ety of dif­fer­ent prob­lems, for a vari­ety of dif­fer­ent peo­ple?

So I did what I do best, and I made a low-res wire­frame. I cre­ate these spec­u­la­tive wire­frames to sort of focus myself on what I thought could be achiev­able, and then I decid­ed to test it against twen­ty dif­fer­ent users I had inter­viewed that had been affect­ed by Gamergate, as well as a hand­ful of Gamergaters them­selves who would inter­act with me on Twitter and I would ask them ques­tions about the ways in which they orga­nized them­selves, the ways in which they talked to oth­er peo­ple, what con­ver­sa­tions they were hop­ing to get out of inter­act­ing with em on Twitter.

What I start­ed to learn is that we need to focus on pri­va­cy in social media. It needs to be as preva­lent and as impor­tant as writ­ing con­tent itself. Do you see that gray box at the top? That is a place­hold­er for a but­ton that sends you to a redesigned pri­va­cy page.

From inter­view­ing these twen­ty users, I got a real­ly robust sense of dif­fer­ent kinds of needs and wants users want­ed out of Twitter. I inter­viewed peo­ple that had over 100,000 fol­low­ers that absolute­ly want to remain com­plete­ly pub­lic. And they want to be reach­able at all times. I inter­viewed some users that had 3,000 fol­low­ers, that want­ed to be com­plete­ly hid­den but still have their tweets treat­ed as media, thus share­able. And I inter­viewed some users that want­ed to not go pri­vate (which is a very pub­lic state­ment on Twitter, to have the lock next to you) but want­ed to have all the affor­dances of pri­va­cy.

What I’ve added as you can see is these check­marks to allow a user to start to change the way their writ­ten con­tent can be accessed and so real­ly actu­al­ly change the way in which the amount of users on Twitter could start to read con­tent they’re post­ing.

One of them is allow fol­low­ers of your fol­low­ers to tweet at you,” so the idea of friends-of-friends. Do not allow accounts with less than X fol­low­ers to fol­low you” or don’t allow accounts less than X days,” mean­ing new accounts are often cre­at­ed in moments of harass­ment cam­paigns. So if an account was a week old with two fol­low­ers, that’s prob­a­bly a troll account. Additionally it also allows users to say, if you’re not on my lev­el you can’t tweet at me.” Not judg­ing; that’s inter­ac­tion some­one want­ed.

And then I start­ed to think more about what does it mean to exist pub­licly as a per­son on a plat­form? Twitter is sort of this mixed iden­ti­ty and mixed emo­tion­al state. It’s both pro­fes­sion­al and per­son­al. It’s used as a net­work­ing tool as well as a social aid and a com­mu­nica­tive tool. So peo­ple either have real­ly per­sis­tent alias­es or avatars that have fol­lowed them from plat­form to plat­form but they’re not using their real name. There’s lev­els of pseudo-anonymity on Twitter. In my case I use my real name and I can’t real­ly undo that. So my needs in using Twitter, espe­cial­ly as a tech­nol­o­gist exists in a much dif­fer­ent way than, per se, some­one who uses it as a casu­al medi­um. And we have very dif­fer­ent needs.

But through these dif­fer­ent kinds of dials, I feel that this serves my needs as well as all of the users I’ve inter­viewed because we’re able to start to tai­lor though UI, and be able to pull from very top-levels of infor­ma­tion, just main­ly around fol­low­ers, as to how acces­si­ble I am.

I added oth­er things such as blocked accounts and blocked tweets. Right now you can only see blocked accounts. You can’t see blocked tweets. What if you could? And I pulled from this because in moments of harass­ment, even if it’s a sus­tained stalk­er, there’s often a tweet that will trig­ger it. It’s nev­er going to some­one’s account, at least with­in a harass­ment cam­paign, you’re not real­ly going to some­one’s account and say­ing, Today I’m block­ing you.” There is often an inter­ac­tion, a tweet, that will trig­ger that response. So what if you could see that, start to group the togeth­er, and maybe send a report to Twitter or to your­self. The user can start to con­tex­tu­al­ize this is a way in which these tweets are linked.” So if there’s a mob harass­ment cam­paign, a user could say, I think all these are linked.” And if Twitter’s imple­ment­ing any kind of machine learn­ing or nat­ur­al lan­guage pro­cess­ing they’d be able to start batch­ing mul­ti­ple reports at once and see how they’re all relat­ed.

Again, what if you could group men­tions togeth­er?

And I added this last night. One thing that I noticed from a lot of my research is that users don’t real­ly have an under­stand­ing as to how their lan­guage is actu­al­ly data and how acces­si­ble their things are.

I’m sure you’ve heard a vari­ety of sto­ries around tweets going viral. Someone tweets some­thing and then months lat­er it’s dug up. Or they tweet some­thing… In the case of this real­ly well-known inci­dent this woman tweet­ed a real­ly off-color joke about AIDS, got on a plane, twelve hours lat­er this tweet had com­plete­ly explod­ed. The back­ground of that sto­ry is that this woman only had a hun­dred fol­low­ers and had nev­er had her tweets inter­act­ed with very much at all. Not in the sense of with strangers. So for her this was a com­plete moment of the sys­tem kind of break­ing. And I won­der if there are ways to start to artic­u­late to users how acces­si­ble you are. Even if you feel small, even if you feel like no one is inter­act­ing with your tweets, you’re still actu­al­ly com­plete­ly open. And the infor­ma­tion you send out into the sys­tem is media that can be iso­lat­ed and shared quick­ly, and that’s sort of the way in which Twitter func­tions.

So I won­dered what if you could just break some­thing down real­ly sim­ply, and just sort of say fol­low­er impres­sions and non-fol­low­er impres­sions to sort of get an idea of as to who’s inter­act­ing with your tweet, and who out­side of your decen­tral­ized social cir­cle on Twitter.

And then I start­ed look­ing at Facebook. Additionally, I did anoth­er round of inter­views, specif­i­cal­ly for this project I’m get­ting into, Social Media Breakup Coordinator, where users actu­al­ly had no idea what the pri­va­cy check­up meant. I think this is a great addi­tion. You add a but­ton. You can say only friends can see this.” But what if Twitter added a pop­up and then said, Great, this con­tent right here, this com­ment. If your friend Jane com­ments on it, her mom can see it.” It start­ed to real­ly show how extend­ed net­works that you’re uncon­nect­ed to, 2nd and 3rd-party rela­tion­ships, can actu­al­ly inter­act with your infor­ma­tion.

I’m real­ly dri­ven by this need and this idea as a design­er, what would it look like to have a semi-private space in a pub­lic net­work, and how could I design that? I think about this a lot because our com­mu­ni­ca­tion on the Internet is asyn­chro­nous, right? But a lot of social media cre­ates things as a time­line. This cre­ates a false idea as to how infor­ma­tion is actu­al­ly accessed, and how data is actu­al­ly stored. And that false infor­ma­tion is artic­u­lat­ed to users. So what feels like safer spaces, even if you’re com­plete­ly pub­lic because you’re not inter­act­ed with, is a lie. It’s a false sense of infor­ma­tion. It’s a false sense of safe­ty.

So I won­der with all these vary­ing lev­els of needs that we have as users, and as we live more and more of our lives dig­i­tal­ly and on social media, what would it look like to design a semi-private space in a pub­lic net­work?

The past two years have real­ly hit this on home that there’s this neb­u­lous­ness sur­round­ing algo­rithms and social media and the way in which our data is saved. And a lot of that hap­pens when Facebook for instance changed their time­line to be algo­rith­mi­cal­ly dri­ven based off con­tent. Then I think it was last sum­mer, or two sum­mers ago, there was this thing called the Ice Bucket Challenge, and these riots in Ferguson, Missouri. And what hap­pened [was] peo­ple real­ized that Ice Bucket Challenge posts were being weight­ed above these oth­er protests. And the way to work around that was to include Ice Bucket Challenge” when you were post­ing about Ferguson to start to flip and change what you were see­ing algo­rith­mi­cal­ly in your time­line. So there’s this idea that users don’t quite know what and why the algo­rithm will weight things over oth­er things. So when you post some­thing on Facebook, the feed­back is, I have no idea when it’s acces­si­ble, how it’s acces­si­ble, and if it will be accessed.”

So that let me do this project that I cre­at­ed. I cre­at­ed a fake per­for­mance art piece—I mean, it’s a real art piece—called Social Media Breakup Coordinator, where I turned a video game art gallery in New York called Babycastles into a doc­tor’s office. And I held fifteen-minute therapy/consulting ses­sions on social media.

I had users fill out a twenty-two point very stan­dard user quiz around why they were show­ing up. But then when they sat with me, I had them sign a terms of ser­vice agree­ment, I lis­tened to them, and then I start­ed to write down notes. But before I start­ed this project, I reached out to a vari­ety of dif­fer­ent peo­ple, because from my research I had sort of start­ed to real­ize that there’s a lot of dif­fer­ent moments where there needs to be human inter­ven­tion with­in algo­rithms with­in social media.

So how do you start to pull away from dif­fer­ent groups that you’ve been asso­ci­at­ed with? How do you start to cut ties? And how do you start to cut ties between infor­ma­tion when you cross-post against dif­fer­ent plat­forms?

A good exam­ple of that is what hap­pens if some­one in your fam­i­ly dies and that ends up in Facebook mem­o­ries because you Instagrammed it? What does that feel like, to have that emo­tion­al trig­ger? What does it feel like to quit a job and not be sure if your new cowork­ers can see your old cowork­ers? Or if you post some­thing neg­a­tive about your old job are you still con­nect­ed to your boss and what can they see? And gen­er­al­ly there is this lack of under­stand­ing that I found that most gen­er­al users (prob­a­bly not most peo­ple in this room) have a lack of under­stand­ing around how much their infor­ma­tion is accessed.

So I was curi­ous on a bunch of lev­els if peo­ple would actu­al­ly pay me to give them advice. If they would trust me as a pro­fes­sion­al. And if they would actu­al­ly engage with my ser­vices. And then I was curi­ous if I could actu­al­ly then covert­ly teach them the pri­va­cy poli­cies of all the dif­fer­ent plat­forms they were on.

When I start­ed this project I real­ized I need­ed to talk to a vari­ety of dif­fer­ent pro­fes­sion­als. I’m a user researcher, so my pro­fes­sion lies in talk­ing to users and design­ing solu­tions for them. But as social media starts to over­take more and more aspects of our lives, I real­ized that there were cer­tain things that I’m not equipped to han­dle. So what hap­pens if some­one has suf­fered trau­ma on social media? As a vic­tim of harass­ment, I still can’t offer any­one feed­back on that, and that’s sort of not my place.

So I spoke to a rape cri­sis coun­selor, an engi­neer, a data sci­en­tist and pro­fes­sor, and a pri­vate ther­a­pist, and a pri­vate psy­chi­a­trist. The take­away I got was main­ly this, and this is some­thing I’d love to impart on most social media engi­neers and design­ers: It’s not my job to nec­es­sar­i­ly tell peo­ple what to do, it’s my job to lis­ten to what peo­ple need to get done.

An exam­ple from that is, let’s say a user came to me for Social Media Breakup Coordinator and said, I have an abu­sive boyfriend and he’s hor­ri­ble and we have a child togeth­er and I want to un-Facebook friend him.” It’s not my place nec­es­sar­i­ly to say, Okay, wait. Can I know more? Are you close with his fam­i­ly? Let’s start to cut down all these ties.” The rea­son I would ask that is, think­ing as a design­er if you’re Facebook friends with some­one, and then you’re Facebook friends with their par­ents, and then it says on that per­son­’s pro­file who their fam­i­ly is, the sys­tem has cre­at­ed more ties to that per­son, even regard­less of if you unfriend them. You would need to block them as well as unfol­low all of these oth­er peo­ple relat­ed to them and tied to their pro­file to actu­al­ly real­ly sep­a­rate.

A lot of the feed­back I got was it’s not nec­es­sar­i­ly your job to tell a user all of that if they’re telling you what exact­ly they need. You sort of need to lis­ten and guide from there and not real­ly get into, Well why are you here? What are all these dif­fer­ent real­ly spe­cif­ic and high­ly per­son­al details?”

So why would I do this?

I was just very inter­est­ed in the ways in which peo­ple live their lives online, and I real­ly want­ed to see if I could also gath­er a lot of data from this project. I had six­teen peo­ple fill out twenty-two dif­fer­ent ques­tions and meet with me and walk through all their dif­fer­ent prob­lems. And I was real­ly curi­ous if I could pro­vide solu­tions the way an algo­rithm would. I out­lined ten dif­fer­ent solu­tions that I could affix to peo­ple based of dif­fer­ent that they answered in a cer­tain order.

And again the covert point of this project was to sort of teach peo­ple about the per­me­abil­i­ty of their posts and real­ly how pri­va­cy is looked at and inter­act­ed with on social net­works. And with the onset of all these dif­fer­ent apps, par­tic­u­lar­ly in America, that are offer­ing to out­source emo­tion­al labor to a per­son, mean­ing there’s all these new apps that’ve been cre­at­ed of, We’ll break up with your boyfriend for you,” I was real­ly sort of curi­ous to see if peo­ple would actu­al­ly engage with me face to face.

So, when I launched the project peo­ple thought it was real. And then the media thought it was real. And it was real­ly hard to explain to, for instance Jezebel, that this was an art project. Because they were like, But you’re charg­ing peo­ple… And you made them sign a con­tract… Is the con­tract legal­ly bind­ing?” Yes, it is. So you charge them mon­ey?” I did. Did you give them fake feed­back?” No, the feed­back was all sin­cere. I real­ly legit­i­mate­ly tried to help solve these prob­lems. But it’s an art project?”

The rea­son that it’s an art project is to me it’s a mas­sive com­ment on the shar­ing econ­o­my that’s in America, and just this idea that I could be an emo­tion­al Mechanical Turk. And I com­plete­ly made that by design and inten­tion. Should peo­ple be trust­ing me with their data? Yes, because I am a pro­fes­sion­al, and I made sure to very very clear­ly artic­u­late the ways in which I would use their data, how they would be pro­tect­ed, and that I would not share any per­son­al infor­ma­tion about them. I went through all of those steps, but is that sort of the nego­ti­a­tion we have with social net­works? Do we have that kind of inter­fac­ing?

And an even big­ger com­ment was no one every com­ment­ed on price. I charged $1 a minute to sit and lis­ten all day to peo­ple. I only gave them fifteen-minute blocks. It was actu­al­ly incred­i­bly tax­ing, phys­i­cal­ly. It was an all-day event where I think I only gave myself fif­teen min­utes for lunch. I def­i­nite­ly have a whole new type of respect for ther­a­pists. That was gru­el­ing.

So before I start­ed the project, I start­ed to break down what plat­forms I would cov­er. These are exam­ples of my Post-It Notes. I had cov­ered Resistor in one evening. And I start­ed to break things down based off the four major plat­forms that are used in America, which is LinkedIn, Facebook, Twitter, and Instagram. I start­ed to break down by what I thought were the four most broad, most uni­ver­sal, social group­ings. So friends, fam­i­ly, work, and roman­tic. Then I start­ed to think about why your roman­tic part­ner would friend you on LinkedIn, for exam­ple. Or why they would fol­low you on Instagram. Or why your boss would friend you on all of those plat­forms.

And I start­ed to attach dif­fer­ent emo­tion­al respons­es. So should you LinkedIn con­nect with your Dad? Maybe? Should you LinkedIn con­nect with your lover? If you want to… But you don’t have to. But then these are connections…if they’re dif­fer­ent par­ty apps that you don’t real­ly use…that you then have to break down lat­er if those rela­tion­ships sour.

So remem­ber when I said that every­one thought that this project was real? It’s because I went to real­ly great pains to also make it look real. When peo­ple showed up, we had a recep­tion­ist who had cof­fee. There was a wait­ing room, and I had peo­ple sign in with the date they arrived, the rea­son, and I the time of their appoint­ment. There was then a paper ver­sion of the quiz if some­one walked in. Sometimes you get walk-ins. The doc­tor gets that all the time.

This is me work­ing. This is what my desk looked like. Everyone got their own fold­er that I would write their name on. I would write out what I called a receipt. It’s all the advice I’m giv­ing them and I’m tak­ing my own notes. We both got a copy of the terms of ser­vice agree­ment. And then I would send them on their way.

So it looked actu­al­ly fair­ly… It looked hacker‑y legit, you know. I mean…a falling-apart build­ing, but I’m giv­ing you legit­i­mate advice and you just paid me $15.

This is our recep­tion­ist, Lauren. This is the wait­ing room. This actu­al­ly was­n’t posed. I popped my head out and saw a bunch of peo­ple sit­ting and read­ing. This is me pro­vid­ing advice. And these are some stu­dents of mine that showed up. I taught a class on visu­al sto­ry­telling with social media, and they had shown up to observe.

This is also what the break­down of the terms of ser­vice agree­ment looked like. This is the first page. It’s pret­ty stan­dard. One of my favorite lines is My obser­va­tions of this per­son­’s behav­ior and respons­es gives me no rea­son to believe that this per­son­’s not ful­ly com­pe­tent to give informed, know­ing, and will­ing con­sent.”

I should also clar­i­fy a real­ly dear friend of mind who’s my col­lab­o­ra­tor, Fred Jennings, works for a law com­pa­ny called Tor Ekeland. They do a lot of dig­i­tal law cas­es. He actu­al­ly draft­ed this up specif­i­cal­ly for me, for the needs of this project. I told him to keep it short, and I told him to bold cer­tain things so users could real­ly see, if they were scan­ning, what I’m talk­ing about. So as you’ll see, var­i­ous social media plat­forms are bold­ed with what I’m giv­ing.

But then I had him out­line the nature of ser­vices. And what you’ll see is that the Client acknowl­edges the Coordinator pro­vides nei­ther con­tent or mate­ri­als intend­ed as finan­cial advice, coun­sel­ing, or ther­a­py.” And I real­ly want­ed to high­light I am not a ther­a­pist, and this is not a ther­a­py ses­sion. If any­thing, I’m like a real­ly weird SEO advi­sor that you’ve con­sult­ed to maybe talk about your per­son­al life with. But I am def­i­nite­ly not a ther­a­pist.

So I had twelve peo­ple fill this out online, four peo­ple do walk-ins. And what’s fas­ci­nat­ing is I only had two peo­ple show up and talk to me about heart­break. This project was not inspired by a breakup. It’s actu­al­ly about break­ing up with social media. I had some­one show up and ask me a lot of real­ly spe­cif­ic ques­tions about LinkedIn and her work­place, and what’s the prop­er way to break up on LinkedIn with your old job. And I was like, You should just prob­a­bly unfol­low them.” I’m like, Do you talk to them on Facebook?”

She’s like, I do.”

I’m like, Well, don’t do that.”

And I start­ed to gath­er all this real­ly fas­ci­nat­ing infor­ma­tion, specif­i­cal­ly around the ways in which my users were using social media. This is some­thing I’m going to open­ly share, prob­a­bly post- this talk, if y’all want to look at what I’m accru­ing.

Different things like, let’s get a lit­tle per­son­al, Why are you here?” Romantic rea­son, work rea­son, friend, fam­i­ly, gen­er­al social media? And maybe these ques­tions seem real­ly innocu­ous, but the way in which I was struc­tur­ing my per­son­al algo­rithm I had built, each of these ques­tions trig­gered a dif­fer­ent kind of answer that I could give some­one, and I could string answers togeth­er. So I could give a com­bi­na­tion of Answer A plus Answer D plus Answer J to sort of give some­one a high­ly per­son­al­ized response to what they’d giv­en me. But this is sort of the way algo­rithm work. It’s not high­ly per­son­al­ized; the com­bi­na­tion to the user just feels per­son­al­ized. And on my end, that was sort of the art project for myself.

And I asked the gen­er­al ques­tion Do you feel safe online?” I was slight­ly sur­prised only two peo­ple said no. But I was more sur­prised that actu­al­ly only two peo­ple said no. I thought it would be less, and then at times I thought it would be more. Given my research in online harass­ment, I was pre­pared for some­one to show up and say, I’m being vic­tim­ized of harass­ment,” and I had a whole dif­fer­ent answer ready for them.

But just the fact that most peo­ple had come with very gen­er­al prob­lems, I was actu­al­ly sur­prised that in gen­er­al 16% of appli­cants don’t feel safe online.

Then I asked how How often do you use social media?” Pretty much every day. How often do you want to be using it?” Pretty much every day.

And the one I found the most fas­ci­nat­ing was when peo­ple described what they used it for. About half of users said they used it for social­iz­ing. And when I asked what do you want to use it for, half of users replied with net­work­ing.” So there’s a sort of pull to actu­al­ly be tak­en off social media.

And this is what I learned from all of this. A lot of advice I gave peo­ple was, Maybe you should just quit Facebook.”

And that was met with a resound­ing, No. How dare you sug­gest that?”

And I was like, Okay, great. Let’s pull back. Let me offer some­thing else. Do you have a smart­phone?”

Of course.”

Delete the app from your phone?”

They’re like, Oh, that’s bril­liant.”

I’m like, I know, right? Whoa.”

But the one thing I actu­al­ly found the most fas­ci­nat­ing was most peo­ple did not under­stand­ing Facebook’s pri­va­cy check­ups. Whenever peo­ple talked about that they want­ed to social­ize less and be less acces­si­ble, the first thing I always said was, Well, what is your pri­va­cy check­up? Have you done one?”

They’re like, Oh my god, what’s that?”

I’m like, We have a prob­lem.”

And the one thing I found super fas­ci­nat­ing is that most users did­n’t realize—and this is actu­al­ly hyper-specific to one user that came through—that you are acces­si­ble even with very pri­vate set­tings on Facebook, to non-Facebook-friends chat mes­sag­ing you. And if you respond to that mes­sage, that chat is moved into your gen­er­al stream of chats and it makes your infor­ma­tion acces­si­ble to that per­son you have not friend­ed.

So I had a friend who was like, I want to be super pri­vate but the rea­son I keep my Facebook open is like what if a young game devel­op­er’s try­ing to reach me?”

And I was like, Did you know that chat does this?”

He was like, I had no idea.”

And I was like, Well, that’s ter­ri­fy­ing, but you could maybe use it this way if you’re not con­cerned about harass­ment but you’re con­cerned about being reached.” Because he was more con­cerned that friends could be exposed through his open­ness, which I was like that’s a very very con­sid­er­ate [?] way to take your Facebook.

One thing I learned is that no one knew any­thing about Instagram’s pri­va­cy poli­cies, nor did they care. They’re like, Eh, Instagram’s fine. We don’t care.”

Again, most peo­ple want­ed to use social media as a net­work­ing tool.

But the biggest take­away was every sin­gle per­son that showed up— And I had a vari­ety of peo­ple that were incred­i­bly savvy; they were engi­neers. They actu­al­ly thought that they did not under­stand social media as well as they could, and that they need­ed some­one to help them bet­ter under­stand. And they need­ed some­one to help them bet­ter under­stand who they paid $15 to in a hack­er video game space.

And I want that to sort of res­onate, because that is a joke. But also real­ly think about the fact that these tools are so neb­u­lous that you would go to a space and pay some­one $15 that you’ve nev­er met before that says they’re an expert, to just han­dle this for you.

And that for me was the biggest take­away. How can we make things feel more acces­si­ble, or bet­ter yet let’s make a new plat­form.

I’m putting this up here because this is one of my biggest pet peeves. You would prob­a­bly nev­er explain how to SMS some­one through screen­shots. You’d prob­a­bly say, Do you see that lit­tle thing on your phone, the chat? Open it up. Write in a num­ber. Say some­thing.”

When this project launched, a friend wrote about me and some work I had been doing, and one of her fol­low­ers legit­i­mate­ly believed that I did not under­stand Facebook. And he took it upon him­self to try to explain Facebook to me. Which oth­er than being kind of insult­ing because I work for a tech com­pa­ny and I have a Masters in inter­ac­tive tech­nol­o­gy, what I found illu­mi­nat­ing is the fact that this is not a weird response. This is not unusu­al, for some­one to say, Oh, right. Facebook is so hard to use when it comes to pri­va­cy and cre­at­ing lists of posts to peo­ple that I’m going to screen­shot every­thing for you.”

And that is nev­er the way in which you should explain a com­mu­ni­ca­tion tool to some­one. If you have to screen­shot some­thing to some­one, you fucked up as a design­er. [clap­ping from audi­ence] And those are my gen­er­al thoughts on that. I can’t even. I just can’t.

So I guess what I impart to you and all of us here is, let’s make some­thing not shit­ty.

And I know the rea­son peo­ple use Facebook, and this is not a talk to get off Facebook. I use Facebook. Facebook will per­sist for a very long time. Think of all the third-party apps that use Facebook as an auto­mat­ic login. That is a design pat­tern that rein­forces the need of Facebook in every­day users’ lives.

But as a design­er and tech­nol­o­gist, I want to make some­thing bet­ter even if it’s just for my friends. We could do socialmedia.onion? Thoughts? And that’s sort of where the future of this project comes in, as I’m actu­al­ly work­ing on a social media co-op with two tech­nol­o­gists in New York, Dan Phiffer and Max Fenton.

I’m doing anoth­er round of Social Media Breakup Coordinator in Oakland in January. And I’m hop­ing to keep gath­er­ing data around the ways users use these plat­forms through my per­for­mance art piece, but also as well as covert­ly keep [impart­ing] infor­ma­tion on pri­va­cy. And what would it be like if we lived our lives just a lit­tle less online. And I’m hop­ing to even­tu­al­ly have a real­ly robust data set that could be used as an actu­al data set and not just a sam­pling.


Audience 1: Did you ever ask the question why those people didn't understand the settings on Facebook, etc. and still used it. I mean, it's like walking into a gun shop, purchasing a firearm, and you have no idea what to do with it.

Sinders: Right. I think… What I said later in the presentation is we have these design patterns in everyday life that really actually enforce this use of Facebook. So this project was really centered around general users, and a general understanding of technology. We are moving to a very highly digitally-literate and data-literate society, but we're not there yet. There's pockets of literacy. This is a really good pocket of literacy right here; we're a really awesome community.

But one thing I strive is like, the people that are "misusing" or "misunderstanding" Facebook, they're not like elderly parents. They're actual cohorts of mine that are my age, people younger, and people even a few years older. And I think the reason is that Facebook is really easy. And it's highly addicting to use. And it's like a phone book; everyone's there. It stores birthdays for you, which is really actually helpful. It's a fast way to talk to people. But I think the bigger thing is it's enforced on other sites. Think of all the web sites you go to during a day and how many of them say "log in with Facebook," "sign up with Facebook." "Log in with Twitter."

And those design patterns, which seem really innocuous to us actually are really important. They further enforce the ubiquity of Facebook, because it makes it easy. I mean, I'm shuddering thinking about all the third-party apps that would be associated right now with a Facebook login if you've done that for every site. But the common user does not know that. And that's sort of the issue, I think.

Audience 2: Hi. Thank you very much for this interesting talk. I have basically two questions that kinda are the same, and they are around the art project part of your talk. And the first one is how did you make sure that people would actually understand that this was a piece of performance art. Did you rely on the absurdity of your proposition, that that would be recognized? Because clearly people thought not, and in terms of satire there needs to be some element of exaggeration or something that makes it clear that this is intended as a piece of art. So I was just wondering what your thought was that.

And the second part was where do you— So the data that you get from your piece of art you presented as a research outcome, almost. So that obfuscates the artiness of your project and turns into real-life data. And isn't that also one of the problems why people are so careless with Instagram, because they see it as art when they photograph their food, whether that's true or not? That's open to debate. But they don't see it as an actual act of data collection.

Sinders: Right. So, I guess to sort of back up. The way in which I describe myself [is] as a speculative designer, and I think about critical design a lot and critical making, and like what is that line. And oftentimes you're making something real that is sort of making a point. Most people seem to sort of get that this was an art project of mine. And it helped that I was in New York doing this, and I was doing it in an art gallery that's a video game gallery. So there were arcades in the back of the space.

But certain people, I actually realized… Because I had a couple people phone in that had sort of seen this and had signed up online and were not in New York. And I realized that they didn't know that this was an art piece. And I kind of went with it. And a lot of that is they were signing a terms of service agreement, I did tell them this is not therapeutic advice, I'm not held liable for any decisions that you make, and I said all that over the phone to make sure that they understood that. And then I told them these are just suggestions. You don't have to follow them, and you are allowed to push back. And that's what I tell every participant. I'm giving you these suggestions based off my best-practice knowledge and this algorithm I've designed that you don't get to see.

So your questions are triggering certain results, but you are also allowed to say, "I don't like that," and I can tailor them slightly. If you don't like the result at all, you should take the quiz again. But the bigger thing is that it walks this really weird line. And this is a weird anecdote, but I'm also a portrait photographer. My background's actually in fine art photography and I got a Masters in interactive technology years later. And my work was my family and I recreating moments post-Hurricane Katrina. So people always ask, "Are these real photographs, Caroline?" Well, they weren't taken on the fly. I set them up. But they were real to me.

And they're saying something. And that's the way in which I would describe this project. It's not real, but it was real to us in the moment, and it's commenting on things and also providing real solution.

Audience 3: Hi. Do you know of some software that shows how open you are to other people? For example, a Facebook app or something that shows you a mirror of yourself, more or less.

Sinders: I don't know of any sort of checker like that. I use a variety of different things. A friend of mind made a really great WiFi sniffer, which is at the extreme end of what you're talking about. Not that you all should do this, I often will unfollow and refollow, and unfriend and refriend people and change my privacy settings, and then I try to log out and get someone else to log in, see if they'll let me audit. And then I will see what I look like to other people. That's a level of insanity that I don't think most people in this room should necessarily engage with. But I don't actually know of a checker that lets you see that.

I know that there are analytics systems you can download with Instagram to see who's unfollowed you and followed you, and who's following you that follows other people that you know, which currently Instagram does not have that analytic feedback for users. It's a third-party app you have to download.

Twitter has started to add analytics on the side to sort of give an idea of how successful your tweets were. But they'll never say like, "This is who didn't follow you that accessed this tweet today." But they give you a more robust analytic breakdown of, "Your tweet about puppy dogs did really well, but your tweet about OpSec did not."

Audience 4: Hi. I really appreciate your insights on visual design and the user experience of data at rest. I'm really curious what your thoughts are on temporal design and the user experience of data in motion. Because you mentioned that one of the things that came out of your interviews was people having a sense of just sort of not understanding social media and feeling like they need help understanding social media.

In programming we talk about code smells, which are sort of features of code and how people use code that are a sign that something's probably not designed right here. And it seems to me that that sense of misunderstanding is a design smell. Maybe that there's just too much trying to consume users' attention and we need to change the rate at which we're delivering. Anyway, it's an open-ended question. I'm just really curious what your thoughts are there.

Sinders: So I've actually thought about that a lot. I actually haven't met with any engineers at Facebook or Twitter, but if you're here I'd love to talk to you. But I met with someone that worked in branding at Twitter and I asked him to just sort of talk about his day job and describe how the branding team targeted ads. Because I figured that was a really good way to get a sense of how the algorithm was working.

He started spouting a lot of buzzwords, as he is prone and wont to do, because he was a coworker of mine from a really old job. But he said something that was really fascinating to me. He was like, "Well you know, Caroline, there's just so much noise. We have all these different algorithms working, but it's just so much noise on top of each other and you're just trying to find this little signal." So I know for instance with Twitter it's exactly that problem, that they infrastructurally designed themselves incorrectly, and to combat it you can't… They're at a point where I feel like they cannot shut it down and rebuild it and become minimal, with a better working codebase. So they're building on top of everything.

The reason I also think that is a lot of anti-harassment that they have they've been rolling out for verified users and not for the common use base. So if you're a verified user, the way in which their anti-harassment initiatives work, it works way different and way better for you. They have an algorithm working where you will never see as a verified use certain harassment tweets. They're catching them before they come to you, and you can look at them later. But there's all these really highly specific changes— And I have not yet seen a verified user account. No one's let me log into theirs. (Again, if you have that, let's chat.) But I've seen enough screenshots and read enough about it, and talked to friends who have it. And it's like Twitter 2.0. It's just slightly better.

So what I think the bigger issue is, there's so much data in motion that they can only isolate it for what they are infrastructurally deciding who is a power user, and that power user infrastructurally actually becomes a better power user.

Audience 4: I guess the hidden question I have there is really more of, is Twitter eventually doomed to tear itself apart because it's triggering people's System 1 responses instead of their reasoning.

Sinders: I really don't have an answer to that question, because as a Twitter who's thought about quitting but I really love the community I have on Twitter, it's kind of a weird emotional negotiation that I have of like, I don't know how accessible I am and I face harassment on a usually monthly basis, for a variety of different things. And it's this weird negotiation I have of, why am I still here? But I actually legitimately like it. And I think that that's the big issue. It's like maybe it will pull itself apart, but harassment while it affects a lot of people is also affecting hyper-specific groups of people. And I think a lot of the fear around it, rightly so, is well if it happens to one person it could happen to you because we're infrastructurally in the same place and we're both equally open.

I guess I'm not sure. I'm interested to see what happens in the next two or three years, because Twitter is not gaining any new followers at this point. They're kind of starting to plateau. So they are not growing at a rate at which other social media networks are growing. And that's a major issue. And dome of that issue could be tied towards bad infrastructural design or really poor code of conduct.

Audience 5: Last year my brother blocked my mom on Facebook and she still vocally beat about it. I mention this because many times our online social network is almost directly mapped or closely intertwined with our offline social network. So did you look into making changes into these social networks that's online, how does that affect our social network offline? As much as you may unfollow and stop talking to your boss or your former colleagues, you'll still meet them at conferences and at tea parties. So how do you deal with that change of this network online which does not actually give a clear picture of how your social network looks like. [?] the interaction between the offline and online after that change.

Sinders: I definitely thought about that a lot. Just in general as a researcher I've always been really intrigued by societal norms and propriety and what is polite behavior across many different cultures. And I'm speaking as an American but I come from a hyper-specific place in the United States. I come from Louisiana, which is the American South. We're a hyper-hyper-specific culture. We speak two languages. It's English and then Cajun, which is an oral-based language. I only know a couple words. But New Orleans where I'm from has the highest rate of birth retention. 75% of people that are born there stay there. So the ways in which I socialize as a New Orleanian is very different than the ways I socialize as a New Yorker.

And that's true even, I think, when you get even more localized. If you look at Americans versus Canadians versus Mexicans and getting into Latin America. So I thought about that a lot, that actually a lot of the interactions you have offline definitely affect and influence the interactions you have online. So a lot of advice I gave people was also having to break down like, how often do you interface with this person, and let's think of the most neutral and polite way to break things down.

So yeah, I thought about that a lot. I haven't yet, with the people I've given advice to, said unfriend someone. Oftentimes unless the relationship has incredibly soured, that's usually the advice. But if it's in the case of a boss, for instance, my reaction is oftentimes, "Why don't you reach out to them if they're an old boss and say, 'I'd love to keep in touch. Here's my email. But I'm keeping my Instagram just for friends only.'"

Audience 6: From your project, I'm curious to know if you think that social breakup is actually possible or if it's not really possible because people end up seeing your stuff anyway.

Sinders: Is it bad if my response is "both?" I think that as social media users, for a really long time we've been taught to interact with social media in a particular way. And I don't think that that way is correct. Facebook actively wants you to post more, as does Twitter, and Instagram wants you to share and accumulate followers. And that's the way in which these networks grow. You're creating content and that content is analytics, and they package and sell that to advertisers.

I'm politically agnostic on that, but I have my own personal thoughts, as a researcher that's just what they do. But I think that that push towards sharing and cross-platform sharing, that you can cross-post, perhaps in terms of privacy is a horrible idea. What are you saying, how are you saying it, are all identifiers, and they're all identifiers that can pinpoint location and who you are, and who you are offline and where you are. And that's something I often do try to impart to people: what are you saying and when, and does it need to be said online?

So I personally, and I always give this example with people that sit with me, I'm like, "I personally try to not post location but I have a very specific reason I can't do that." And I had a lot of internal dialogue of should I post that I'm at CCC? What if someone's here and they want to talk to me about something that I don't want to talk about? Or what if I say I'm home, does that make my mom more of a target if someone wanted to try to swat us again?

And those are extreme examples. But it's also important to think about like, are you accidentally doxxing yourself? If you're saying, "I'm at the bar downstairs from my apartment. Let's check in on Foursquare and post that on Instagram," you've pinpointed where you live. And that's information that people don't actually need. So I always try to walk this line of like, I think it's totally find to post pictures of food and family and friends and to do it frequently. But I think it's important to know are you highlighting where you are and are you highlighting regular patterns in your lives? And are you then amplifying that to a variety of people that you don't know and you have no idea how many people are accessing it?

Audience 7: [Angel] I've got a question from the Internet. Yesterday there was a talk titled "The Possibility of an Army" by Constant Dullaart, who bought thousands of fake accounts. What do you think about these actions?

Sinders: I guess I need a little more context. This person bought thousands of fake accounts to…?

Audience 7: [Angel] Actually, I don't have any context for that for you.

Sinders: In graduate school, this fantastic ethnographer, Trisha Wang, came to speak to us and the professor, Clay Shirky, at the time was saying he bought her 50,000 followers in a day, and I think he paid like $100. I think it's really fascinating the ways that that bumps you up into a different sort of social strata, and how it presented her in a completely different way online. That changed the way in which people interacted with her and the amount of followers she started to accrue on a daily basis.

Oh, I think I'm… So, this person created like a thousand different accounts. I think that that's what that question may have meant.

Audience 7: [Angel] From what I read, he bought them.

Sinders: Oh, he bought them? I would be curious to know why. Like if he was looking at data or if he just bought a thousand followers. But I guess I need a little more information.

Introducer: Alright. Well the person's not here so we don't know. We have another question from mic #3.

Audience 8: Hi. My sister had an occasion where someone who she sort of…became a stalker, and didn't really know her very well, but then started to send really weird messages to her. And it got to a point where they were following her on Instagram and she can't really control…because this person knows who she is and her friends. She couldn't control her information and so this person would send stuff based on, "Oh, we bought this blender," and would deliver it to our house along with letters about like how they would have sex even though they had never really interacted before. And it got really scary and unsafe. And it's sad for that person but also it became really scary for my sister and she didn't know where to go and what to do. And when she went to the police and said, "I'm scared that this person might come and touch me when I'm going home late at night. What should I do?" They said, "Well, until something happens we really can't do anything for you." So there may be resources out there for people who are facing this, but for those watching this video, what would you recommend them to do?

Sinders: First of all I want to say I'm so sorry for your sister. That's horrible. And secondly what I would suggest doing is, there are a variety of different non-profits that exist. Crash Override is one once you've been harassed in a really specific way. What I would suggest is Smart Girl's Guide to Privacy has this. It's this really fantastic book, and they list where you can access, I think, lawyers that are more digitally savvy around digital crimes.

And my recommendation in a case of that with that kind of persistence where it's a regular person, meaning a regular stalker, it's one entity and they're actually starting to sort of move away from social media and moving into letters, you should get a lawyer. And from there figure out ways to assign space between you and the other person across state lines, even. I don't know the particulars of this case. If this person is in a different state than your sister, that gets a little bit trickier. Are they in the same city? [reply inaudible] So they're in the same city. There's a lot more you can do. My recommendation would be to immediately find a lawyer who is well-versed in online harassment. But if that person's in the same city and they're sending letters, I think that's a pretty good reason to start pressing charges. That would be my immediate reaction.

Audience 9: Yes. Thank you for your wonderful talk, first of all. One thing I find myself personally very preoccupied by is not just the question of how to act on social media in the present, but also how to clean up after my past. Certainly things I've written or posted before. And I actually find that the obstacle towards doing that is frequently infrastructural. It's really hard to sort of have an oversight of everything you've done in the past. What do you sort of see as the future of design on these platforms? Are they intentionally making it difficult, or have they coded themselves into a corner, and is this going to become a bigger problem?

Sinders: I guess I would say, basing off the way the design is now, I think it's a mixture of having coded into a corner and also trying to make design minimal. So, a lot of trends in design are around optimization and usability. But we're optimizing for speed and we're making things more usable for mobile. But we're not optimizing or designing for safety. And we're not optimizing or designing for longevity of life within interacting on these platforms. So I would say it's a misuse of design priorities. And I think that now there is some pushback with people saying, "How's this being accessed? There's harassment persisting on this platform. How's this happening? Oh, it's happening because of these reasons," etc.

I would say that it's just a misalignment of priorities within a design hierarchy and a coding hierarchy.

Further Reference

This presentation page at the CCC media site, with downloads available.

Help Support Open Transcripts

If you found this useful or interesting, please consider supporting the project monthly at Patreon or once via Square Cash, or even just sharing the link. Thanks.