Sebastian Schmieg: Thank you all so much for hav­ing me. So what am I going to talk about tonight? I will of course being an artist talk a bit about some art projects, but most­ly I will speak about the research I do as part of my prac­tice, specif­i­cal­ly about humans as soft­ware exten­sions. And towards the end I will share some thoughts on why being a human soft­ware exten­sion could actu­al­ly be some­thing maybe not pos­i­tive but bring some new pos­si­bil­i­ties. So I will try to end the talk on a pos­i­tive note.

In 2008, the sci­ence fic­tion movie Sleep Dealer spec­u­lat­ed about a future which could­n’t be more time­ly. The bor­der between Mexico and the US has been closed, there­fore immi­grant work­ers in the US have been replaced by robots. However, the robots are remote­ly con­trolled by peo­ple in Mexico who have their bod­ies plugged direct­ly into the net­work. Then two years lat­er in 2010, the CEO of CrowdFlower, which is a crowd­sourc­ing and crowd­work plat­form, Lucas Biewald speaks of a sim­i­lar sit­u­a­tion:

Before the Internet, it would be real­ly dif­fi­cult to find some­one, sit them down for ten min­utes and get them to work for you, and then fire them after those ten min­utes. But with tech­nol­o­gy, you can actu­al­ly find them, pay them the tiny amount of mon­ey, and then get rid of them when you don’t need them any­more.

Biewald’s remarks how­ev­er are not sci­ence fic­tion. Instead they describe a con­tem­po­rary con­di­tion.

So humans as soft­ware exten­sions. What is this con­di­tion? I would sum­ma­rize it as peo­ple extend­ing com­pu­ta­tion­al sys­tems by offer­ing their bod­ies, their sens­es, and their cog­ni­tion. And specif­i­cal­ly, bod­ies and minds that can be eas­i­ly plugged in and lat­er eas­i­ly be dis­card­ed. So bod­ies and minds algo­rith­mi­cal­ly man­aged and under the per­ma­nent pres­sure of con­stant avail­abil­i­ty, effi­cien­cy, and per­pet­u­al self-optimization.

So as such, humans as soft­ware exten­sions are both the foun­da­tion and the result of a megas­truc­ture which Benjamin Bratton calls The Stack. It’s a com­pu­ta­tion­al total­i­ty of plan­e­tary scale. So some­how we can imag­ine it as a planetary-scale com­put­er con­sist­ing of a stack of lay­ers from rare earth min­er­als to data cen­ters to bots and peo­ple. Which in this mod­el are exact­ly the same.

So in this com­pu­ta­tion­al total­i­ty, even the small­est nodes can be addressed and can be pro­grammed.

What this tech also describes is a new geog­ra­phy. So Google Maps defin­ing bor­ders, or as just seen in Sleep Dealer, the US build­ing a wall while still being able to plug right into the bod­ies of Mexican work­ers.

Two images: Video clip of a man disguising himself as a car seat as part of research into how drivers react to sharing the road with an autonomous vehicle; a computer station set up to remote control a self-driving car.

But to be clear, from my point of view the mod­el of planetary-scale com­pu­ta­tion as a total­i­ty is as much a real­i­ty as it is also a gigan­tic fan­ta­sy and ide­ol­o­gy of pow­er opti­miza­tion and effi­cien­cy. So this is the state of self-driving cars right now, at least from the per­spec­tive of Ford.

So I would now like to give some exam­ples about what I mean by humans as soft­ware exten­sions and what effects this way of man­ag­ing dig­i­tal labor has.

Three years ago I worked on a piece in which I explored dig­i­tal colo­nial­ism. And among oth­er things explored Google’s and Facebook’s attempts to inte­grate into their ser­vices those two-third of the world’s pop­u­la­tion who are not online yet. As you all know, Google wants to use bal­loons, Facebook wants to use drones, and they want to have them cir­cle above these areas that do not have any Internet con­nec­tion, basi­cal­ly suck­ing that which is below up into the net­work.

So the piece of which you see a lit­tle bit here is called How to Appear Offline Forever, and it con­sists of a mix of found mate­r­i­al like videos, images, and ques­tions. And there’s also lay­er of sto­ries writ­ten by peo­ple from Silicon Valley, Sri Lanka, and Zambia which are all loca­tions of impor­tance to this piece.

So in order to get in con­tact with peo­ple in Sri Lanka, I end­ed up using the out­sourc­ing plat­form Upwork, which offers a high­ly effi­cient inter­face for hir­ing free­lancers from all around the world. On their plat­form you can sort free­lancers by price, skills, rat­ing, and then you can pick who­ev­er you think fits the job best. And their user expe­ri­ence off hire and fire is well-crafted, soft­ware exten­sions that can be plugged in and removed again eas­i­ly. It does­n’t mat­ter where they are, who they are, as long as they get it done.

So once the free­lancers in Sri Lanka got to work, I noticed that I was not only able but also encour­aged to spy on them. Upwork records every key­stroke and reg­u­lar­ly takes screen­shots while free­lancers work, build­ing a grow­ing diary of their activ­i­ty. so I found myself in a sit­u­a­tion in which I was­n’t only being sur­veilled by cor­po­ra­tions or states, I was also doing the same myself, man­ag­ing” my exten­sions, spy­ing on them in order to mon­i­tor their per­for­mance. So this is not them—like cor­po­ra­tions or states—spying on us, us pro­tect­ing our­selves against them through encryp­tion. This is all of us fight­ing for our place in the net­work, try­ing to be valu­able nodes

So this is me and one of the free­lancers, which is a very love­ly lady I got to meet in Sri Lanka lat­er.

So if you look at this his­tor­i­cal­ly, and if you go back just fif­teen, maybe twen­ty years, out­sourc­ing via the Internet was a prac­tice that could only be employed by big IT com­pa­nies, most­ly from the US to India. Today it is cheap and easy, and it can be done by any­body.


A new ser­vice by Amazon called Amazon Key illus­trates this rather new sit­u­a­tion per­fect­ly. With Amazon Key you can remote­ly grant access to your apart­ment while you’re not at home. Using the Amazon Key cam­era, lock, and app, you can spy on the oth­er­wise com­plete­ly invis­i­ble work­ers from your smart­phone. So this is the lock, remote­ly unlocked. So here, out­sourc­ing does­n’t gen­er­ate free time. Instead it is born out of neces­si­ty. It is mar­ket­ed as mak­ing pos­si­ble the trans­for­ma­tion and a lib­er­a­tion from being man­aged to also being able to man­age oth­ers. And in this case you’re able to deal with them with­out ever hav­ing to meet them in per­son.

So now every­body not only can but has to, and actu­al­ly of course wants to. We all want to use peo­ple as soft­ware exten­sions. And with that comes also that we have to remote­ly track and rate their per­for­mance.

So let’s recap. Factory work­ers are extend­ing machines with their bod­ies. Freelancers have escaped the fac­to­ry but have to offer them­selves as flex­i­ble exten­sions to the mod­ern media assem­bly line. For exam­ple like we just saw on Upwork.

So now micro entre­pre­neurs have to invent their own shops, offer­ing their cre­ativ­i­ty in the form of lit­tle pack­ages that are called gigs. You can buy such gigs at a fixed price on plat­forms like Fiverr. For those who do not know fiverr, ini­tial­ly each gig on Fiverr was priced at exact­ly five dol­lars, of which the plat­form kept one dol­lar.

So some months ago I found a way to direct­ly access all videos uploaded to the plat­form in real-time, includ­ing every sin­gle video that peo­ple on Fiverr are pro­duc­ing for their clients. Through this crack in the surface—or you could also call it a secu­ri­ty prob­lem or a pri­va­cy issue they have—I could look at the leaked stream of videos, and I did this for days and weeks and months and I down­loaded videos worth of like more than 100 giga­bytes. And I was look­ing for some pat­terns to under­stand this mar­ket­place.

Screenshot of an Excel spreadsheet with a text editor window overlaid and a not reading 'Hello here is your files i did not sleep; work is important and you said your business may affect'

So, to give you some idea about what I saw there. On the plat­form it is dog eat dog. Be the best, the cheap­est, the most cre­ative, the most effi­cient. Be just like a prop­er soft­ware exten­sion: nev­er sleep, work all the time.

At the same time, every­body is also fight­ing against the plat­for­m’s algo­rithms and clean inter­faces that hide gig work­ers on page 2, 3, 4, and so on.

As many gigs offer unre­al­is­ti­cal­ly short deliv­ery times for cre­ative work, it becomes clear that they them­selves use bots, gen­er­a­tors, and tem­plates, sim­u­lat­ing” cre­ative work and cre­at­ing yet anoth­er lay­er of man/machine com­plex­i­ty. So, using automa­tion in order to not be replaced by automa­tion. Their biggest sell­ing point seems to be their low cost cou­pled with a tru­ly nat­ur­al inter­face: a human being.

In con­trast, there is anoth­er group of peo­ple con­scious­ly offer­ing their bod­ies, often as car­ri­ers of mes­sages like screens. So here the fan­ta­sy of the uni­ver­sal address­abil­i­ty and avail­abil­i­ty of all nodes man­i­fests itself in the dis­tant and often exot­ic body that acts as a screen. This goes hand in hand with the kicks offer­ing per­son­al porn or erot­ic videos and fetish videos.

Screenshot of a gig project titled 'I Will Make Your Face Pizza" with a headshot of a man whose face has been replaced with a pizza except for the mouth and eyes.

So here I want to sum­ma­rize. Being a soft­ware exten­sion on a hyper­com­pet­i­tive plat­form fos­ters and demands some­thing that I will call sur­vival cre­ativ­i­ty.” That means com­ing up with what­ev­er it takes to sur­vive in a com­pet­i­tive envi­ron­ment. And as a reminder, Fiverr might be an extreme exam­ple but it exem­pli­fies a devel­op­ment that has become a real­i­ty for many already, and it’s not like it’s them and us; we are all human soft­ware exten­sions.

So far I’ve man­aged to talk about soft­ware with­out men­tion­ing arti­fi­cial intel­li­gence even once. That’s nice I think. Instead I’ve drawn this bleak pic­ture of a quasi-totality of work and exploita­tion. However, automa­tion and arti­fi­cial intel­li­gence sup­pos­ed­ly imply a future with­out work, right?

So in the pre­vi­ous exam­ples plat­forms, soft­ware, arti­fi­cial intel­li­gence act­ed as sci­en­tif­ic man­age­ment,” the Taylorist boss algo­rith­mi­cal­ly dis­trib­ut­ing and mod­u­lat­ing human work­ers as soft­ware exten­sions.

Image of the book Inventing the Future, by Nick Srnicek and Alex Williams

Now, one great post-work idea is to not only auto­mate the man­age­ment of work­ing bod­ies and minds but instead to com­plete­ly replace all human nodes with AI as well. While I think this is an excel­lent foun­da­tion for dis­cussing our soci­ety’s obses­sion with work, I would also argue this hypoth­e­sis is as appeal­ing as it is flawed, unfor­tu­nate­ly.

So, my obser­va­tion is this. AI—artificial intelligence—is an appro­pri­a­tion and a pos­si­ble extrap­o­la­tion of exist­ing knowl­edge and skills, yes. And as such it might as well do our jobs. But it is first and fore­most used as a scheme to frag­ment work into tasks that can be done any­where, 247, and to make this labor invis­i­ble.

What we see here is a piece called Segmentation Network which I made last year. It plays back over 600,000 seg­men­ta­tions man­u­al­ly cre­at­ed by Mechanical Turk crowd­work­ers for Microsoft’s COCO image recog­ni­tion dataset. These so-called seg­men­ta­tions are based on pho­tos we have uploaded to Flickr, and they are used in machine learn­ing, train­ing AI what it can see and what not. So you can auto­mate as much as you want, but at some point you will have to train and espe­cial­ly main­tain the machines and the soft­ware.

Images of crowdwork tasks asking people to select images from a group containing particular objects or animals.

So, I would say AI cre­ates yet anoth­er lay­er of bad­ly or unpaid care and main­te­nance work, which is often invis­i­ble on pur­pose. I would say this needs to change. And I think fem­i­nist the­o­ry and prac­tices have have a lot to say about this issue, but that’s anoth­er talk and I would like to hear it.

So here is a point in case and may be a solu­tion. At the end of 2011 while still being stu­dents and shar­ing a stu­dio, Silvio Lorusso a friend an artist and me, we start­ed to take a screen­shot of every sin­gle CAPTCHA that we had to solve while nav­i­gat­ing the Web. So over the years prov­ing that we are human time and time again, we cap­tured hun­dreds of CAPTCHAs.

This year, we thought about how to make this lit­tle thing, or these lit­tle things, the col­lec­tion of these things into some­thing which is as valu­able and as expen­sive as pos­si­ble. And so we pub­lished the com­plete col­lec­tion as a series of five hand­made lep­orel­lo books. So each of these books is one year. And if you if you expand all these lep­orel­los they have a total length of nine­ty meters, chron­i­cling five years of micro­la­bor as well as the his­to­ry of CAPTCHAs.

So if you look at it, CAPTCHAs start­ed as a tech­nique to mere­ly pre­vent spam. And then they kin­da mor­phed into a method for deci­pher­ing house num­bers and tran­scrib­ing books. And then late­ly it’s become a means of teach­ing image recog­ni­tion to AI soft­ware.

While we were col­lect­ing these CAPTCHAs, Gabriela Rojas-Lozano in 2015 filed a class action law­suit against Google. And she claimed that Google oper­ates a high­ly prof­itable tran­scrip­tion busi­ness built upon free labor, which it decep­tive­ly and unfair­ly obtains from unwit­ting web­site users.” Unfortunately her claims were reject­ed. So here the judge states like, you spent only a few sec­onds on this, you can­not be expect­ed to pay for such a small job. However, her attempt to sue Google was still a suc­cess, I would say, because it led to the proof that Google has per­fect­ed a mag­i­cal process in work is trans­formed into lit­er­al­ly noth­ing. So wel­come post-work soci­ety.

How does this mag­i­cal trick work? It’s rather sim­ple. You take a job, let’s say tran­scrib­ing books, and you frag­ment it, and you frag­ment it more and more and more, until sud­den­ly the job is mag­i­cal­ly done with­out any­one hav­ing ever worked on it. Because if nobody has to get paid then nobody had to work either, right? Hence the judge’s state­ment is proof that this mag­ic actu­al­ly works. It gets bet­ter. In the end, Google still ends up being paid even though they have just made the job dis­ap­pear mag­i­cal­ly.

So what I want to sug­gest now is to seize the means of mag­ic. What about this: frag­ment­ing those plat­forms that algo­rith­mi­cal­ly man­age us to such a degree that they sim­ply do not exist any­more. Medically their job will still be done and in the end, we get the mon­ey. I guess in a less mag­i­cal ver­sion. We could call this plat­form coop­er­a­tivism.

If you do not believe in mag­ic then I have two more sug­ges­tions about what we as soft­ware exten­sions can do. So I will show you two clips from my laces video piece which is called I Will Say Whatever You Want in Front of a Pizza. You can see the full ver­sion online on this web site, or I think there’s going to be a screen­ing some­where here in the video lodge. Anyway I will show you two bits even though it’s a twelve-minute video and just look­ing at two bits does­n’t make too much sense. Still I think they show a point I want to make pret­ty nice­ly.

So, the video is nar­rat­ed from the per­spec­tive of a cloud work­er. And this is the pro­tag­o­nist. [plays first ~1:00 of the project video] So that’s the pro­tag­o­nist.

To give you some back­ground, in 2016 Donald Trump’s team hired a Singaporean teenag­er through Fiverr, which is the plat­form I talked about before, and they hired her to con­vert a PowerPoint into a Prezi, which is the soft­ware I’m using for the video, basi­cal­ly out­sourc­ing the Make America Great Again cam­paign. It’s a true sto­ry.

So, at some point in the video the pro­tag­o­nist, who is now work­ing not only as a piz­za deliv­ery bot but also as a cloud­work­er on Amazon’s Mechanical Turk plat­form prepar­ing datasets for AI, at some point he gets to know the Singaporean teenag­er as a fel­low work­er. And she’s got an idea. [plays first 3:455:36 of the project video]

Okay so the idea is this: when we are extend­ing soft­ware with our bod­ies and minds, we are also extend­ing our reach into the soft­ware and reach­ing into the soft­ware, being part of the soft­ware, you can start to manip­u­late these sys­tems that gov­ern us and that we have to use to gov­ern oth­ers. And once we’re plugged in, we can manip­u­late data, we can cre­ate new and weird and slow and inef­fi­cient soft­ware from with­in. So it can be fun like leav­ing Easter eggs for oth­ers to find real­iz­ing yes, there are actu­al peo­ple inside these sys­tems.

Which brings me to my third and last thought why being a soft­ware exten­sion also has some chances or pos­si­bil­i­ties. I will end the talk by talk­ing about Mark Zuckerberg, which will make me look like a fool. Anyway. Being a soft­ware exten­sion can also offer a new aes­thet­ic and a new way of being. And I think this video, which could be called the father of all stu­pid demos illus­trates this in a rather inter­est­ing way.

Here for what­ev­er rea­son, Zuckerberg is demon­strat­ing Facebook’s vir­tu­al­i­ty by vis­it­ing Puerto Rico in the wake of Hurricane Maria. So why do I show this video? Contrary to what he had intend­ed, Zuckerberg as a crudely-abstracted ver­sion of him­self turned into a soft­ware exten­sion, detach­es and dis­so­ci­ates him­self from the real world. This is what I like.

Because I think and you will agree with me, soft­ware is not per­fect. It’s full of bugs It often behave in unex­pect­ed and weird and glitchy ways, doing stu­pid things often and often over and over again in an infi­nite loop. Therefore embrac­ing the weird and abstract aes­thet­ic of a human as a soft­ware exten­sion could actu­al­ly allow us to detach our­selves from cir­cum­stances under which we are required to be our best work­ing selves all the time. So, being avail­able all the time, address­able, pro­gram­ma­ble, to update our­selves all the time. We could use being a soft­ware exten­sions, the aes­thet­ic of being a soft­ware exten­sion, as a mask behind which we can hide, pre­tend­ing to be a bot. Thank you.

Further Reference

Session page, and media

34C3 Wiki

Help Support Open Transcripts

If you found this useful or interesting, please consider supporting the project monthly at Patreon or once via Cash App, or even just sharing the link. Thanks.