Golan Levin:Hello and wel­come back to our Art && Code: Homemade. My name’s Golan Levin. I’m pro­fes­sor of elec­tron­ic art at Carnegie Mellon University and direc­tor of the Frank-Ratchye STUDIO for Creative Inquiry. And this is the third and final day of Art && Code: Homemade 2021. 

The Art && Code fes­ti­val series is con­cerned with democ­ra­tiz­ing the cul­tur­al and cre­ative poten­tials of emerg­ing tech­nolo­gies. We have a Discord serv­er to sup­port com­mu­ni­ty con­ver­sa­tions, by the way, if you’re watch­ing this. And if you’re inter­est­ed in par­tic­i­pat­ing in those chats we ask you to please reg­is­ter at artand​code​.com, which is our fes­ti­val web site. And you can specif­i­cal­ly go to artand​code​.com/​h​o​m​e​m​a​d​e​/​r​e​g​i​s​ter to receive the link for the Discord chats. 

It’s a pan­dem­ic. It’s win­ter. We’re quar­an­tined in our rooms. And the world is in tur­moil. How can we stay vital­ly cre­ative and con­nect­ed at this moment? We present Art && Code: Homemade, a free online fes­ti­val fea­tur­ing inspi­ra­tional talks by cre­ators we admire who work with dig­i­tal tools and crafty approach­es to make things that pre­serve the mag­ic of some­thing homemade. 

Art && Code: Homemade has been made pos­si­ble by an award from the Media Arts Program of the National Endowment for the Arts, by the Sylvia and David Steiner Speaker Series at Carnegie Mellon University, and by sup­port from gen­er­ous dona­tions to the Director’s Fund at the Frank-Ratchye STUDIO for Creative Inquiry. 

This morn­ing, it’s my ter­rif­ic plea­sure to intro­duce Cyril Diagne. He’s a Paris-based inter­ac­tion design­er and cre­ative coder spe­cial­ized in the appli­ca­tion of machine learn­ing and com­put­er vision to human-computer inter­ac­tion. He’s a co cre­ator of ClipDrop, an aug­ment­ed real­i­ty app for cut­ting and past­ing between phys­i­cal and dig­i­tal worlds. Cyril Diagne. 

Cyril Diagne: Thank you Golan. A few words about myself. I’ve always been a bad stu­dent. This is my final grade at a French bac­calau­re­ate. I had three out of twen­ty, which is…really bad. I know that folks in the US use a dif­fer­ent nota­tion system—A, B, C, D—so to sort of give a com­mon ground, if we apply this ratio to a key­board this is keys that you’re left with: 

So I’ve always been a bad stu­dent. This was my own native lan­guage, the French test. I got three out of twen­ty. I felt like okay, nor­mal­ly the path is you’re a good kid, you have good grades, and you have a good job. I failed at the first two; there was no way I would get a good job. So I was like fuck it, I’m just going to be an artist because I guess that’s the only thing that’s left. 

So I did that. I learned how to code and became an artist at the art col­lec­tive LAB212 in France. Which was a phe­nom­e­nal expe­ri­ence. We got to cre­ate inter­ac­tive instal­la­tions that we exhib­it­ed around the world. 

This is my very first project in the col­lec­tive, called POP for ​“People on Pop” in 2009 with young school stu­dents from the Meknes school in Morocco. And we cre­at­ed this instal­la­tion lets you take a pose from a chore­o­g­ra­ph­er, take a pic­ture when you reach that pose, an add it to a col­lec­tive video dance. 

And just with the theme ​“home­made,” this showed how we record­ed the chore­og­ra­phy. Because I was so broke I had a blue roll of screen. And the shop did­n’t have blue, they only had green. And I only had the mon­ey to buy one roll. So I had the bril­liant idea ​“Oh, it’s no prob­lem. I’ll just put both togeth­er and I’ll keep both col­ors.” Of course if you’ve done video key­ing you know how stu­pid an idea that was. And you can imag­ine the pain I had to key that video. But you know, at the end of the day it does­n’t mat­ter so much. It was painful, but it worked. And it was enough to get started. 

And for about sev­en years I just did that, real­ly, hack­ing around at home. Here you can see my for­mer home. For a quick anec­dote, because rent in Paris is so expen­sive I did­n’t have space for a lab, so I had to get rid of my couch. So I did­n’t have a liv­ing room, I only had a lab instead of the liv­ing room. But it was just so fun, every day just going on about what­ev­er it was felt that inter­est­ing at the time, not wor­ry­ing if it fit any box­es, If I was going to be able to exhib­it this, to show it even. Didn’t real­ly mat­ter. It was just about hack­ing all day long and exchang­ing ideas with the friends of the col­lec­tive and see­ing what could be done with it. 

And strange­ly and iron­i­cal­ly, just doing that even­tu­al­ly led me to an aca­d­e­m­ic job. Because ECAL, the uni­ver­si­ty of design in Lausanne, Switzerland offered me the role of pro­fes­sor and two lead the bach­e­lor of media and inter­ac­tion design. Which felt absolute­ly crazy because as I men­tioned, I’ve always been a bad stu­dent. But they said actu­al­ly this kind of prac­tice is what we do in the bachelor. 

So I was a bit skep­ti­cal at the begin­ning but then I real­ized that it’s true. It’s real­ly the way that the bach­e­lor is sort of struc­tured and the way that stu­dents go about cre­at­ing and under­stand­ing, and yes cre­at­ing knowl­edge in the field of media and inter­ac­tion design, which is an extreme­ly hands-on, prac­ti­cal approach to inter­ac­tion design. 

So I had three won­der­ful years there. This is one project which I real­ly liked, which we exhib­it­ed at the Milan Design Fair, which is When Objects Dream. So here, the stu­dents turned objects of every­day life and tried to explore what could be their inner life and how we could get a peek on that inner life. 

So it was won­der­ful and it rec­on­cil­i­at­ed me with acad­e­mia. But I fell in love, both lit­er­al­ly and fig­u­ra­tive­ly. Literally because I got mar­ried that year, and my wife was doing long-distance. She was in Paris. And we thought it’d be nice to live togeth­er, so I moved back to France, to Paris. And fig­u­ra­tive­ly because— [laughs]

Okay. That was an ambi­tious con­nec­tion to make, I real­ize now. I fell in love with— [laughs] I fell in loves with AI. I think that was in 2014. I saw the incep­tion paper, and real­ly the same way that I felt when I saw code and I felt how it was going to change the world and com­plete­ly change my life, I felt the same thing with AI. I felt the same spark again that this is going to be incred­i­bly important. 

This shows an exam­ple about from a beau­ti­ful paper from Shan Carter and Michael Nielsen that sort of goes through how AI can help cre­ate more nat­ur­al inter­ac­tion with con­tent and with data. So this real­ly shows strik­ing­ly how user inter­faces are going to be trans­formed by this technology. 

And yes, I was lucky to enter a very long res­i­den­cy of six years at the Google Arts & Culture Lab, which I just stopped in December. And I was able to focus entire­ly on that, so that was real­ly real­ly real­ly nice. And of course some crazy, unex­pect­ed things hap­pened. For instance show­ing my hacky openFrameworks visu­al­iza­tion of art­works to Steven Spielberg and James Cameron at TED. It was a bit ran­dom but it hap­pened. So that was real­ly an unex­pect­ed jour­ney through the cor­po­rate world of big FAANG companies. 

I was able at the Google Arts & Culture Lab to real­ly apply machine learn­ing to inter­ac­tion design. This is a project called Portrait Matcher which is very sim­i­lar to Sharing Faces from Kyle McDonald but here applied to art­works. And it was fun, it was nice. But it felt a lit­tle bit gim­micky, a lit­tle bit— The con­nec­tion with the art­works was a bit shal­low and I want­ed to see if there was a way to cre­ate a deep­er con­nec­tion with art­works and using the face. 

And so that was in 2015. The FaceNet paper was out but there was no pub­lic imple­men­ta­tion of it. But being at Google, I was able to use their inter­nal imple­men­ta­tion because it’s Googlers who made this paper. And so it was crazy, I was able to just put all the art­works of Google Arts & Culture through the mod­el, get the embed­dings, and then do nearest-neighbor search from any image. And it was mind­blow­ing. The results we were get­ting were… I was absolute­ly shocked by some of the con­nec­tions that it would make. And not just on por­traits but on any per­son that would appear in an art­work. So that was pret­ty insane. 

And I real­ized that as I was show­ing it, peo­ple were send­ing me their pic­tures and they would ask me to send them their match. The more I would send back, the more would come. And at the time I had to do them one by one man­u­al­ly on a Jupyter note­book. And it became quite obvi­ous that peo­ple were cre­at­ing a super strong con­nec­tion with what­ev­er match they were getting. 

And it was quite amaz­ing that this project actu­al­ly became pub­lic and became a fea­ture with­in the Google Arts & Culture app, which became viral and pushed the app as num­ber on on all the stores. It was pret­ty insane. 

Now, I think that was in 2018. And of course there’s been many devel­op­ment and it’s still crazy to me today that despite all the crazy issues that were present in that app, in that fea­tures, that it still man­aged to get sort of a pos­i­tive nar­ra­tive attached to it. 

‘It Took Us by Surprise’: Even Google Was Shocked by the Success of Its Megaviral Face-Matching Art App

Of course, in part I would say thanks to this proxy, it helped fos­ter the con­ver­sa­tion on all the issues that it had. And I would­n’t think that the same thing would hap­pen again today. But it was sort of a lucky moment to have released that project. But any­way, I won’t go too long on this. 

We did many exper­i­ments that are acces­si­ble on the exper­i­ments web site. But then I start­ed to feel a lack of some­thing that was real­ly miss­ing and was increas­ing­ly painful, which was agency. Being able to choose and to bring a project to light. 

For instance this isn’t one exam­ple that I still to this day find absolute­ly delight­ful and I adore, which is a watch face that shows you dancers that match the posi­tion of the watch hands. And it was a bit strange that I felt no one real­ly cared to make this app pub­lic. And yeah, it made me feel this lack of agency that you have in the cor­po­rate world, where of course there are gate­keep­ers. And it remind­ed me of all the issues I had at school with peo­ple that are not nec­es­sar­i­ly the peo­ple who the art­work is for but are still the ones to judge if they’re gonna get to see it at all. And I felt that felt wrong. 

And so that real­iza­tion coin­ci­dent­ed with the lock­down. And it was an oppor­tu­ni­ty to force myself to cre­ate inde­pen­dent pro­to­types and research that would only ful­fill my own interests. 

And so I set up this sort of struc­ture where in a max­i­mum one week—but as it turns out most of the projects actu­al­ly were done over the week­end. But there would be this hard lim­it of one week in order to be able not to even think if what I’m doing makes any sense. But the goal is real­ly the sprint. Someone who sprints does­n’t stop in the mid­dle to think if he’s going fast or not—he just goes and looks back after. This is why I don’t like the idea of a two-week sprint, which hap­pens a lot in cor­po­rate. Because I think it sort of fails the pur­pose to have this sort of week­end break in the mid­dle. So no: max­i­mum one week, and the most impor­tant thing is no pause. You go as far as you can, you go to the end, and after you, look back and you real­ize oh, what have I done? This is stu­pid, I should absolute­ly not show that. Or, the oppo­site, you’re like well, now that it’s done you know what? Let’s just show it and maybe it will inter­est some­one. And this exact process has changed my life. And maybe it can change yours too. 

So yes, the process was very sim­ple. Because in [tune?] toward AI plus inter­ac­tion design. So you basi­cal­ly mix one mod­el, and there’s plen­ty avail­able on Papers With Code, Github. A dataset. So again, you have visu­al data, Google dataset search. Plenty of freely-accessible datasets avail­able. And an inte­gra­tion. So the inte­gra­tion, you come up with your own sort of expe­ri­ence and cul­ture. And whether it’s an app, a tool, web site, what­ev­er. A phys­i­cal object, a piece of elec­tron­ics, whatever. 

But then these sort of spark an exper­i­ment idea. Then you can process it and fine-tune a mod­el using this unique mix with this unique mix as intent. And then you can deploy the mod­el and share a demo, and very impor­tant­ly, share it. If you feel like you there is no risk of caus­ing harm with what you’ve done…at least this is what I was apply­ing. If there’s no risk to cause harm, then I should­n’t try to judge too hard. I should share it and let oth­er peo­ple find maybe some­thing that would res­onate with­in it. includ­ing the code. Because the rea­son why this research exist­ed was thanks to open source, so it was absolute­ly crit­i­cal to make all those pro­to­types open source. 

So this is one exam­ple called Instagram 3D Chrome Extension. It was done on a week­end but just apply­ing the 3D Photography Using Context-aware Layered Depth Inpainting paper from CVPR 2020 to Instagram. So using this super-hackish way where you don’t need a GPU because it’s using Colab, you can have a Chrome exten­sion that turns all images on Instagram into 3D auto­mat­i­cal­ly. So I thought this was an inter­est­ing way to sort of revis­it exist­ing con­tent, here the beau­ti­ful pho­tos of Martin Parr. 

This was anoth­er exam­ple of a week­end exper­i­ment again, Face Doodle using MediaPipe. So here this ongo­ing obses­sion with the face and inter­ac­tion, inter­act­ing with your face but here being able to just doo­dle and draw sim­ple shapes and col­ors on your face. This just shows a lit­tle bit of how it works. But again, just using open source tech­nolo­gies— One of the most beau­ti­ful things for me about open source is that you don’t need per­mis­sion. This is such an under­es­ti­mat­ed aspect of open source, which is that because there is no price, because there is no license, because there is no ​“con­tact us” but­ton to get a tri­al… You just get the code, you don’t ask per­mis­sion from any­body. You just get going. And I find that extreme­ly powerful. 

And for the anec­dote, it can have such unex­pect­ed rip­ple effects. I mean, I don’t think Golan Levin knew he knew that he was going to change my life, me Cyril in France, when he zipped a few C++ files and sent them over to Zach Lieberman. And yet, this is just the way this code goes when it’s open source. 

This is anoth­er exam­ple, Freestyle Flip Detection using Tensorflow.js run­ning in the brows­er. Here the idea was to do a sort of jug­gling game where you could play with your phone with­out look­ing at your screen. So it was a bit short in the week­end. I did­n’t fin­ish it but I man­aged to do a sort of func­tion­ing pro­to­type. But it was fun.

And then there was this one. AR Copy Paste. Just so you know— I’ll let you see the video. But just so you know, for me this was real­ly one pro­to­type like the oth­er. I thought it was cool because oth­er­wise I would not make it, but for me it was­n’t some­thing par­tic­u­lar­ly dif­fer­ent from the oth­ers. It was just anoth­er week­end. I start­ed on Friday at home using what­ev­er equip­ment I had. And released the video on Sunday. So real­ly home­made, extreme­ly scrap­py. The code is open source, you can see how badly-coded it is. But it worked. And it res­onat­ed with the folks who got to see it online. 

This is a lit­tle bit of how it works. So, using SIFT—I won’t go into too much tech­ni­cal detail, but using OpenCV it’s pos­si­ble to know which loca­tion on the screen the phone is point­ing at. And the pro­to­type was using BASNet, a mod­el from Xuebin Qin and oth­er researchers from the University of Alberta, it’s pos­si­ble to detect which pix­els of an incom­ing image are part of the main sub­ject and which ones are part of the back­ground using a con­vo­lu­tion­al neur­al net, sort of a U‑net. And com­bin­ing the two, it’s pos­si­ble to cre­ate fair­ly eas­i­ly this tool, this app, that can copy any­thing around you and past it direct­ly in what­ev­er tool you’re using at whichev­er loca­tion you need it. 

So the cool thing is that it reduces— Even in the best sce­nario, if you have AirDrop… So you have an iPhone, you have a Mac, you have every­thing set up, it requires eleven oper­a­tion. With this it requires two operations. 

And a crazy thing hap­pen again, which that the world took notice, and even acad­e­mia took notice. Because I got to my first pub­lic paper on Arxiv. Remember, I’m a bad stu­dent so it’s a bit weird. And even invit­ed to talk dur­ing the Black in AI Workshop at the NeurIPS, which is an extreme­ly for­mal aca­d­e­m­ic conference—the oppo­site of this one. 

And so it’s real­ly… Yes, the world react­ed in unex­pect­ed ways. And anoth­er anec­dote for that is LAB212, we only start­ed exhibit­ing in France once we got inter­na­tion­al recog­ni­tion. And we’re like ​“Guys, we’ve been there the whole time. What took you so long?” But it’s this thing that peo­ple will look some­where else, maybe exter­nal, for val­i­da­tion. But it showed the ben­e­fits of not only being in one place. 

And yes, the world took notice. The pro­to­type went kind of viral. It showed up in many blogs, and my neigh­bor came and was like ​“What’s hap­pen­ing? I see your name on LinkedIn. I don’t under­stand.” It real­ly blew out of proportion. 

work­ing on a way to copy & paste mar­gar­i­tas irl #ar #ml pic​.twit​ter​.com/​n​h​Y​P​e​L​T​5Pq

— Matt Reed (@mcreed) August 17, 2020

And because the code was open source peo­ple were able to tweak it and turn it into their own ver­sion. Here is one of my favorite exam­ple, some­one mak­ing cock­tails from the pho­tos that they find online. So here mar­gar­i­tas with what I guess is a mar­gari­ta machine. I did­n’t know that exist­ed but appar­ent­ly. So, I found that super cool. And I would have def­i­nite­ly nev­er had the idea to make that. That’s anoth­er beau­ty of open source. 

Now, with every­thing that gains pub­lic vis­i­bil­i­ty, of course you get the trolls. And inter­est­ing­ly with this project, the trolls were com­ing with the idea that it’s fake. That it was a fake video done in 3D. And the crazy thing was, some argu­ments seemed to real­ly make sense. I mean I’m like no, objec­tive­ly if I had not built my pro­to­type myself and I just read this com­ment see­ing that video, I could believe this. It’s fair­ly elab­o­rate. And so there were two routes. Either I would fight online with every sin­gle one of them and it would have led nowhere. Or, I could just make it pub­lic and have that any­one can test it and see that it actu­al­ly works. 

And so thanks to the help of Jonathan Blanchet and Damien Henri, two friends, we turned it into a pub­lic app. So some­thing that start­ed extreme­ly scrap­py in a week­end hack just because why not, pret­ty much, became an app that peo­ple were actu­al­ly using in their work­flows. And it’s being used by hun­dreds of thou­sands of peo­ple, which is still crazy to me. Because as I men­tioned again, it did­n’t start with a sort of grand vision. It was a small, fun pro­to­type amongst many. 

And it keeps going on, because now we are a team of six peo­ple. Because a prod­uct requires a com­pa­ny, because now we have so many aspects going on. We need help so we are six. And we’ve joined a start­up accel­er­a­tor called Y Communicator. So in pret­ty much all aspects we are effec­tive­ly in a start­up now, which is again real­ly fun­ny because it’s nev­er been a dream of mine to become an entre­pre­neur. I would even say I would maybe a bit of a neg­a­tive view of the start­up world. So, it’s crazy to real­ize that now effec­tive­ly I’m in a start­up. So it’s quite funny. 

And again it’s unex­pect­ed and it kind of does­n’t make sense. But at the end of the day, as long as I feel that my integri­ty and per­son­al con­vic­tions are main­tained, why not? 

And okay, now I real­ize that this tran­si­tion is a bit hard­er than I thought, with Marco Polo but I— Okay. Do I have time? A lit­tle bit of time. 

So, the idea is it’s nev­er been that easy to change life. Our envi­ron­ment shapes us in many ways. Of course the phys­i­cal environment—the cul­tur­al envi­ron­ment shapes us in many ways. And in many cas­es this can be lim­it­ing for cre­ativ­i­ty, for express­ing our­selves and who we are. And espe­cial­ly dur­ing lock­down, a huge part of that envi­ron­ment is dig­i­tal. And it’s incred­i­bly easy to change our life dig­i­tal­ly. You remove a few apps, you install a few oth­ers, you unfol­low some peo­ple, you fol­low some oth­ers. And that’s it, you’ve changed lives. You don’t need any­more to go like Marco Polo, go through the Silk Road. Which at the time was an insane idea. You had to leave your fam­i­ly, you have to rad­i­cal­ly change so many things. But then he came back from that jour­ney with the first writ­ing coal, porce­lain, gun­pow­der. So all these incred­i­ble things that he found. Back then to have these crazy sto­ries, to have these crazy adven­tures, you had to give up on many things. Right now you don’t have to give up on so many things. You can entire­ly change your life by just switch­ing tabs. 

So, I guess my point is that because our envi­ron­ment shapes us in so many ways and can lim­it us so strong­ly, and because it is so easy to rad­i­cal­ly change this envi­ron­ment, it seems like there are a lot of incen­tives to try. And if that’s just one week and see how it feels, for me it has changed my life. So yeah, maybe it can change yours too. Thank you. And I guess now we have time for ques­tions, if there are any ques­tions. I’ll see if I can see the Discord or if Golan has questions. 

Golan Levin: Thank you so much, Cyril. This was an amaz­ing pre­sen­ta­tion, full of a lot of ideas. We’re a tiny bit short on time and there’s a lot of ques­tions that are in the Discord for you. Many peo­ple are ask­ing you know, how do you get your ideas, and do you keep a note­book. And you know, how do you nav­i­gate social media and viral­i­ty, because it seems like you’re not just good at com­ing up with ideas but you’re good at com­ing up with ideas to have viral poten­tial. Unfortunately I’m going to have to ask you to maybe talk to folks in the chat because we have to get our next speak­er on deck.

Cyril Diagne: Yeah.

Levin: But this was a great pre­sen­ta­tion. I know I’m going to real­ly think about this notion of a sprint real­ly hav­ing to go with­in one week. I think that’s super impor­tant, because the break of the week­end changes things. Thank you so much, and every­one we’re gonna resume in just a cou­ple min­utes with Lee Wilkins. There’s a lot of clap­ping on the Discord. Thank you so much Cyril, and we’ll all see you in two or three min­utes for Lee Wilkins.

Diagne: Thank you.