Golan Levin: Our first speak­er, Patricio Gonzalez Vivo, is an artist and engi­neer from Buenos Aires who makes dig­i­tal art from repur­posed e‑waste as an alchem­i­cal prac­tice. His work is tech­ni­cal­ly sophis­ti­cat­ed meets play­ful tin­ker­ing. He’s the author of The Book of Shaders, glslViewer, and the PixelSpirit Deck. Patricio’s work has been shown at Eyeo, Resonate, GROW, FRAMED, FILE, Espacio Fundación Telefónica, and FASE. He’s taught at Parsons The New School, NYU ITP, and the School for Poetic Computation. Patricio Gonzalez Vivo.

Patricio Gonzalez Vivo Thank you Golan. I’m turn­ing my cam­era on and…hope you can see me now? 

Levin: Yes.

Vivo: Perfect. Thank you. So I’m going to share the screen next. Here it goes. 

So I want to start by say­ing how grate­ful I am to be here. Thank you Golan and every­body that made this pos­si­ble. For me it has been a great cou­ple of months. Having the oppor­tu­ni­ty to real­ly work on glslViewer and some impor­tant fea­tures and also to med­i­tate on the tool togeth­er with oth­er tool­mak­ers. So it has been a joy to experience. 

Today I kind of want to share a path that I went through from tak­ing a per­son­al tool which is…like a per­son­al toy tool to the point that it’s a pro­fes­sion­al tool that I can use in my every­day work for what­ev­er kind of work I do. So with that in mind I want to start by kind of defin­ing some pat­terns that I have found dur­ing work­ing with code as a medi­um. And that pat­tern, which we will refer to as The Pattern, I think the start­ing point will be…here.

It will be mak­ing the algo­rithm. And hope­ful­ly every­body can relate to this and take this to their own per­son­al expe­ri­ence. It does­n’t have to be one-to-one but I hope every­body can find it res­o­nant. So, you start by cod­ing your pro­gram. It could be a visu­al expe­ri­ence or a gen­er­a­tive piece of work. It could be a par­ti­cle sys­tem. And you kind of devel­op this inti­mate con­nec­tion to your code, right. You know how it behaves. It came out of your mind. You based it on some paper that you found. 

And after cre­at­ing it, you dis­cov­er and you have a sense of what are the pres­sure points. What are those are vari­ables here and there that when you change the val­ues, it real­ly kind of makes some­thing dif­fer­ent. It could be some­thing inter­est­ing, some­thing that looks dif­fer­ent, or some­thing that real­ly speaks to you. And at this point our work becomes a lit­tle like tin­ker­ing and like pok­ing at your code. And kind of see­ing what hap­pens. So par­tic­u­lar­ly in this step I real­ly enjoy kind of how you’re dri­ven by curiosity. 

And I think the imme­di­ate next step, at least for me, what hap­pens is that I have this urge of like, put on the Bret Victor hat and I’m going to put some slid­ers into that. And here you can use a lot of tools. In the past I’ve actu­al­ly used Reza’s ofUI when I was work­ing most­ly on openFrameworks. And here a series of chal­lenges comes, right. You’re try­ing to—like you have this kind of pup­pet and you’re pulling the strings, and you want to define what is the sweet spot of each one. So you have to define the ranges that they work on, and which will be the nice defaults to have when the thing opens. And once you’re done, you go through anoth­er series of kind of curios­i­ty dri­ving us. Like now there’s a slid­er, how can I make this code and this algo­rithm real­ly speak through me, right, ampli­fy my voice. How can I find my own voice in this or some­thing that I’m inter­est­ed in pursuing. 

And that leads us to the fourth step which I have found, which is you want to save it. You want to kind of make an amber copy of that and store it some­where. And the step here brings anoth­er chal­lenge. So if you’re not using a mature UI you will be faced with like, which for­mat should I make it? Should I make it some­thing that is easy for GitHub to source con­trol, or should it be some­thing bina­ry, and I don’t care so much about that. But this part real­ly is kind of like a moment of col­lect­ing dif­fer­ent iter­a­tions and vari­a­tions of your own work. Which is fascinating. 

But that brings us to the next step, which is your pro­gram now needs to deal with mul­ti­ple states. So you need to load it cor­rect­ly, and it needs to tran­si­tion cor­rect­ly. At this point you also have to deal with the source con­trol, hav­ing a way of keep­ing up with all these slight­ly dif­fer­ent vari­a­tions and hav­ing ways of pri­or­i­tiz­ing and kind of like…you know, some­times we fall into the typ­i­cal file for­mat­ting of like ​“finalFINAL/final-version1.” Everyone has a dif­fer­ent ver­sion of that, but I usually…that has happened.

And I think this step kind of opens up the final one, which is you real­ize that very inter­est­ing things hap­pen in between those stages. And this real­ly opens to kind of mak­ing com­po­si­tions over time, right. My par­ti­cle sys­tem is all clus­tered togeth­er, very tight­ly tied, and then I release it. And they come out very fast into the space. Or some­thing like that. And in the past I have been tempt­ed to also imple­ment a whole like time­line for that, in order to be able to do some sto­ry­telling with this composition. 

So, this is the over­all pat­tern that I have fol­lowed, and hope­ful­ly oth­er peo­ple feel that they have been there too. 

And at this point I want to pro­pose a dif­fer­ent pat­tern I have found with glslViewer. It’s not par­tic­u­lar­ly bet­ter or worse, but it kind of has helped me to decon­struct this in order to not spend so much time reim­ple­ment­ing UIs. And it’s a very old pat­tern, so I’m using the word ​“new” here tak­en with a grain of salt because it’s new for me but it’s very old. Actually it came from the Unix prin­ci­ples of design, which is old, as old as like C and mainframes. 

And what real­ly struck me from that mod­el is this idea of like, instead of hav­ing one pro­gram that does it all…like it has all the but­tons and has all the whis­tles, and all the fea­tures, you go more for sim­ple, very atom­ic or mod­u­lar struc­tures where each one of the pro­grams does one thing but it does it real­ly well. 

And nowa­days there’s a dif­fer­ent par­a­digm, a more mod­ern name for this, espe­cial­ly when you devel­op pro­grams, which is this idea of sep­a­rat­ing the mod­el, the view, and the con­trollers. And with this I’m kind of get­ting clos­er to glslViewer. 

So the mod­el in our case when you are… And to give a small intro­duc­tion, glslview­er is shad­er sand­box. And the code is gonna be the mod­el, the view is gonna be the ren­der­ing of that, and the con­trol is the para­me­ters that we feed into it, right. And hope­ful­ly you can grab this mod­el and think in your own appli­ca­tions, like we could say with Processing or with openFrameworks or what­ev­er pat­tern you’re using. 

So let’s get into glslViewer. And glslViewer, to give con­text, was born while I was devel­op­ing The Book of Shaders. The book real­ly clicked for me when I start­ed putting the text togeth­er with these sand­box­es. Because it real­ly helps in this jump— The con­cepts can be very abstract and new for some peo­ple, so shar­ing this path between learn­ing a new con­cept and tin­ker­ing with it and pok­ing into some code, I thought it would real­ly help. And I think that it does, and part of the pop­u­lar­i­ty of the book comes from that. 

So for that I made my first sand­box, which is the glslCanvas. And for here there’s already some design deci­sions. Because it’s an edu­ca­tion­al text, for me was it impor­tant for the text to be the pri­ma­ry inter­face. And this is kind of major. If you go to oth­er plat­forms like Shadertoy that are like, wild­ly more well-known, they real­ly hide all the com­plex­i­ty and you are work­ing in a wrap­per of the main shad­er that runs.

So, I made glslViewer, which is the sand­box. And around that I made this lit­tle edi­tor. And I want to talk a lit­tle more about this. So note here that when you touch a vari­able or a col­or, these wid­gets come up. And I want to empha­size how for me, the impor­tance was to put the result of these slid­ers and col­or pick­ers back into the code. Because the code is the main inter­face. So it’s like the hood of a car. And I want the uni­forms to be there. I want all the weird things that you have to put at the top of a shad­er to be there. And then, that code, as is with these val­ues hard­cod­ed, you can then take it any­where else. Because it’s just a plain GLSL shad­er, you don’t have to do any weird tin­ker­ing or chang­ing. You pret­ty much can take it any­where else. Which is some­thing that on oth­er plat­forms, because they have to deal with these wrap­pers, it gets com­pli­cat­ed and obfus­cat­ed. So I’m gonna keep mov­ing forward.

So, from my own neces­si­ty of doing more com­pli­cat­ed work, or not rely­ing on the Internet— At that moment I was com­mut­ing between Brooklyn and Manhattan, so I had all these hours of com­mute time. And I was want­i­ng to play with shaders on my lit­tle Raspberry Pi com­put­er. So for me it was very impor­tant to make it as sim­ple as pos­si­ble and real­ly fast on some­thing super inex­pen­sive like the com­put­er that I can car­ry with me that does­n’t belong to the cor­po­ra­tion where I’m work­ing or what­ev­er. And just enjoy kind of like the sim­plic­i­ty and abstrac­tion of being just in a small com­put­er which does very lit­tle but it does it well.

And here you can see the first design deci­sion also is to dis­en­tan­gle glslViwer from the edi­tor. So you can pick up any edi­tor, the code is your inter­face. This is the way that you access the thing. And you can choose what­ev­er edi­tor you want to. 

Then this expands more, and I made an openFrameworks addon because I feel very strong­ly for that com­mu­ni­ty. So I want to port this code, too. 

And then this is kind of like an overview of how it grows. It went from being a 2D can­vas with some tex­tures, to the abil­i­ty of pass­ing through con­sole in, which is the pipe in…the POSIX Unix way of pass­ing data from one pro­gram to anoth­er one. Then I added the abil­i­ty to have mul­ti­ple buffers. Then I added the abil­i­ty to add 3D mod­els, a cam­era descrip­tion, a back­ground pass, a cube­map, spher­i­cal har­mon­ics, lights, shad­ows. And post-processing. So this kind of like, puts it all togeth­er. And this I think is kind of like the process of my own learn­ing, right. 

So here you can see a more mature and actu­al ver­sion of glslViewer. In this par­tic­u­lar case I was work­ing on some VR con­tent for run­ning on the Quest. So I’m kind of try­ing to fig­ure out what is the bare min­i­mum of code and the most effec­tive and effi­cient way of doing some shaders for this very under­pow­ered device.

And you can see that there is a con­sole that’s down there, and this con­sole is what allows me to do debug­ging, to turn lights on and off, and so forth, to check the gen­er­at­ed light maps, etc. 

Moving for­ward, same. In this I’m doing more like post-processing passes. 

And here I’m doing images. And this is most of what I do for a liv­ing. There’s this amount of work that hap­pens in the code, but I found that I have the need to com­mu­ni­cate pret­ty often on how I’m work­ing on some­thing and how it’s look­ing. Especially to check with cre­ative direc­tors and so forth. So one of the abil­i­ties is that glslViewer allows me to make real­ly qual­i­ty videos real­ly fast. I make a PNG sequence and it’s [indis­tinct] to ffm­peg and I can have a video of any kind of qual­i­ty that I want. I end­ed up inte­grat­ing this fea­ture back into the GLSL edi­tor in the book. Because I have found that com­mu­ni­cat­ing with oth­ers is prob­a­bly the most cru­cial part of my work, and I have some lim­i­ta­tions in terms of lan­guage because English is not my first lan­guage. So this real­ly helps. 

So let’s go back to this pre­sen­ta­tion of the mod­el and let’s cov­er some ground that we already cov­ered. glslViewer is the view­er, the code hap­pens in the code edi­tor; it’s a dif­fer­ent pro­gram, and then we have the con­trols that would hap­pen through the con­sole in. 

So we went through through the his­to­ry and we learned that besides code, we can drop some things in also as inputs and as inter­faces; can be images, and can be geome­tries. That means that if the image changes or the code changes, both things will change inside the view­er. So the view­er will pick it up.

And then I was show­ing how the con­sole is a con­sole in. And then most recent­ly, I think a cou­ple of years ago, OSC. So I can com­mu­ni­cate with any pro­gram that uses OSC in my own com­put­er or from the net­work. And this was for me a piv­otal moment because this allowed me to stop think­ing of my own pro­gram as an iso­lat­ed island and start think­ing of all the pro­grams that are around and that can com­mu­ni­cate with it. 

So the image can come from a game, or Blender, or the geom­e­try can come from Blender. The code can be edit­ed in any any code edi­tor, like vim or Visual Studio or what­ev­er. And from the con­sole in, I can send com­mands from Python, from oth­er pro­grams like ORCA, or [TouchOSC], Ossia, it could be a MIDI device. So this opens to using instru­ments. And this real­ly put me on the path of what real­ly inspires me about Unix, right. It’s this ecosys­tem of apps that are designed to talk very well with each oth­er. And it’s kind of like an orches­tra. There’s a gen­eros­i­ty between pro­grams of being good col­lab­o­ra­tors. So that’s one of the things I tried to be thought­ful of when I devel­oped glslViewer, is how can I make a good col­lab­o­ra­tor that sends some­thing that is read­able or easy to parse. 

And I have found some great friends along the way, and these are friends I derive inspi­ra­tion from so I want to give a free shout out to like Syphon, that Anton Marini is involved with. When I learned about Syphon, it was like this should be the par­a­digm instead of hav­ing these super nar­row sand­box­es that try to put every­thing there, hav­ing these more dis­persed and decen­tral­ized ways of working. 

Then you have Ossia, which appar­ent­ly is a lit­tle less known. But it’s just a time­line that exports OSC and you can put a lot of things inside it. It can sup­port col­ors, and you can real­ly orches­trate and define your nar­ra­tive there. 

ORCA, and every­thing real­ly that Hundredrabbits does is amaz­ing. And they real­ly are com­pos­ing a whole oper­a­tion sys­tem of inter­con­nect­ed apps. And I think that ORCA as a flex­i­ble sequencer real­ly expos­es that. Because it real­ly lets you export to dif­fer­ent formats. 

And then the oth­er one is TouchOSC, which is a great app to have because it real­ly helps you with OSC. 

So, back to glslViewer and how it works, most of my work hap­pens and looks like this when it’s not code. And this is just a shad­er that turns on and runs a pro­gram and loads a video, and it’s going to ren­der it headless­ly, and it’s going to cre­ate a PNG sequence between this moment 0 and the sec­ond 3.23. So I use this for my art prac­tice, and so for me this is my work­shop. Because this is the place that I come here with these ideas and I can med­i­tate and habi­tate them. 

Sometimes I have to do more…work work, and so I have to design a VFX for VR. 

Sometimes I’m dri­ven more by curios­i­ty and like, how I can imple­ment a GPU sim­u­la­tion on glslViewer run­ning a cou­ple of shaders and I jump to do it. Sometimes it’s more fun as a mul­ti­verse and I try to match aesthetics. 

I have been moti­vat­ed by Doré and want­ed to do some engraving-style shaders. 

Part of my art­work is also geospa­tial, so I made own LIDAR and I visu­al­ized it with glslViewer. And then I com­posed some crazy…I don’t know, landscapes. 

More recent­ly I have been try­ing, using OSC, to devel­op part­ner apps. One is MidiGyver, which allows you to con­nect things that are MIDI to OSC so I can con­nect it to glslViewer. So I’m kind of expand­ing my own ecosystem. 

This is anoth­er video of me using a nanoKONTROL2 to con­trol these shapes that I do in [indis­tinct].

And more recent­ly I have being tin­ker­ing with an OP‑Z and send­ing the com­mands. And this small device real­ly excites me because what I’ve seen on the OP‑Z, on these kind of like musi­cal instru­ments, is that they’re instru­ments. They’re already designed to work over time on a com­po­si­tion, and acces­si­bil­i­ty. And they’re designed as instru­ments, to ampli­fy our voice. 

And thanks to this res­i­den­cy I have been able to work on the video fea­ture, which this will kind of like, wrap it up and I feel that this pro­gram’s becom­ing fea­ture com­plete. So I’m very grate­ful for this res­i­den­cy because I have been able to put some time into it. And I have been hav­ing a lot of fun doing dif­fer­ent post-processing effects just with glslViewer and a cou­ple of shaders. 

And also because my artwork…there’s a lot of it but it’s very spa­tial, I have been work­ing with satel­lite data and deriv­ing depth from it. So I’ve also been doing these explo­rations about clouds and com­pound­ing these dif­fer­ent satel­lite views of spe­cif­ic days. And all this is glslViewer. This is how flex­i­ble and use­ful for me it is, and I hope oth­er peo­ple find it too. 

This is way more recent. I have been kind of going back to this old project where I had been dis­as­sem­bling old LCD mon­i­tors that I sal­vaged from being thrown away. And I put a lit­tle Raspberry Pi in. Because glslViewer has been designed for run­ning on these small devices, I can make lit­tle tiny shaders that run well on them, and that’s part of my prac­tice. So I have been work­ing on this med­i­ta­tion about bring­ing val­ue to these old metal­lic struc­tures and kind of ele­vat­ing them through art. 

So before leav­ing and, in this lit­tle space that have, if I still have some space, I want to point out that there’s also this thing about glslViewer that you can pipe it. So this I men­tioned before is part of the abil­i­ty of send­ing one com­mand from a dif­fer­ent app. So I want to point out one project, one big project where I was able to kind of use this. 

And this is PixelSpirit, which is a tarot deck that has GLSL code on the back and that com­piles what is in the front. And I pre­sent­ed dif­fer­ent cards, and on each card the code gets a lit­tle more com­pli­cat­ed. I intro­duce dif­fer­ent func­tions and those func­tions then get called by oth­er cards. So the more you go into the deck, the abil­i­ties of what is pos­si­ble to ren­der becomes more com­pli­cat­ed and intricate. 

And in order to make this device I actu­al­ly used glslViewer. So I made a lit­tle Python script which grabbed the text of the code—the cards are basi­cal­ly frag­ment shaders. Part of it, the shad­er gets ren­dered through glslViewer, the text gets com­posed by Pandoc and con­vert­ed to LaTeX, and LaTeX makes a PNG from it. And then I run this through ImageMagick, aggre­gat­ing all the dec­o­ra­tions on the edges. And this gets print­ed. And this is actu­al­ly the make rou­tine. So I basi­cal­ly com­pile a tarot deck. 

And this real­ly speaks to my heart of what a cre­ative oper­at­ing sys­tem of inter­con­nect­ed cre­ative pro­grams means. Like the abil­i­ty to—okay, you want to do some­thing mate­r­i­al, you want to play with satel­lites, you want to send some­thing to a print­er. And the tool is able to adapt to these dif­fer­ent workflows. 

And then it becomes phys­i­cal objects. And these are very per­son­al objects, so I’m very grate­ful when peo­ple get it. 

One more thing that I want to talk about is I spent some time in 2017 doing AR fil­ters. And one of the chal­lenges of work­ing with big tech com­pa­nies is that some­times you have the priv­i­lege to use the lat­est iPhone, and some­times you have to devel­op some­thing that has to run on a super obscure device on Android devices some­where in the world. And that requires some dis­ci­pline, and that means bench­mark­ing usu­al­ly, in the com­put­er graph­ics scene. 

So I want to show a lit­tle video of how I’m using glslViewer run­ning on a Raspberry Pi Zero, which is like a $5 com­put­er, to basi­cal­ly test all the func­tions that I’ve made in com­pil­ing a library. This library’s become kind of my life and I have been adding into it. But while this is doing the test­ing mak­ing sure that these func­tions real­ly run cor­rect­ly, it’s also doing some bench per­for­mance test­ing on them—benchmarking. So this bench­mark­ing for me is great because when when I have to design I can actu­al­ly have a rough idea of how expen­sive it will be to do this or that effect.

And see here I made this kind of web site that’s just to visu­al­ize what the extent of the library is, but also to see the met­rics of how much each func­tion costs. 

And what I want to say is that I’m open sourc­ing this, and I’m very excit­ed. I named it Lygia, after Lygia Clark’s work. It’s a muti-language GLSL shad­er library. It’s trans­lat­ing GLSL, HSLSL, and it’s going to be in Metal, too. And there are some plans for oth­er things. People are step­ping up. I haven’t talked pub­licly, but I’m very excit­ed about it. 

And I’m hop­ing that peo­ple love it and sup­port it. So I added a lit­tle sup­port but­ton to all my GitHub things. So if you’re a user, sponsor.

And…that’s all. Thank you so much. 

Golan Levin: Patricio, thank you so much. We have maybe thir­ty sec­onds for one ques­tion, and there’s so many ideas in this, from your eco­log­i­cal prac­tice recov­er­ing LCD mon­i­tors to sort of the new kinds of tools and new ways of pro­gram­ming we see you explor­ing, and the ecolo­gies of small tools. This has been a very tech­ni­cal talk, though, and I guess my quick ques­tion if you can give it just a moment is, your biog­ra­phy men­tions alche­my. You know, you made a Pixel Spirit deck sim­i­lar to tarot. You even used the ImageMagick library. And I won­der if you could briefly com­ment on the rela­tion­ship of your work to magic. 

Patricio Gonzalez Vivo: Well, in a pre­vi­ous life I was a clin­i­cal psy­chol­o­gist. And I did a spe­cial­iza­tion in expres­sive art ther­a­py. When you work with expres­sive art ther­a­py, you’re dan­ger­ous­ly close to Carl Jung. And when you’re read­ing Carl Jung’s books you get real­ly exposed to mythol­o­gy and basi­cal­ly to this uni­verse of sym­bols. And you under­stand that our way of learn­ing is through sym­bols and arche­types. And I think you devel­op a spe­cial love for those. And so when I tran­si­tioned into code, it was hard not to see that the same mys­ti­cism was around these slight­ly more kind of hard sym­bol­ic struc­tures, which are noth­ing more than just a mir­ror of our mind, just as mythology.

Levin: Thank you so much Patricio.