Golan Levin: I’m thrilled now to wel­come Reza Ali, who is a Los Angeles-based design­er, soft­ware engi­neer, and mak­er. He is pas­sion­ate about mak­ing nov­el cre­ative tools and explor­ing the kinds of new visu­al aes­thet­ics that can be made using these tools. Reza is the cre­ator of Satin, a new 3D graph­ics frame­work built on Apple’s Metal library. Reza Ali. 

Reza Ali: Okay. Thank you for the intro­duc­tion Golan. Appreciate it. Thank you for hav­ing me. I’m stoked to be here. Really stoked to share what I’ve been work­ing on for the last cou­ple years. So, super excit­ed to share, and let’s just get started. 

So, I made a lit­tle out­line for the talk. And here’s a quick run-through of what I’ll be talk­ing about. So, lit­tle dis­claimer, some back­ground about me. I’ll be talk­ing about Satin, the open source project I’ve been work­ing on. I’ll show some exam­ples, talk about why I made it. How to use it, the work­flow. Some things that have been made with it. What I accom­plished dur­ing this res­i­den­cy. And there’ll be a sec­tion for par­tic­i­pa­tion if any­body wants to join in the fun. And I’ll be giv­ing thanks afterwards. 

So, yeah a dis­claimer. These are my thoughts, my opin­ions and ide­olo­gies. They don’t rep­re­sent my past or cur­rent employ­ers, or clients, or whatever. 

So a lit­tle bit of back­ground about me. My name is Reza Ali. I con­sid­er myself a design­er, direc­tor, devel­op­er. And I’ve been involved in open source since 2010. As Patricio men­tioned, I worked on ofxUI, which is basi­cal­ly an inter­face library for openFrameworks that helps peo­ple con­trol their instal­la­tions or art pieces, or just sketch­ing around. And it became one of the top three addons for OF. So I’m super pas­sion­ate about con­tribut­ing back and using oth­er peo­ple’s code to make ideas real and test things out and learn. 

And my career’s suc­cess is def­i­nite­ly built on the shoul­ders of all these open source frame­works and giants and com­mu­ni­ties. So always love pay­ing back the favor and mak­ing new friends. 

So, over the last decade I’ve worked for agen­cies and star­tups and big tech com­pa­nies. And recent­ly I start­ed my own com­pa­ny to help serve my clients and work in a more sus­tain­able way. So my com­pa­ny is Hi-Rez. We design and devel­op appli­ca­tions for MacOS and iOS. And specif­i­cal­ly I love work­ing on high-end real-time graph­ics, visu­al­iza­tion, aug­ment­ed real­i­ty, com­put­er vision, and inter­face design. 

And essen­tial­ly the core of what I do is I just like to make cool new things with nov­el tech­nolo­gies and APIs. And I’ve open sourced many of my frame­works recent­ly because I just want them to become bet­ter. I don’t nec­es­sar­i­ly want to write the same code over and over like I did in my 20s. My clients save time, I save time, every­one wins. And I love using these frame­works for my per­son­al design and art­work, so I def­i­nite­ly hand­craft them to be fun to use and easy to integrate. 

I've open sourced a bunch of Swift frameworks. Youi (UI), Force (Sketch), Satin (3D), Juicer (Animation), Easings (Easing), Spectra (FFT). Satin, a 3D graphics framework, is the most valuable.

I’ve open sourced a bunch of Swift frame­works in the last cou­ple years. But I’ll be talk­ing about one in par­tic­u­lar called Satin. So what is Satin? 

Satin is a 3D graph­ics frame­work inspired by Three.js that helps design­ers and devel­op­ers that work with Apple’s Metal API via Swift to cre­ate beau­ti­ful real-time graphics. 

So what does that mean? It means Satin pro­vides class­es for actu­al­ly cre­at­ing geome­tries, and mesh­es, and mate­ri­als, and buffers, and uni­forms, and cam­eras, and shaders, and all sorts of stuff that you would need to actu­al­ly put pix­els on a screen and make a beau­ti­ful scene. So, inspired by Three, Satin’s Renderer just allows you to pass in a scene, pass in a cam­era, and it’ll ren­der it to screen or a tex­ture with­out hav­ing to deal with the com­plex­i­ties of the Metal API. Which is a very clean API, but also can get hairy if you are try­ing to do a lot of cus­tom things. 

So, a lit­tle bit about Satin. Its archi­tec­ture and com­po­nents are designed to max­i­mize flow and min­i­mize fric­tion. I think while I’m work­ing on things I’m also learn­ing by cod­ing. So I think if you can enjoy that process as much as pos­si­ble you’re gonna real­ly have a lot of fun but also learn as much as pos­si­ble. And also you’re more like­ly to exper­i­ment a bit more and make more mis­takes, but also find these beau­ti­ful moments through experimentation.

So, unlike Unity or Unreal, Satin only works on MacOS, iOS, and tvOS. And this is by design. I love these plat­forms and I love work­ing on them, so the frame­work is real­ly easy to inte­grate, real­ly fast, small, flex­i­ble, as bug-free as pos­si­ble, and per­for­mant. And I think it’s real­ly fun to work with. 

So what can you do with Satin? These are just exam­ples that actu­al­ly live in the repo, so you can run and test these out your­self. I’ve just made some videos. So this is show­ing a 2D exam­ple, a 2D cam­era, cam­era con­troller, just ren­der­ing a quad with a UV tex­ture material.

And we have the same for 3D. So this is just an icosa­he­dron ren­dered with a per­spec­tive cam­era. And the per­spec­tive cam­era con­troller’s pan­ning around, mov­ing around, rotat­ing. So all this comes for free. So it’s real­ly easy to get up and running. 

And this is just show­ing like a debug scene show­ing the piv­ot point of the camera. 

Satin is writ­ten Swift, but some parts of it that are super hard­core are writ­ten in C. So you can do real­ly fun things with C like gen­er­ate geome­tries and then use that with Satin with no issue. 

There’s also tri­an­gu­la­tion in the library. So you can do amaz­ing things with text and 3D text. 

There’s a cou­ple of mate­ri­als that’re built in. So when you first start your jour­ney in graph­ics pro­gram­ming you’re gonna have to write a shad­er. And that can be intim­i­dat­ing. But I’ve cre­at­ed a cou­ple of mate­ri­als that just allow you to hit the ground run­ning. So this is a depth mate­r­i­al based off of how far you are in scene and will just sort of apply a depth shader. 

And you can do things like import 3D mod­els and apply mat­caps. And you can also do things like do com­pute shaders, to make par­ti­cle sys­tems come come to life. And Satin makes this as sim­ple as pos­si­ble, so you don’t have to wor­ry about ping-ponging, or triple buffer­ing, or all these kind of more advanced con­cepts that you might not be aware of. 

In addi­tion there’s cube­maps and live­cod­ing shaders. There’s a PVR exam­ple that shows like, you can make real­ly sim­ple stuff with Satin, but you can also go super super hard­core and have a full physically-based ren­der­ing pipeline. 

Who doesn't love reinventing the wheel? I love taking things apart, learning how things work, solving hard problems, architecting code, math, geometry, and designing APIs.

And I think the ques­tion I get a lot is like why do this? I think I’m just curi­ous as a per­son. Like I love tak­ing things apart and learn­ing how things work, and just kind of archi­tect­ing code. To me it’s all just Legos and I have a real­ly fun time rein­vent­ing wheels. 

So, the oth­er rea­son I made it was as a user of Processing, openFrameworks, and Cinder, and Three, these frame­works are real­ly real­ly amaz­ing. They’ve taught me so much about good design pat­terns and graph­ics tech­niques, and I’ve met a bunch of cool peo­ple. And that’s real­ly why I’m talk­ing to you guys here today. But when I start­ed ship­ping and main­tain­ing apps actu­al­ly built on these frame­works it became real­ly tedious to main­tain their code because there’s always some­thing crash­ing, or there’s no pack­age man­ag­er, or there’s an OS update that breaks it. 

So things broke often when using blocks or addons or upon OS updates, language updates, etc. And if you run into a bug in the framework and you can't fix it because of its complexity, you are stuck until someone fixes it. Sometimes it never gets fixed.

And all code is sub­ject to this, but I think I ran into enough bugs and enough hur­dles where I just decid­ed I think I might need to make my own. 

A cou­ple years ago I worked on WebXR at Google. And WebXR is an API that expos­es VR and AR APIs to web pages through Javascript. And I thought this was the future of AR and VR. Because the Web is such a nat­ur­al plat­form. But after about a year going super hard on this I real­ized that I was actu­al­ly wrong, and the Web was­n’t the ide­al can­di­date for the future of the XR metaverse.

And I learned a lot about browsers and the Web, and I real­ized there’s just too much bag­gage the meta­verse to be built on that. And this is obvi­ous­ly my opin­ion. Browsers take months if not years to land APIs because of secu­ri­ty pol­i­tics and code com­plex­i­ty. So just to give you an exam­ple, one com­mit to Chromium might take a year to get inte­grat­ed into a sta­ble release. 

WebGL exposed via Javascript is performant, but there is always some jank because of garbage collection. The WebXR API is years behind the cutting edge ARKit APIs. There are too many security issues to navigate around to make persistent AR applications (which are the ones that will matter most).

And there’s always some jank, there’s always too many secu­ri­ty issues. So I decid­ed like, I’m gonna go and just think dif­fer­ent­ly about how that could be made. 

Around that same time I got an oppor­tu­ni­ty work for Apple to…I can’t real­ly say much about what it did but this was part of the job descrip­tion, to cre­ate ambi­ent pro­ce­dur­al ani­ma­tions” using Metal or SceneKit. And at the time I had no idea— I did­n’t real­ly know Metal at all, nor did I remem­ber any Objective C. The one con­straint that was real­ly killer was like, no frame­works were allowed. So I basi­cal­ly start­ed with an emp­ty fold­er, and that was a real­ly great expe­ri­ence to learn how to build one of these crazy openFrameworks, Cinder-like frame­works that allows you to be cre­ative. And after about a year there I was super inspired. And I want­ed to keep mak­ing things with Metal because the API’s so clean com­pared to OpenGL.

So, I had learned a lot about what was need­ed to make things, what mat­tered most, how to archi­tect things so they could be reusable, extend­ed, and real­ly kind of induce flow in the process and real­ly kin­da have fun. So I start­ed to learn Swift, because Objective C is great but it’s not real­ly that much fun. And I loved writ­ing Javascript but I want­ed Javascript with­out all the garbage. So Swift was the choice. 

And then nat­u­ral­ly after about two years of Javascript and Metal…you know, you look for jobs some­times and you real­ize it’s all Unity and Unreal. And I real­ly did­n’t want to go there because I just knew that I was just so far behind all my tal­ent­ed friends work­ing in these frame­works. And anoth­er rea­son is I just did­n’t want to learn anoth­er closed frame­work that was giant and mono­lith­ic. I did some real­ly deep think­ing and reflec­tion and read­ing and real­ized that there might be a mar­ket that nobody was chas­ing after. 

And real­ly this is me pitch­ing this as like a busi­ness thing, but at the same time I self­ish­ly used this all the time for my art­work. But let’s say BMW or Ikea or Porsche or Audi wants to build an AR expe­ri­ence into their iOS app, or even their Android app, they’re not gonna rebuild their entire app with Unity or Unreal. It’s just not prac­ti­cal. I think com­pa­nies want more con­trol over what’s going on. So, they’re gonna use native tech­nolo­gies, and you can see Google did the same thing with Daydream Home. 

But not every­body will have a Metal expert at their dis­pos­al. Metal’s a pret­ty clean API but it is big, and it is very gen­er­al pur­pose. So I was think­ing an open source frame­work that makes it easy and more fun to build these expe­ri­ences would be valu­able and need­ed at some point to build all these awe­some expe­ri­ences, regard­less of whether they’re AR or not. 

So there was lit­er­al­ly not much resources. I think it was like five peo­ple that I found on the Internet a cou­ple years ago doing stuff, but not real­ly cre­at­ing any big frame­works to help peo­ple. So I kept think­ing like, I’ll build some­thing and maybe some­thing will come of it, and real­ly I just want­ed to do my own research and devel­op­ment and make my own apps with it. So I set off to make it a real­i­ty and have worked on it for about a year and a half. And every project I did, it got bet­ter and faster and stronger and more flex­i­ble. So it’s at a point now where it’s get­ting some stars on GitHub and that’s real­ly exciting. 

So, I’ll talk about work­flow a lit­tle bit. I think that’s impor­tant to under­stand how it dif­fers from oth­er frame­work. So, it’s fun because it’s opin­ion­at­ed. It’s not the most hard­core frame­work writ­ten in C++, but it’s a lot more fun to write code in Swift. Especially if you’re just sketch­ing. Satin is fun because it’s built for imme­di­ate feed­back and play­ful dis­cov­ery, like Patricio was talk­ing about. It’s real­ly impor­tant to be able to like, noo­dle with things right then and there to see what’s hap­pen­ing and to under­stand what your code is doing. And with Satin I want­ed to make easy things easy. Like I want a box on screen, easy. But I want­ed to make hard things doable. Like I wan­na send this buffer to the shad­er and do some­thing crazy. 

The way Satin does this is it basi­cal­ly allows you to live­code shaders and com­pute sys­tems, and seam­less­ly cre­ate UI so you can noo­dle with para­me­ters and learn the dark arts of shad­er programming. 

And there are a cou­ple main class­es that are super use­ful for doing this. There’s the FileWatcher that basi­cal­ly watch­es any file and emits an update when­ev­er things change. There’s the MetalFileCompiler; typ­i­cal­ly when work­ing with Metal, you have to just load a shad­er and then it just should work. If it does­n’t work you have to stop the pro­gram, edit, recom­pile. The file com­pil­er basi­cal­ly allows you to use oth­er Metal files, reuse func­tions real­ly eas­i­ly, it watch­es all these files, it emits changes. So you can have some­thing like LiveMaterial that automag­i­cal­ly pars­es a struct which rep­re­sent uni­forms and watch­es the shad­er and it informs you of updates. 

A cou­ple oth­er fun things. There’s the Metal Library that has tons of fun and use­ful func­tions. It’s real­ly excit­ing to hear Patricio talk about his library. I’m def­i­nite­ly going to take a look at that. 

There are Satin Parameters that rep­re­sent floats, float2’s, float3’s, [vec­tors?], booleans, etc. And you can hook into events when things change, so it makes it real­ly easy to cre­ate code that’s like Hey, do this when this changes.” 

And ParameterGroups rep­re­sent structs. So you no longer have to have a copy of your struct on the Swift side and the GPU side. And this becomes impor­tant because when you want to max­i­mize flow you want to min­i­mize the num­ber of times you’re hit­ting stop and play. So this allows you to just keep writ­ing shaders with­out hav­ing to stop your app and recompile. 

So this is an exam­ple of the typ­i­cal process. This is a blob shad­er. It’s just run­ning on a sphere. And you can see that there’s no ver­tex shad­er, so Satin will inject a ver­tex shad­er for you. And here I am just kind of cre­at­ing a boil­er­plate ver­tex shad­er. So essen­tial­ly I’m just writ­ing give me the ver­tex uni­form, some blob uni­forms, and the LiveMaterial in Satin will take care of wiring up all these uni­form buffers to your ver­tex shad­er and your frag­ment shad­er. So you can real­ly just focus on live­cod­ing the shad­er and just mak­ing it as fun as possible. 

So I’m just writ­ing some boil­er­plate here. You can see that in the struct above, Satin will parse that struct and if there’s com­ments next to vari­ables, it will cre­ate a UI ele­ment that rep­re­sents that para­me­ter. So you’ll see when I switch back over to the app there’s a lit­tle pan­el that has a col­or uni­form and you can see that I’ve changed it. And so, any errors that will hap­pen will be in the con­sole so you can see what’s happening. 

So now I’m just includ­ing a noise func­tion and then essen­tial­ly off­set­ting the sphere’s posi­tion based off that noise func­tion. And time is auto­mat­i­cal­ly fed in, so you can do some fun things with time. You can set the range of slid­ers over there and you can start to just use these things instant­ly to start design­ing and not nec­es­sar­i­ly lose any flow in the process.

So this is typ­i­cal­ly how I write shaders now. There’s para­me­ters, these para­me­ters can be saved to a file, reloaded. 

Here what I’m doing is just cre­at­ing a cus­tom ver­tex struct that basi­cal­ly gets passed to the frag­ment shad­er so I can do some fun col­or stops. And you can see I def­i­nite­ly make mis­takes all the time. But what’s use­ful is that this pro­gram’s not crash­ing and I’m just like, fig­ur­ing out why it’s not crash­ing. And then I’m able to go back and keep writ­ing my shad­er. And it’s a real­ly fun way to com­pose things with­out hav­ing to stop all the time. And giv­en all the library func­tions it’s real­ly easy to start to make things that look pret­ty real­ly quickly. 

So, that was that. This is a more com­plex exam­ple where I was under­stand­ing how do you use a cube map, and how do you refract and reflect and do all these crazy things. How do you take tex­tures from your pro­gram and then use that to do all these sort of gnarly shad­er effects. And all this I think is me say­ing it’s real­ly easy and fun to hook all these things togeth­er and be cre­ative. And Satin pro­vides that under­ly­ing base, that core, that allows you to just be cre­ative and just work, ver­sus try­ing to fig­ure out okay, how does this API work or how do I do this and this and this. 

So this is just an exam­ple from some of my work. I’ll show things that I’ve been mak­ing with it. It’s still a very young frame­work, but there are things I did like for exam­ple ren­der point clouds and styl­ize them; have shadows. 

I’ve been mak­ing my own art­work, which is just a lot of spheres with nois­es in it and play­ing with shad­ing and blend­ing. Another exam­ple, just on black. 

And what’s nice about work­ing with Satin is like, if you’re in the Mac or iOS ecosys­tem, you can do real­ly fun things like make screen­savers pret­ty quick­ly, and sort of ship apps on that plat­form that are high­ly graph­ic very very fast. 

Let’s see, this is anoth­er screen­saver. And all these are just things I’ve made like…maybe not the reac­tion dif­fu­sion one but this I made just like in a day just sort of play­ing around. So it’s def­i­nite­ly become a real­ly use­ful tool in my arse­nal to just quick­ly sketch and make things. 

This is a clock I was design­ing, inspired your Dieter Rams. 

I start­ed design­ing some soap with a cou­ple friends. And I was just think­ing about the design process. So here’s this ray march­ing form sim­u­lat­ing reac­tion dif­fu­sion. And then I take those things and write code to export things. And it’s become real­ly crit­i­cal in my process for just sketch­ing but also actu­al­ly exe­cut­ing on design things. 

So here, at some point every soft­ware engi­neer or coder will even­tu­al­ly start mak­ing fur­ni­ture. This is my attempt to make fur­ni­ture. It’s Voronoi shelves. It’s a little…I dun­no, over­played, but I just fig­ured I’d take a crack at it. And this is like a real­ly fun way of just debug­ging, try­ing to fig­ure out form-making and you know, Satin is crit­i­cal in this process of actu­al­ly tak­ing ideas and then mak­ing them into real­i­ty. So for me I think it’s core to my com­pu­ta­tion­al thinking.

You can also use it for ray trac­ing and crazy, awe­some, gnarly par­ti­cle effects. You can also do audiore­ac­tive things with it. 

And I’ve been recent­ly kind of get­ting back into form-making. And so you can use Satin as a ren­der engine and use oth­er libraries to be gen­er­at­ing forms. 

So I’ve been work­ing on this 3D mod­el­er for a cou­ple months. And Satin’s used here to do all the ren­der­ing. Like the grid in the back­ground, the giz­mos, the actu­al mesh. The tum­bler in the top right. The oth­er UI is just Swift UI. So you can see it’s very pow­er­ful. You can start to build these real­ly crazy 3D appli­ca­tions, and Satin’s been real­ly crit­i­cal just to be the under­ly­ing tech­nol­o­gy behind a lot of that stuff. 

So doing the res­i­den­cy, I saw it as an oppor­tu­ni­ty to kind of take a step back from the frame­work and learn from all these open source lead­ers, and learn how to cre­ate com­mu­ni­ty. So I cre­at­ed a Discord, and that’s been a great place where the com­mu­ni­ty has been telling me We’d love to see this,” or How do you do this.” And this has been real­ly help­ful in try­ing to under­stand what prob­lems exist and what users actu­al­ly want to do with it. And this has helped me quite a bit. 

And so because of that and the res­i­den­cy, I’ve writ­ten a bunch of code to make it eas­i­er to expose para­me­ters on iOS. I fixed a bunch of things that were bro­ken because of OS updates. I refac­tored a bunch of its shad­er sys­tem and file com­pil­er, and made it super easy to use Satin to make AR apps and experiences. 

And I’ll show just a cou­ple of exam­ples. I’ve cre­at­ed an XCode tem­plate that shows you how to cre­ate a scene, live­code a shad­er, expose para­me­ters, have those things auto­mat­i­cal­ly update when a shad­er para­me­ter’s been updated. 

I also got a request to…how does one take a buffer of data and then draw tons of points and lines and what­ev­er. So this exam­ple shows you how to take the SARS COVID-19 genome sequence, pipe it into a Metal buffer, and then use that in a ver­tex and frag­ment shad­er to change the col­ors of these dots, but also to lay them out. 

It also shows you how to export an high-resolution image. So if you’re an artist work­ing with code, I think it’s impor­tant to be able to make prints or what­ev­er have you. 

Then I cre­at­ed a general-purpose GPU com­put­ing exam­ple just to show hey, this is how you use the com­pute sys­tem to do some­thing a lit­tle bit more com­pli­cat­ed. So this is just show­ing a Flocking exam­ple inspired Dan Shiffman.

And get­ting these things to work both on MacOS and iOS was real­ly impor­tant, because it’s real­ly easy to pro­to­type things on MacOS, because it’s just…you don’t have to deploy on device. And now my Youi frame­work has been updat­ed so now you can very quick­ly with the same code deploy that on iOS and have all those con­trols accessible. 

So, if you’d like to par­tic­i­pate, thank you for lis­ten­ing. The frame­work is slow­ly grow­ing. You can con­tribute code, or ideas, or fund­ing. I’ll have a spon­sor but­ton on the Satin repo soon. 

If you’re inter­est­ed in build­ing some­thing, or you think Satin can help you in your mis­sion, please reach out. And if you’re a stu­dent and you want to learn Metal and Swift and Satin just to like gain some career cap­i­tal or even­tu­al­ly get a job job or just build some­thing fun, I’m all ears. 

And I want to say some thanks. Thank you to the STUDIO for Creative Inquiry and Carnegie Mellon for hav­ing me. Thank you Golan, and Thomas, and Linda, and Chris, and the res­i­dents. Your sup­port, sto­ries, ques­tions, and feed­back have been real­ly inspi­ra­tional and I’m con­stant­ly think­ing about new things like codes of con­duct and build­ing com­mu­ni­ty and all these oth­er fac­tors that I did­n’t even think about before. 

And I want to thank my younger broth­er Haris, who is wise beyond his years and has helped me count­less times archi­tect­ing code, and problem-solving, and just being a bet­ter per­son. So, thank you. 

@rezaali (twitter), @syedrezaali (instagram), reza@hi-rez.io (work), https://github.com/hi-Rez/Satin (code)

Thank you all for lis­ten­ing, and these are some places where you can find me on the Internet, on Twitter, Instagram. Here’s my work email. And of course where you can find Satin online. 

And with that I think my talk is done. Thank you for hav­ing me.