Golan Levin: I’m thrilled now to welcome Reza Ali, who is a Los Angeles-based designer, software engineer, and maker. He is passionate about making novel creative tools and exploring the kinds of new visual aesthetics that can be made using these tools. Reza is the creator of Satin, a new 3D graphics framework built on Apple’s Metal library. Reza Ali.
Reza Ali: Okay. Thank you for the introduction Golan. Appreciate it. Thank you for having me. I’m stoked to be here. Really stoked to share what I’ve been working on for the last couple years. So, super excited to share, and let’s just get started.
So, I made a little outline for the talk. And here’s a quick run-through of what I’ll be talking about. So, little disclaimer, some background about me. I’ll be talking about Satin, the open source project I’ve been working on. I’ll show some examples, talk about why I made it. How to use it, the workflow. Some things that have been made with it. What I accomplished during this residency. And there’ll be a section for participation if anybody wants to join in the fun. And I’ll be giving thanks afterwards.
So, yeah a disclaimer. These are my thoughts, my opinions and ideologies. They don’t represent my past or current employers, or clients, or whatever.
So a little bit of background about me. My name is Reza Ali. I consider myself a designer, director, developer. And I’ve been involved in open source since 2010. As Patricio mentioned, I worked on ofxUI, which is basically an interface library for openFrameworks that helps people control their installations or art pieces, or just sketching around. And it became one of the top three addons for OF. So I’m super passionate about contributing back and using other people’s code to make ideas real and test things out and learn.
And my career’s success is definitely built on the shoulders of all these open source frameworks and giants and communities. So always love paying back the favor and making new friends.
So, over the last decade I’ve worked for agencies and startups and big tech companies. And recently I started my own company to help serve my clients and work in a more sustainable way. So my company is Hi-Rez. We design and develop applications for MacOS and iOS. And specifically I love working on high-end real-time graphics, visualization, augmented reality, computer vision, and interface design.
And essentially the core of what I do is I just like to make cool new things with novel technologies and APIs. And I’ve open sourced many of my frameworks recently because I just want them to become better. I don’t necessarily want to write the same code over and over like I did in my 20s. My clients save time, I save time, everyone wins. And I love using these frameworks for my personal design and artwork, so I definitely handcraft them to be fun to use and easy to integrate.
I’ve open sourced a bunch of Swift frameworks in the last couple years. But I’ll be talking about one in particular called Satin. So what is Satin?
Satin is a 3D graphics framework inspired by Three.js that helps designers and developers that work with Apple’s Metal API via Swift to create beautiful real-time graphics.
So what does that mean? It means Satin provides classes for actually creating geometries, and meshes, and materials, and buffers, and uniforms, and cameras, and shaders, and all sorts of stuff that you would need to actually put pixels on a screen and make a beautiful scene. So, inspired by Three, Satin’s Renderer just allows you to pass in a scene, pass in a camera, and it’ll render it to screen or a texture without having to deal with the complexities of the Metal API. Which is a very clean API, but also can get hairy if you are trying to do a lot of custom things.
So, a little bit about Satin. Its architecture and components are designed to maximize flow and minimize friction. I think while I’m working on things I’m also learning by coding. So I think if you can enjoy that process as much as possible you’re gonna really have a lot of fun but also learn as much as possible. And also you’re more likely to experiment a bit more and make more mistakes, but also find these beautiful moments through experimentation.
So, unlike Unity or Unreal, Satin only works on MacOS, iOS, and tvOS. And this is by design. I love these platforms and I love working on them, so the framework is really easy to integrate, really fast, small, flexible, as bug-free as possible, and performant. And I think it’s really fun to work with.
So what can you do with Satin? These are just examples that actually live in the repo, so you can run and test these out yourself. I’ve just made some videos. So this is showing a 2D example, a 2D camera, camera controller, just rendering a quad with a UV texture material.
And we have the same for 3D. So this is just an icosahedron rendered with a perspective camera. And the perspective camera controller’s panning around, moving around, rotating. So all this comes for free. So it’s really easy to get up and running.
And this is just showing like a debug scene showing the pivot point of the camera.
Satin is written Swift, but some parts of it that are super hardcore are written in C. So you can do really fun things with C like generate geometries and then use that with Satin with no issue.
There’s also triangulation in the library. So you can do amazing things with text and 3D text.
There’s a couple of materials that’re built in. So when you first start your journey in graphics programming you’re gonna have to write a shader. And that can be intimidating. But I’ve created a couple of materials that just allow you to hit the ground running. So this is a depth material based off of how far you are in scene and will just sort of apply a depth shader.
And you can do things like import 3D models and apply matcaps. And you can also do things like do compute shaders, to make particle systems come come to life. And Satin makes this as simple as possible, so you don’t have to worry about ping-ponging, or triple buffering, or all these kind of more advanced concepts that you might not be aware of.
In addition there’s cubemaps and livecoding shaders. There’s a PVR example that shows like, you can make really simple stuff with Satin, but you can also go super super hardcore and have a full physically-based rendering pipeline.
And I think the question I get a lot is like why do this? I think I’m just curious as a person. Like I love taking things apart and learning how things work, and just kind of architecting code. To me it’s all just Legos and I have a really fun time reinventing wheels.
So, the other reason I made it was as a user of Processing, openFrameworks, and Cinder, and Three, these frameworks are really really amazing. They’ve taught me so much about good design patterns and graphics techniques, and I’ve met a bunch of cool people. And that’s really why I’m talking to you guys here today. But when I started shipping and maintaining apps actually built on these frameworks it became really tedious to maintain their code because there’s always something crashing, or there’s no package manager, or there’s an OS update that breaks it.
And all code is subject to this, but I think I ran into enough bugs and enough hurdles where I just decided I think I might need to make my own.
A couple years ago I worked on WebXR at Google. And WebXR is an API that exposes VR and AR APIs to web pages through Javascript. And I thought this was the future of AR and VR. Because the Web is such a natural platform. But after about a year going super hard on this I realized that I was actually wrong, and the Web wasn’t the ideal candidate for the future of the XR metaverse.
And I learned a lot about browsers and the Web, and I realized there’s just too much baggage the metaverse to be built on that. And this is obviously my opinion. Browsers take months if not years to land APIs because of security politics and code complexity. So just to give you an example, one commit to Chromium might take a year to get integrated into a stable release.
And there’s always some jank, there’s always too many security issues. So I decided like, I’m gonna go and just think differently about how that could be made.
Around that same time I got an opportunity work for Apple to…I can’t really say much about what it did but this was part of the job description, to “create ambient procedural animations” using Metal or SceneKit. And at the time I had no idea— I didn’t really know Metal at all, nor did I remember any Objective C. The one constraint that was really killer was like, no frameworks were allowed. So I basically started with an empty folder, and that was a really great experience to learn how to build one of these crazy openFrameworks, Cinder-like frameworks that allows you to be creative. And after about a year there I was super inspired. And I wanted to keep making things with Metal because the API’s so clean compared to OpenGL.
So, I had learned a lot about what was needed to make things, what mattered most, how to architect things so they could be reusable, extended, and really kind of induce flow in the process and really kinda have fun. So I started to learn Swift, because Objective C is great but it’s not really that much fun. And I loved writing Javascript but I wanted Javascript without all the garbage. So Swift was the choice.
And then naturally after about two years of Javascript and Metal…you know, you look for jobs sometimes and you realize it’s all Unity and Unreal. And I really didn’t want to go there because I just knew that I was just so far behind all my talented friends working in these frameworks. And another reason is I just didn’t want to learn another closed framework that was giant and monolithic. I did some really deep thinking and reflection and reading and realized that there might be a market that nobody was chasing after.
And really this is me pitching this as like a business thing, but at the same time I selfishly used this all the time for my artwork. But let’s say BMW or Ikea or Porsche or Audi wants to build an AR experience into their iOS app, or even their Android app, they’re not gonna rebuild their entire app with Unity or Unreal. It’s just not practical. I think companies want more control over what’s going on. So, they’re gonna use native technologies, and you can see Google did the same thing with Daydream Home.
But not everybody will have a Metal expert at their disposal. Metal’s a pretty clean API but it is big, and it is very general purpose. So I was thinking an open source framework that makes it easy and more fun to build these experiences would be valuable and needed at some point to build all these awesome experiences, regardless of whether they’re AR or not.
So there was literally not much resources. I think it was like five people that I found on the Internet a couple years ago doing stuff, but not really creating any big frameworks to help people. So I kept thinking like, I’ll build something and maybe something will come of it, and really I just wanted to do my own research and development and make my own apps with it. So I set off to make it a reality and have worked on it for about a year and a half. And every project I did, it got better and faster and stronger and more flexible. So it’s at a point now where it’s getting some stars on GitHub and that’s really exciting.
So, I’ll talk about workflow a little bit. I think that’s important to understand how it differs from other framework. So, it’s fun because it’s opinionated. It’s not the most hardcore framework written in C++, but it’s a lot more fun to write code in Swift. Especially if you’re just sketching. Satin is fun because it’s built for immediate feedback and playful discovery, like Patricio was talking about. It’s really important to be able to like, noodle with things right then and there to see what’s happening and to understand what your code is doing. And with Satin I wanted to make easy things easy. Like I want a box on screen, easy. But I wanted to make hard things doable. Like I wanna send this buffer to the shader and do something crazy.
The way Satin does this is it basically allows you to livecode shaders and compute systems, and seamlessly create UI so you can noodle with parameters and learn the dark arts of shader programming.
And there are a couple main classes that are super useful for doing this. There’s the FileWatcher that basically watches any file and emits an update whenever things change. There’s the MetalFileCompiler; typically when working with Metal, you have to just load a shader and then it just should work. If it doesn’t work you have to stop the program, edit, recompile. The file compiler basically allows you to use other Metal files, reuse functions really easily, it watches all these files, it emits changes. So you can have something like LiveMaterial that automagically parses a struct which represent uniforms and watches the shader and it informs you of updates.
A couple other fun things. There’s the Metal Library that has tons of fun and useful functions. It’s really exciting to hear Patricio talk about his library. I’m definitely going to take a look at that.
There are Satin Parameters that represent floats, float2’s, float3’s, [vectors?], booleans, etc. And you can hook into events when things change, so it makes it really easy to create code that’s like “Hey, do this when this changes.”
And ParameterGroups represent structs. So you no longer have to have a copy of your struct on the Swift side and the GPU side. And this becomes important because when you want to maximize flow you want to minimize the number of times you’re hitting stop and play. So this allows you to just keep writing shaders without having to stop your app and recompile.
So this is an example of the typical process. This is a blob shader. It’s just running on a sphere. And you can see that there’s no vertex shader, so Satin will inject a vertex shader for you. And here I am just kind of creating a boilerplate vertex shader. So essentially I’m just writing give me the vertex uniform, some blob uniforms, and the LiveMaterial in Satin will take care of wiring up all these uniform buffers to your vertex shader and your fragment shader. So you can really just focus on livecoding the shader and just making it as fun as possible.
So I’m just writing some boilerplate here. You can see that in the struct above, Satin will parse that struct and if there’s comments next to variables, it will create a UI element that represents that parameter. So you’ll see when I switch back over to the app there’s a little panel that has a color uniform and you can see that I’ve changed it. And so, any errors that will happen will be in the console so you can see what’s happening.
So now I’m just including a noise function and then essentially offsetting the sphere’s position based off that noise function. And time is automatically fed in, so you can do some fun things with time. You can set the range of sliders over there and you can start to just use these things instantly to start designing and not necessarily lose any flow in the process.
So this is typically how I write shaders now. There’s parameters, these parameters can be saved to a file, reloaded.
Here what I’m doing is just creating a custom vertex struct that basically gets passed to the fragment shader so I can do some fun color stops. And you can see I definitely make mistakes all the time. But what’s useful is that this program’s not crashing and I’m just like, figuring out why it’s not crashing. And then I’m able to go back and keep writing my shader. And it’s a really fun way to compose things without having to stop all the time. And given all the library functions it’s really easy to start to make things that look pretty really quickly.
So, that was that. This is a more complex example where I was understanding how do you use a cube map, and how do you refract and reflect and do all these crazy things. How do you take textures from your program and then use that to do all these sort of gnarly shader effects. And all this I think is me saying it’s really easy and fun to hook all these things together and be creative. And Satin provides that underlying base, that core, that allows you to just be creative and just work, versus trying to figure out okay, how does this API work or how do I do this and this and this.
So this is just an example from some of my work. I’ll show things that I’ve been making with it. It’s still a very young framework, but there are things I did like for example render point clouds and stylize them; have shadows.
I’ve been making my own artwork, which is just a lot of spheres with noises in it and playing with shading and blending. Another example, just on black.
And what’s nice about working with Satin is like, if you’re in the Mac or iOS ecosystem, you can do really fun things like make screensavers pretty quickly, and sort of ship apps on that platform that are highly graphic very very fast.
Let’s see, this is another screensaver. And all these are just things I’ve made like…maybe not the reaction diffusion one but this I made just like in a day just sort of playing around. So it’s definitely become a really useful tool in my arsenal to just quickly sketch and make things.
This is a clock I was designing, inspired your Dieter Rams.
I started designing some soap with a couple friends. And I was just thinking about the design process. So here’s this ray marching form simulating reaction diffusion. And then I take those things and write code to export things. And it’s become really critical in my process for just sketching but also actually executing on design things.
So here, at some point every software engineer or coder will eventually start making furniture. This is my attempt to make furniture. It’s Voronoi shelves. It’s a little…I dunno, overplayed, but I just figured I’d take a crack at it. And this is like a really fun way of just debugging, trying to figure out form-making and you know, Satin is critical in this process of actually taking ideas and then making them into reality. So for me I think it’s core to my computational thinking.
You can also use it for ray tracing and crazy, awesome, gnarly particle effects. You can also do audioreactive things with it.
And I’ve been recently kind of getting back into form-making. And so you can use Satin as a render engine and use other libraries to be generating forms.
So I’ve been working on this 3D modeler for a couple months. And Satin’s used here to do all the rendering. Like the grid in the background, the gizmos, the actual mesh. The tumbler in the top right. The other UI is just Swift UI. So you can see it’s very powerful. You can start to build these really crazy 3D applications, and Satin’s been really critical just to be the underlying technology behind a lot of that stuff.
So doing the residency, I saw it as an opportunity to kind of take a step back from the framework and learn from all these open source leaders, and learn how to create community. So I created a Discord, and that’s been a great place where the community has been telling me “We’d love to see this,” or “How do you do this.” And this has been really helpful in trying to understand what problems exist and what users actually want to do with it. And this has helped me quite a bit.
And so because of that and the residency, I’ve written a bunch of code to make it easier to expose parameters on iOS. I fixed a bunch of things that were broken because of OS updates. I refactored a bunch of its shader system and file compiler, and made it super easy to use Satin to make AR apps and experiences.
And I’ll show just a couple of examples. I’ve created an XCode template that shows you how to create a scene, livecode a shader, expose parameters, have those things automatically update when a shader parameter’s been updated.
I also got a request to…how does one take a buffer of data and then draw tons of points and lines and whatever. So this example shows you how to take the SARS COVID-19 genome sequence, pipe it into a Metal buffer, and then use that in a vertex and fragment shader to change the colors of these dots, but also to lay them out.
It also shows you how to export an high-resolution image. So if you’re an artist working with code, I think it’s important to be able to make prints or whatever have you.
Then I created a general-purpose GPU computing example just to show hey, this is how you use the compute system to do something a little bit more complicated. So this is just showing a Flocking example inspired Dan Shiffman.
And getting these things to work both on MacOS and iOS was really important, because it’s really easy to prototype things on MacOS, because it’s just…you don’t have to deploy on device. And now my Youi framework has been updated so now you can very quickly with the same code deploy that on iOS and have all those controls accessible.
So, if you’d like to participate, thank you for listening. The framework is slowly growing. You can contribute code, or ideas, or funding. I’ll have a sponsor button on the Satin repo soon.
If you’re interested in building something, or you think Satin can help you in your mission, please reach out. And if you’re a student and you want to learn Metal and Swift and Satin just to like gain some career capital or eventually get a job job or just build something fun, I’m all ears.
And I want to say some thanks. Thank you to the STUDIO for Creative Inquiry and Carnegie Mellon for having me. Thank you Golan, and Thomas, and Linda, and Chris, and the residents. Your support, stories, questions, and feedback have been really inspirational and I’m constantly thinking about new things like codes of conduct and building community and all these other factors that I didn’t even think about before.
And I want to thank my younger brother Haris, who is wise beyond his years and has helped me countless times architecting code, and problem-solving, and just being a better person. So, thank you.
Thank you all for listening, and these are some places where you can find me on the Internet, on Twitter, Instagram. Here’s my work email. And of course where you can find Satin online.
And with that I think my talk is done. Thank you for having me.