Golan Levin: I’m thrilled to introduce our second presenter this evening, Bomani Oseni McClendon, who is a Brooklyn-based engineer and educator working with software and electronic crafts. A contributor to the ml5.js project, Bomani builds software to support artists, and teaches electronic circuitry and physical computing to undergraduates at Parsons School of Design and through free programs for primary school students in Red Hook, Brooklyn. In his creative practice Bomani studies the ways that Black health outcomes are influenced by a history of scientific racism, examining his own proximity to a technosolutionist monoculture and the medical industry as a starting point. He holds a Bachelor of Science in Computer Science from Northwestern University, where he focused on human-computer interaction and learning sciences. Bomani McClendon.
Bomani Oseni McClendon: Hey everyone. Good to see you all. My name is Bomani Oseni McClendon. Thank you all so much for coming to this talk. I’m really excited to be here with you all. And I want to start by just thanking Golan and the STUDIO for Creative Inquiry as well for holding space for us to speak about our work today and for bringing us together in this amazing community of open source artists and technologists over the last few months of the OSSTA residency.
Today I’m going to talk about a project called ml5.js, an open source project which has been the focus of my work throughout the residency. But first I’ll just introduce myself to provide some context about the direction from which I’m approaching this work.
So, I’m an engineer, educator, and health worker living in Brooklyn, New York. I’ve always been excited about circuits and software. I studied computer science in university, with a focus on human-computer interaction and learning sciences and media technologies. And I think that one of the most amazing things about software in particular is the fact that with only a laptop and an Internet connection, you can build so much. And so using software as a medium, I found it deeply gratifying, actually, to be able to experiment and to play in this space.
I became primarily interested in working creatively with technology after being the lead software developer for a large-scale interactive art piece called Shrumen Lumen, which is here, as a member of the Foldhaus Collective based in the Bay Area.
And so since then I’ve had the opportunity to work in support of artists and educators. To name a couple of examples, I built an app to support an exhibition on predictive policing for an American artist who was doing a show at Queens Museum, and a small company I cofounded called Dialup Digital collaborated with an artist named Ryan Kuo to build an online conversational chatbot called Baby Faith which is focused on online disinformation.
I worked on an interface called Genius Home Studio, which helps new producers craft beats and learn how to do that all through an online workstation.
And I currently work as a senior software engineer at a company called Duolingo that’s actually not so far from the Carnegie Mellon campus building tools to support language educators.
On that topic of education, I’m really passionate about teaching. I began my teaching practice through a workshop series called Out of the Black Box I created with my friend and coteacher Neta Bomani with support from Pioneer Works Art Foundation and Good Shepherd services here in New York City. And through this program we have the opportunity to teach primary school students living in Red Hook, Brooklyn about electronic circuitry through arts and crafts activities. And I currently teach about electronic circuitry and physical computing at Parsons School of Design in the undergraduate design and technology major.
My personal practice and community work has centered around topics of social health and medical disparities. And this is a space that I’ve come to through my own personal experience supporting family members who’re deeply affected by the medical system, or through talking with family members who’re health workers. And specifically I’m interested in the ways that scientific racism is used as an ideological basis for a society that turns black people into clinical material.
For example, in one project I was interested in the ways that online media is used as a prominent vector for spreading medical narratives that pathologize Blackness. Many statements on Black health outcomes are often overlooking all of the roles that socioeconomic disparities play in those outcomes, and anti-Blackness plays in those outcomes, and it leads people to believe that these disparities are accounted for by genetic or evolutionary differences and deficiencies associated with our race. And so they kind of position race as the disease. So for this project, I wrote a program that automatically crawled through hundreds of articles on WebMD to collect every sentence containing the word “black” or “African” and I printed the resulting data set into a book called Black Health. And flipping through the book you can kind of feel the weight of how these issues and these health disparities are affecting our community, but you can also kind of start to critique the ways in which these topics are discussed and how they lack so much necessary context.
And sometimes I also like to think about these ideas more symbolically. This is a piece called Informed Consent, which is an emotional remembrance of an experience supporting my parent through his process of signing an informed consent contract for multiple myeloma cancer treatments. And in this project I laid out the pages of his actual contracts so that viewers could sort of see the medical sterility with which this so-called informed consent process is actually approached in real modern studies. But I also wanted to overlay this visual with hands kind of beating against the pages to kinda illustrate this idea that many people have been caught behind these types of documents in the past, and that the informed consent process is actually also a way to exploit people and misinform them in some cases.
And most recently I’ve been studying these concepts practically. At the beginning this year I earned my state national certifications to practice as an emergency medical technician. And over the last few months I’ve been volunteering through NYC Medical Reserve Corps as a COVID-19 vaccinator administering the Moderna vaccine here in Bushwick, Brooklyn. And I’m also a new member at Bedford-Stuyvesant Volunteer Ambulance Corps, which is a Black-led ambulance service providing emergency medical services and free medical training to the Bed-Stuy, Brooklyn community.
One of the things that I’ve learned through these experiences is that the work, or care work perhaps, associated with supporting and engaging in these communities that I’m a part of has really been deeply restorative to my own spirit, especially through the last eighteen or so months. And through creating tools to support artists, teaching youth, or through medical preparedness and response, I’ve found that identifying ways to be a part of shared collaborative and community projects that produce knowledge or build infrastructure that has been influential and instructive for me has connected me to a lot of really amazing people and ideas. And so, most relevant to this presentation I want to talk about how my more recent work as an open source maintainer has actually helped me learn more about how to be present in communities that are important to me.
So you’ve heard me reference it a few times, but let me tell you what ml5 is and kind of what it contributes to the open source community. We typically describe ml5.js as a neighborly approach to creating and exploring artificial intelligence in the browser. In other words, ml5.js aims to make machine learning approachable for a broad audience of artists, students, educators, creative coders and others, who are interested in using machine learning to make creative projects on the Web. ml5.js is a beginner-focused project, meaning that our goal is to make the library as friendly as possible to people who don’t have much or any prior experience with math, ML, or coding.
And a primary benefit of ml5.js is the simplicity. Though our libraries internally use complex machine learning algorithms and models built with TensorFlow.js, we wrap the internal logic with our simplified models to make it easier to use.
We also have a wide variety of educational resources from sources like The Coding Train and through various open source ml5.js curriculums, tutorials, demos, examples and so on that have been shared by contributors and maintainers. And since we’re deeply influenced by the Processing Foundation and the work of p5.js, the libraries, documentation, and examples are actually built to express how easy it is to use ml5 and p5.js together.
As a community, we really encourage ethical and friendly uses of machine learning. And overall I would say that if you’re new to ML, or if you’re new to programming but you have an interest in leveraging ML techniques in your work, ml5.js might be a good place for you to start.
So what can ml5.js do? At the moment ml5.js supports over twenty different machine learning models which enable a variety of different features. You can use models like BodyPix, PoseNet, FaceMesh, Handpose, etc. to create creative sort of computer vision projects. Alternatively you can use models like CharRNN, Sketch-RNN, CartoonGAN and so on to generate text or images using machine learning techniques.
I’m gonna just go over a couple of examples here of how simple it kind of is to use the software and create things like this. Mere’s a MobileNet image classifier.
Here’s an example of style transfer. Again, just a couple of lines of code to accomplish this.
And here’s PoseNet kind of showing full-body skeletal tracking, which you can then pair with other techniques to create kind of interesting creative projects.
For example, people have used things in our library to create things like an ASL learning tool using machine learning computer vision. Or have used the library to bring together literary works from a variety of Black scholars and used that to kind of generate speculative new texts that combine some of the ideas that are coming from writings by a variety of different thinkers.
The library began in 2018 at NYU’s ITP program, with the initial groundwork laid by Cristobal Valenzuela. And since then the library, software, and community have been advanced by many contributors and maintainers including people like Joey Lee, Dan Shiffman, Yining Shi, Ashley Jane Lewis, Christina Dacanay, Lydia Jessup, Sam Krystal, Michael Weinberg, and myself just to name a few of us. And you can see a more complete list of these folks on our Github page. But I wanted to highlight this because even though I’m here speaking on behalf of the library today, it’s important to remember that this is really a community-built project and there’s tons of people who have been involved with shaping this work.
I personally first joined the ml5 team at the beginning of 2020, and since then I’ve been primarily focused on maintenance and improvements for the library that aim to make the ml5.js community and codebase more approachable to people with a wide variety of backgrounds. And over the last eight or so months, I’ve operated as the lead software maintainer for ml5.js. I respond to Github issues, coordinate releases of the library, and I recently added support for the new TensorFlow hand pose and face mesh models within ml5.js. I’m also part of the working group that’s developing the ml5.js code of conduct and software license, and I’ll speak a little bit more about that code of conduct in depth kind of later in the talk.
So, through these experiences there have been a couple of things that I’ve learned. You know, definitely valuable skills in software development or open source workflows, etc. But the most important things that I’ve learned have actually been related to how I think about my roles in supporting communities, contributors, and also particularly in upholding community values as well. And there are a few primary insights, or instead I’d actually like to call them personal reminders, for myself at least, from my experience that I’d like to talk with you all today.
So, the first kind of personal reminder is to remember that contributions mean more than just Github commits. And I’ll illustrate this reminder anecdotally through a case where I actually forgot this. So one of the things that I deeply appreciate about my experience within the ml5.js community is that we are deeply beginner-focused. When I say beginner-focused I don’t just mean that the library’s easy for beginners to use. I also mean that we really welcome contributions to the library from beginners as well. Many people have actually taken their first steps into open source by making changes in the ml5.js code base or within our community.
And this is personally why I’m excited about the library, but I also recognize that this kinda means we’re not always easily able to meet the highest bar of pure engineering quality or software maintenance. And to address this we’re in this general that is focused on making improvements to the stability and quality of the library. Over the first few years of the project, we had a lot of growth and a lot of change, new contributors, new models and features, lots of growth in usage, etc. And over the last year or so, we’ve generally slowed down the addition of new features to focus on stabilizing and improving what we have at the moment.
So when I started the OSSTA residency, I decided that the best way of setting a goal around “cleaning up our library” was to focus on closing issues which contained old questions, or bugs, or features, and to-dos that we hadn’t yet been able to our address. And at that time, you know, I wrote down that my goal was to take us from I don’t know, around like 170 open issues or something to around 50 by the end of the residency period, mostly by fixing or addressing these issues by myself.
Hm. And I know I can’t see y’all’s faces, but I do hope some of y’all are laughing at me right now because it was actually a pretty ridiculous goal to try to take that on. And as a person with an engineering background who also got my start in the ml5.js library by making major changes to the codebase, I sort of inadvertently perceived my role as a software engineer to be the person who could figure out and fix you know, most of the issues. And what I realized is that not only is my knowledge like, limited…for example I’m not an expert on TensorFlow.js at the moment, but so is my time. Pretty busy, got a lot of different things going on just like we all do. And so it actually wasn’t tractable for me as an individual with the current structure of my life to set a goal around closing issues fast enough to reach fifty open issues.
And so I began to feel really swamped by the responsibility involved in this, and started to find myself almost procrastinating on my work with the library. Maybe this is something that happens to some of y’all as well. And so at this impasse, you know, it really pushed me to deeply reflect on what was happening. In the ml5.js community, we frequently, frequently highlight that contributions can come in literally any form: research, writing, design, community development, instructional support, not just coding. But in my own mind I had boxed myself into a position where I could only view my contributions as software changes. And this was stressful. It was also strategically wrong. And it limited our library’s ability to reach the quality and stability goals that we had envisioned.
So, I had to change approaches. And with the support of my teammates and collaborators I returned to our core values of openness and beginner-friendliness and recognized that as a lead software maintainer, what I actually can contribute is a supportive entrance into our project, and into coding for some people, into open source for some people. And so I started to experiment with ways to help make more people feel excited to contribute to ml5.js.
I started a maintenance channel in our community Discord server where I gave updates about what I was changing in the library and invited community members to chime in and ask questions. I did a two-hour livestream on Twitch during which I narrated myself working on just day-to-day maintenance tasks; labeling issues, reviewing PRs, updating documentation, coordinating releases and so on. And I also committed to literally just asking for more help. From friends, from students, in talks I gave, in Discord, on Twitter and so on. And I started labeling things like good first issues on our Github, and creating project plans that I could actually share with other people who I was requesting help from as well.
I’ve recently been meeting with and chatting with a lot of the new community members who have an interest in helping maintain the library, just to understand how I can support them in finding areas of the ml5.js project and community that they’d like to advance. And I’ve really redefined my understanding of contribution to the library from fixing all the issues, to helping create a community that can improve the library together as a collaboration.
And so even though we haven’t reached my earlier goal of only having fifty open issues on our repository, I fundamentally feel a lot more excited and engaged in the work and in this new way of contributing to the project. I believe it deeply aligns with much more of the values that we have set for the project.
Anyways, I say all that to say that if you’re listening to this talk and you’re remotely interested in contributing to an open source creative tool and community, we would absolutely appreciate your help in whatever form that takes. After the talk I’ll share links to our Discord server in Zoom and in the YouTube chat as well. And I’d love to talk with you about maybe your interests or some of the things that you think could be improved within the library or the community, just to help you find what you’d like to shape.
The second personal reminder I have is that clear values are really critical infrastructure in any project, and I think especially in open source projects. For more than a year now, the team has really been working to understand a central question, which is essentially like, how can we use the rules that govern the ml5.js community to encourage friendly, creative applications of machine learning? And so one of the things that’s important to us is that we recognize this space of machine learning is quite fraught. You know, there are a variety of issues that are happening in this space, and I think machine learning as we all know, or as many of us know, is a tool that can advance a lot of significant violence to a variety of communities that affect many of us who are contributors and people who are using also the library and the project.
And so what we’ve been working on over the course of the last year or so is a code of conduct that essentially clarifies our values as a library and as a community and a project. This is a screenshot of the first version, which we shared a few months ago and has been undergoing a round of reviews from various creators and technologists in the community. We’ll be sharing an updated version of this soon, but I just wanted to quickly touch on some of the immediate benefits that I think starting to think about this has had for us.
In general it really sets a standard for how we conduct ourselves in the library, and it’s forced me to ask questions like okay, all of our documentation right now is in English. How many people are we leaving out of the picture, and what can we do moving forward to change that? And I think while the code of conduct is still new, I’ve personally understood that this process of kind of thinking about our values has really greatly influenced how I think about my own work and my presence within the library and within the community.
And lastly I think it’s set us on a course as well about thinking on how the work that we’re doing creates relationships with major technology companies or entities whose technology we’re using, and I think that this kind of question around how open source technologies strike up relationships with varieties of entities across this sort of technology landscape is a question that we want to think about. For example, what would it mean for us to think about maybe supporting more models that aren’t created by Google, and how might that help this library be more didactic in the ways that it can communicate the space and span of this domain?
Briefly I just want to share one final personal reminder, and that’s that open source is really a durational practice rather than a momentary one. And strong open source communities and technologies that enable them are products of commitment and responsibility. Every contribution that ml5.js has received is an instance of someone taking responsibility to share their time and energy with the broader community. And as a maintainer I believe we have a responsibility to honor those contributions through our commitments to the future of the library, and most importantly to the people who comprise our open source communities. And in practice I believe that this looks like writing clear documentation of our tools and ideas, or seeing out our changes to the end, asking for lots of help from the community, and supporting all types of contributions.
And even though this is a set of values that I’ve held for a while, I’ve recognized that also thinking about this, we really need to double down on the fact that when I say like, open source is a durational practice rather than a momentary one, I also mean to say that open source, or at the least the open source that I aspire to be involved with, requires durable communities as well. And so these prior reminders about contribution and about values and codes of conduct I think really feed into this broader goal of creating an open source community that can exist as a healthy space for beginners to take their first steps in this domain.
Ten years from now I can’t promise that ml5.js or any given open source project or library will be actively maintained or even useful in the technology landscape, but in the present I do believe that the decisions that we make about how we support our open source communities, how we uphold values within our open source communities, and about how we build up our collaborators will influence decisions that people ten years from now make about how they bring new open source technologies into the world. And I think that that’s the effect that the ml5.js community has had on me. That’s the effect that some of my peers here in the OSSTA residency have had on me. And I think through this residency I’ve realized that that’s the effect that I’d like for my open source efforts to have on others as well.
That’s it for me. Thanks all for listening.
Golan Levin: Bomani, thank you so much for sharing that with us. I use ml5 of course with my students. And I think it’s… I see this again and again in the presentations from the OSSTA residency group that contributions are not just writing code but also managing a community. What I see coming out of your presentation is this amazing sentence you had that values are critical infrastructure. You know, that tools have politics, and I think when one is “messing around” with machine learning, that politics is really you know, on your sleeve. Because these tools as you point out can be used to create great harm, and these tools are very fraught. So recognizing the politics of the tools you’re working on and the ways in which the values you’re embedding in them are critical to the infrastructure both in terms of how people use it and how the community wants to participate in it is really incredible.
Could you give maybe a quick example or two from the codes of conduct and the license you’re creating that you think kind of embed some of these values in particular?
Bomani Oseni McClendon: Yeah, I mean I think that one of the things that we have been discussing a lot is just our focus on the beginner-friendliness aspects. And I’ve recognized that that has come up a lot in decisions that I’ve made as a software maintainer for example. We frequently receive requests to switch the entire library to TypeScript, for example, or to change some particular pattern about how we make things within the project. And I think that we’ve actually held off on a lot of those things because we are really really committed to this particular experience of people who are not experts, who are not you know, full-time professional coders being able to like, come into this project and understand it from a perspective of usage but also to understand it behind the scenes, to be able to go and potentially read the code and glean some insights from it as well. So I think that’s just one of many examples where I’ve noticed that come up a lot in my day-to-day decisionmaking.