Golan Levin: I’m thrilled to intro­duce Luis Morales Navarro. He’s a grad­u­ate stu­dent in the Learning Sciences and Technologies pro­gram at the University of Pennsylvania, and a con­trib­u­tor to p5.js. His back­ground is in mid­dle school com­put­er sci­ence, open source soft­ware devel­op­ment, and phys­i­cal com­put­ing. He’s inter­est­ed in issues of access, inclu­sion, and moti­va­tion in learn­ing to code, and the role com­mu­ni­ties play in the devel­op­ment of com­pu­ta­tion­al flu­en­cy. Luis Morales Navarro. 

Luis Morales Navarro: Thank you Golan. Let me get my slides on screen and start. There we go.

Hello! Thank you Golan, again. I’m very grate­ful that I got the oppor­tu­ni­ty to par­tic­i­pate in this pro­gram and learn so much from all of you. I do not describe myself as an artist, so this res­i­den­cy was a great oppor­tu­ni­ty to engage with artists and oth­er folks that are work­ing on what I pre­fer to call open source soft­ware tools for cre­ative expres­sion and cre­ative com­put­ing. If any­one is inter­est­ed in down­load­ing the slide deck that I’m going to be using, I post­ed a link on the chat, and it has screen reader-accessible alt text descrip­tions for images. 

I am very inter­est­ed in how peo­ple learn to code. And for some peo­ple when they’re learn­ing to code, it is a very excit­ing oppor­tu­ni­ty where [indis­tinct] are chal­lenges that have to be solved and puz­zles to dis­cov­er new things and try new things. But for oth­ers it can be a very frus­trat­ing expe­ri­ence, espe­cial­ly when learn­ers are not sup­port­ed by the tools that they’re using, by their peers, their teach­ers, or by the com­mu­ni­ties around them. 

I’m not an artist, but I spend most of my time study­ing and research­ing how novices learn to code, con­sid­er­ing the cog­ni­tive, the socio­cul­tur­al, the emo­tion­al and moti­va­tion­al fac­tors of learn­ing. But also think­ing about how we can improve the expe­ri­ence of learn­ing to code and make it more engag­ing for novices, and youth in par­tic­u­lar, as they make personally-relevant cre­ative com­put­ing projects. 

Debugging by Design: A Constructionist Approach to High School Students’ Crafting and Coding of Electronic Textiles as Failure Artifacts

At Penn, I’m cur­rent­ly work­ing with a team research­ing how youth design and solve bugs in phys­i­cal com­put­ing projects. And although this is a top­ic I can talk about for quite some time, today I’m going to talk about some­thing else. 

I’m going to talk about p5.js and web acces­si­bil­i­ty, and par­tic­u­lar­ly about cre­at­ing infra­struc­ture for acces­si­bil­i­ty in an open source cre­ative cod­ing com­mu­ni­ty. So, is p5.js acces­si­ble to peo­ple who are blind and visu­al­ly impaired. That’s a ques­tion that a lot of peo­ple ask. And the quick answer is no. Very few sketch­es are. Yet, we have been work­ing towards mak­ing p5.js more acces­si­ble to peo­ple who are blind over the past five years. And this is what I’m going to talk about today, about the work that we have done and how it has evolved over time towards a community-centered approach, and also towards cre­at­ing infra­struc­ture that can sup­port bet­ter acces­si­bil­i­ty fea­tures in the future. 

Six workshop participants sit around a table with laptops in front of them. They discuss and try a coding exercise.

It all start­ed in 2016, when Chancey Fleet, a blind tech­nol­o­gy instruc­tor at the New York Public Library’s Talking Book Library, she want­ed to cre­ate a dia­gram for a ban­quet. And she reached out to Claire Kearney-Volpe and they start­ed look­ing for tools to make the dia­gram. Chancey is blind, and they found that most of the tools out there were not acces­si­ble to peo­ple who were blind. Claire reached out to the Processing Foundation and the folks that were devel­op­ing p5.js. And they start­ed work­ing togeth­er to improve the acces­si­bil­i­ty of p5.js sketch­es. In the pic­ture we actu­al­ly can see Chancey sit­ting over here, and here is Claire Kearney-Volpe. 

So to give you a lit­tle bit of con­text, peo­ple who are blind nav­i­gate com­put­ing tech­nolo­gies using screen read­ers, which are soft­ware that read out loud what is dis­played on the screen. But to do this the con­tent dis­played on the screen has to fol­low cer­tain acces­si­bil­i­ty guide­lines, so that the screen read­er can access it. For exam­ple screen read­ers rec­og­nize images but can not describe them. We need to add alter­na­tive text or image descrip­tions for screen read­ers to com­mu­ni­cate the mean­ing of what an image says. 

So Claire start­ed work­ing on adding alt text to the ref­er­ence exam­ples of the p5.js web site. And actu­al­ly, sim­ple web con­tent that is well-written is very acces­si­ble to screen read­ers. But the canvas ele­ment which is at the core p5.js is the most inac­ces­si­ble web ele­ment, because it is a bitmap. A screen read­er only knows that there’s a can­vas ele­ment, but it can­not tell what shapes are being drawn and what is going on inside of the can­vas. And because it’s a can­vas ele­ment and not an image ele­ment, the alt text fea­ture for images does­n’t real­ly work for the canvas. 

I met Claire in late 2016, when Scott Fitzgerald who is lis­ten­ing to this talk intro­duced us. And they were work­ing with Mathura Govindarajan to add acces­si­bil­i­ty fea­tures to the p5.js edi­tor that Cassie was start­ing to work on. I joined them, and togeth­er with the sup­port of a ded­i­cat­ed group of con­trib­u­tors and advisors—Cassie, Lauren McCarthy, Josh Mielle, Sina Bahram, and Chancey fleet—we implant­ed three acces­si­ble can­vas outputs—a text out­put, a grid out­put, and a sound output—that soni­fied the move­ment of shapes. So if a shape moved up and down, the pitch of the sound would change, and then if it moved left or right the sound would pan. And all that was work that we did in the alpha edi­tor, so the very ear­ly p5.js web editor. 

At the time, Claire, Mathura, and I were all work­ing at NYU, and that was very help­ful because this project became part of NYU’s Ability Lab and it was some­thing that we could do dur­ing our day job. We test­ed and improved the out­puts. Claire was teach­ing a class on acces­si­ble tech at NYU. And that was also great. Because we could do user test­ing and involve stu­dents in the process of devel­op­ing these outputs. 

Our efforts were lim­it­ed to the edi­tor. And we start­ed think­ing, how could we use screen reader-accessible out­puts out­side of the edi­tor? Mathura and I got a Processing Foundation fel­low­ship with Clair as our men­tor to fur­ther devel­op the out­puts into an addon that could be used to make any p5 sketch on the Web acces­si­ble. This addon could be includ­ed into the HTML of any p5 project and make the out­puts acces­si­ble. So for this, what it was doing was kind of read­ing from the sketch of p5 and inter­pret­ing that to cre­ate a mean­ing­ful descrip­tion of what was going on in the can­vas. We call this kind of mon­key patch­ing because there was no real con­nec­tion between the p5.js library and what the addon did. And for that we got a lot of help from NYU stu­dents— Antonio Guimaraes, Elizabeth G. Betts, Mithru Vigneshwara… And we actu­al­ly got our first con­trib­u­tor who was some­one who we did­n’t know when were we work­ing in this. Yossi Spira just showed up and start­ed con­tribut­ing with code, and that was a won­der­ful thing that hap­pened. And we cre­at­ed this library that is called p5.accessibility.js. As you can see, it has been archived. And I’m going to talk more about why it has been archived in a few minutes. 

By the time we fin­ished the fel­low­ship that helped us cre­ate p5.accessibility.js, none of us worked at NYU any­more. And we could­n’t ded­i­cate as much time to work on this and to work on p5 acces­si­bil­i­ty. So the project kind of died out for a lit­tle bit. 

Until 2019, when some of us went to the p5.js Contributors Conference at the STUDIO for Creative Inquiry at Carnegie Mellon. And that was a real­ly impor­tant moment. Because as a com­mu­ni­ty as Cassie described, we came togeth­er and we reflect­ed on what access meant for our com­mu­ni­ty and on what the pri­or­i­ties were for p5.js. I’m show­ing an image again, and I’m pret­ty sure you’re going to see it for a third time today in a few min­utes in eve­lyn’s pre­sen­ta­tion. At the 2019 p5.js Contributors Conference, we rein­forced our com­mit­ment to access and inclu­sion. And togeth­er with Claire, Sina Bahram, Lauren, Kate Hollenback who is here too, Olivia McKayla Ross, and eve­lyn, we out­lined a path­way for­ward. Which in terms of web acces­si­bil­i­ty meant a cou­ple of short-term actions.

First we iden­ti­fied the need for func­tions that allow users to write their own descrip­tions. And sec­ond the impor­tance of merg­ing the addon into the p5.js library. So we did not want to con­tin­ue work­ing on p5.accessibility.js, because acces­si­bil­i­ty should not be an addon but some­thing that is part of the core library of p5.js.

And this idea of two types of out­puts, that kind of was the begin­ning of rede­vel­op­ing what acces­si­bil­i­ty looks like in p5.js. Having func­tions for user-generated out­puts and func­tions for library-generated outputs. 

I worked on this with Kate Hollenbach part­ly sup­port­ed by Google Summer of Code last year. And we worked on four dif­fer­ent func­tions. For user-generated out­puts we cre­at­ed describe(), that allows peo­ple to cre­ate a short descrip­tion of what their sketch is doing and what’s going on on the can­vas; describeElement(), that helps peo­ple cre­ate short descrip­tions of ele­ments or groups of shapes that when they come togeth­er they cre­ate mean­ing also on the canvas. 

And then library-generated out­puts. A textOutput(), a text descrip­tion of what’s going on on the can­vas. And a gridOutput(). All these func­tions that are mak­ing this pos­si­ble are dis­trib­uted through­out the p5.js library. And that is real­ly excit­ing. Because you can find them in many files that have noth­ing to do with acces­si­bil­i­ty. You will find things that have to do with web acces­si­bil­i­ty. And then if you mess up some­thing some­where else, it’s going to mess it up in anoth­er place. 

And we decid­ed to do that so that it’s ful­ly inte­grat­ed, but we also decid­ed to put it in the ref­er­ence in the sec­tion called Environment.” Because we want­ed acces­si­bil­i­ty to be a core part of the envi­ron­ment of learn­ing to code with­in p5.js

And I think what I’m going to do now is actu­al­ly show you… Yes! I’m going show you what it looks like, so I’m going to stop shar­ing my screen and share my screen in one second. 

So I’m going to start by show­ing you an exam­ple. So this is an exam­ple of a sketch using describe(). There you can see what we are doing here is just writ­ing a lit­tle descrip­tion of what the sketch does. And I’m gonna play the sketch. And there you can see it. It’s a pink square with a red heart in the bot­tom right cor­ner. And this is what a screen read­er, what the assis­tive tech­nol­o­gy that peo­ple who are blind would get out of this sketch: [A machine-generated female voice reads out, Pink square with red heart in the bot­tom right cor­ner. You are cur­rent­ly on the text ele­ment.”]

So in a very sim­ple way, any pro­gram­mer can just write a short descrip­tion inside of the func­tion describe(), putting the descrip­tion between quo­ta­tion marks. 

Let me show you anoth­er exam­ple of describe(). In this case I’m also dis­play­ing the descrip­tion on the can­vas on the HTML. And you can see how the descrip­tion is also chang­ing. So the val­ue of the ellipse that we’re draw­ing here is being shown, too. And I’m gonna play again. This is what a screen read­er would get out of this can­vas: A machine-generated female voice reads out, Canvas descrip­tion: region: green cir­cle at x pos 5 mov­ing to the right. Canvas descrip­tion: green cir­cle at x pos 49 mov­ing to the right. You are cur­rent— Canvas descrip­tion: region: green cir­cle at x pos 79 mov­ing to the right.”

And that gives you an idea. So there what I was doing is I was access­ing the can­vas sev­er­al times, and every time I would access it it would give me a dif­fer­ent descrip­tion because the ellipse is mov­ing and the descrip­tion is chang­ing. So it’s kind of a dynam­ic descrip­tion. And again, this is cre­at­ed by using describe(), a very sim­ple function. 

Here is anoth­er exam­ple of using describe() ele­ments. Here we’re using describe() to describe the whole can­vas and then describeElement() to describe lit­tle ele­ments inside of the can­vas, in this case a heart and a cir­cle. And also instead of read­ing it or telling you about it, I’m gonna play— A machine-generated female voice reads out, You are cur­rent­ly on a tex— Canvas: heart and yel­low cir­cle over pink back­ground. Canvas ele­ments and their descrip­tions: cir­cle. You are— Yellow cir­cle in the top left cor­ner. Heart. You are cur­rent­ly on the— Red heart in the bot­tom right cor­ner. End of can­vas descrip­tion.”

So that gives you an idea of how this can be help­ful so that peo­ple can describe the can­vas as a whole but then also the lit­tle things that are hap­pen­ing there. 

And here’s a sketch that is a lit­tle bit more com­plex. So in this one we have all these things hap­pen­ing at the same time. And what is great about it is that we can describe the thing as a whole but then go into the lit­tle ele­ments and say a lit­tle bit about them. So this is what the screen read­er would get out of this descrip­tion that I wrote in this sketch. A machine-generated female voice reads out, Main can­vas descrip­tion: region: a red wheel­bar­row beside white chick­ens, inspired by The Red Wheelbarrow’ by William Carlos Williams.” Canvas ele­ments and their descrip­tions: wheel­bar­row: A red wheel­bar­row with a gray wheel rests on the brown ground. Chicken 1: a white chick­en in front of the wheel­bar­row. Chicken 2: a white chick­en to the right of the wheel­bar­row. Chicken 3: a white chick­en stand­ing between Chicken 1 and Chicken 2. End of can­vas descrip­tion.”

So it gives you an idea of how screen read­ers read the con­tents of these func­tions. And for the oth­er two func­tions I’m not gonna play the screen read­er. I’m just gonna show you what it looks like. So this is text out­put. Again, this is a library-generated out­put that has a gen­er­al descrip­tion of what the can­vas has, then the con­tent of the can­vas with a lit­tle bit more detail. And then a table with the shapes. And what is excit­ing about this is that if the ele­ments on the can­vas are changing—and again right now sup­port is lim­it­ed so we’re only sup­port­ing basic shapes. But infra­struc­ture is there so that this can be expand­ed. If the shapes on the can­vas are mov­ing, the descrip­tions are also mov­ing. So you can see it said top mid­dle” and now it says top right.” So the descrip­tion is dynam­ic. And it’s being gen­er­at­ed as the ellipse is being gen­er­at­ed in the draw() loop. So that’s real­ly excit­ing because the descrip­tion and the ellipse are being cre­at­ed at the same time. 

And let me show you now the table out­put. So table out­puts are cool, cause they kind of cre­ate a table where the ele­ments are mapped. So it cre­ates a table that is ten by ten, and then depend­ing on the posi­tion of the ele­ments on the can­vas, that’ll be the posi­tion of the ele­ments on the table or on the grid. So here you can see how the red cir­cle is kind of mov­ing inside of the table as it is mov­ing on the can­vas. Very excit­ing and this is use­ful because screen read­er users can actu­al­ly kind of get an idea of spa­tial dis­tri­b­u­tion between the shapes on the canvas. 

And I’ll show you this one where noth­ing is chang­ing but again it just gives you an idea of how things are dis­trib­uted spa­tial­ly on the canvas. 

So yeah, we decid­ed to inte­grate the acces­si­bil­i­ty fea­tures into the can­vas and to cre­ate these user-generated out­puts. And we’ve been work­ing on it for a while. It’s been fun. The func­tions that make this pos­si­ble are dis­trib­uted across more than twenty-six files in the p5.js source. And that’s also excit­ing because peo­ple that are work­ing on some­thing that has noth­ing to do with acces­si­bil­i­ty have to encounter lines of code that deal with acces­si­bil­i­ty, and then that involves the com­mu­ni­ty in think­ing about it. 

And that’s actu­al­ly what I want to talk more about. And acces­si­bil­i­ty’s not real­ly a task for the few. It is and it should be a com­mu­ni­ty endeav­or. And a cou­ple of folks ear­li­er talked about “#noCodeSnobs” as a p5.js kind of mantra. And that’s def­i­nite­ly part of this project, but also #noAccessibilitySnobs. So, try­ing to think about acces­si­bil­i­ty as some­thing that any­one and every­one can engage with, and some­thing that should be easy. And as a com­mu­ni­ty respon­si­bil­i­ty where because we care about each oth­er we want to make our work, our projects, more acces­si­ble to more people. 

So I think that acces­si­bil­i­ty fea­tures should be acces­si­ble. That’s the first thing. They should be acces­si­ble to screen read­er users, of course. They should be acces­si­ble to con­trib­u­tors. And they should be acces­si­ble to peo­ple who use p5.js. In the Web Content Accessibility Guidelines there is a prin­ci­ple that is called under­stand­able.” And that is that infor­ma­tion must be under­stand­able. So it’s not enough to say look, this exists and here is the infor­ma­tion. But it should be up pre­sent­ed in a way that the per­son that needs the infor­ma­tion can under­stand it. And I think that that’s real­ly impor­tant for the com­mu­ni­ty. It is real impor­tant for us to think about acces­si­bil­i­ty in a way that we can con­vince con­trib­u­tors, even con­trib­u­tors that are not work­ing on things relat­ed to acces­si­bil­i­ty, to con­sid­er acces­si­bil­i­ty as they work. But also so that we can con­vince peo­ple who are mak­ing sketch­es in a class­room, or sketch­es for fun, to also con­sid­er the acces­si­bil­i­ty in their sketch­es. So that’s kind of what I’m think­ing a lot about these days, is how do we make these efforts about acces­si­bil­i­ty under­stand­able to more and more people. 

And at the same time, make them more acces­si­ble. To screen read­er users of course, because we want them to be able to under­stand the con­tent that the p5.js com­mu­ni­ty is cre­at­ing. To con­trib­u­tors, so that they have to think about acces­si­bil­i­ty when­ev­er they make a change to the library even if it’s not relat­ed to acces­si­bil­i­ty. And actu­al­ly dur­ing OSSTA res­i­den­cy some­thing very fun­ny hap­pened and some­thing that made me real­ly hap­py hap­pened. And it’s the WebGL fea­tures of p5.js were bro­ken for about two weeks, and we were get­ting all these mes­sages on GitHub of peo­ple say­ing like [in a mock­ing voice], WebGL is not work­ing, help us.” 

And the rea­son why it was­n’t work­ing was because of acces­si­bil­i­ty. Some lit­tle bug in one of our acces­si­bil­i­ty func­tions that declared glob­al vari­ables was break­ing WebGL. And for me that was a huge accom­plish­ment. Because that meant that acces­si­bil­i­ty was ful­ly inte­grat­ed into the library and there was a con­sid­er­a­tion that we as the p5.js com­mu­ni­ty had to take when­ev­er think­ing about the library as a whole and not an iso­lat­ed thing that only a few peo­ple care about. 

As part of this work done on web acces­si­bil­i­ty, con­trib­u­tor docs. So again doc­u­ment­ing what is hap­pen­ing, where, and why is it hap­pen­ing there. I also think that acces­si­bil­i­ty should be acces­si­ble to every­one. To peo­ple like any of us, and peo­ple that are learn­ing to code, espe­cial­ly novices. And that’s why we cre­at­ed these describe() and describeElement() func­tions to empow­er users to cre­ate their own descriptions. 

And a lot of peo­ple ask Why don’t you use image recog­ni­tion soft­ware to cre­ate descrip­tions of what the can­vas says, or what the can­vas has?” And it is a pos­si­bil­i­ty, and we are think­ing about explor­ing it and it’s def­i­nite­ly down the pipeline of the things that we want to try and explore. But the real­i­ty is that the best descrip­tions of a sketch come from the peo­ple that cre­ate them. And that’s why when we’re think­ing about acces­si­bil­i­ty, and that’s why I get very excit­ed when I talk about describe() and describeElement(), because it’s a very sim­ple func­tion that any begin­ner can use to make their sketch­es a lit­tle bit more accessible. 

So there’s a lot of work to do. User test­ing, test­ing, and test­ing. That’s some­thing that has to hap­pen. And these lit­tle things that I showed you today are not near per­fect, and they will prob­a­bly nev­er be. And I’m very glad that I have the sup­port of Claire Kearney-Volpe. And togeth­er with Chancey Fleet, Claire is doing a lot of the test­ing so that then we can get that feed­back and inte­grate it into how these func­tions with­in p5.js work. 

We’ve also got­ta work on tuto­ri­als, improv­ing the ref­er­ence, and on learn­ing mate­ri­als. And that’s one of the things that I’ve been try­ing to tack­le dur­ing this res­i­den­cy work­ing on the reference. 

And we’ve got­ta work on acces­si­bil­i­ty in the edi­tor, and that’s a con­ver­sa­tion that we keep hav­ing, par­tic­u­lar­ly because CodeMirror is not the most acces­si­ble tool and it’s cur­rent­ly used in the editor.

But the most impor­tant thing per­haps is that we’ve got to keep going. And we’ve got­ta push for­ward and try dif­fer­ent approach­es and new things. And anoth­er thing that we can do and then you can also do, is you can use describe() and describeElement() in your sketch­es. And you can use it with your stu­dents. And that could be a real­ly cool way of engag­ing with acces­si­bil­i­ty as a com­mu­ni­ty effort, an effort of care, and of mak­ing our work more acces­si­ble to more people. 

Big thanks to: Claire Kearney-Volpe, Kate Hollenbach, Lauren McCarthy, Mathura Govindarajan, Chancey Fleet, Cassie Tarakajian, Josh Mielle, Sina Bahram, Antonio Guimaraes, Elizabeth G Betts, Mithru Vigneshwara, Yossi Spira, Olivia McKayla Ross, Evelyn Masso, Taeyoon Choi, and Atul Varma.

So, a lot of this work has been pos­si­ble because of the p5.js com­mu­ni­ty and I just want to say a big thanks to a bunch of peo­ple that have con­tributed through through­out the past five years to the way that we think about acces­si­bil­i­ty, but also to the test­ing, and of course to the devel­op­ment of these func­tions, and of the library that is archived and is not used any longer. 

Yeah. Thank you so much. I think that that’s it for now, Golan.


Golan Levin: And I’m back. Hi, Luis. Thank you so much for that won­der­ful pre­sen­ta­tion. We have a minute for a ques­tion or two, and I thought I’d ask, actu­al­ly. So, I have to say that the first time I saw the describe() func­tion, I was actu­al­ly kin­da teared up. I was real­ly moved by the way that it made the art­works avail­able to new peo­ple, and to peo­ple who could­n’t expe­ri­ence them otherwise. 

But I won­der if you could talk about… I mean, these func­tions like describe(), they’re for the audi­ence of the art­works. But I won­der if you could talk about enhanc­ing acces­si­bil­i­ty for the cre­ators of the art­works, for pro­duc­tion not just perception. 

Luis Morales Navarro: Yeah. So that’s the rea­son why we have two sets of func­tions. The user-generated descrip­tions, we’re think­ing of those as func­tions that can sup­port con­sump­tion or inter­act­ing with art­work that already exists. And then we have the library-generated out­puts. And those are still…in baby steps. The library-generated out­puts cur­rent­ly only sup­port sim­ple shapes, and it is real­ly hard to actu­al­ly get a mean­ing­ful descrip­tion of what’s hap­pen­ing on the can­vas to describe how shapes relate one to anoth­er, that’s some­thing that we def­i­nite­ly have to improve. And WebGL is a com­plete like, new world that we haven’t even start­ed work­ing towards tackling. 

But using these three out­puts that we used to have—and now we only have two out­puts, the table out­put and the text out­put, we’ve actu­al­ly seen novice pro­gram­mers who are blind cre­ate projects and play with col­or. And again, I’m talk­ing about very few peo­ple, because this still has not been test­ed wide­ly. But I can think about like Antonio Guimaraes, who was a stu­dent at NYU ITP cre­at­ing a very col­or­ful chess­board and being very excit­ed by being able to code some­thing that peo­ple could see and then read­ing or lis­ten­ing to the descrip­tion of what they were creating. 

So that’s a very excit­ing exam­ple that I saw hap­pen. And it’s some­thing that we def­i­nite­ly have to work on. How do we improve the tools for cre­at­ing? And I think that’s a big part of it is—and it’s a con­ver­sa­tion that a lot of folks have been hav­ing, is think­ing about tran­si­tion­ing from CodeMirror to some­thing that is bet­ter in the edi­tor. And that some­thing that is bet­ter, we don’t real­ly know what it is. [laughs] That’s a big prob­lem. Because it’s not only about how do you under­stand the out­put, but also how do you nav­i­gate code in a way that is easy and simple.