Golan Levin: Welcome back every­one. Our tenth and final Shall Make, Shall Be pre­sen­ta­tion is by arts​.codes, an artist col­lec­tive and open source dis­tri­b­u­tion plat­form co-directed by Margaret Schedel and Melissa F. Clarke that cel­e­brates art with com­pu­ta­tion­al underpinnings. 

Melissa F. Clarke is a Brooklyn-based inter­dis­ci­pli­nary artist whose work often employs gen­er­a­tive and inter­ac­tive custom-programmed envi­ron­ments. In her work she extrap­o­lates research into mul­ti­me­dia instal­la­tions, gen­er­a­tive envi­ron­ments, audio­vi­su­al sculp­tures, per­for­mances, and print­ed images. A grad­u­ate of NYU ITP, Clarke has taught media arts at SUNY Stony Brook, and has par­tic­i­pat­ed in res­i­den­cies at Pioneer Works, the Simons Center for Geometry and Physics, and Visible Future Labs at the School of Visual Arts. 

Margaret Schedel tran­scends the bound­aries of dis­parate fields to pro­duce inte­grat­ed work at the nexus of com­pu­ta­tion and the arts. With an inter­dis­ci­pli­nary career blend­ing clas­si­cal train­ing in cel­lo and com­po­si­tion, dig­i­tal audio research, and com­pu­ta­tion­al arts edu­ca­tion, she is inter­na­tion­al­ly rec­og­nized for the cre­ation and per­for­mance of fero­cious­ly inter­ac­tive media. Her research in the soni­fi­ca­tion of ges­ture and data takes form in inter­ac­tive opera, vir­tu­al real­i­ty, and video games. Schedel is pro­fes­sor and co-director of com­put­er music at SUNY Stony Brook.

Folks, I’m pleased to wel­come arts​.codes.

Melissa F. Clarke: Hello!

Margaret Schedel: Thank you very much.

Clarke: Okay! Hi, my name is Melissa. My pro­nouns are she and her. Welcome to our project, v.erses, a playable audio­vi­su­al instal­la­tion inspired by the Tenth Amendment. 

So the Tenth Amendment has been called both mys­te­ri­ous and remark­ably clear. In one sense it is but a tru­ism, yet over the last sev­er­al decades cas­es decid­ed by the judi­cial inter­pre­ta­tions of the Tenth Amendment are becom­ing more numer­ous and par­ti­san. Over the years the Supreme Court has found judicially-enforceable lim­its on the pow­er of the fed­er­al gov­ern­ment to reg­u­late the states. By 2011, the Court decid­ed that indi­vid­u­als can change fed­er­al laws on the basis of the Tenth Amendment. Currently we are wit­ness­ing states pass laws that seem to over­ride Supreme Court prece­dents and deci­sions, ie. abor­tion and LGBTQ rights. 

So, here is an early…or was, will be again—an ear­ly con­cep­tu­al sketch of v.erses in its entire­ty. Inspired by the sep­a­ra­tion of pow­ers in the Tenth Amendment, v.erses is a playable inter­ac­tive sound and visu­al art­work with two dis­tinc­tive inter­faces: one rep­re­sent­ing the fed­er­al gov­ern­ment, and one rep­re­sent­ing the states or the people. 

Schedel: So, hi every­one. I’m Meg; she or they. As Melissa said we were real­ly inspired by the sep­a­ra­tion of pow­ers in the Amendment to cre­ate a sys­tem where votes on the judi­cia­ry side would con­trol what kind of inter­ac­tions could take place on the state side. So essen­tial­ly the votes on the col­umn deter­mine the rules of the game.

This is the Judiciary col­umn, which has been com­plet­ed, that has nine inter­ac­tive sides where the judges begin the fate of the game­play by voting. 

And all the elec­tron­ics are housed in this judi­cia­ry col­umn, includ­ing the tablets, sen­sors, a com­put­er, an audio inter­face, six speak­ers, and the reac­tive lighting. 

On the col­umn we have nine kiosk tablets con­nect­ed to a cus­tom con­tent man­age­ment sys­tem run­ning the game. So, this is a high-definition iter­a­tion of the tablet inter­face. The tablets dis­play Amendment infor­ma­tion, then a spe­cif­ic case of a few pre-selected cas­es that users have the abil­i­ty to vote on. 

So when we were devel­op­ing the con­cept for v.erses, we were par­tic­u­lar­ly struck by the final clause of the Tenth Amendment, which reads are reserved to the States respec­tive­ly, or to the peo­ple. This unam­bigu­ous­ly reserved all non-delegated pow­er to its orig­i­nal source, the peo­ple. We then designed the State side as flow­ing flag-like rib­bons that would inter­act with the judicially-decided algo­rithms of the sound and the visuals.

Schedel: Because of the instal­la­tion con­straints at Federal Hall we’re going to be using prox­im­i­ty sen­sors with­in the col­umn instead of ribbons. 

Clarke: So, here’s a lit­tle screen cap­ture [4:43–] of our judi­cial game inter­face on the nine columns. All nine tablets are timed in sync with each oth­er so that the users for each game must decide and sub­mit their votes con­cur­rent­ly before the sec­ond part of the game begins. 

So they’re intro­duced to the Tenth Amendment and the facts of a case—we have sev­er­al cas­es in the queue. And then they’re prompt­ed with a ques­tion. All the while they know that this is a timed deci­sion. They scroll down, and they decide yes or no with the prob­lem, and they see the timer is run­ning, and this per­son says ​“no.”

And then they’re told the sound and visu­als of the game that com­mence have been decid­ed by their votes. 

Looking under the hood of our piece, we have the con­tent man­age­ment sys­tem. It’s basi­cal­ly a cloud-based sys­tem where it’s stor­ing the cas­es that we have pre­loaded. And it takes the vot­ing data of the judges. And once it’s com­piled it applies it to the algo­rithms of the sound and the visu­al out­put for the game interaction. 

Schedel: So essen­tial­ly the cas­es decide which set of sounds will or will not be affect­ed by the sen­sors. So if the par­tic­i­pants del­e­gate a pow­er to the fed­er­al gov­ern­ment, that track becomes sta­ble. So here, a syn­the­siz­er either repeats every 8^th note if it is a fed­er­al con­trol, or can get up to 196^th notes if under state con­trol. [6:19–6:32]

Some of the inter­ac­tions are more sub­tle. So for the drums it’s the amount of reverb that is con­trolled. [6:42–6:58]

So the idea is you need a bal­ance of fed­er­al and state con­trol in order to make a piece that is not too bor­ing or too active. 

For our vocals, we enlist­ed the help of Toni Blackman, who freestyled about the Bill of Rights and the Tenth Amendment. So if the vot­ers give the pow­er to the fed­er­al gov­ern­ment on the case that decides the vocal inter­ac­tion, there are no vocals. But if they give the pow­er to the states, the amount of the vocals—from just inter­jec­tions, to only vers­es, to only cho­rus­es, to the whole sound­scape at once—are con­trolled. [7:35–8:04]

Clarke: Nice. So again, we’re arts​.codes. We’re an artist col­lec­tive and open source dis­tri­b­u­tion plat­form that cel­e­brates with com­pu­ta­tion­al under­pin­nings of art. 

Schedel: And some of our ear­li­er projects includ­ed háček, which was a piece that we designed for ShmooCon, where we used actu­al ISP data from a con­test to get a tick­et to the con to cre­ate an inter­ac­tive expe­ri­ence. So essen­tial­ly users can switch which pearl that you ride in an attempt to get to the cen­ter of the land­scape and get a tick­et to the con­fer­ence. It was both a VR expe­ri­ence and then a mesh sculp­ture with the abstract­ed pro­jec­tions of the video and audio hap­pen­ing live in VR behind it. 

And here’s what’s hap­pen­ing inside the VR. So the height of the land­scape, the paths of the pearl, and all of the sounds are con­trolled by that ISP data which had a time ele­ment. So every­thing is basi­cal­ly cre­at­ed from this data. 

Clarke: So after háček we cre­at­ed Glass Menagerie, which pre­sent­ed a portable instal­la­tion and vir­tu­al real­i­ty expe­ri­ence in col­lab­o­ra­tion with Brookhaven National Laboratory cen­ter for Functional Nanomaterials. An ear­ly and con­cep­tu­al sketch here demon­strates the VR por­tion, which we want­ed to present the nano data to be reversed in scale to the per­cep­tion of the par­tic­i­pant. So basi­cal­ly big­ger than life. 

As part of the instal­la­tion expe­ri­ence we 3D-printed the data so that the folks with­in the expe­ri­ence would have a tan­gi­ble expe­ri­ence with the data. So that was basi­cal­ly show­cased along­side the VR. And here’s a demo of the VR expe­ri­ence, using the data for the 3D objects which are the visu­als and the sound:

Menagerie is also a plat­form for cre­at­ing frame­works so that sci­en­tists can cre­ate their own three-dimensional forms and vir­tu­al real­i­ty iter­a­tions using their data sets. So here we are shar­ing the prod­uct with the sci­en­tists and explor­ing that with them dur­ing their tenth anniversary. 

Schedel: So, thank you to our fel­low pre­sen­ters, our orga­niz­ers, and our audi­ence. We’re real­ly look­ing for­ward to play­ing all the games in Federal Hall in July. Thank you.

Clarke: Thank you.