Tal Zarsky: What I’ll try and do now is address the ana­lyt­i­cal argu­ment pre­sent­ed in Frank’s paper, which to a great degree touch­es upon a vari­ety of prob­lems aris­ing from the use of pre­dic­tive algo­rithms when what they’re try­ing to pre­dict is human-related deci­sions and actions. 

Now, anoth­er sub­set of Frank’s paper is not only that this dynam­ic has prob­lems, but some of these prob­lems arise from two traits. One is that this process is auto­mat­ed. And the oth­er that this process is opaque. And when you look at these two ele­ments, then the next argu­ment [that] fol­lows is if we gen­er­ate a greater extent of trans­paren­cy and per­haps lim­it the lev­el of automa­tion, some of these prob­lems will be mit­i­gat­ed, a ques­tion I would like to touch upon. 

Now my main con­tri­bu­tion to the dis­cus­sion would be try­ing to look at Frank’s over­all dis­cus­sion that he had—here are sev­er­al ele­ments in the slide—and try and break down four main top­ics which are forms of four dif­fer­ent attacks or prob­lems with these process­es, and run this ana­lyt­i­cal process of look­ing to what extent are these prob­lems exac­er­bat­ed by automa­tion and lack of trans­paren­cy. And if we take away and have less automa­tion and more trans­paren­cy how are these prob­lems pos­si­bly mitigated? 

Now also what I intend to focus on are less on the obvi­ous insights and more on some­what more provoca­tive points. And the rest of these issues I touch upon on in var­i­ous arti­cles that I wrote and you could eas­i­ly find them online. 

Now, before I go fur­ther, I already made one mis­take. Because I not­ed automa­tion and trans­paren­cy as two dif­fer­ent issues, and to a great extent they’re inter­wo­ven and con­nect­ed at var­i­ous con­texts. And I’ll talk about one con­text now. 

When you make a deci­sion to opt for an auto­mat­ed process, to some extent you’re already by doing so com­pro­mis­ing trans­paren­cy. Or you could say it the oth­er way around. It’s pos­si­ble to argue that if you opt for extreme­ly strict trans­paren­cy reg­u­la­tion, you’re mak­ing a com­pro­mise in terms of automa­tion. And let’s try to demon­strate this with a point from Frank’s paper. He talks about cred­it rat­ings, the FCRA, and in a recent amend­ment he notes that you must inform the indi­vid­ual of the four dom­i­nant fac­tors which led to the rat­ing deci­sion about him. 

Now, if the process is ful­ly auto­mat­ed, which means that a data-driven process with no ana­lyst involve­ment takes all the data, crunch­es it, comes up with var­i­ous pat­terns and cat­e­go­riza­tions, it might be impos­si­ble to indi­cate these four lead­ing fac­tors. There might be thou­sands of such fac­tors, each one inter­act­ing dif­fer­ent­ly with the oth­er. So, once you make this law that you must have the abil­i­ty to indi­cate these four dom­i­nant factors—and Frank cor­rect­ly notes that also this rule is watered down. But even when you make this law, there be a com­pro­mise here in the lev­el of automa­tion, and some would say in the lev­el of accu­ra­cy the process requires. 

Now this is a fair point. It’s fair for the reg­u­la­tor to come in and say, ​“We find trans­paren­cy more impor­tant than hav­ing full automa­tion.” But it’s a point that we have to remem­ber. There are oth­er inter­ac­tions between these terms of automa­tion and trans­paren­cy but we’ll set that aside for the moment. 

Now, I want to go back to the open­ing point here. And we’re hav­ing your dis­cus­sions about using algo­rithms, and this is close­ly relat­ed to a term I don’t think I’ve heard this morning—maybe I missed it—the term of ​“big data” that many peo­ple are talk­ing about that to over­come this big data issue you need to run these algo­rithms, and you have big data in var­i­ous con­texts. But we’re talk­ing here, espe­cial­ly Frank in the first part of his sec­tion, we’re talk­ing about infor­ma­tion which relates to human behav­ior. And when you make pre­dic­tions we’re not mak­ing pre­dic­tions about physics, Frank points out, we’re mak­ing pre­dic­tions about how peo­ple will behave. 

To make such pre­dic­tions, you need to have cer­tain assump­tions about the con­sis­ten­cy of human con­duct. That peo­ple will car­ry through their behave. There if such a thing as a human trait. And it’s a very uncom­fort­able feel­ing because this builds into our under­stand­ing of free will, right. If you could pre­dict how some­one will behave, should we actu­al­ly judge him neg­a­tive­ly if he does some­thing wrong? So lawyers and philoso­phies from var­i­ous schools might feel very intim­i­dat­ed by this notion, and I think that this leads to the clash that many peo­ple have with this notion of pre­dict­ing and gov­ern­ing algo­rithms. And it’s very…in my opin­ion it’s very close to this notion of pri­va­cy, also. From a dif­fer­ent per­spec­tive, that when it comes to pri­va­cy, this notion that not every­one could always see what we’re doing and track us at all times, is some­thing that’s part of lib­er­al soci­ety. And when it’s com­ing under attack we have a very vis­cer­al feel­ing that some­thing’s wrong. We can’t put a fin­ger on it. And it might come from this inter­nal clash. What we’ll try to do now is under­stand where it’s com­ing from. 

So, we could have four gen­er­al per­spec­tives on prob­lems of pre­dic­tions. I’m not going to go through the list now, let’s start out with the first. Now the first argu­ment, which is a run­ning theme in Franks paper paper and also pre­sent­ed now is that when you use these pre­dic­tive algo­rithms, you’re decreas­ing over­all wel­fare. What does that mean, it means that we have a process and this process does­n’t real­ly work. It’s rid­den with errors from var­i­ous forms—errors in infor­ma­tion, errors in the process. And at the end of the day, no one is going to gain out of this sys­tem at all. Okay. 

So this is one argu­ment that you see in the lit­er­a­ture. But when you think of this argu­ment you need to think of two oth­er ele­ments. One is you have to think of the alter­na­tives, and you have to remem­ber the pre­vi­ous prac­tices that peo­ple writ­ing spe­cif­ic reports about indi­vid­u­als. Everything was done in hand. They were look­ing at their per­son­al traits. Do we do we real­ly want to move into that alter­na­tive world? And in addi­tion, we have to talk about what this process facil­i­tates. It facil­i­tates over­all a low lev­el of cred­it in the United States which uses this sys­tem. So this has ben­e­fits to all of soci­eties, and it’s espe­cial­ly a ben­e­fit to those with lim­it­ed access to cap­i­tal. So this is a com­pro­mise we make with­in society.

Now, Frank explains that trans­paren­cy plays a pow­er­ful role, or lack thereof—a lev­el of opac­i­ty, it leads to the prob­lems of this process. Now, that could be argued from var­i­ous direc­tions. One argu­ment is we don’t real­ly know what’s hap­pen­ing and there­fore we can’t make cor­rec­tions. Or there’s a lack of an incen­tive, a lack of an incen­tive to these firms to man­age their prac­tice’s cor­rect­ly because we don’t know what they’re doing. And there are argu­ments to be made as to how much mer­it this argu­ment has because even if we know what is hap­pen­ing it’s not sure that we have reporters going after that. And gov­ern­ment will be involved because this is a very com­plex and tech­ni­cal topic. 

But even set­ting that aside, and our next pan­elist will talk about this to a greater extent. Once you have trans­paren­cy, so…there is this fear of gam­ing, right? And what does gam­ing actu­al­ly mean? It means that if peo­ple under­stand how these pre­dic­tive process­es work, peo­ple would under­stand what the proxy is, and they’ll work around the proxy but still at the end of the day they reach the prob­lem­at­ic out­come. That means that if there are var­i­ous indi­ca­tors for prob­lem­at­ic behav­ior, they will engage in dif­fer­ent behav­ior, but still the out­come will be the same. And we’ll hear lat­er about how a prob­lem­at­ic that is. And the prob­lem might be that this under­mines the entire sys­tem of low­er­ing the lev­el of credit. 

Now that is true. But I want to point out an even greater prob­lem. Which is once you have trans­paren­cy with regard to these prox­ies, peo­ple are going to work around and try and game the sys­tem. And the sys­tem might crash and we might not care because we might have a high­er lev­el of cred­it. But this might lead to mas­sive neg­a­tive exter­nal­i­ties because peo­ple will engage in actions that are deemed prob­lem­at­ic to cred­it. So one exam­ple from the paper is that cer­tain dis­count stores are an indi­ca­tion of a low­er lev­el of cred­it. That means that peo­ple will stop shop­ping in these low­er dis­count stores. Which might lead over­all to prob­lems in the econ­o­my and decrease over­all wel­fare because we do want peo­ple to pur­chase at dis­count stores. And this is only one exam­ple. We don’t want them con­stant­ly act­ing think­ing about how they’ll work through their cred­it score because that will have over­all neg­a­tive impact. So this is one point. We have more on that but I’ll skip ahead.

Now the next point…I think this will be my final point here, is that we talk about…in the paper, that regard­less of the fact that this enhances or decreas­es over­all wel­fare, there are var­i­ous unfair out­comes. Because you have trans­fers of wealth among groups. And we’re point­ing to two dif­fer­ent groups here. There’s a trans­fer of wealth between con­sumers to firms. And there’s a trans­fer of wealth between more pow­er­ful groups to the dis­en­fran­chised, or the oth­er way around. And how will a high­er lev­el of trans­paren­cy affect these issues? 

So, this is some­thing that’s worth think­ing about, and let me give you two quick intu­itions. One intu­ition is that because the firms have so much infor­ma­tion about us, they could entice us into these unfair deals. And Oren Bar-Gill writes about this exten­sive­ly and he also talks about the effect of dis­clo­sure on this process, so this is anoth­er inter­est­ing realm of lit­er­a­ture in the law and econ world that actu­al­ly looks at the same prob­lem and offer the same solu­tion. And it’s com­pelling to fig­ure out if real­ly trans­paren­cy will low­er this risk of the trans­fer of wealth from weak­er con­sumers to the larg­er firms because we have so much infor­ma­tion and could struc­ture our pre­dic­tions based upon that. 

But the oth­er point is what will be the effect of hav­ing more infor­ma­tion about the process about the fact that we’ll have trans­fer of wealth from var­i­ous groups to anoth­er. Sot the basic intu­ition is that what’s hap­pen­ing now is that we’re able to put our fin­ger about peo­ple that there’s a high­er chance that they’re not going to pay. And we’re low­er­ing the risk from them and trans­fer­ring the wealth to these strong groups. And if I’m sophisticated—I have access to knowl­edge, I have access to edu­ca­tion, I could make sure I’m not going to be in that weak­er group. And if there’s transparency…so gov­ern­ment and firms will say, ​“No no, we can’t do that. You can’t allow them to do that,” so that’s how trans­paren­cy will solve this. 

But a pos­si­ble the­o­ry would be if we have trans­paren­cy this prob­lem will only get worse. Why would that be? Now let’s think about this world. We have full infor­ma­tion, and we see the process and we see that var­i­ous actions, var­i­ous groups, are indi­cat­ed as high­er lev­els of risk. So think of Walmart, for instance. Data min­ing is indi­cat­ing Walmart, peo­ple shop­ping at Walmart, high­er lev­el of risk. What is Walmart going to do? It’s going to put pres­sure through Washington or direct­ly on these groups. Think about it that there’s no indi­ca­tion about lawyers or doc­tors and their lev­el of cred­it. What will groups of lawyers and doc­tors do? Put pres­sure through Washington or direct­ly, ​“Indicate us in the list as low­er lev­el or risk.” And there­fore, the fact that your have trans­paren­cy as opposed to a opac­i­ty will bring in all these dis­cus­sions of polit­i­cal econ­o­my and pri­vate inter­est into this realm of auto­mat­ed pre­dic­tion, and make them even more biased and there­fore trans­paren­cy will be a prob­lem and not a solu­tion. So I encour­age you to look at this paper, which I hope to open up lat­er. And oth­er papers I wrote about this issue. And for more dis­cus­sions on this point

Do I have a thir­ty sec­onds for a last point? Okay.

So I focused most­ly on the cred­it point. I want to talk briefly about a very inter­est­ing seg­ment in the paper which Frank refers to as ​“the need for speed,” right. That you have great incen­tives cur­rent­ly in Wall Street that the banks are using tech­nol­o­gy and telecom­mu­ni­ca­tion infra­struc­ture to allow them to engage in faster trans­ac­tions, even a split sec­ond before the oth­er, and in that way they get the deal first and there­fore they have advan­tages. And this is some­thing that secu­ri­ty reg­u­la­tors all around the world are think­ing about. And recent­ly I with a col­league wrote a paper called Queues in Law and I found this is a very inter­est­ing exam­ple. That’s the entire premise, that why you have this need for speed is this notion of first in time, first in right. And we real­ly have to think of this very basic notion in soci­ety and why are we say­ing that in this spe­cif­ic instance, first in time should be first in right? Perhaps this is a point that there’s no rea­son to accept this notion, which is very basic to our under­stand­ing of fair­ness and effi­cien­cy, and set that aside. 

And I just want to say that at this point here actu­al­ly, this fact that you have always first in time, first in right, it’s allow­ing wealth trans­fer to sophis­ti­cat­ed play­ers. So there’s a prob­lem with using this mod­el. But on the oth­er hand this is some­thing we need to think about. There are again mas­sive pos­i­tive exter­nal­i­ties. Because you have firms like Wall Street putting in a lot of mon­ey into ICP, telecom­mu­ni­ca­tion infra­struc­ture, com­put­er sys­tems that are lat­er used in oth­er realms which assist soci­ety. So in fact, all of us invest­ing in stock mar­kets and allow­ing some of our fund­ing to pass on to Goldman Sachs, that put the mon­ey into inven­tions in telecom­mu­ni­ca­tions, which come back to us in an infra­struc­ture advan­tage, maybe that’s a deal that we real­ly need in soci­ety when gov­ern­ment does­n’t have incen­tives to invest in these forms of tech­nolo­gies. Thank you very much, and ques­tions I’ll be hap­py to receive at this address.