Ethan Zuckerman: One of our ambi­tions for this con­fer­ence is a min­i­mum of one law­suit per pan­el, so we’re off to a very good start here. And per­haps try­ing to keep this in the same tone, Cory men­tioned leg­endary secu­ri­ty researcher Bunnie Huang. He’s going to be tak­ing the stage next. You may know Bunnie from his time at MIT, where he was both a hack­er in the MIT sense, and a hack­er in the much broad­er sense, includ­ing some real­ly ground­break­ing work on reverse engi­neer­ing on the Xbox; one of the world’s lead­ing hard­ware hack­ers. He’s also going to be talk­ing with anoth­er hack­er leg­end, Edward Snowden, who is going to be join­ing us via video to talk about a project that they are work­ing on togeth­er. So, Bunnie and Edward.

Edward Snowden: This is the first time that I’ve giv­en an actu­al aca­d­e­m­ic talk. This is excit­ing. One of the inter­est­ing things about Cory’s talk—it’s great to fol­low him up—is that he brought up an idea that is great for this form because it’s some­thing we don’t talk enough about, which is the idea that laws are actu­al­ly a weak guar­an­tee of out­come. We out­law mur­der, we out­law theft. They still hap­pen. We out­law many oth­er behav­iors, they still hap­pen. This is not to say they’re bad. This is not say we don’t want any rules. But there are bet­ter guar­an­tees. And we should con­sid­er when they are appro­pri­ate and when they in fact can pro­vide a greater enforce­ment for indi­vid­ual or human rights. Then the actu­al laws are poli­cies them­selves left naked and sort of [belong in the world?].

So let’s let’s get start­ed with the talk. A quick intro­duc­tion. My name is Edward Snowden. I’m Director of the Freedom of the Press Foundation. Some years ago I told the truth about a mat­ter of pub­lic impor­tance, and as a result a war­rant was issued for my arrest and I’m no longer able to trav­el freely. But today is a great exam­ple of why that does­n’t mean exact­ly what it once did.

And because of that, I’d like to thank very much MIT for orga­niz­ing this con­fer­ence, and the oppor­tu­ni­ty to speak with every­body here today. For jour­nal­ists in the audi­ence, that’s not a small thing. I should point out that they deserve cred­it for liv­ing up to that com­mit­ment to knowledge. 

Now, no orga­ni­za­tion is per­fect. Everyone makes mis­takes. But that is quite a risk. And this may be the first time an American exile has been able to present orig­i­nal research at an American uni­ver­si­ty. So it’s hard to imag­ine, I would say, a more apt plat­form for this talk than the Forbidden Research con­fer­ence.

But that’s enough pre­am­ble. The guid­ing theme of many of the talks today, I think, is that law is no sub­sti­tute for con­science. Our inves­ti­ga­tion regards coun­ter­ing what we’re call­ing law­ful abus­es of dig­i­tal sur­veil­lance. Lawful abuse, right, what is that? It does­n’t seem to make a lot of sense. It seems like it might be a con­tra­dic­tion in terms. When I announced the talk on Twitter, some­body imme­di­ate­ly was like, Lawful abuse, isn’t that a con­tra­dic­tion?” But if you think about it for just a moment it might seem to be a lit­tle bit more clear. After all, the legal­i­ty of a thing is quite dis­tinct from the moral­i­ty of it. 

And I claim no excep­tion­al exper­tise on any of this, but hav­ing worked at both the NSA and the CIA I do know a lit­tle bit about what I would con­sid­er to be law­ful abus­es. After all, mass sur­veil­lance was argued to be con­sti­tu­tion­al and yet the courts found very dif­fer­ent­ly despite the fact that it was hid­den and was occur­ring for more than a decade.

Lawful abuse is some­thing that I would define as an immoral or improp­er activ­i­ty per­pet­u­at­ed or jus­ti­fied under a shel­ter of law. Can you think of an exam­ple of that? I mean, it does­n’t take long to look back in his­to­ry and find them, I think. But what about things that are more recent? Mass sur­veil­lance, of course, is the exam­ple that’s near­est my own expe­ri­ence, but let’s set that aside. What about tor­ture? The Bush admin­is­tra­tion aggres­sive­ly argued that tor­ture could be legal­ized. What about indef­i­nite deten­tion? The intern­ment of indi­vid­u­als for years with­out access to tri­al or due process. Extrajudicial killing. the tar­get­ed assas­si­na­tion of known indi­vid­u­als far from any war zone, often by drone in today’s world.

Now, they may be crim­i­nals. They may be even peo­ple who are armed com­bat­ants. In many cas­es, but not all. And the fact that these things are chang­ing, often in secret, often with­out the pub­lic’s aware­ness or their knowl­edge or con­sent, should be dis­turb­ing. Given that there are sort of covert legal pro­tec­tions for these engage­ments. Now, such abus­es aren’t lim­it­ed strict­ly to nation­al secu­ri­ty. And that’s impor­tant, right, because we don’t want this to entire­ly be this big par­a­digm of pol­i­tics between sort of doves and hawks. Segregation, slav­ery, geno­cides. These have all been per­pet­u­at­ed under frame­works that said they were law­ful as long as you abid­ed by the reg­u­la­tions that are sort of man­ag­ing those activities.

Lawful abuse of sur­veil­lance could also be more dif­fi­cult to spot, not some­thing that’s as obvi­ous. How about a restric­tion on who and how you can love some­one that’s enforced by vio­lence? Or some­thing as sim­ple as an inten­tion­al tax loop­hole. Or dis­crim­i­na­tion. Lawful abuse.

So we’ve defined the term, right? But what is the actu­al prob­lem? Well, advances in the qual­i­ty of our tech­nol­o­gy, com­bined with a retreat in the qual­i­ty of our legal frame­works have cre­at­ed a par­a­digm in which our dai­ly activ­i­ties pro­duce an end­less wealth of records which can and are being used to do harm to indi­vid­u­als. Including those who have them­selves done no wrong.

If you have a phone in your pock­et that’s turned on, a long-lived record of your move­ments has been cre­at­ed. As a result of the way the cell­phone net­work func­tions, your devices are con­stant­ly shout­ing into the air by means of radio sig­nals a unique iden­ti­ty that sort of val­i­dates you to the phone com­pa­ny. And this unique iden­ti­ty is not only saved by that phone com­pa­ny, but it can also be observed as it trav­els over the air by inde­pen­dent, even more dan­ger­ous, third parties.

Now, due to the pro­lif­er­a­tion of sort of an ancient third-party doctrine-style inter­pre­ta­tion of law, even the most preda­to­ry and uneth­i­cal data col­lec­tion regimes are often entire­ly legal. And effec­tive­ly what this means is that if you have a device, you have a dossier. They may not be read­ing it, they may not be using it, but it’s out there.

Now, why should we care? Even if there are these com­pre­hen­sive records being cre­at­ed about your pri­vate activ­i­ties, right. Where you are. Who you went with. How long you were there. Did you meet with any­one? So on and so forth. Or were any pur­chas­es made. Any sort of elec­tron­ic activ­i­ty record, when things are at it.

I can think of a thou­sand and sev­en­ty rea­sons why it mat­ters. According to the fig­ures of the com­mit­tee to pro­tect jour­nal­ists, more than one thou­sand and sev­en­ty jour­nal­ists or media work­ers have been killed or gone miss­ing since January of 2005. And this is some­thing that might not be as intu­itive as you might expect. People go, Well, we had a lot of wars going on. Surely it’s com­bat relat­ed. These are com­bat deaths.” But when you look at these same fig­ures, mur­der is actu­al­ly a more com­mon cause of death than com­bat. And amongst this num­ber, pol­i­tics was a more com­mon news beat than war correspondence.

Now, why is this? It’s because one good jour­nal­ist, in the right place at the right time, can change his­to­ry. One good jour­nal­ist can move the nee­dle in the con­text of an elec­tion. One well-placed jour­nal­ist can influ­ence the out­come of a war. This makes them a tar­get. And increas­ing­ly, the tools of their trade are being used against them. Our tech­nol­o­gy is begin­ning to betray us not just as indi­vid­u­als, but as class­es of work­ers, par­tic­u­lar­ly those who are putting a lot on the line, at risk for the pub­lic interest. 

Speaking specif­i­cal­ly here about jour­nal­ists, who by virtue of their trade rely upon com­mu­ni­ca­tion in their dai­ly work. And unfor­tu­nate­ly, jour­nal­ists are begin­ning to be tar­get­ed on the basis of specif­i­cal­ly those com­mu­ni­ca­tions. A sin­gle mis­take can have a major impact. A sin­gle mis­take can result in a deten­tion, as was the case in the case of David Miranda, who was pass­ing through London Heathrow, actu­al­ly, in the report­ing that was relat­ed to me and my archive mate­r­i­al that was passed to jour­nal­ists. His jour­nal­is­tic mate­ri­als were seized by the British gov­ern­ment. And this was after they inter­cept­ed com­mu­ni­ca­tions regard­ing his plans to trav­el through their country.

But it can also result in far far worse than deten­tion. In the Syrian con­flict, the Assad regime began shelling the civil­ian city of Homs to extend that almost all for­eign jour­nal­ists were forced to flee. Now, the gov­ern­ment stopped accred­it­ing jour­nal­ists, and those who were accred­it­ed and were report­ing their loca­tions were being harassed. They were being beat­en. They were being dis­ap­peared. So only a hand­ful remained, includ­ing a few who actu­al­ly head­ed to this city, par­tic­u­lar­ly to the Baba Amr dis­trict to doc­u­ment the abus­es that were being vis­it upon the pop­u­la­tion there.

Now, typ­i­cal­ly in such cir­cum­stances a jour­nal­ist work­ing in these kind of dan­ger­ous con­di­tions would­n’t file their reports until after they had left the con­flict area, because they don’t want to invite any kind of reprisal. It is dan­ger­ous. But what hap­pens when you can’t wait? What hap­pens when there are things that a gov­ern­ment is sort of argu­ing aren’t hap­pen­ing and in fact are hap­pen­ing? The Syrian gov­ern­ment at the time said of course that they weren’t tar­get­ing civil­ians, civil­ians were being impact­ed, these were ene­my com­bat­ants. And it’s impor­tant to under­stand these law­ful abus­es of activ­i­ties hap­pen in many dif­fer­ent places. You might be going, Oh well, this isn’t law­ful. Surely this isn’t law­ful.” And of course by an inter­na­tion­al law con­text you’re absolute­ly right. By any sort of mean­ing­ful inter­pre­ta­tion of the Universal Declaration of Human Rights, this is a human rights vio­la­tion. It is a ware crime. 

But, domes­tic laws are hell of a thing. And you’ve got to remem­ber that while you might trust American courts, China has courts. Russia has courts. North Korea has courts. Syria has courts. They have lawyers, they have offices of gen­er­al coun­sel, who cre­ate poli­cies to over­see and reg­u­late these kind activ­i­ties and cre­ate frame­works to jus­ti­fy what­ev­er it is that the insti­tu­tions of pow­er actu­al­ly want to do.

Now, in this moment, in that Syrian city of Homs, the gov­ern­ment was lying in a way that actu­al­ly affect­ed inter­na­tion­al rela­tions. They were say­ing this was a jus­ti­fied offen­sive against ene­my forces, and yet there was a reporter there by the name of Marie Colvin who infil­trat­ed the city. She actu­al­ly crawled in I believe through a tun­nel, in the dark. Had to climb stone walls and things like that. They could­n’t speak because they were afraid about being fired upon. And she said this was not the case. She actu­al­ly filed this report live, despite the fact that they were wor­ried that there might be some kind of gov­ern­ment reprisal. She spoke four times to four dif­fer­ent news agen­cies on a sin­gle day, and they sound­ed some­thing like this. 

I’m at ground zero, and I’m see­ing what is being hit. Civilian build­ings are being hit. I’m on a street. The hous­es on this street have been hit, includ­ing the one I’m in. They blew off the top floor last week. There are only civil­ian hous­es here.

Secondly, the civil­ians can’t leave. You know, you may say, Well, if it’s so bad why are you stay­ing there?” The Syrians are not allow­ing the civil­ians to leave. Anyone who gets on the street, if they’re not hit by a shell, they’re sniped. There’s snipers all around Baba Amr on the high buildings.

I think the sick­en­ing thing is the com­plete mer­ci­less nature of this bomb­ing, whether or not—what is the tar­get they are hit­ting, civil­ian build­ings absolute­ly mer­ci­less­ly and with­out caring.
[clip of Marie Colvin report­ing, at ~49:4850:42]

Now, this might sound like just anoth­er war sto­ry. But the next day, the makeshift media cen­ter that she was oper­at­ing from, the one where the build­ing, the top floor had been hit the week before, was repeat­ed­ly and pre­cise­ly shelled by the Syrian army. She died as a result of this shelling, as did a French jour­nal­ist. The pho­tog­ra­ph­er that she was work­ing with was also wound­ed. And it was­n’t until some­time lat­er that we found, based on Lebanese sig­nals intel­li­gence col­lec­tion and some oth­er report­ing, that the Syrian army had actu­al­ly giv­en the order to specif­i­cal­ly tar­get jour­nal­ists who were break­ing sort of a no news black­out in this organization.

But how did they dis­cov­er her? How did they know where to aim their shells? Well, accord­ing to report­ing that occurred just this week, actu­al­ly, the week pri­or I believe, her fam­i­ly has filed a law­suit against the Syrian gov­ern­ment. And they have evi­dence alleg­ing that the radio fre­quen­cy emis­sions off her com­mu­ni­ca­tions that she used to file those news reports were inter­cept­ed by the Syrian army. They used direction-finding capa­bil­i­ties to track and locate this ille­gal, unlaw­ful media cen­ter, and then walk artillery fire toward it. 

Now, walk­ing artillery fire is sort of how you re-aim artillery when it falls short or when it goes far of where you’re actu­al­ly try­ing to hit. You have a spot­ter some­where in the city who goes, Oh, you did­n’t quite hit the media cen­ter. You hit the hos­pi­tal next door. Move it a lit­tle bit to the right, a lit­tle bit to the right.” And they heard these shells coming. 

By the time the sec­ond shell hit, they knew they were in trou­ble. This hap­pened at six o’clock in the morn­ing. She was going to grab her shoes, because as is cus­tom in the region you have to enter the house with bare feet, and she was caught by a shell and killed at that point.

Now, there’s a ques­tion here among many pol­i­cy offi­cials, where they go, Was this legal? What process­es do we use to sort of reme­di­ate these kind of threats when these things hap­pen? What hap­pens when the poli­cies fail?” And of course this is an argu­ment that the Syrian gov­ern­ment itself would say is mis­un­der­stood. These were actu­al­ly attacks that were occur­ring by ter­ror­ists, or what­ev­er. Or we if did these oper­a­tions they were lawful. 

But there’s a larg­er ques­tion of does it mat­ter? Does it mat­ter whether it was autho­rized by law, or not? Was this a moral action, regard­less of whether it was law­ful or unlaw­ful? And, are these kind of things pre­ventable? Can we enforce some stronger guar­an­tee of the kind of loca­tion­al indi­ca­tors of our activ­i­ties that we’re putting out there? Perhaps in the case of Marie Colvin, we could not.

But what about the case of future jour­nal­ists? What about a jour­nal­ist who has to meet with a source in a denied area, and they don’t want their phone to be shout­ing into the air, to be giv­ing up some kind of loca­tion­al indi­ca­tor of their move­ment? This is an area that is the focus of our research. Can we detect if the phone starts break­ing the rules, and for exam­ple if you turn off your your WiFi indi­cat­ing, you put your phone in air­plane mode, you try to turn off GPS. You get a lit­tle icon that lights up and says I’m off.” But is that actu­al­ly the case? Can you trust the device? What if the device has been hacked? What if some­thing else is going on? 

So we want­ed to inves­ti­gate, can we use these same devices that are so fre­quent­ly used against us as a kind of canary to detect these new tar­get­ed attempts for mon­i­tor­ing com­mu­ni­ca­tions, not just based on the ema­na­tions that go out on our own phones, but mal­ware attacks, inten­tion­al efforts to com­pro­mise the phone. For exam­ple, there was an Argentinian pros­e­cu­tor who, after he was mur­dered, when he was inves­ti­gat­ing whether the state had been engaged in seri­ous vio­la­tions of law, they recov­ered a mal­ware sam­ple from his phone. Now, that mal­ware sam­ple did not match the oper­at­ing sys­tem of his phone, so it was not respon­si­ble in that case. But it was clear that an attempt had been made to com­pro­mise his devices and use them against him. This same mal­ware was found tar­get­ing oth­er activists, oth­er jour­nal­ists, oth­er lawyers in the Latin American region.

If we can start to use devices, again as a kind of canary, to iden­ti­fy when these phones have been com­pro­mised, and we’re able to get these to a tar­get­ed class of indi­vid­u­als such as jour­nal­ists, such as human rights work­ers, they can detect that these phones are break­ing the rules, they’re act­ing in unex­pect­ed ways, what we can do is we can begin affect­ing the risk cal­cu­la­tion of the offensive actors in these cas­es. The NSA, for exam­ple, is very ner­vous about get­ting caught red-handed. They don’t want to risk the polit­i­cal impact of being seen tar­get­ing groups like jour­nal­ists, like American lawyers, despite the fact that they have been engaged in such oper­a­tions. In rare cas­es. It’s not their meat and pota­toes, but it does happen.

Other gov­ern­ments are not so care­ful. But, if we can cre­ate a track record of com­pro­mise, if we can cre­ate a track record of unlaw­ful or uneth­i­cal activ­i­ty, we can begin cre­at­ing a frame­work to over­turn the cul­ture of impuni­ty that affects so many of these lost jour­nal­ists’ lives. In those thou­sand and sev­en­ty cas­es of dead jour­nal­ists, or the dis­ap­peared, impuni­ty was the most com­mon outcome.

But I want to make it clear here that the idea is not just to pro­tect an indi­vid­ual jour­nal­ist’s phone, which is a wor­thy cause, but to again increase the cost of engag­ing in these kinds of activ­i­ties, engage in the cost of car­ry­ing out law­ful abus­es of dig­i­tal sur­veil­lance. And with­out sort of bela­bor­ing the point here, let’s go to the actu­al tech­ni­cal side of this and talk about what we’ve actu­al­ly done. Bunnie, let’s talk about ini­tial results.

Bunnie Huang: Sure. Thanks for set­ting all that up, Ed, and moti­vat­ing the back­ground for why we’re try­ing to do what we’re try­ing to do.

When we start­ed out the project, the basic chal­lenge that was out­lined is how do we take a venue where reporters meet with sources, and secure it against state-level adver­saries? There’s a lot of peo­ple who are smarter than me who are work­ing extreme­ly hard to turn your smart­phones into mini cyber fortresses.

The prob­lem is that phones are a very large, com­pli­cat­ed attack sur­face. You have email, you have web, you have mes­sag­ing, you have the abil­i­ty to install apps. Trying to secure this against a state-level adver­sary’s very chal­leng­ing, just like try­ing to cre­ate a city that’s robust against land, sea, and air attack.

But turn over the phone, and look on the back side. This is a sur­face that’s much sim­pler, and some­thing that I feel more com­fort­able with as a hard­ware guy. And there’s only two real­ly notable fea­tures on the back, the anten­nae. Those form sort of a choke point that we can look at to see if any­thing’s going in or out of the device. And so if you want to go ahead and make sure that your phone isn’t send­ing sig­nals, you say, Well, why don’t we just go ahead and put it in air­plane mode.” Turns out the ques­tion is can you trust the gate­keep­er? Can you trust the UI

If you go to the Apple web site and you read the lit­tle thing about air­plane mode, it actu­al­ly says that since iOS 8.2, air­plane mode does not turn off GPS. In fact, when you have your device in air­plane mode, the GPS is con­stant­ly on and can be pinged with­out any indi­ca­tion on the UI at all. And that’s a pol­i­cy that they have for the phones. You can also turn on WiFi and Bluetooth in air­plane mode acci­den­tal­ly, or inten­tion­al­ly. And that lit­tle icon is still there mak­ing you think that your device should­n’t be receiv­ing or trans­mit­ting radio signals.

So the ques­tion is is there a way we can inde­pen­dent­ly mon­i­tor that gate? Can we install, effec­tive­ly, a closed-circuit TV cam­era of our own design, of our own con­struc­tion, and in our own instal­la­tion, to audit and ver­i­fy that this is actu­al­ly happening?

So the tech­ni­cal goal here is to make sure the radios are real­ly off. We want to look at the cel­lu­lar modem, WiFi, Bluetooth, GPS, and NFC (Apple Pay). It’s a tech­nique that we call direct intro­spec­tion.” And we have a set of eight prin­ci­ples that we came up with for this project that we use to eval­u­ate dif­fer­ent approaches.

First is that we want to make sure that what­ev­er we come up with is com­plete­ly open source and inspectable. You don’t have to trust us or what we say. 

Second is we want to cre­ate a par­ti­tioned exe­cu­tion envi­ron­ment for the intro­spec­tion engine. You can think of the thing we’re doing as like a des­ig­nat­ed dri­ver for the phone. The phone may be giv­en a spiked drink and unable to assess its own secu­ri­ty sta­tus. So we have a com­plete­ly physically-separated exe­cu­tion envi­ron­ment for eval­u­at­ing the signals.

We also want to make sure that the prop­er oper­a­tion is field-verifiable. You need to guard against a hard­ware fail­ure. So if the cable falls out dur­ing intro­spec­tion, that’s real­ly bad. You want to be able to check that that’s still there. And you also want to guard against poten­tial so-called evil maid” attacks.

It’s also want­i­ng to make sure that it’s dif­fi­cult to trig­ger a false pos­i­tive. If the thing’s always warn­ing you that your phone is going off and it’s actu­al­ly not true, you’re going to start ignor­ing the warn­ings. And this cri­te­ri­on made us rule out a bunch of more pas­sive approach­es like sens­ing the anten­nas through the RF ema­na­tions, because if you hap­pened to walk by a very strong emit­ter like a WiFi access point or some­thing, your phone would trip and you would [start] ignor­ing the alarms.

We also want­ed to make sure it’s dif­fi­cult to induce false neg­a­tives. It’s quite pos­si­ble, for exam­ple, a sys­tem ven­dor can be com­pelled to push an update to your phone through a com­plete­ly secure mech­a­nism. And so even the sys­tem ven­dor can go ahead and put holes in the walls that you thought were once intact.

We also want­ed to avoid leav­ing a sig­na­ture that’s easy to pro­file. So, we don’t want to have some­thing where some­one says, Okay, let’s look for peo­ple who have intro­spec­tion engine on their phone and tar­get those guys because they have some­thing to hide.” So we have to cre­ate some­thing that’s essen­tial­ly very strong­ly cor­re­lat­ed at the hard­ware lev­el with the acti­va­tion of the radios. These are sig­nals which even a firmware update or some oth­er kind of remote mod­i­fi­ca­tion to the phone can’t bypass. So, they’re a very strong indi­ca­tor of radio activity.

We came up with a list of can­di­dates that are here. I’m not going to go through all of them in detail for lack of time, but if you’re inter­est­ed there’s a blog post live on PubPub now. And I go through the details of what the sig­nals are, why we chose them, and what we plan to do. But the basic idea is if you see these wires wig­gle and you think your phone is in air­plane mode, there’s a prob­lem. Something is turn­ing on the radios in that mode, and you know your your phone has been compromised.

So in terms of next steps, of course we actu­al­ly need to devel­op the hard­ware. We don’t expect jour­nal­ists to car­ry around oscil­lo­scopes and hack their phones and so on and so forth. And the basic approach—this is a pure­ly a con­cept ren­der­ing, but the idea is to try to cre­ate a bat­tery case-style add-on to the back of a phone which con­tains the intro­spec­tion engine. It has its own UI, because you can’t trust the UI in the front of the phone. You have input and out­put to that device. And there’s a cable that goes between the intro­spec­tion engine to the phone through the SIM card port on the iPhone 6.

So the solu­tion here is spe­cif­ic to the iPhone 6, but the tech­nique should be extend­able to oth­er makes and mod­els of phones. That’s basi­cal­ly it for our pre­sen­ta­tion. So thanks a bunch, Ed, for set­ting this up. And I look for­ward to work­ing on this.

Snowden: My plea­sure. It’s been amaz­ing. If I could just say one thing real quick for the room, as this was my first aca­d­e­m­ic col­lab­o­ra­tion. Having Bunnie as your pri­ma­ry col­lab­o­rate on the very first time is amaz­ing. He is one of the indi­vid­u­als whose com­pe­tence gives peo­ple impos­tor syn­drome. So I’ll do my best to live up to it. Thank you so much.