Ethan Zuckerman: One of our ambi­tions for this con­fer­ence is a min­i­mum of one law­suit per pan­el, so we’re off to a very good start here. And per­haps try­ing to keep this in the same tone, Cory men­tioned leg­endary secu­ri­ty researcher Bunnie Huang. He’s going to be tak­ing the stage next. You may know Bunnie from his time at MIT, where he was both a hack­er in the MIT sense, and a hack­er in the much broad­er sense, includ­ing some real­ly ground­break­ing work on reverse engi­neer­ing on the Xbox; one of the world’s lead­ing hard­ware hack­ers. He’s also going to be talk­ing with anoth­er hack­er leg­end, Edward Snowden, who is going to be join­ing us via video to talk about a project that they are work­ing on togeth­er. So, Bunnie and Edward.

Edward Snowden: This is the first time that I’ve giv­en an actu­al aca­d­e­m­ic talk. This is excit­ing. One of the inter­est­ing things about Cory’s talk—it’s great to fol­low him up—is that he brought up an idea that is great for this form because it’s some­thing we don’t talk enough about, which is the idea that laws are actu­al­ly a weak guar­an­tee of out­come. We out­law mur­der, we out­law theft. They still hap­pen. We out­law many oth­er behav­iors, they still hap­pen. This is not to say they’re bad. This is not say we don’t want any rules. But there are bet­ter guar­an­tees. And we should con­sid­er when they are appro­pri­ate and when they in fact can pro­vide a greater enforce­ment for indi­vid­ual or human rights. Then the actu­al laws are poli­cies them­selves left naked and sort of [belong in the world?].

So let’s let’s get start­ed with the talk. A quick intro­duc­tion. My name is Edward Snowden. I’m Director of the Freedom of the Press Foundation. Some years ago I told the truth about a mat­ter of pub­lic impor­tance, and as a result a war­rant was issued for my arrest and I’m no longer able to trav­el freely. But today is a great exam­ple of why that doesn’t mean exact­ly what it once did.

And because of that, I’d like to thank very much MIT for orga­niz­ing this con­fer­ence, and the oppor­tu­ni­ty to speak with every­body here today. For jour­nal­ists in the audi­ence, that’s not a small thing. I should point out that they deserve cred­it for liv­ing up to that com­mit­ment to knowledge. 

Now, no orga­ni­za­tion is per­fect. Everyone makes mis­takes. But that is quite a risk. And this may be the first time an American exile has been able to present orig­i­nal research at an American uni­ver­si­ty. So it’s hard to imag­ine, I would say, a more apt plat­form for this talk than the Forbidden Research con­fer­ence.

But that’s enough pre­am­ble. The guid­ing theme of many of the talks today, I think, is that law is no sub­sti­tute for con­science. Our inves­ti­ga­tion regards coun­ter­ing what we’re call­ing law­ful abus­es of dig­i­tal sur­veil­lance. Lawful abuse, right, what is that? It doesn’t seem to make a lot of sense. It seems like it might be a con­tra­dic­tion in terms. When I announced the talk on Twitter, some­body imme­di­ate­ly was like, Lawful abuse, isn’t that a con­tra­dic­tion?” But if you think about it for just a moment it might seem to be a lit­tle bit more clear. After all, the legal­i­ty of a thing is quite dis­tinct from the moral­i­ty of it. 

And I claim no excep­tion­al exper­tise on any of this, but hav­ing worked at both the NSA and the CIA I do know a lit­tle bit about what I would con­sid­er to be law­ful abus­es. After all, mass sur­veil­lance was argued to be con­sti­tu­tion­al and yet the courts found very dif­fer­ent­ly despite the fact that it was hid­den and was occur­ring for more than a decade.

Lawful abuse is some­thing that I would define as an immoral or improp­er activ­i­ty per­pet­u­at­ed or jus­ti­fied under a shel­ter of law. Can you think of an exam­ple of that? I mean, it doesn’t take long to look back in his­to­ry and find them, I think. But what about things that are more recent? Mass sur­veil­lance, of course, is the exam­ple that’s near­est my own expe­ri­ence, but let’s set that aside. What about tor­ture? The Bush admin­is­tra­tion aggres­sive­ly argued that tor­ture could be legal­ized. What about indef­i­nite deten­tion? The intern­ment of indi­vid­u­als for years with­out access to tri­al or due process. Extrajudicial killing. the tar­get­ed assas­si­na­tion of known indi­vid­u­als far from any war zone, often by drone in today’s world.

Now, they may be crim­i­nals. They may be even peo­ple who are armed com­bat­ants. In many cas­es, but not all. And the fact that these things are chang­ing, often in secret, often with­out the public’s aware­ness or their knowl­edge or con­sent, should be dis­turb­ing. Given that there are sort of covert legal pro­tec­tions for these engage­ments. Now, such abus­es aren’t lim­it­ed strict­ly to nation­al secu­ri­ty. And that’s impor­tant, right, because we don’t want this to entire­ly be this big par­a­digm of pol­i­tics between sort of doves and hawks. Segregation, slav­ery, geno­cides. These have all been per­pet­u­at­ed under frame­works that said they were law­ful as long as you abid­ed by the reg­u­la­tions that are sort of man­ag­ing those activities.

Lawful abuse of sur­veil­lance could also be more dif­fi­cult to spot, not some­thing that’s as obvi­ous. How about a restric­tion on who and how you can love some­one that’s enforced by vio­lence? Or some­thing as sim­ple as an inten­tion­al tax loop­hole. Or dis­crim­i­na­tion. Lawful abuse.

So we’ve defined the term, right? But what is the actu­al prob­lem? Well, advances in the qual­i­ty of our tech­nol­o­gy, com­bined with a retreat in the qual­i­ty of our legal frame­works have cre­at­ed a par­a­digm in which our dai­ly activ­i­ties pro­duce an end­less wealth of records which can and are being used to do harm to indi­vid­u­als. Including those who have them­selves done no wrong.

If you have a phone in your pock­et that’s turned on, a long-lived record of your move­ments has been cre­at­ed. As a result of the way the cell­phone net­work func­tions, your devices are con­stant­ly shout­ing into the air by means of radio sig­nals a unique iden­ti­ty that sort of val­i­dates you to the phone com­pa­ny. And this unique iden­ti­ty is not only saved by that phone com­pa­ny, but it can also be observed as it trav­els over the air by inde­pen­dent, even more dan­ger­ous, third parties.

Now, due to the pro­lif­er­a­tion of sort of an ancient third-party doctrine-style inter­pre­ta­tion of law, even the most preda­to­ry and uneth­i­cal data col­lec­tion regimes are often entire­ly legal. And effec­tive­ly what this means is that if you have a device, you have a dossier. They may not be read­ing it, they may not be using it, but it’s out there.

Now, why should we care? Even if there are these com­pre­hen­sive records being cre­at­ed about your pri­vate activ­i­ties, right. Where you are. Who you went with. How long you were there. Did you meet with any­one? So on and so forth. Or were any pur­chas­es made. Any sort of elec­tron­ic activ­i­ty record, when things are at it.

I can think of a thou­sand and sev­en­ty rea­sons why it mat­ters. According to the fig­ures of the com­mit­tee to pro­tect jour­nal­ists, more than one thou­sand and sev­en­ty jour­nal­ists or media work­ers have been killed or gone miss­ing since January of 2005. And this is some­thing that might not be as intu­itive as you might expect. People go, Well, we had a lot of wars going on. Surely it’s com­bat relat­ed. These are com­bat deaths.” But when you look at these same fig­ures, mur­der is actu­al­ly a more com­mon cause of death than com­bat. And amongst this num­ber, pol­i­tics was a more com­mon news beat than war correspondence.

Now, why is this? It’s because one good jour­nal­ist, in the right place at the right time, can change his­to­ry. One good jour­nal­ist can move the nee­dle in the con­text of an elec­tion. One well-placed jour­nal­ist can influ­ence the out­come of a war. This makes them a tar­get. And increas­ing­ly, the tools of their trade are being used against them. Our tech­nol­o­gy is begin­ning to betray us not just as indi­vid­u­als, but as class­es of work­ers, par­tic­u­lar­ly those who are putting a lot on the line, at risk for the pub­lic interest. 

Speaking specif­i­cal­ly here about jour­nal­ists, who by virtue of their trade rely upon com­mu­ni­ca­tion in their dai­ly work. And unfor­tu­nate­ly, jour­nal­ists are begin­ning to be tar­get­ed on the basis of specif­i­cal­ly those com­mu­ni­ca­tions. A sin­gle mis­take can have a major impact. A sin­gle mis­take can result in a deten­tion, as was the case in the case of David Miranda, who was pass­ing through London Heathrow, actu­al­ly, in the report­ing that was relat­ed to me and my archive mate­r­i­al that was passed to jour­nal­ists. His jour­nal­is­tic mate­ri­als were seized by the British gov­ern­ment. And this was after they inter­cept­ed com­mu­ni­ca­tions regard­ing his plans to trav­el through their country.

But it can also result in far far worse than deten­tion. In the Syrian con­flict, the Assad regime began shelling the civil­ian city of Homs to extend that almost all for­eign jour­nal­ists were forced to flee. Now, the gov­ern­ment stopped accred­it­ing jour­nal­ists, and those who were accred­it­ed and were report­ing their loca­tions were being harassed. They were being beat­en. They were being dis­ap­peared. So only a hand­ful remained, includ­ing a few who actu­al­ly head­ed to this city, par­tic­u­lar­ly to the Baba Amr dis­trict to doc­u­ment the abus­es that were being vis­it upon the pop­u­la­tion there.

Now, typ­i­cal­ly in such cir­cum­stances a jour­nal­ist work­ing in these kind of dan­ger­ous con­di­tions wouldn’t file their reports until after they had left the con­flict area, because they don’t want to invite any kind of reprisal. It is dan­ger­ous. But what hap­pens when you can’t wait? What hap­pens when there are things that a gov­ern­ment is sort of argu­ing aren’t hap­pen­ing and in fact are hap­pen­ing? The Syrian gov­ern­ment at the time said of course that they weren’t tar­get­ing civil­ians, civil­ians were being impact­ed, these were ene­my com­bat­ants. And it’s impor­tant to under­stand these law­ful abus­es of activ­i­ties hap­pen in many dif­fer­ent places. You might be going, Oh well, this isn’t law­ful. Surely this isn’t law­ful.” And of course by an inter­na­tion­al law con­text you’re absolute­ly right. By any sort of mean­ing­ful inter­pre­ta­tion of the Universal Declaration of Human Rights, this is a human rights vio­la­tion. It is a ware crime. 

But, domes­tic laws are hell of a thing. And you’ve got to remem­ber that while you might trust American courts, China has courts. Russia has courts. North Korea has courts. Syria has courts. They have lawyers, they have offices of gen­er­al coun­sel, who cre­ate poli­cies to over­see and reg­u­late these kind activ­i­ties and cre­ate frame­works to jus­ti­fy what­ev­er it is that the insti­tu­tions of pow­er actu­al­ly want to do.

Now, in this moment, in that Syrian city of Homs, the gov­ern­ment was lying in a way that actu­al­ly affect­ed inter­na­tion­al rela­tions. They were say­ing this was a jus­ti­fied offen­sive against ene­my forces, and yet there was a reporter there by the name of Marie Colvin who infil­trat­ed the city. She actu­al­ly crawled in I believe through a tun­nel, in the dark. Had to climb stone walls and things like that. They couldn’t speak because they were afraid about being fired upon. And she said this was not the case. She actu­al­ly filed this report live, despite the fact that they were wor­ried that there might be some kind of gov­ern­ment reprisal. She spoke four times to four dif­fer­ent news agen­cies on a sin­gle day, and they sound­ed some­thing like this. 

I’m at ground zero, and I’m see­ing what is being hit. Civilian build­ings are being hit. I’m on a street. The hous­es on this street have been hit, includ­ing the one I’m in. They blew off the top floor last week. There are only civil­ian hous­es here.

Secondly, the civil­ians can’t leave. You know, you may say, Well, if it’s so bad why are you stay­ing there?” The Syrians are not allow­ing the civil­ians to leave. Anyone who gets on the street, if they’re not hit by a shell, they’re sniped. There’s snipers all around Baba Amr on the high buildings.

I think the sick­en­ing thing is the com­plete mer­ci­less nature of this bomb­ing, whether or not—what is the tar­get they are hit­ting, civil­ian build­ings absolute­ly mer­ci­less­ly and with­out caring.
[clip of Marie Colvin report­ing, at ~49:4850:42]

Now, this might sound like just anoth­er war sto­ry. But the next day, the makeshift media cen­ter that she was oper­at­ing from, the one where the build­ing, the top floor had been hit the week before, was repeat­ed­ly and pre­cise­ly shelled by the Syrian army. She died as a result of this shelling, as did a French jour­nal­ist. The pho­tog­ra­ph­er that she was work­ing with was also wound­ed. And it wasn’t until some­time lat­er that we found, based on Lebanese sig­nals intel­li­gence col­lec­tion and some oth­er report­ing, that the Syrian army had actu­al­ly giv­en the order to specif­i­cal­ly tar­get jour­nal­ists who were break­ing sort of a no news black­out in this organization.

But how did they dis­cov­er her? How did they know where to aim their shells? Well, accord­ing to report­ing that occurred just this week, actu­al­ly, the week pri­or I believe, her fam­i­ly has filed a law­suit against the Syrian gov­ern­ment. And they have evi­dence alleg­ing that the radio fre­quen­cy emis­sions off her com­mu­ni­ca­tions that she used to file those news reports were inter­cept­ed by the Syrian army. They used direction-finding capa­bil­i­ties to track and locate this ille­gal, unlaw­ful media cen­ter, and then walk artillery fire toward it. 

Now, walk­ing artillery fire is sort of how you re-aim artillery when it falls short or when it goes far of where you’re actu­al­ly try­ing to hit. You have a spot­ter some­where in the city who goes, Oh, you didn’t quite hit the media cen­ter. You hit the hos­pi­tal next door. Move it a lit­tle bit to the right, a lit­tle bit to the right.” And they heard these shells coming. 

By the time the sec­ond shell hit, they knew they were in trou­ble. This hap­pened at six o’clock in the morn­ing. She was going to grab her shoes, because as is cus­tom in the region you have to enter the house with bare feet, and she was caught by a shell and killed at that point.

Now, there’s a ques­tion here among many pol­i­cy offi­cials, where they go, Was this legal? What process­es do we use to sort of reme­di­ate these kind of threats when these things hap­pen? What hap­pens when the poli­cies fail?” And of course this is an argu­ment that the Syrian gov­ern­ment itself would say is mis­un­der­stood. These were actu­al­ly attacks that were occur­ring by ter­ror­ists, or what­ev­er. Or we if did these oper­a­tions they were lawful. 

But there’s a larg­er ques­tion of does it mat­ter? Does it mat­ter whether it was autho­rized by law, or not? Was this a moral action, regard­less of whether it was law­ful or unlaw­ful? And, are these kind of things pre­ventable? Can we enforce some stronger guar­an­tee of the kind of loca­tion­al indi­ca­tors of our activ­i­ties that we’re putting out there? Perhaps in the case of Marie Colvin, we could not.

But what about the case of future jour­nal­ists? What about a jour­nal­ist who has to meet with a source in a denied area, and they don’t want their phone to be shout­ing into the air, to be giv­ing up some kind of loca­tion­al indi­ca­tor of their move­ment? This is an area that is the focus of our research. Can we detect if the phone starts break­ing the rules, and for exam­ple if you turn off your your WiFi indi­cat­ing, you put your phone in air­plane mode, you try to turn off GPS. You get a lit­tle icon that lights up and says I’m off.” But is that actu­al­ly the case? Can you trust the device? What if the device has been hacked? What if some­thing else is going on? 

So we want­ed to inves­ti­gate, can we use these same devices that are so fre­quent­ly used against us as a kind of canary to detect these new tar­get­ed attempts for mon­i­tor­ing com­mu­ni­ca­tions, not just based on the ema­na­tions that go out on our own phones, but mal­ware attacks, inten­tion­al efforts to com­pro­mise the phone. For exam­ple, there was an Argentinian pros­e­cu­tor who, after he was mur­dered, when he was inves­ti­gat­ing whether the state had been engaged in seri­ous vio­la­tions of law, they recov­ered a mal­ware sam­ple from his phone. Now, that mal­ware sam­ple did not match the oper­at­ing sys­tem of his phone, so it was not respon­si­ble in that case. But it was clear that an attempt had been made to com­pro­mise his devices and use them against him. This same mal­ware was found tar­get­ing oth­er activists, oth­er jour­nal­ists, oth­er lawyers in the Latin American region.

If we can start to use devices, again as a kind of canary, to iden­ti­fy when these phones have been com­pro­mised, and we’re able to get these to a tar­get­ed class of indi­vid­u­als such as jour­nal­ists, such as human rights work­ers, they can detect that these phones are break­ing the rules, they’re act­ing in unex­pect­ed ways, what we can do is we can begin affect­ing the risk cal­cu­la­tion of the offensive actors in these cas­es. The NSA, for exam­ple, is very ner­vous about get­ting caught red-handed. They don’t want to risk the polit­i­cal impact of being seen tar­get­ing groups like jour­nal­ists, like American lawyers, despite the fact that they have been engaged in such oper­a­tions. In rare cas­es. It’s not their meat and pota­toes, but it does happen.

Other gov­ern­ments are not so care­ful. But, if we can cre­ate a track record of com­pro­mise, if we can cre­ate a track record of unlaw­ful or uneth­i­cal activ­i­ty, we can begin cre­at­ing a frame­work to over­turn the cul­ture of impuni­ty that affects so many of these lost jour­nal­ists’ lives. In those thou­sand and sev­en­ty cas­es of dead jour­nal­ists, or the dis­ap­peared, impuni­ty was the most com­mon outcome.

But I want to make it clear here that the idea is not just to pro­tect an indi­vid­ual journalist’s phone, which is a wor­thy cause, but to again increase the cost of engag­ing in these kinds of activ­i­ties, engage in the cost of car­ry­ing out law­ful abus­es of dig­i­tal sur­veil­lance. And with­out sort of bela­bor­ing the point here, let’s go to the actu­al tech­ni­cal side of this and talk about what we’ve actu­al­ly done. Bunnie, let’s talk about ini­tial results.

Bunnie Huang: Sure. Thanks for set­ting all that up, Ed, and moti­vat­ing the back­ground for why we’re try­ing to do what we’re try­ing to do.

When we start­ed out the project, the basic chal­lenge that was out­lined is how do we take a venue where reporters meet with sources, and secure it against state-level adver­saries? There’s a lot of peo­ple who are smarter than me who are work­ing extreme­ly hard to turn your smart­phones into mini cyber fortresses.

The prob­lem is that phones are a very large, com­pli­cat­ed attack sur­face. You have email, you have web, you have mes­sag­ing, you have the abil­i­ty to install apps. Trying to secure this against a state-level adversary’s very chal­leng­ing, just like try­ing to cre­ate a city that’s robust against land, sea, and air attack.

But turn over the phone, and look on the back side. This is a sur­face that’s much sim­pler, and some­thing that I feel more com­fort­able with as a hard­ware guy. And there’s only two real­ly notable fea­tures on the back, the anten­nae. Those form sort of a choke point that we can look at to see if anything’s going in or out of the device. And so if you want to go ahead and make sure that your phone isn’t send­ing sig­nals, you say, Well, why don’t we just go ahead and put it in air­plane mode.” Turns out the ques­tion is can you trust the gate­keep­er? Can you trust the UI

If you go to the Apple web site and you read the lit­tle thing about air­plane mode, it actu­al­ly says that since iOS 8.2, air­plane mode does not turn off GPS. In fact, when you have your device in air­plane mode, the GPS is con­stant­ly on and can be pinged with­out any indi­ca­tion on the UI at all. And that’s a pol­i­cy that they have for the phones. You can also turn on WiFi and Bluetooth in air­plane mode acci­den­tal­ly, or inten­tion­al­ly. And that lit­tle icon is still there mak­ing you think that your device shouldn’t be receiv­ing or trans­mit­ting radio signals.

So the ques­tion is is there a way we can inde­pen­dent­ly mon­i­tor that gate? Can we install, effec­tive­ly, a closed-circuit TV cam­era of our own design, of our own con­struc­tion, and in our own instal­la­tion, to audit and ver­i­fy that this is actu­al­ly happening?

So the tech­ni­cal goal here is to make sure the radios are real­ly off. We want to look at the cel­lu­lar modem, WiFi, Bluetooth, GPS, and NFC (Apple Pay). It’s a tech­nique that we call direct intro­spec­tion.” And we have a set of eight prin­ci­ples that we came up with for this project that we use to eval­u­ate dif­fer­ent approaches.

First is that we want to make sure that what­ev­er we come up with is com­plete­ly open source and inspectable. You don’t have to trust us or what we say. 

Second is we want to cre­ate a par­ti­tioned exe­cu­tion envi­ron­ment for the intro­spec­tion engine. You can think of the thing we’re doing as like a des­ig­nat­ed dri­ver for the phone. The phone may be giv­en a spiked drink and unable to assess its own secu­ri­ty sta­tus. So we have a com­plete­ly physically-separated exe­cu­tion envi­ron­ment for eval­u­at­ing the signals.

We also want to make sure that the prop­er oper­a­tion is field-verifiable. You need to guard against a hard­ware fail­ure. So if the cable falls out dur­ing intro­spec­tion, that’s real­ly bad. You want to be able to check that that’s still there. And you also want to guard against poten­tial so-called evil maid” attacks.

It’s also want­i­ng to make sure that it’s dif­fi­cult to trig­ger a false pos­i­tive. If the thing’s always warn­ing you that your phone is going off and it’s actu­al­ly not true, you’re going to start ignor­ing the warn­ings. And this cri­te­ri­on made us rule out a bunch of more pas­sive approach­es like sens­ing the anten­nas through the RF ema­na­tions, because if you hap­pened to walk by a very strong emit­ter like a WiFi access point or some­thing, your phone would trip and you would [start] ignor­ing the alarms.

We also want­ed to make sure it’s dif­fi­cult to induce false neg­a­tives. It’s quite pos­si­ble, for exam­ple, a sys­tem ven­dor can be com­pelled to push an update to your phone through a com­plete­ly secure mech­a­nism. And so even the sys­tem ven­dor can go ahead and put holes in the walls that you thought were once intact.

We also want­ed to avoid leav­ing a sig­na­ture that’s easy to pro­file. So, we don’t want to have some­thing where some­one says, Okay, let’s look for peo­ple who have intro­spec­tion engine on their phone and tar­get those guys because they have some­thing to hide.” So we have to cre­ate some­thing that’s essen­tial­ly very strong­ly cor­re­lat­ed at the hard­ware lev­el with the acti­va­tion of the radios. These are sig­nals which even a firmware update or some oth­er kind of remote mod­i­fi­ca­tion to the phone can’t bypass. So, they’re a very strong indi­ca­tor of radio activity.

We came up with a list of can­di­dates that are here. I’m not going to go through all of them in detail for lack of time, but if you’re inter­est­ed there’s a blog post live on PubPub now. And I go through the details of what the sig­nals are, why we chose them, and what we plan to do. But the basic idea is if you see these wires wig­gle and you think your phone is in air­plane mode, there’s a prob­lem. Something is turn­ing on the radios in that mode, and you know your your phone has been compromised.

So in terms of next steps, of course we actu­al­ly need to devel­op the hard­ware. We don’t expect jour­nal­ists to car­ry around oscil­lo­scopes and hack their phones and so on and so forth. And the basic approach—this is a pure­ly a con­cept ren­der­ing, but the idea is to try to cre­ate a bat­tery case-style add-on to the back of a phone which con­tains the intro­spec­tion engine. It has its own UI, because you can’t trust the UI in the front of the phone. You have input and out­put to that device. And there’s a cable that goes between the intro­spec­tion engine to the phone through the SIM card port on the iPhone 6.

So the solu­tion here is spe­cif­ic to the iPhone 6, but the tech­nique should be extend­able to oth­er makes and mod­els of phones. That’s basi­cal­ly it for our pre­sen­ta­tion. So thanks a bunch, Ed, for set­ting this up. And I look for­ward to work­ing on this.

Snowden: My plea­sure. It’s been amaz­ing. If I could just say one thing real quick for the room, as this was my first aca­d­e­m­ic col­lab­o­ra­tion. Having Bunnie as your pri­ma­ry col­lab­o­rate on the very first time is amaz­ing. He is one of the indi­vid­u­als whose com­pe­tence gives peo­ple impos­tor syn­drome. So I’ll do my best to live up to it. Thank you so much.


Help Support Open Transcripts

If you found this useful or interesting, please consider supporting the project monthly at Patreon or once via Square Cash, or even just sharing the link. Thanks.