J. Nathan Matias: I’m Nathan Matias, the founder of CivilServant.

Mason English: My name is Mason English, I’m an MD/MBA stu­dent, and I’m one of the mods over at r/politics. We actu­al­ly have a cou­ple here as well if you have any oth­er questions. 

So, a lit­tle bit about r/politics. r/politics is one of the ear­li­er sub­red­dits cre­at­ed. And I say that it’s one of the defaults. It was a default through 2013. The rea­son­ing for that is that it has a unique link with the type of users that are mem­bers of r/politics. So Reddit, being a young technology-savvy kind of draw for a user base, r/politics has the same kind of thing, lead­ing to those kind of users being a lit­tle bit more lib­er­al. We have more lib­er­al users, a high­er percentage. 

Kind of a Graph overview table here. Liberal users com­prise a larg­er per­cent­age of these r/politics users, while con­ser­v­a­tives will com­prise a small­er per­cent­age. This being a huge sim­pliza­tion of this thing. But, through those users and through their vot­ing, they can con­trol what is seen and what is not seen. So a lib­er­al user, as a block, will down­vote more often than not some­thing they don’t agree with necessarily. 

Just for exam­ple, two types of com­ments up here. That ​“Rules and laws apply only to poor peo­ple and lib­er­als, duh!” ver­sus ​“What’s the actu­al proof for the Russian hack­ing?” One’s ask­ing a good ques­tion, one is…just…kind of a troll. But because of that type of user vot­ing, one’s going to be seen, and the one ask­ing pos­si­bly a legit­i­mate ques­tion’s not going to be seen.

Going on, lib­er­al users, less asso­ci­at­ed with inci­vil­i­ty just as a whole. While the con­ser­v­a­tive users, just from what we’ve seen mod­er­at­ing, are more asso­ci­at­ed with this incivility.

Liberal users feel like they have lost. Throughout this past elec­tion, through­out the inau­gu­ra­tion, over the last year of pol­i­tics, lib­er­al users feel like they’ve lost and con­trol the voting. 

This has led to a large degree of inci­vil­i­ty through­out the elec­tion and up to now. We have attempt­ed to com­bat this with big, giant sticky posts address­ing the com­mu­ni­ty direct­ly. We even stole the sticky com­ment thing from [r/]science to try and help with that. We weren’t able to see much change but we weren’t run­ning a sta­tis­ti­cal mod­el and data to actu­al­ly observe that kind of thing. 

Reddit can be great. At its best it has great com­mu­ni­ty report­ing. Users that will go in and build amaz­ing com­ments that will source them­selves and real­ly show what Reddit can do. At its worst, par­ti­san vot­ing and these par­ti­san times we cur­rent­ly live in are lead­ing to a hin­drance to the discussion. 

So work­ing with Nate, we want­ed to check in specif­i­cal­ly on down­votes. Do down­vote but­tons because unruly online behav­ior? And can hid­ing these down­votes pre­vent unruly behavior? 

Slash(dot) and burn: dis­trib­uted mod­er­a­tion in a large online con­ver­sa­tion space [free]

Matias: So let me unpack that puz­zle. So, why would we think that down­vote but­tons make things worse? And part of that comes back to a ques­tion that we are still ask­ing, that’s still kind of an open ques­tion about how we design online plat­forms. Early in the 2000s, when the down­vote but­ton was invent­ed, a researcher named Cliff Lampe (who’s here in the room) did one of the ear­li­est stud­ies of these vot­ing sys­tems and found that yes, if you ask peo­ple to vote on con­tent, on aver­age you know, peo­ple can sep­a­rate high and low qual­i­ty com­ments. Analysis sug­gests a qual­i­fied ​“yes.” And Cliff, we’re still try­ing to fig­ure out exact­ly what that ​“qual­i­fied” means, as often hap­pens with research.

How Community Feedback Shapes User Behavior

Many years lat­er, a fur­ther study by Justin Cheng and a num­ber of col­lab­o­ra­tors looked at polit­i­cal dis­cus­sions across four news sites and actu­al­ly found that when peo­ple get down­vot­ed, they come back more fre­quent­ly, and their future com­ments are low­er qual­i­ty. They feel pun­ished, like a minor­i­ty, and perhaps…the study did­n’t nec­es­sar­i­ly look at their moti­va­tions, but per­haps peo­ple come back and then kind of lash out or com­plain, and it esca­lates from there. 

So, we want­ed to work with the r/politics com­mu­ni­ty to find out if hid­ing the down­vote but­ton would actu­al­ly improve the qual­i­ty of con­ver­sa­tion. And par­tic­u­lar­ly to look at the effect on the future behav­ior of peo­ple who were com­ment­ing for the first time in the community. 

Now, Reddit mod­er­a­tors are able to hide the down­vote, but it’s a bit com­pli­cat­ed because they’re only able to do it for some users. And also, they’re only able to do it for the whole com­mu­ni­ty at once. 

So the first thing we did was we basi­cal­ly set up a ran­dom­ized tri­al where on some days the down­vote but­ton was vis­i­ble in com­ments and oth­er days it wasn’t. 

And then we also kind of acknowl­edged that desk­top users would­n’t see the down­vote and con­se­quent­ly we hoped that there would be a low­er share of down­votes on those days, even if mobile users who are about…we esti­mat­ed about 65% of par­tic­i­pants dur­ing that peri­od would still be able to down­vote as nor­mal. So it’s kind of a messy first pass, but at the same time this is the tool that mod­er­a­tors had at their dis­pos­al and we want­ed to find out what if any dif­fer­ence this made, espe­cial­ly because a lot of com­mu­ni­ties have tried this tac­tic of hid­ing down­votes over the years. 

Does Hiding Downvotes Improve Behavior in r/politics?

So here are some of the ear­ly find­ings that we have from this pilot study, which is not in any way the kind of final word on this. The first thing we did was to look at whether hid­ing down­votes actu­al­ly affect­ed the vote scores for com­ments. Because the way that Reddit works, if your com­ment received more upvotes, you’re giv­en some­thing called a high­er score—it’s a lit­tle obfus­cat­ed by the plat­form but it still is use­ful some­times for analy­sis. And then the vis­i­bil­i­ty of your com­ment, it’s pushed high­er in the dis­cus­sion if you have more upvotes; it’s fur­ther down in the dis­cus­sion if you have more down­votes. And in some cas­es it’s even hid­den by the platform. 

So the key to our research was to find out if hid­ing this but­ton would even affect the score at all. And we found that when we mon­i­tored the first ten com­ments in dis­cus­sion and took snap­shots of their score over time, hid­ing the down­votes changed the score a lit­tle bit. It made the score a lit­tle high­er, it reduced the share of down­votes. But not…quite as much as we were hop­ing for. 

The next thing we looked at was the effect on whether a per­son­’s com­ment would ever get a neg­a­tive score. And there we found a more sharp dis­tinc­tion, that in that com­mu­ni­ty some­where around 6% of the first ten com­ments would get a neg­a­tive score at some point. And we were able to reduce that by half by hid­ing the down­vote. So were able to see that hid­ing the down­votes does have an effect on peo­ple’s vot­ing behavior. 

And that’s the first part of the study. We then want­ed to know if peo­ple who com­ment­ed for the first time in the com­mu­ni­ty behaved dif­fer­ent­ly in the future if they com­ment­ed on a day where the share of down­votes was reduced. And we did find that sim­i­lar­ly to what Justin Cheng found, peo­ple who do com­ment for the first time on a day where there is no down­vot­ing, they actu­al­ly are less like­ly to come back. That peo­ple who are more like­ly to receive down­votes are much more like­ly to come back and com­ment a sec­ond time. I should say step that back. Not much more like­ly, slight­ly more like­ly to come back. It’s actu­al­ly a fair­ly small per­cent­age point change. 

And then we were curi­ous. Like, does this add up to any dif­fer­ence in how peo­ple behave in the future? And we looked at the chance of that per­son­’s future com­ments to be removed by mod­er­a­tors. And there we could­n’t dis­tin­guish any effect that was dif­fer­ent from chance. So it might have been pos­si­ble with a larg­er sam­ple size, or with bet­ter mea­sure­ments of the qual­i­ty of peo­ple’s com­ments. It’s pos­si­ble that the effect is real­ly small and we just weren’t able to see it. But using the sta­tis­ti­cal tools that we had, we weren’t able to observe any sys­tem­at­ic dif­fer­ences in the future behav­ior of first-time com­menters com­ing from days where we hid the down­votes than oth­er days. 

So, what do we learn from a null result like this? In our dis­cus­sion with the com­mu­ni­ty, many peo­ple asked why do it if you don’t nec­es­sar­i­ly get the answer you’re look­ing for. There are a few rea­sons why we might val­ue this study. The first is that we learned that hid­ing the down­vote but­ton does­n’t have the dra­mat­ic effect on the ulti­mate votes that are giv­en that I think a lot of peo­ple hoped for. 

And that’s real­ly valu­able. Moderators hope that maybe hid­ing that but­ton will real­ly affect the votes. Not so much. It’s pos­si­ble that if we were able to affect vot­ing across mobile and desk­top it would make more of a dif­fer­ence. We don’t know. 

On top of that, we found that while hid­ing down­votes at least has some effects, it does­n’t have very much of an effect. So it’s not a panacea to the con­flicts that we’ve already heard described hap­pen­ing in the com­mu­ni­ty. So as the com­mu­ni­ty thinks about oth­er things that it could do, down­vot­ing might be part of it. It may be pos­si­ble that with a more reli­able way to remove the down­vote but­ton there might be new ways to study this issue. But for what com­mu­ni­ties can do now, this does­n’t seem to have the effects that peo­ple are look­ing for. 

And on top of that we learned a lot about how to mea­sure these things. The things that peo­ple care about. And how to do work with a com­mu­ni­ty in a polit­i­cal­ly con­tentious sit­u­a­tion so that it can hap­pen with the con­sent and trans­paren­cy of the com­mu­ni­ty, which I’m incred­i­bly grate­ful for the r/politics sub­red­dit for. 

So as you have ques­tions about how we can improve polit­i­cal con­ver­sa­tion online, we’d love to talk to you, and you can reach us at civilser​vant​.io Thanks.