Piper Below: Hi, my name is Piper Below. I’m a pro­fes­sor of human genet­ics at Vanderbilt University in the Vanderbilt Genetics Institute. And I’m one of the head mods of r/science. This is going to be such a short talk that I’m gonna most­ly let Nate here run the mic. But I’ll be up here and if you want to ask me ques­tions lat­er I’ll be around. 

Nathan Allen: My name’s Nathan Allen. I’m user ​“nate” on Reddit, Principal Scientist at the Sigma-Aldrich cor­po­ra­tion. I’m group leader for new prod­uct research there. 

J. Nathan Matias: And I’m Nathan Matias, the founder of CivilServant.

Allen: Alright, so. A lit­tle intro­duc­tion about what r/science is. So r/science is real­ly the largest sci­ence forum on the Internet. We say that we have more than 18 mil­lion sub­scribed users. For a point of ref­er­ence, the total com­bined sub­scriber base of the top ten news­pa­pers in the United States is around ten mil­lion. So we’re rapid­ly approach­ing twice the sub­scribed user base of all of the news­pa­pers in the coun­try. We had about 3.6 mil­lion uniques in December. And about near­ly 14 mil­lion page views. So, pret­ty big forum. We deal with a lot of traf­fic. And we’ve adapt­ed to deal with this in many ways. 

There are two basic types of sub­mis­sions that you get on r/science. There’s user-submitted con­tent. These are sum­maries of peer-reviewed jour­nal arti­cles; things that sum­ma­rize a jour­nal that was recent­ly pub­lished. We put in hard binds to deter­mine what those are so we are not decid­ing what sci­ence is. Science is decid­ed by the jour­nal ref­er­ees in the peer review sys­tem. If it gets pub­lished in a legit­i­mate peer-reviewed jour­nal and some­body writes up a sum­ma­ry of it, you may post it and we will allow it. 

The oth­er type of con­tent we allow are AMAs, where we go out and we recruit sci­en­tists who have enough promi­nence or inter­est to speak direct­ly to the gen­er­al pub­lic. The aim of these is to dis­in­ter­me­di­ate the rela­tion­ship between sci­en­tists and the gen­er­al pub­lic. We rec­og­nize that the trust in insti­tu­tions is wan­ing in our soci­ety, but insti­tu­tions are made up of indi­vid­u­als. As peo­ple under­stand that indi­vid­u­als make up insti­tu­tions their faith in insti­tu­tions can be rede­vel­oped to a point at which we maybe will have faith in sci­en­tif­ic insti­tu­tions again. This is our long-term goal.

In order to accom­plish this how­ev­er, if we ask sub­stan­tial guests who are of promi­nence and not real­ly used to the Internet, there’s a great deal of hes­i­ta­tion to come to the Internet and speak. Nobody is going to show up for their own lynch­ing, after all. 

In order to do this, you may rec­og­nize the state bird of r/science. We have rules. And we enforce these rules. We…bru­tal­ly enforce these rules some­times, and we’ve kind of become known for that. 

Audience mem­ber: So basi­cal­ly like AskHistorians.

Allen: Uh, not quite that bad… But, you know… They’ve found out the ulti­mate solu­tion. [?] is an AskHistorians mod. 

So. We have the largest mod­er­a­tor team on Reddit by far. No one’s even close. The ModEveryone sub­red­dit has 50% less mod­er­a­tors than we do. We have more mod­er­a­tors than the top fifty sub­red­dits on Reddit com­bined. We have almost dou­ble that num­ber. We have a lot of moderators. 

So, we have the band­width to enforce our rules bru­tal­ly and with­out any mer­cy. We don’t want to do that. We want peo­ple to want to not break the rules and inter­act with our com­mu­ni­ty in a way that’s con­struc­tive and civ­il. We want to shape our com­mu­ni­ty with cul­tur­al norms and sug­ges­tions. And this is what led us to work with Nate Matias to test how we can influ­ence that, because these things are dif­fi­cult to get a han­dle on. So let Nate tell you about what we did on that.

Matias: So, just about two years ago, the mod­er­a­tors of r/science came to me and asked this ques­tion, can we can we post the rules in a way that actu­al­ly pre­vents peo­ple from vio­lat­ing them and get­ting into trou­ble with the mod­er­a­tors? We were also wor­ried about a side-effect, that it might push new­com­ers away at the same time. 

Now, the kind of the­o­ries behind this that come from social psy­chol­o­gy are ideas about social norms. That when we post rules to the top of a com­mu­ni­ty, we influ­ence peo­ple’s beliefs about what’s approved and what’s dis­ap­proved by the peo­ple around us. And so we thought that by post­ing the rules to the top of dis­cus­sions, we would influ­ence those beliefs and con­se­quent­ly influ­ence peo­ple’s behavior. 

We were also wor­ried about some oth­er side-effects. That some­one might come to the com­mu­ni­ty, they might have pre­pared a post, they might see these rules, and then decide not to par­tic­i­pate at all. Either because they felt like their com­ment was­n’t okay there. Because they felt that maybe it would be too much work to alter what they were going to say to fol­low the rules. Or because maybe they felt that this isn’t the place for me. And as a com­mu­ni­ty that wants to broad­en who has a sci­en­tif­ic con­ver­sa­tion with schol­ars, with oth­er peo­ple who are curi­ous about sci­ence, that would not be a good outcome. 

So with that in mind, here’s what we did. We cre­at­ed a set of mes­sages that we used a piece of soft­ware to post to the top of dis­cus­sion threads. So when some­one shared a link to a jour­nal arti­cle or host­ed a live Q&A with a sci­en­tist, we would use this bot to ran­dom­ly assign some dis­cus­sions to receive a mes­sage that wel­comed peo­ple, explained the rules, described the con­se­quences, explained that lots of peo­ple agree with the rules. And that there were enough mod­er­a­tors to enforce them. All things that have kind of ties to the­o­ries in social psychology. 

And then we mea­sured a bunch of things to see what the effect would be on whether first-time com­menters would post things that fol­lowed the rules. And also looked at the impact on the per­cent­age of first-time commenters. 

So, we found when we con­duct­ed this test in 2016, we ran it over 2,200 dis­cus­sions includ­ing many live Q&As. The study end­ed up includ­ing over 20,000 new­com­er com­ments. And we found that as you can see, post­ing this mes­sage to the top of dis­cus­sion actu­al­ly did increase the chance that new­com­ers would par­tic­i­pate in a way that com­plied with r/science’s rules. 

And we also found that, con­trary to our expec­ta­tions that it would maybe push peo­ple away, that it actu­al­ly increased the rate of new­com­er par­tic­i­pa­tion. That when peo­ple saw that the rules were there and they would be rein­forced, more peo­ple par­tic­i­pat­ed in those con­ver­sa­tions. In fact, at the vol­umes that r/science was at at that time, on aver­age the com­mu­ni­ty could pre­vent over 1,800 new­com­ers a month from some kind of unruly behav­ior. And would also gain over 9,000 first-time com­menters a month, just by post­ing the mes­sages to the top of the community. 

And this study did­n’t just stop with r/science. It end­ed up being picked up and adopt­ed by a num­ber of oth­er plat­forms. The Discus plat­form which is used on lots of sites across the Web has point­ed to our research as a moti­vat­ing fac­tor behind fea­tures that they’ve released to tens of thou­sands of web­sites across the Internet. And our friends at The Coral Project have cit­ed our research as one influ­ence in their wider efforts to reengi­neer how com­ments hap­pen online. 

For us, we’re glad when peo­ple take what we’ve learned in one con­text and hope that peo­ple will learn from them. We’re also espe­cial­ly excit­ed when peo­ple respond to our find­ings with skep­ti­cism and say, ​“But what about my con­text? Will that real­ly work here?” And our answer at CivilServant is ​“We don’t know. That’s a great ques­tion to do your own study on.” 

And that’s why we sup­port repli­ca­tions, where com­mu­ni­ties can do their own research and then con­tribute to a shared pool of knowl­edge. So if you’re inter­est­ed to test ideas about harass­ment pre­ven­tion in your com­mu­ni­ty and on your plat­form, we’d love to talk. And you should feel free to reach out to the mod­er­a­tors of r/science lat­er and chat with them about the amaz­ing work that they do. Thank you.