Piper Below: Hi, my name is Piper Below. I’m a pro­fes­sor of human genet­ics at Vanderbilt University in the Vanderbilt Genetics Institute. And I’m one of the head mods of r/science. This is going to be such a short talk that I’m gonna most­ly let Nate here run the mic. But I’ll be up here and if you want to ask me ques­tions lat­er I’ll be around.

Nathan Allen: My name’s Nathan Allen. I’m user nate” on Reddit, Principal Scientist at the Sigma‐Aldrich cor­po­ra­tion. I’m group leader for new prod­uct research there.

J. Nathan Matias: And I’m Nathan Matias, the founder of CivilServant.

Allen: Alright, so. A lit­tle intro­duc­tion about what r/science is. So r/science is real­ly the largest sci­ence forum on the Internet. We say that we have more than 18 mil­lion sub­scribed users. For a point of ref­er­ence, the total com­bined sub­scriber base of the top ten news­pa­pers in the United States is around ten mil­lion. So we’re rapid­ly approach­ing twice the sub­scribed user base of all of the news­pa­pers in the coun­try. We had about 3.6 mil­lion uniques in December. And about near­ly 14 mil­lion page views. So, pret­ty big forum. We deal with a lot of traf­fic. And we’ve adapt­ed to deal with this in many ways.

There are two basic types of sub­mis­sions that you get on r/science. There’s user‐submitted con­tent. These are sum­maries of peer‐reviewed jour­nal arti­cles; things that sum­ma­rize a jour­nal that was recent­ly pub­lished. We put in hard binds to deter­mine what those are so we are not decid­ing what sci­ence is. Science is decid­ed by the jour­nal ref­er­ees in the peer review sys­tem. If it gets pub­lished in a legit­i­mate peer‐reviewed jour­nal and some­body writes up a sum­ma­ry of it, you may post it and we will allow it.

The oth­er type of con­tent we allow are AMAs, where we go out and we recruit sci­en­tists who have enough promi­nence or inter­est to speak direct­ly to the gen­er­al pub­lic. The aim of these is to dis­in­ter­me­di­ate the rela­tion­ship between sci­en­tists and the gen­er­al pub­lic. We rec­og­nize that the trust in insti­tu­tions is wan­ing in our soci­ety, but insti­tu­tions are made up of indi­vid­u­als. As peo­ple under­stand that indi­vid­u­als make up insti­tu­tions their faith in insti­tu­tions can be rede­vel­oped to a point at which we maybe will have faith in sci­en­tif­ic insti­tu­tions again. This is our long‐term goal.

In order to accom­plish this how­ev­er, if we ask sub­stan­tial guests who are of promi­nence and not real­ly used to the Internet, there’s a great deal of hes­i­ta­tion to come to the Internet and speak. Nobody is going to show up for their own lynch­ing, after all.

In order to do this, you may rec­og­nize the state bird of r/science. We have rules. And we enforce these rules. We…bru­tal­ly enforce these rules some­times, and we’ve kind of become known for that.

Audience mem­ber: So basi­cal­ly like AskHistorians.

Allen: Uh, not quite that bad… But, you know… They’ve found out the ulti­mate solu­tion. [?] is an AskHistorians mod.

So. We have the largest mod­er­a­tor team on Reddit by far. No one’s even close. The ModEveryone sub­red­dit has 50% less mod­er­a­tors than we do. We have more mod­er­a­tors than the top fifty sub­red­dits on Reddit com­bined. We have almost dou­ble that num­ber. We have a lot of mod­er­a­tors.

So, we have the band­width to enforce our rules bru­tal­ly and with­out any mer­cy. We don’t want to do that. We want peo­ple to want to not break the rules and inter­act with our com­mu­ni­ty in a way that’s con­struc­tive and civ­il. We want to shape our com­mu­ni­ty with cul­tur­al norms and sug­ges­tions. And this is what led us to work with Nate Matias to test how we can influ­ence that, because these things are dif­fi­cult to get a han­dle on. So let Nate tell you about what we did on that.

Matias: So, just about two years ago, the mod­er­a­tors of r/science came to me and asked this ques­tion, can we can we post the rules in a way that actu­al­ly pre­vents peo­ple from vio­lat­ing them and get­ting into trou­ble with the mod­er­a­tors? We were also wor­ried about a side‐effect, that it might push new­com­ers away at the same time.

Now, the kind of the­o­ries behind this that come from social psy­chol­o­gy are ideas about social norms. That when we post rules to the top of a com­mu­ni­ty, we influ­ence people’s beliefs about what’s approved and what’s dis­ap­proved by the peo­ple around us. And so we thought that by post­ing the rules to the top of dis­cus­sions, we would influ­ence those beliefs and con­se­quent­ly influ­ence people’s behav­ior.

We were also wor­ried about some oth­er side‐effects. That some­one might come to the com­mu­ni­ty, they might have pre­pared a post, they might see these rules, and then decide not to par­tic­i­pate at all. Either because they felt like their com­ment wasn’t okay there. Because they felt that maybe it would be too much work to alter what they were going to say to fol­low the rules. Or because maybe they felt that this isn’t the place for me. And as a com­mu­ni­ty that wants to broad­en who has a sci­en­tif­ic con­ver­sa­tion with schol­ars, with oth­er peo­ple who are curi­ous about sci­ence, that would not be a good out­come.

So with that in mind, here’s what we did. We cre­at­ed a set of mes­sages that we used a piece of soft­ware to post to the top of dis­cus­sion threads. So when some­one shared a link to a jour­nal arti­cle or host­ed a live Q&A with a sci­en­tist, we would use this bot to ran­dom­ly assign some dis­cus­sions to receive a mes­sage that wel­comed peo­ple, explained the rules, described the con­se­quences, explained that lots of peo­ple agree with the rules. And that there were enough mod­er­a­tors to enforce them. All things that have kind of ties to the­o­ries in social psy­chol­o­gy.

And then we mea­sured a bunch of things to see what the effect would be on whether first‐time com­menters would post things that fol­lowed the rules. And also looked at the impact on the per­cent­age of first‐time com­menters.

So, we found when we con­duct­ed this test in 2016, we ran it over 2,200 dis­cus­sions includ­ing many live Q&As. The study end­ed up includ­ing over 20,000 new­com­er com­ments. And we found that as you can see, post­ing this mes­sage to the top of dis­cus­sion actu­al­ly did increase the chance that new­com­ers would par­tic­i­pate in a way that com­plied with r/science’s rules.

And we also found that, con­trary to our expec­ta­tions that it would maybe push peo­ple away, that it actu­al­ly increased the rate of new­com­er par­tic­i­pa­tion. That when peo­ple saw that the rules were there and they would be rein­forced, more peo­ple par­tic­i­pat­ed in those con­ver­sa­tions. In fact, at the vol­umes that r/science was at at that time, on aver­age the com­mu­ni­ty could pre­vent over 1,800 new­com­ers a month from some kind of unruly behav­ior. And would also gain over 9,000 first‐time com­menters a month, just by post­ing the mes­sages to the top of the com­mu­ni­ty.

And this study didn’t just stop with r/science. It end­ed up being picked up and adopt­ed by a num­ber of oth­er plat­forms. The Discus plat­form which is used on lots of sites across the Web has point­ed to our research as a moti­vat­ing fac­tor behind fea­tures that they’ve released to tens of thou­sands of web­sites across the Internet. And our friends at The Coral Project have cit­ed our research as one influ­ence in their wider efforts to reengi­neer how com­ments hap­pen online.

For us, we’re glad when peo­ple take what we’ve learned in one con­text and hope that peo­ple will learn from them. We’re also espe­cial­ly excit­ed when peo­ple respond to our find­ings with skep­ti­cism and say, But what about my con­text? Will that real­ly work here?” And our answer at CivilServant is We don’t know. That’s a great ques­tion to do your own study on.”

And that’s why we sup­port repli­ca­tions, where com­mu­ni­ties can do their own research and then con­tribute to a shared pool of knowl­edge. So if you’re inter­est­ed to test ideas about harass­ment pre­ven­tion in your com­mu­ni­ty and on your plat­form, we’d love to talk. And you should feel free to reach out to the mod­er­a­tors of r/science lat­er and chat with them about the amaz­ing work that they do. Thank you.


Help Support Open Transcripts

If you found this useful or interesting, please consider supporting the project monthly at Patreon or once via Square Cash, or even just sharing the link. Thanks.