Reputation or peer review? The role of outliers

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events-papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.

Publication type: 
Articolo
Author or Creator: 
Grimaldo, Francisco
Paolucci, Mario
Sabater-Mir, Jordi
Publisher: 
Springer, Budapest , Ungheria
Source: 
Scientometrics (Print) 116 (2018): 1421–1438. doi:10.1007/s11192-018-2826-3
info:cnr-pdr/source/autori:Grimaldo, Francisco; Paolucci, Mario; Sabater-Mir, Jordi/titolo:Reputation or peer review? The role of outliers/doi:10.1007/s11192-018-2826-3/rivista:Scientometrics (Print)/anno:2018/pagina_da:1421/pagina_a:1438/intervallo_pagin
Date: 
2018
Resource Identifier: 
http://www.cnr.it/prodotto/i/398266
https://dx.doi.org/10.1007/s11192-018-2826-3
info:doi:10.1007/s11192-018-2826-3
Language: 
Eng
ISTC Author: 
Mario Paolucci's picture
Real name: