So, I’m starting to lose faith in the whole blind peer review system.
All this started when I was assigned several reviews by a session chair outside of my specified area of expertise. Ostensibly, the session chair for each track area should look over the reviewers credentials, evaluate their trustworthiness, and choose appropriate weights for the reviews. But, in this case, the session chair didn’t look at my specified areas of expertise, much less put any effort into determining whether I was capable of providing valuable insight. I declined the off-topic reviews, as I have been taught, but the experience was somewhat troubling.
It lead me to believe that there’s a relatively high probability that any paper you submit will not get reviewed by someone who knows much about your field. In fact, I would venture to bet that anyone with a dot-edu e-mail address can sign up to review papers submitted to most of the big conferences. Actually, if you’re a curious undergrad or a non-research university staff member, you might want to try registering to be a reviewer on some of the big journals and conferences. If you get assigned lots of reviews, or any off-topic reviews, that means the session chairs aren’t even trying to do their job. (Disclaimer: If you do choose to do this, it’s your fault if you get in trouble. Also, please don’t actually submit any reviews. Just click decline on everything.)
Of course, I realize that science is supposed to be self-regulating, but since there is no way to know who has reviewed a particular paper, how are we supposed to be confident that it really is self-regulation, instead of random weed-out or worse yet, regulation by idiots? I’ve been thinking about publishing some of my work in Philica. Does anyone have any other options for open peer review?