Monday, November 30, 2009

Is blind review blind?

I think that some sort of blind review in philosophy is the norm, but I think that blind review comes in different forms. Apart from AJP, I don't know of many journals that have blind referees and editors. Actually, I don't think I know that this is AJP's policy, but I seem to remember reading that this is their policy. (Full disclosure, I have fond feelings towards AJP because they've published my work, their editor writes you nice little notes thanking you for refereeing, and the editor will also return submissions to you within ten minutes of submission with a note that says 'I've spotted three typographical errors on the first page alone that you should fix them before resubmitting' (Ryan and I only found two after hours of searching, but I suspect that was all part of the fun.) There are always going to be problems with googly eyed referees trying to subvert the blind review process, but I'm really interested in the idea of blinding the editors today. (While I'm on the subject, don't referees know they know we know that they're looking? It's one of the reasons I take the opportunity to address referee's criticisms in posts here after I receive rejections. It's rare that you get the opportunity to stand up for your work when a referee criticizes it, but when you know that refs are checking in it's nice to be able to take the opportunity to explain why you think your work is more defensible than an unhappy referee suggests or say how you plan on fixing the paper to address the referee's criticisms. You can't make them accept your work or take back their criticism, but at the very least you hope they can respect the responses. (And, yes, fwiw, I know that some refs only look after the review is completed.))

I figured I'd post this because we might see the launch of an exciting new journal and the discussion has thus far has largely focused on open access and there's not been much discussion of the review process. I don't know who the editors for NIP will be apart from one and I have nothing but good things to say about this person. It seems that I'm not the only one who thinks that NIP could be a good model for other journals, but I'm less concerned about the open access stuff than I am about review practices. I've heard various rationales for having editors that know the identity of an author, but they seem weak. I've heard that it would be a hassle to blind the papers, but that seems unlikely if we're talking about journals that use online submissions. If TAs responsible for hundreds of students in intro classes can blind before grading, I have a hard time believing that editors can't look at a blinded version of a paper while deciding whether to send the paper out and which referee(s) to use. I've heard that the editors benefit from knowledge of the author's identity so they can use that knowledge to decide which referees will be appropriate for the paper. I don't know why we need to do this on the front end. Ask the referees whether they know the identity of the author and after the refereeing process is completed, the editor can then decide whether the referee's relationship to the author was problematic. I suspect that this will be rare. We all know the work of our friends and enemies rather well and I can't imagine that we'd deceive some editor by saying that we didn't know who authored a paper to help friends or harm enemies. I've heard that editors will use their knowledge of the author's talents and abilities to select referees, but isn't this precisely the sort of thing that should be discouraged? I've also heard in various blog threads that if you fall out of favor with an editor, you can expect to get bad treatment in the future. I've also heard that writing to the editor to address a referee's report will put you in bad standing with an editor. I don't know how much of this to believe, but if there is some truth to this, I think this is just another reason to blind editors. I don't like annoying people (but I'm sure someone near and dear to me will say that I love annoying people lately) and I can imagine that editors develop a dislike for authors who pester them, but I don't think this should affect how someone's work is handled by a journal. Going back to the grading example, if you have a particularly obnoxious student (it does happen sometimes, admit it) it's probably best that you try to blind student work before grading it. That just seems sort of obvious, so there must be some important difference between grading and the review process that I don't get. Throw in additional stuff about biases that are favorable to an author or unfavorable to an author, and it seems like there's a good case to be made for keeping editors blind. But, they aren't. Not to my knowledge, at least. I have to confess that my knowledge is quite limited.

10 comments:

Aidan said...

I'm fairly sure that both Phil Quarterly and Mind have a practice of not letting the editor know the author. Certainly the later does. I agree entirely that it's a good practice.

M said...

Actually, this is a problem that is pretty easy to solve in these tech-saturated days. A simple web-based interface should allow an author to submit a document to a server that gives a unique tracking ID that doesn't give away the name of the author until the reviews are in. The editor could correspond with the author prior to the return of the reviews via a facebook-like private message system (i.e., which only reveals the tracking ID, not the name).

Clayton said...

Hey Aidan,

That's good to know. I send a fair amount of stuff to PQ, so it would be good if my annoying habit of sending everything I write to them won't adversely affect my chances of getting them to finally say 'yes'.

Hey M,
Agreed, it's hard to see how some journal could accept electronic submissions and not be able to institute (real) blind review.

Anonymous said...

Philosophy and Public Affairs uses fully blind review, where the journal's editor doesn't know the author.

John Turri said...

Clayton,

This might be a relevant difference between grading papers in a course and refereeing: A grader gets paid and grading is part of what it takes to keep her job, whereas a referee doesn't get paid and refereeing isn't part of what it takes to keep his job.

I think this is partly why editors sometimes use knowledge of the author to select a ref. They probably want to avoid alienating reliable referees by sending them papers from authors who (are believed to) write less than impressive stuff.

I don't claim that any of this speaks to fairness. But it seems to speak to something broadly relevant to adopting the policy.

Anyway, in the end, I'm with you. Keeping submissions anonymous to the handling editors is good.

Joe said...

Although I like the idea of blinded editors, doesn't it present a bit of logistical problem for a journal's editor? Editors risk sending the manuscript to the author asking her to referee the paper.

I'm interested in learning how journals, such as those mentioned by Clayton and other commentators, avoid such an embarrassing situation.

Clayton said...

Hey Joe,

I think there's a high tech way and a low tech way. The high tech way is have a system like Synthese does where everyone has a single user number and can log in either as an author or reviewer. The low tech way is just to tell reviewers that the review process is thoroughly blinded and say that there's a chance that you know the author of the paper being sent. Just say that if that's the case, disclose the nature of the relationship (e.g., identity or something else like constitution) and I think authors will be understanding.

Aidan said...

I think the one's I mentioned both run everything through a editorial assistant, who does see who the author is. There are various ways one can arrange things so that the editor doesn't ever have to find out who the author is even if s/he tries to use the author as a referee for their own paper. Of course, this means that the process isn't fully blind. At PQ, the editorial assistant isn't a philosopher, which should ensure considerably less bias that if the editor were to know the names of the authors. (When I say 'the editor', I mean the office - I don't mean to suggest anything about Katherine Hawley in particular!) I think things are different at Mind, since the EA is a philosopher.

Anonymous said...

To be pedantic, I think what you're discussing is called 'double-blind review'. It is single blind when the author doesn't know who the reviewers are. Every journal is blind-review.

Clayton said...

Fair enough. So, blind review is sort of like review with one eye open.