Have you controlled for those respondents that submit a response to the “quality of referee comments” question but admit that they didn’t receive any comments?

Here’s why I ask. The survey uses a 5-point Likert scale asking respondents about the quality of comments they received. 1 = very poor/ very unhelpful and 5 = very good / very helpful. Although the survey asks respondents not to answer this question if they did not receive comments, I believe some might still respond. To the respondent’s mind, receiving no comments scores a 1 on the Likert scale because receiving no comments is “very unhelpful.” PR, Mind, JPhil, PPR, and Nous seem especially more likely to reject without providing the author any substantive comment because of the volume of mss they receive.

Relatedly, there may be respondents that believe receiving an email from the editor indicating that their manuscript has been rejected is a comment. So, respondents may believe they have received comments and respond very negatively (perhaps) to the quality of comments question. (There’s no way for you to control for this other than rephrasing the question or explaining what you mean by “comments;” instead of “comments”, might I suggest using “referee reports” or “referee reviews?”)

Finally, although I don’t believe that Mark is wrong that some authors feel “spiteful” after receiving a rejection of their manuscript, I believe there might be a strong correlation between negative view of the quality of comments and the length of time it took for the author to receive initial decision on the manuscript. JPhil, which is notorious for taking an unconscionably long time to review mss, seems to rank lowest among the other prominent journals (BTB, I’m not judging JPhil here; I’m just pointing out a fact that I’ve heard from others who have submitted to JPhil). Perhaps you can test for this correlation too.

These are just some thoughts on the matter.

Best,
Joe

That would be very easy to do. I’ll go that route.

However, I think it’s also important to keep track of what authors of accepted papers, or of papers that have received a verdict of revise & resubmit, have to say. Certainly the quality of comments accompanying a verdict of revise & resubmit is an important matter. And sometimes a journal’s referees and editors can do an important job providing comments improving an accepted paper.

Why not just provide a few different averages? That way more relevant information is available.

I think that’s a great idea, and I think I’m going to do that.

The policy is working out really well.