Referees, reports, and how to survive them

In this post, I first voice some basic complaints about the referee process. Then I end with some positive notes.

The complaints

I think that PhD students should read some harsh referee reports during their course of study. I will show some reports I received in a later post. Referees are the main readers of our papers and, some would argue, the only readers that matter. It is critically important to know how this group of readers evaluates papers.

If you want to join academia because your university professors seem nice and wise, and you want to work with them or be like them, you will likely be disappointed when you read your first referee report. Make no mistake, referees are harsh. They have literally caused depression for many and, in extreme cases, premature death.

Here is a set of low expectations that I think are good to have to avoid referee-induced depression.

1. Referees are harsh to everyone like you, not just you.

2. Referees always want to reject because rejecting is costless.

3. Referees never want to accept because accepting can be very costly.

4. Referees will never fully understand your paper because they barely read it.

Economists study incentives, and you bet that most people in the field know about these problems. If you are a researcher in the purest sense, you may just shrug and say, “Who cares about the referees? Rejecting my work is the journal’s loss.” This mentality is good for health, but some young scholars may not have this. They may believe in the referees’ most unfair comments and fall into self-doubt or even self-destruction. One caveat, of course, is that you need to have a fair assessment of your own work. I assume this for most readers because, after all, if your work is as bad as the referee says, it would be easy for you to figure out because you have worked on the paper for so long.

Ironically, people with the best papers get hurt the most by unfair referees. Suppose that, in the first case, you’ve spent five years solving a big problem. The referee then tells you the paper is worthless (perhaps without trying to understand your work). In the second case, you wrote a piece of short work that is somewhat useful. The referee tells you the same thing: that the paper is worthless. In the first case, because your expectations are much higher, the word “worthless” is going to bite into your soul. In the second case, it still hurts, but much less so because you didn’t expect much to begin with.

This means that if you are depressed after a rejection, chances are you have an excellent paper. If you have an excellent paper, well, then don’t feel depressed. Clean it up just slightly and send it out again. You are testing the referees and editors as much as they are testing you. You are about to identify an emerging influential journal with your great work. Good journals are journals that have good referees and that accept good papers. It is true that tenure requirements and journal tier lists limit the range of journals in which you can publish (another big problem that people are aware of and I will write about in the future). But I presume that if you chose academia, your aim is to do good research that hopefully can benefit the people, not just to get tenured, right? Many people around me who deserve tenures do not really need tenure. They just keep producing good work (not just papers) no matter what. They will do well wherever they go. Having them around is the institution’s fortune.

Of course, there might be some valid points in the reports. However, when the points are valid, you will notice them quickly, and you will not feel depressed about those valid points. So often, however, those valid points are so minor that even after addressing them, the message of the paper remains 99.9% the same. Also, you have likely thought of them before and decided to drop them because they are not essential.

In sports, if a referee makes a wrong judgment, s/he loses the job. In academia, if a referee makes a wrong/misleading recommendation, there isn’t even a way to complain.

Some positive notes

After understanding all the above, if you still want to be in academia, congratulations, you are in the right profession for life. Here are some encouraging quotes from the Bible:

“If you do what is right, won’t you be accepted? But if you do not do what is right, sin is crouching at the door. Its desire is for you, but you must rule over it.”

“The stone which the builders rejected has become the chief cornerstone.”

Someone senior told me that “you should write papers so good that they cannot reject.” Academia, just like any other field, is thorny. What you are truly working on is yourself, not the papers. You need to set your mind straight by doing what is upright and then let the good work be produced through you. This is hard because you have to defeat all the worldly nonsense and get to the essence of matters. But if it were easy, you would be bored, wouldn’t you? The name of the game is not to publish good work because the referees are good judges. It is to produce good work despite the referees being imperfect, as all humans are. This requires hard work and a whole lot of understanding and sympathy for humanity --- the people you said you set out to serve.

Dealing with rejection is a course that all people must take. Here is a helpful video on this subject. It explains to you the traps that cause people to fall. You do not have to agree with everything in the video for it to be helpful: https://www.youtube.com/watch?v=aYLa-4eYllY