Up
Bias in peer review
Review process

Simulation & Gaming:
An Interdisciplinary Journal

+++

From  http://www.mang.canterbury.ac.nz/courseinfo/AcademicWriting/Sadomaso.htm

A Note on Sadomasochism in the Review Process: I Hate When That Happens

Morris B. Holbrook

This article describes an apparent sadomasochism in the review process for journals of marketing and consumer research. It provides vignettes representing both the author’s and the reviewer’s perspectives. It then discusses two sets of seven suggestions intended to encourage reconciliation. 

Willie and Frankie

Billy Crystal, a comedian from "Saturday Night Live," performs a routine in which characters named Willie and Frankie entertain each other by describing in vivid detail the horrifyingly brutal things that they do to themselves when they are alone in their spare time. As examples of sick jokes, Crystal’s imaginative exercises in painful self-mutilation set new standards for sadomasochistic comedy. Their humor lies mostly in their repeated refrain when, after one character recounts some particularly ghastly episode with his friend telepathically interjecting many of its most gruesome features, the narrator explains with mock understatement, "I hate when that happens."

In a recent phonograph album (Mahvelous!, A&M Records, SP 5096), Willie and Frankie run through an extensive repertoire of grizzly exercises in self-torture. Structurally, Willie usually begins with an ominous introduction, such as, "The other day, I was sittin’ on the stoop — not doin’ much, just burnin’ the hair off my arms with my magnifyin’ glass — and I reached for that . . . uuh . . . ." Frankie supplies the crucial piece of missing information — for example, "Meat thermometer?," "Self-threading movie projector?," or "Six-inch replica of the Empire State Building?" Willie replies, "Yeah" or "Right," and then proceeds to describe the grotesque manner in which he uses the instrument in question to inflict pain on himself. Invariably, he gets stuck again, and Frankie supplies another critical detail like "Hammer?," "Red-hot projector bulb?," or "Living unicorn in the Ringling Brothers, Barnum, and Bailey Circus?" The two friends then conclude by commiserating as follows:

WILLIE: "Yeah. I hate when that happens . . . ."

FRANKIE: "I know what you mean."

BOTH: "Ooh! Ow! Ouch!"

WILLIE "I hate when that happens."

FRANKIE: "Tell me about it."

BOTH: "Ooh! Ow! Ouch!"

WILLIE: "I hate when I do that."

FRANKIE: "I know what you mean."

Professor Maas and Dr. Hart

Several years ago, in an incident not entirely unrelated to Billy Crystal’s comedic revelations, one of my counterparts on the marketing faculty at a major school of business (let’s call him Professor X. Maas) dreamed up an imaginative and relatively harmless practical joke to play on one of the more experienced senior professors at the same institution (whom, for short, we shall call Dr. B. N. Hart). It happened that Dr. Hart had been working for many years on a stream of research dealing with the diffusion of an innovation (that we shall refer to as Product I). Professor Maas somehow contrived to obtain some stationery from a major marketing journal and wrote a bogus letter from the editor to Dr. Hart, saying (1) that the editor had just heard about his diffusion research on Product I, (2) that the journal had already published more than enough work on related topics, and (3) that he therefore wanted to rule in advance against the possible publication of any papers on this subject.

To everyone’s surprise, Dr. Hart (who had earned wide recognition for his well-developed sense of humor) did not find this practical joke the least bit funny. Indeed, rather than responding with his customary good-natured chuckle, he became extremely angry and turned an apoplectic shade of crimson. However, this reaction did not reveal any inability to find humor in a joke skillfully played at his own expense. Rather, it reflected the fact that he accepted the preposterous letter at face value and took it completely seriously. In short, Dr. Hart-the wise and experienced veteran-had grown so inured to the vagaries of the review process at the various journals of marketing and consumer research that he instinctively interpreted Professor Mass’ outrageous scam as a genuine example of the real thing.

Those who heard about this sad episode knew that it reflected something quite dismal about the review process in marketing and consumer research. To put this problem in its most charitable terms and speaking as one who has played on both ends of the field, it appears that, in mind and spirit, if not in body, those who offer their work like sacrificial lambs to this rapacious review process behave like masochists, while those who inflict their subsequent torment play the role of sadists. In short, it appears that those who participate in the review process that precedes publication in the fields of marketing and consumer research engage in a socially approved form of intellectual sadomasochism.

In an effort to clarify this situation and in hopes of alleviating it somewhat, this article explores both sides of the review process-the author’s and the reviewer’s perspectives — and draws on my own experiences from both vantage points to report what I believe are some common concerns felt by representatives of each viewpoint. After voicing the author’s and reviewer’s complaints, I shall offer some suggestions for prescriptions that I believe could encourage reconciliation. 

The Author and Reviewer Speak

You want to talk about some pain? One day, I was sitting in my study — not doing much, you know, just grading a pile of 123 exams — and I got one of those . . . uuh . . . .

Brilliant ideas that come to you once in a lifetime in a sudden, blinding flash of insight?

Yeah. So I thought for about two weeks about this problem that had never before been addressed, let alone solved, in the history of marketing — you know, just to see how far I could get with it. Well, pretty soon, I thought I had . . . uuh . . . .

A solution that would advance the state-of-the-art by a big quantum jump?

Right. So I took about two years designing a study, collecting some data, analyzing the results, and writing this 50-page paper reporting my findings. And then I sent this paper to . . . aah . . . .

The leading journal in the field?

Yeah. So I waited for about seven months and kept hoping that I’d hear from the journal and everything like that — you know — until, finally, when I was about to lose interest in the paper entirely, I got one of those . . . uuh . . . .

Letters from the editor saying they would reconsider the paper if you could shorten it by 50% while also satisfying 12 pages of technically incompetent and hopelessly unsympathetic criticisms from three mutually inconsistent reviews?

Right. So I took four more months to revise the manuscript and wrote an 18-page letter in which I explained how I had handled every one of the reviewers’ helpful comments and sent it back to the editor and then waited for another five months until finally I got . . . uuh. . . .

A rejection letter?

No . . . uuh . . . .

An unconditional letter of acceptance?

No . . . aah . . . .

A letter saying that the paper was much improved but still required some additional material suggested by the reviewers, that it must better explain its managerial relevance, and that it must be shortened from 25 to 15 pages before it could be reconsidered for possible publication as a brief note?

Yeah. I hate when that happens.

Tell me about it.

I hate when they do that.

I know what you mean.

 The Reviewer’s Perspective

It’s like, the other day, I was sitting around my office — not doing much, you know — just coding some data on 657 questionnaires that I collected two years ago and haven’t had time to look at yet — when I decided to get my . . . uuh . . . .

Mail that had been accumulating in the box for about 10 days because you were too busy to read it?

Yeah. So I went downstairs and got this big stack of envelopes about 16" thick and found. . . aah . . . .

Papers from four different journals, each requesting reviews with a turnaround time of less than a month?

Yeah. So I put these papers at the bottom of a gigantic pile of things I desperately needed to do and went back to coding questionnaires for about three or four more weeks until, one day, I got . . . uuh . . . .

A letter from one of the journals saying that a manuscript had been with you for over a month and that you should review it and return it immediately or you would be dropped from the editorial board?

Right. So I spent about two hours hunting for that thing and finally found it at the bottom of my pile and took the rest of the day to read it. Well, I must have underestimated my task because it was over 50 pages long, stupendously boring, filled with all sorts of funny Greek symbols that didn’t make any sense, completely incoherent, and . . . uuh . . . .

Poorly written?

Yeah. So I wrote the most polite review I possibly could, under the circumstances, and sent it back to the editor. Well, then I must have forgotten about the review because I figured I’d never see that paper again. But, practically before I’d even had a chance to start thinking about doing some of the other things I needed to finish, I got . . . aah . . . .

A revision of that same paper reduced to about 25 pages in length with very few improvements but including 18 pages of notes on why the author couldn’t make most of the changes you had asked for?

Right. And the paper was still terrible. So I wrote back, you know, and said I still think that the paper stinks but that it does have some nearly interesting aspects that might go into a really short 15-page note or something like that, you know. And then I sent that letter back to the editor and practically the next day I got. . . uuh. . .

A 15-page version of the same paper with smaller type and narrower margins?

No, not that quick. No, I got . . . uuh. . . .

A thank-you note from the editor, expressing gratitude for all the time and trouble you had spent on the paper?

No, what I got was . . . aah . . . .

Two more 50-page manuscripts to review for the same journal?

Yeah. Boy, is that painful. I hate when that happens.

Tell me about it.

I hate when they do that.

I know what you mean. 

Suggestions for Possible Conciliatory Gestures

As these two vignettes vividly illustrate, both authors and reviewers seem to encounter legitimate gripes by virtue of their participation in the review process that precedes publication in scholarly journals. Anyone who has worn both hats knows that the frustrations can be painful and almost equally distressing on both sides of the fence, though the grass usually looks greener in the other person’s yard. I claim no special expertise in these matters except by virtue of having myself played the roles of author, reviewer, and editor from time to time. Based on those sometimes frustrating experiences, I wish to offer two sets of seven suggestions that might contribute toward a partial reconciliation.

Suggestion 1. Do not hesitate to make your paper interesting. Avoid use of the dull, lifeless, empty-sounding style that some writers affect when they want to sound scientific or scholarly. Try to use active verbs, colorful language, and vivid imagery. Remove passive verbs from your prose as carefully as you would remove bits of broken egg shell from an omelet that you plan to eat or pieces of slivered glass from an ice-cream cone that you intend to lick. Up to the limits imposed by science and scholarship, try to make your writing come alive. Try to make it dance and sing. In short, strive for a more poetic and less prosaic voice.

Suggestion 2. Remember that brevity is the soul of wit. In each section, paragraph, and sentence of your paper, shun excessive length as diligently as you would avoid paying extra interest expenses on your credit card. Say exactly what you need to say, no more, and then stop. Like this.

Suggestion 3. Adjust the overall length of your paper to the magnitude of its contribution. Do not attempt to inflate a routine replication of some small study into a 50-page magnum opus in which you dutifully repeat all previous researchers’ citations of every extant reference that bears even tangentially on the topic at issue. Conversely, do not strive to condense your life’s work into a short 5-page note that just skims lightly across the surface of profound questions that you prefer not to address and, therefore, leave unresolved. In other words, match the size of your report to the importance of your story.

Suggestion 4. Don’t carve your work too thin and thereby sell yourself short. Specifically, don’t go for "twofers" by chopping one cohesive study into several pieces that you then attempt to publish separately in hopes of extending the length of your vita. For example, don’t treat the 10 chapters of your dissertation as 10 distinct working papers, each sent to a different journal. Remember that your audience will grow tired of reading about the same piece of research reported from every possible angle.

Suggestion S. Speak clearly. Don’t write in a style that sounds like mumbling. Don’t let your thoughts come tumbling out in the order in which they occur to you. Rather, impose order on them. Shape your paper into a coherent structure and do everything possible to reveal that structure to your reader through the use of an introductory preview, clear headings and subheadings, and a transparent concluding summary.

Suggestion 6. Accept the reviewers’ criticisms gracefully and behave as if every word of their comments offered a pearl of wisdom. Act in good faith by trying to fix all aspects of the paper with which they have found fault. If they disagree with each other, seek a resolution of this conflict. Forget your righteous indignation over what seems like the incompetence, insensitivity, and mutual inconsistency of the reviewers’ comments. Cultivate a posture of humility. Refuse to make suggested changes only if they are, literally, either impossible or destructive to the paper’s contribution.

Suggestion 7. Write a polite letter — as long as, but no longer than, necessary — in which you explain, point by point, how you have responded to every one of each reviewer’s helpful criticisms. Find some way to make it easy for the reviewers to relate your responses directly to their original comments, one by one. Put almost as much effort into this letter as you do into the revision itself. Know in your heart that, in making their final judgments, the reviewers will rely on the letter at least as heavily as on the paper.

 Suggestions for Reviewers

Suggestion 1. Set aside enough time to read the paper in depth and to provide a full set of comments in one sitting. Remember that if you carry the manuscript around with you on the subway, read it during halftime at the football game or in the waiting room at your dentist’s office, and then dictate your critique to a tape recorder while driving to work on the thruway, your comments will likely appear incoherent, unsympathetic, and self-contradictory. These qualities should rank high on your list of flaws to be avoided.

Suggestion 2. Struggle to find a way to make your review positive in tone. Even when you find fault with the paper, favor the use of adjectives like interesting, imaginative, and ambitious. If, by some miracle, you happen to like the paper, don’t shrink from being enthusiastic in your praise. Recall that authors hardly ever hear kind remarks about their work. The human tendency toward criticizing others dwells so strongly in our constitutions that we find it much easier to recognize flaws than to discover virtues. Try to repress that tendency. Allow yourself the pleasure of saying something nice or paying a compliment once in a while.

Suggestion 3. Avoid rejecting the basic premise or purpose of the work. Don’t decide that a whole stream of research activity is inherently worthless or unimportant. Don’t assume that just because a particular direction of inquiry does not happen to interest you, it also will fail to interest all other potential readers. Rather, try to accept the author’s stated purpose and, instead of attacking the avowed objectives of the research, concentrate on evaluating how successfully it accomplishes those goals. In other words, judge the work’s execution as a means, not its value as an end. Leave the latter issue to editors and other godlike creatures.

Suggestion 4. Ask for additional material that will strengthen the paper, not that which will merely gratify your own ego. For example, do not request the inclusion of references to your own work or your own pet theories unless these really will deepen or otherwise improve the author’s treatment of the research topic. Do not parade your own erudite command of the literature by asking for extra citations unless these will enrich an otherwise thin list of references or bolster an otherwise weak position. Conversely, do not tell the author to remove direct quotations if they really do advance the argument better than the author’s own words could, no matter how skillfully written.

Suggestion 5. Allow yourself the option of asking for no revisions in those rare cases when the law of diminishing returns would render any further work on the paper of minimal incremental value. Remember that, occasionally, someone does submit a draft that is already well-conceived, well-executed, and well-written. Do not be so arrogant as to suppose that you can improve further on such a well-conducted effort. Rather, display your good judgment by pronouncing the paper ready for publication in its present form without need for revision. Why not be the first reviewer in history to demonstrate good sense and good taste in this hitherto unprecedented manner?

Suggestion 6. When you do make extensive detailed suggestions, try to commit yourself to an essential list of necessary changes the first time around. Read the paper carefully enough on the initial try to find or at least to anticipate everything that you will eventually think needs changing. Avoid discovering more and more new suggestions on each succeeding revision in a manner that throws the review process into a never-ending cycle of revision after revision. Especially avoid this tendency where it has the effect of leading the author through a circular sequence of alterations that ultimately render the final version of the paper nearly identical to the first.

Suggestion 7. During the sometimes inevitably lengthy series of revisions, pay attention to what the other reviewers have to say. Recognize that, in trying to please another reviewer, the author may be forced to compromise somewhat in following your own inspired suggestions. Further, be a little humble. Realize, to your horror, that the other reviewers just might know more about a particular topic than you do. 

Conclusion

I believe that, if authors and reviewers would heed these two sets of seven suggestions, their mutual cooperation would foster a conciliatory atmosphere and increase the spirit of peace and harmony in the fields of marketing and consumer research. I believe that, by following my 14 prescriptions, we might better rediscover the truism that as authors, reviewers, and editors, we are really the same people and, therefore, share mutually interdependent needs that lend themselves to rapprochement. I believe that such reconciliation will help us renounce the devil of sadomasochism in the review process and that, someday, such a renunciation will permit us to stand Billy Crystal on his ear by saying . . . uuh . . . .

"Authors and reviewers can work together constructively?"

"Yeah. I like when that happens."

"Tell me about it."

"I like when they do that."

"I know what you mean."

 • Bias in peer review • Review process •

 Recent issues of Simulation & Gaming: An Interdisciplinary Journal