[See Update Below]
Stan Liebowitz has posted a reply to Oberholzer-Gee/Strumpf’s (O/S) referee report/ reply to Liebowitz’s original comment submitted and rejected by the JPE for publication (got all that?) (HT: Newmark and Peter). Stan includes email exchanges between himself and OS concerning access to the data (O/S did not allow access), copies of the rejection letter from Steve Levitt, and responds to the O/S reply on the merits. Whatever one thinks about the underlying merits of the econometric debate or the appropriateness of using authors of the primary paper as referees for a comment (this doesn’t bother at all, actually), it does strike me as quite odd — to say the least — that Liebowitz’s comment was apparently rejected from a journal with a data policy which generally requires that underlying data be will be made “readily available to any researcher for purposes of replication” primarily because Liebowitz did not respond to data he asked for but did not receive from O/S.
Here is the abstract:
Through a stroke of luck, a referee report in the review process at the JPE has been positively identified as the Oberholzer-Gee/Strumpf (O/S) response to my earlier comment. Regardless of the response’s provenance, what counts is whether it solidly refuted my comment. This ‘sequel’ analyzes the O/S response. The O/S response only deals with four of the nine points discussed in my comment, leaving the five remaining critiques unchallenged. The conclusion of my review is that the O/S response fails as a defense of these four points and contains many of the same types of errors that marred their original paper. This sequel also discusses the history of this dispute including O/S’ various reasons for not making their data available. Finally, this sequel provides full documentation on the JPE’s decision not to publish the comment.
UPDATE: Stan emailed me the following making the case that the use of original authors as anonymous referees for comments should bother me more than it does.
I realize that asking the authors to respond to a comment in the guise of a referee report is apparently not that unusual, but I do not think it is a good practice. I have two reasons for this view.
First, the reason for anonymity for a referee is to allow the referee to feel free say what he wants without a fear of retribution that might otherwise limit what he says. I agree with this logic for actual referees. I don’t think it should apply to responses for comments however. The author of the comment is known to the authors of the paper being commented upon and vice-versa. The original authors already have every incentive to defend their work as strongly as possible and thus don’t seem to need to be cloaked behind anonymity. If anything, I think that keeping their identity clear forces them to make statements which they are willing to stand behind. Allowing them anonymity, in this case, allows them to say things that they otherwise might not want attributed to them—i.e., to overstate their case.
Second, the author of the comment will also take what is said in a response differently than he would if it were a report from a neutral third party. If a neutral party claims you have made a major error and that the original authors are correct, you will take it more seriously than if the original authors say it. It is important to know that a report is biased so you can properly parse what it is saying. Given that the original authors are likely to overstate their case, particularly if their identity is cloaked, reading such a report as if it were from a neutral third party might leave some authors too distraught to properly continue with their comment.
Authors of the original paper are supposed to write a response, in their own name. That has been the norm, even if it is not always followed. And in my opinion it should be the norm.
