A comment over at writedit's unending thread on NIH grant paylines caught my attention.
Hello All,I just received my summary statement for an R15 I submitted in October and found out that I had four reviewers score my proposal. I thought that three reviewers was routine. Would anyone know why a fourth was included?
Thanks for your help.
I had the following response, slightly expanded for this context.
Any member of the panel can review any proposal (save conflicts) assigned to that panel and write a critique if they so choose. In theory you could end up with 20 or more of them!
Of course this never happens. Although I will say that it seems to me that the chances of picking up extra reviews is enhanced when your application is reviewed in a tiny Special Emphasis Panel convened to cover just a handful of applications. (I think I've had a 5-reviewer summary statement as my high water mark.)
What I have seen occur at a normal study section meeting is that someone not assigned to the application will get really involved in some point of discussion (pro or con, people, pro OR con). The system recognizes that the summary statement is supposed to reflect the discussion. Revision of the critique to more accurately depict the discussion is highly encouraged.
So sometimes the SRO or Chair or a panel member will say "Hey, will you 1) write that up as a critique; 2) write up a brief blurb for the SRO to include when prepping the resume or 3) write up a point for one of the other reviewers to include when they are editing their review?"
Or, occasionally, a reviewer will just be so ticked about where the assigned reviewers and panel votes went that s/he will just write up a dissenting view on his/her own hook. I'd suggest that if you see a fourth, highly disparate review, perhaps focused on a single issue, that you recognize that this probably had very little effect on your score... Nice to get the warm fuzzies if the outlier loved your proposal, I suppose.
One final scenario may also be at work. The way I understand it* the official rules require that at least two people assigned to the application have to be in the room if it is to be discussed. So if for some reason 2 of the 3 are phone reviewers, the SRO may tap a 4th "discussant" reviewer to fulfill this requirement.
*it is possible that this is only a strongly encouraged practice and/or a practice limited to selected study sections. I'm not easily finding instructions-to-SROs on the CSR website.
Education
Jobs
Comments
When I was on study section, when the SRO thought there might be controversy with a proposal, she would peremptorily assign a fourth reviewer. This happened about 1 in 20-25 proposals. Part of the message here is: Do not underestimate the power of the SRO. In assigning reviewers he/she has a lot of sway on the outcome.
Posted by: arrzey | March 4, 2010 1:07 PM
Do not underestimate the power of the SRO
I suspected this from the first time a revised app of mine seemed to be mysteriously missing the clearly outlying third reviewer (It's always Reviewer #3!).
When I sat on the same section for a few rounds...whew!
Now, SROs are very professional and dedicated to fairness in my experience. But boyoboyoboy. If one wanted to wield power, s/he could have a huge effect. It would take very little thinking whatsoever for an SRO to match what they know about a given reviewer to a given app to significantly shift the outcome.
So, you know, if you postdocs are thinking that the PO job is more high falutin' than the SRO job....in terms of real and immediate impact on the conduct of NIH funded science you are better off going SRO.
just sayin'
Posted by: DrugMonkey | March 4, 2010 1:39 PM
SRO has the most influence how your application is reviewed. SRO in fact is "the" person who can truly tell you (though they are not allowed) what went wrong with the review. Your assigned PO seldom is the one that sits there and writes notes on your grant. So when you talk to PO (the only person you are allowed to talk about what went wrong with your application" for most part, is just reading notes taken by other POs. So unless your assigned PO is actually there, you really not not getting any useful information than what is written by the reviewers. Good SRO can summarizes the sentiment well so you can get some good information, but there is only so much room for them to write a paragraph. Besides, SRO has been pushed so hard to crank up the summary statement nowadays with the new rule, it is hard for them to spend that much time to mull over each application like the older days. PO on the other hand can flex muscle when it comes to "ordering" with the new system (since there will be many applications with equal scoring). PO also can makes a huge difference in picking out special consideration grant for council. Bottom line: it is always wise to treat these guys with respect.
BTY..many these problems stem from the gigantic change in the review process ALL AT ONCE. There is no time to get well adjust to one change before the other comes in. Scientific community tend to support changes that sound good at the first suggestion because it increases turn around time and it shortens reading burden by the reviews, it shortens the blah blah on the critique sheet, and so on. What people seem to overlook is the underlying effect on the overall quality of review because of time restriction, page restriction, and so on. These unfortunate side effects do not affect the best and the worst of applicants, but do tremendous disservice for the middle pot... majority of the scientists. I can not wait to find out how fantastic the 12 page application work in the next few rounds (with only one resubmission allowed). I would also like to learn the trick in turing a near miss A1 into a "brand new" grant for next submission.
Posted by: JAT | March 4, 2010 2:28 PM
I would also like to learn the trick in turing a near miss A1 into a "brand new" grant for next submission
You'd better learn it. Or you aren't going to get a grant funded, lemme tell you.
Posted by: Cashmoney | March 4, 2010 2:32 PM
It's sort of entertaining to see the disgruntlement fly when summary statements come out. However, I am not highly entertained by my own pink sheet and need some DrugMonkey wisdom. Reviewers #2 and #3 were very enthusiastic, but wanted more of a particular kind of preliminary data, which we can certainly work on. However, the primary reviewer obviously was unhappy with many aspects of the grant and went on at length. Really. At lengthy length. But I can live with that.
What I'm struggling with is several opinionated comments from R #1 that were listed as "weaknesses" where this person completely misrepresented our hypothesis. I was extraordinarily careful to stay away from making certain statements, which this reviewer went ahead and assumed I made anyway and strongly disagreed with that (I would disagree too if I had actually said those things). So....is it worth asking the SRO not to assign this reviewer in the next round? I have never done such a thing, and again am not overly concerned with all the criticism, but rather with the "artistic license" that was used to provide interpretations that we ourselves carefully did not make. Clearly I will have to do a better job next time around, not just to say "this" but to explicitly state that "I am NOT saying that". Still.....Grrrr.
Posted by: BugDoc | March 4, 2010 3:16 PM
@JAT In my experience, if one is polite and not pushy one can talk with the SRO. My advice for approaching either the SRO or the PO is to send them an email, include the title & number of the proposal in the subject asking them if "it would be possible for you to call them". Yes it is groveling, humiliating, etc etc, but you might get something very useful for changing the grant. Never call without an invite from the SRO (and they do say no). Also, never call without having seen your pink sheets and thought about them. Read them twice, get someone else to look at them with you. When you talk with NIH staff say something along the lines of "I'm trying to improve my grant and would appreciate any insight you can give me for the rewrite". If you have a specific question, make sure it is phrased in a non-angry tone of voice. "Do you think that the state ment XYZ in the review means I should umpty-ump more experiments?" Do not argue with them, listen carefully and take notes. Thank them copiously. Getting any of these people angry will not help you get funded now or in the future. Remember these people get yelled a lot. They don't like it any more than the rest of us.
Posted by: arrzey | March 4, 2010 3:17 PM
BugDoc - you definitely can ask the SRA (in the cover letter) for reassigning the application. If you justify this request well, they will reassign.
This touches another subject though - why the NIH system allows a reviewer to misrepresent someone's proposal with total impunity... This happens more often than you think.
Posted by: Gummibears | March 4, 2010 4:15 PM
arrzey, yes; Gummibears, no.
I wouldn't put anything in the cover letter about reviewers. That's for requesting a study section and an IC for possible funding in my view.
BugDoc: two bits here. The total misrepresenting of your hypothesis needs to be covered in the standard revision process. Use the comments and your local friends to see where you can re-word or re-state your application to make it emphatically clear what you do indeed mean. (try not to fall into the trap of concentrating too much on rebutting that particular misinterpretation with a lot of "and we are not stating XYZ here, for any boneheaded reviewers on the panel")
You also seem to be touching a little bit on one of my bugaboos about grant review. Which is the substitution of judgment about empirical prediction (i.e., "I hypothesize something different from your hypothesis") for judgment about the proposal's ability to generate an empirical test (i.e., "Your hypothesis is reasonably supported by the lit and data but your approach cannot provide a meaningful test of this hypothesis") If this is what is going on, I really have no solution other that to slip in as many comments as you can to the effect that even if your hypothesis is wrong, the experiments will give good interpretable falsification anyway, lead to blah, blah, possibly support the alternative hypothesis, etc.
Posted by: DrugMonkey | March 4, 2010 5:16 PM
A couple of times I got interested by the title/PI of a non-assigned grant, intended to just take a quick look, gotten drawn in to reading an entire grant, and ended up writing a review. Then you just e-mail the SRO and say, "Dude, I just read this grant and wrote a fucking review", and she'll be all like, "Coolio! I'll assign you as a discussant (used to be called "tertiary reviewer")." And then you just go to IAR and upload your review.
Posted by: Comrade PhysioProf | March 4, 2010 8:09 PM