I've been meaning to pick up on a comment made by a reader over at writedit's epic thread on NIH paylines, scores and whatall. (If you want to swap war stories and score/IC payline grumbling, that is the hot place in town.) The guy was ticked off about a recent review he received and had a question:
I am an establishe investigator. I subnitted a competing renewal ... I got a score of 40 (37 percentile). I was very shocked and dissapointed to find out that my application had a preliminary score of 2.7 (which would have been fundable) but it seems one negative reviewer carried the day, and convinced others to pull down the score. I have not yet seen the comments, but if the comments have factual errors, especially from the negative errors, can I appeal the review and request a re-review?
Recently, as luck would have it, a loyal reader of the blog submitted the following scores, received on the review of her R01 grant proposal. Under the new scoring procedures in place since last June, these are scores which each reviewer suggests for criteria of Significance, Investigator, Innovation, Approach and Environment. I may have slightly re-ordered specific scores for concealment purposes but this is essentially the flavor.
rev#1: 2,1,1,1,1
rev#2: 2,2,3,3,1
rev#3: 3,2,5,4,2
It really is always Reviewer #3, isn't it?
Although we have more detail in the second case, let us credit the first person's description of events, leading to more-or-less equivalent scenarios. The appearance that two of three reviewers loved the proposal a whole lot and the third managed to torpedo it.
My first response to the specific scores would be "Congratulations! You must have written a pretty good proposal!" if you managed to get someone throwing down the 1 scores with a 2 tossed out so they don't look like a total homer- that's good stuff. You have an advocate like that pulling for your app and it is hard to make the case you got a raw deal. The way I'm looking at scores these days, the 2s and 3s of the next-fondest reviewer are pretty schweet too. This is my point about the strong advocate- it could be that this next-fondest person is just looking for a reason to improve the scores. After all, he/she could be sitting on an app that for random reasons he/she felt was better and was trying to spread the scores out. Absent a strong advocate, you are stuck with 2-3 range. If another reviewer is pressing for better, you might just get two pulling toward the 1-2 range. That, in my current understanding, being the entry card for a shot at a fundable score.
But...dum..da..dum..dum what's up with that third reviewer?
4th, 3rd, tomato tomahtoCalm down, calm down. Those scores aren't all that bad really. The allowable range goes up to 9 and there are only 1-2 pt gaps across the ordered reviewers. So the score disparity isn't huge. Functionally, these scores can mean all the difference in the world of course. Unless the odd reviewer finds that s/he just completely misread the app in some particular, no way s/he is going to be talked down below a 3 on those scores.
You will note at this point that I am assuming that the preliminary overall scores are somewhat related to the individual scores. There is not supposed to be any specific relationship, which is a topic of rant-inducing proportion for another day. Suffice it to say that to a first approximation I am comfortable making the leap. So the initial preliminary overall score identified by these reviewers were probably 1 or 2 (2 unlikely), 2 or 3 (about equi-likely) and a 3 or 4. (Here I am making the further assumption that the scores were not substantially edited after the meeting.) What are the possible post-discussion scenarios and voting outcomes?
Well, it could have stayed similar to initial. Perhaps the advocate talked the middle one down so you ended up with 1, 2, 4 or even 1, 1, 4. Maybe the middle or bad one talked the advocate up. So it was 2,3,4 or worse. And then we have to make assumptions about which reviewer was most convincing to the mean of the panel itself. Some 20 more reviewers would be voting, typically within the post-discussion range. Did they lean to the good side, minimizing the contribution of the detested Reviewer #3? Or did they lean towards spiking the app? Were they split?
Getting back to the comment waaaay up at the top of the post, it is generally ridiculous to claim that one reviewer ruined your chances of funding in a way that is unfair or shows that the system is broken. After all, I don't ever (and I mean ever) hear anyone claiming the system is broken or screaming about appeal because of receiving an outlying score in the favorable direction.
Since you've been so patient, the reader was kind enough to relate the app ended up with a 26 priority score (i.e., the vote averaged 2.6). So looks like the panel voted smack dab in the middle of the range identified in the reviewer's original critiques. But so what? If it had ended up toward a 3.5 or so, we would only conclude that the "bad" reviewer was convincing to a whole panel of people. That is no flaw in the system. And if it had trended more toward a 2.1 or so? Everyone would be jumping around high-fiving each other. Except that one outlying reviewer perhaps. But s/he shouldn't be grousing about the system either.
Final thought on reading the tea leaves. There will be additional clues in the "resume" section of the summary statement if your SRO is any good. Best case scenario is that a single issue, maybe two, identifies the core problem. Worst case, there are several things mentioned without a lot of clarity. But this probably means that the panel did not coalesce into a single viewpoint. Sometimes this difference can help you decide what to spend the most time on during your revision process.
Education
Jobs
Comments
I have been the victim of "the third reviewer" but on one occasion I was that third reviewer. In that case, the first two reviewers felt the proposal was pretty good. Maybe not ready for prime time, but pretty good. The problem was they were doing something that violated the standard of care in a vulnerable population. Bringing that up, even as the third reviewer, pretty much trashes anything good about the proposal.
Of course, when a reviewer did unto me recently it was completely unwarranted- no fatal flaw, just a difference of opinion of what factor might be most important in the experimental system. Less impressive than an ethics violation, but enough to blast the score to unfundable in the current climate.
Peer review is not a good system; it's just better than any other we have come up with to date.
Posted by: Pascale | March 10, 2010 9:35 PM
In my recent study section experience, I WAS the dreaded reviewer #3. Others were giving the proposal a 1, but I stuck to my guns at a 5, because despite the high quality of the investigators and even of the project itself, it just was not responsive to the RFA. I felt really strongly about that and felt I needed to explain why to the other reviewers who would be voting. I don't know how the final aggregate score turned out, but the whole room was pretty divided. I also think the main reason that the people who liked it were so enthusiastic was some unconscious biasing towards the glamour of the investigator and institution, being swept away by the snazzy proposal, and not seeing that it didn't fit the fundamentals of the particular funding mechanism.
Posted by: Arlenna | March 10, 2010 9:39 PM
I don't ever (and I mean ever) hear anyone claiming the system is broken or screaming about appeal because of receiving an outlying score in the favorable direction.
This is silly. No grant ever gets funded because of one outlier score in the favorable direction, but grants go unfunded all the time because of one outlier score in the unfavorable direction.
Posted by: whimple | March 10, 2010 11:39 PM
Whimple is correct. The problem is that the ridiculously low paylines cannot tolerate a single negative review, no matter how idiosyncratic.
I was recently in the position of the positive "reviewer 1" on Study Section, and I was infuriated with "reviewer 3," who was really a bit of a crank and was trying to spike a great proposal from a promising young investigator.
I was tempted to revise my score to a 1 (from a 2), just to counteract reviewer 3's score, but I thought that would be wrong. Have you ever done that?
I made my advocate's case as calmly and as factually as I could, but I'm really not sure how the rest of the panel ended up -- this was a phone review, so I couldn't read the body language. I guess I will have to check Project Reporter in a few months to see how it all turned out.
Posted by: Neuro-conservative | March 11, 2010 12:33 AM
What? Of course you should have given more range to counter the reviewer you thought was off the mark. Where's the problem? The whole point is that you are trying to persuade the panel to vote a certain outcome. The preliminary and post discussion scores are merely part of the vocabulary...
Posted by: DrugMonkey | March 11, 2010 1:01 AM
The problem was I couldn't honestly say that the application met the NIH criteria for a 1. I also was (perhaps overly) self-conscious about getting into a pissing match with reviewer 3. Instead, I just tried to keep the focus on the merits of the proposal.
Posted by: Neuro-conservative | March 11, 2010 1:33 AM
Oh, well, sometimes being all reasonable makes the crank look like an axegrinder by contrast and so your view wins out anyway.
If you are talking about guidelines to save the 1 score for once in a lifetime greatness, that is utter nonsense. When it takes a panel vote of close to 2.0 to secure funding, blocking of the 1 is just stupid. Range compression stinks.
Posted by: DrugMonkey | March 11, 2010 2:03 AM
Yeah, I have really struggled with the 1 score ever since the new system went into effect.
I agree that it can't be reserved for Nobel-level genius, but I also am trying to avoid gaming or overthinking the new system. Part of the intention of the new system was to overcome the tricks that reviewers had learned over the years. (Of course this is futile in an age of 10% paylines.)
Posted by: Neuro-conservative | March 11, 2010 2:15 AM
Folks,
The overwhelming problem with peer review right now is the payline. That 10% is suicidal not only for science but also for peer review itself.
Would it not be worthy and more productive right now to start a discussion proposing practical ways to NIH to increase the payline to 20% and request institutes to set up an additional 3-4% aside for disputed reviews (i.e. do not reach the payline).
Could we possibly come up with practical ways to restructure or reorganize the budget as to achieve the goal in 6-months (pilot). if those ideas are doable then it might be a question of going to Congress and request that they approve that change and commit ourselves to go back and show the results in a reasonable period of time.
Thinking about appeal is just impractical and a waste of time. I have had that experience.
Posted by: anonymous | March 11, 2010 3:40 AM
The problem is not that third review. There have been third reviews since the dawn of time. Its the funding level. 10%, however, is not suicidal for science. NSF paylines in life sciences have been
Posted by: arrzey | March 11, 2010 8:54 AM
sorry ... post cut off mid-stream.
The problem is not that third review. There have been third reviews since the dawn of time. Its the funding level. 10%, however, is not suicidal for science. NSF paylines in life sciences have been 10% for at least 30 years, and gosh, evolution, ecology, comparative physiology are still ongoing endevours. Yes, there are far fewer scientists in these fields, and they don't get paid like NIH-funded basic scientists. Hyperbole isn't going to persuade anyone. The concern is that the survival/washout point won't be graduate school, but in very good junior people who are struggling right now to get funded. There are many things to do to help such folks RIGHT NOW. Besides being a national advocate for science funding (what if congress critters got more letters about NIH than say, flag burning?), in my view the most important thing that mid to senior people can do is foster careers. Help the juniors. Read their grants, be on their grants (if possible), share lab space, resources and advocate advocate advocate for them at your university. If you're junior, go find a mentor. Not necessarily in your subfield - just a senior faculty who can talk to you about what to do and how to do it.
Posted by: arrzey | March 11, 2010 8:55 AM
sorry ... post cut off mid-stream.
The problem is not that third review. There have been third reviews since the dawn of time. Its the funding level. 10%, however, is not suicidal for science. NSF paylines in life sciences have been 10% for at least 30 years, and gosh, evolution, ecology, comparative physiology are still ongoing endevours. Yes, there are far fewer scientists in these fields, and they don't get paid like NIH-funded basic scientists. Hyperbole isn't going to persuade anyone. The concern is that the survival/washout point won't be graduate school, but in very good junior people who are struggling right now to get funded. There are many things to do to help such folks RIGHT NOW. Besides being a national advocate for science funding (what if congress critters got more letters about NIH than say, flag burning?), in my view the most important thing that mid to senior people can do is foster careers. Help the juniors. Read their grants, be on their grants (if possible), share lab space, resources and advocate advocate advocate for them at your university. If you're junior, go find a mentor. Not necessarily in your subfield - just a senior faculty who can talk to you about what to do and how to do it.
Posted by: arrzey | March 11, 2010 9:02 AM
#11 & #12
***In my view the most important thing that mid to senior people can do is foster careers. If you're junior, go find a mentor. Not necessarily in your subfield - just a senior faculty who can talk to you about what to do and how to do it.***
Very good advice. Except that in practical terms, those excellent mentors, outside my field and my department, are themselves struggling to survive ( while doing excellence science).
I am not familiar with NSF; in biomedical sciences (with a 10% payline) is very hard to survive both for junior and less junior investigators (mid and senior)
Posted by: michelle | March 11, 2010 10:11 AM
OK, I'm going to use this post as an opportunity to vent. The example you gave didn't seem that outlier-ish to me. On a recent review, I received all 1s and 2s from 2 reviewers and "reviewer #3" gave me all 5s except for an 8! on one criteria. (An 8 seems like the equivalent of, "This scientist can't figure out which end of a test tube should be on top.") The final score given by the review section was slightly better than the average of the reviewer scores, but, yes, it's remarkable how different people can have such radically different reactions to a proposal.
Posted by: Animatronic | March 11, 2010 10:13 AM
Obviously, reviewer # 3 does not like you, no matter how many improvements you might make. You're better off requesting your SRO to consider eliminating that anonymous reviewer (for your review) in the next round.
I can't find any other explanation to "such radically different reactions". I agree that it is very discouraging.
Posted by: marlene | March 11, 2010 10:50 AM
I am going to make a radical suggestion to improve the payline: Decrease indirects! Indirects at my institution are almost 50% of the direct costs, and these are negotiated at higher levels at more prominent unis. Making a fixed rate (25-30% perhaps) would increase the amount of funding available for the research!
Of course, universities will not support this plan, since it takes away a major source of relatively unencumbered funds. I am not convinced that institutions are using these funds to support the research infrastructure as intended, though. Frankly, this seems like the easiest way to keep reviewers from having to decided "levels of excellence" in grant proposals.
Posted by: Pascale | March 11, 2010 10:59 AM
Absolutely! SROs and chairs have been explicitly instructed to disabuse study section members of the idea that a 1 in the new system is like a 1.0 in the old system. It is not intended to be that way, and there should be a *lot* more 1s assigned in the new system than 1.0s in the old.
The way I see it is that in the old system, no one *ever* got a 1.0. In the new system, the best grant at each round of study section should get a 10.
Posted by: Comrade PhysioProf | March 11, 2010 11:12 AM
Are universities supposed to dictate how and why taxpayers money are used or misused in order to ensure the future of biomedical research or science in general in the US ?.
Universities can certainly dictate how to pay their administrators and look for the money elsewhere, not at NIH !!!!.
Posted by: anonymous | March 11, 2010 11:15 AM
Well the Federal auditors are "convinced" Pascale. Have you ever sat down and really tried to grasp what it costs to do the research you do? From buying the land, to capital expense on building .....down to the janitorial service. ALL costs?
Someone has to pay the bill. States are in bad shape right now. Market is down so moneybags endowments can't make it up either.
I hear this cry about indirects a lot but I find it rare that someone has priced out starting their own dinky little research institute (apart from those who actually do it, I've seen several in my location) or even SBIR funded business. Once you've totted up the numbers (and I have, as it happens) those indirects don't look so bad.
Posted by: Cashmoney | March 11, 2010 11:18 AM
Well... so the PI with multiple grants should be pulling the same indirects on all those grants? Same building, same space, same janitorial staff, same lighting, same administrative parasites... They could just pay indirects on the one largest grant per PI to cover all that.
Posted by: whimple | March 11, 2010 11:27 AM
Cashmoney,
How many times do you buy the land to build a university, particularly the big unis in the country?. I think that Federal auditors should have a harder look at the real cost that Pascale's and other scientist's research cost their universities.
And I agree with wimple in the comment about multiple grants PIs ?. Why do you think multiple grants PI are so adored by the administrators?. Money talks and should talk. But it should talk for the right reasons.
Posted by: Cheapskate | March 11, 2010 12:32 PM
I've seen some "reviewer 3" scores because the reviewer is stuck on the definitions assigned to the numbers. People are still getting used to the new scoring system, and a reviewer may pull out his list and say the application is "very strong with only some minor weaknesses. That's a 3."
Also, I've noticed a trend of panel members scoring with higher numbers in the allowed range for the applications they did not review. Are they protecting their own field? Is this an indirect way of championing the application they like? Anyway, it means that if one of the reviewers sticks with a high score, more of the other panel members may trend toward that high score (without having to explain why).
Posted by: Joe | March 11, 2010 12:47 PM
And yet, CPP, and yet. I had a Chair explicitly tell the panel that a CSR Chair's instruction session said the 1.0 should be reserved for the very unusual application. This seemed to contrast with all the use-the-range instruction but what can you do? The Chair's role is to get the panel all on the same page and unless someone else was at the meeting to provide a different view, you are stuck.
Posted by: bikemonkey | March 11, 2010 1:08 PM
I agree that cutting the budgets of all grants by even a small amount could result in better paylines. For the most part, I think most labs would easily adjust to a 15-20% cut. This might raise more complaints from folks with purely soft money positions at places with ridiculously high indirect cost rates. For example, I was looking at indirect rates (using the tool that DM posted a while back in his blog) for places such as the Scripps Institute (86%) or the Salk Institute (92% !!!) in California and was really surprised that they were that high, or that NIH actually agreed to negotiate such a rate. I think smaller grants across the board and reasonable indirect costs would allow NIH to spread the wealth a little more, and if a PI can make the case that she needs a larger budget to perform the research, then this can be applied for in the form of supplements.
That being said, does anyone know why the indirect costs at those places are so high?
Posted by: Namnezia | March 11, 2010 1:17 PM
Namnezia, what makes you think that reviewers and POs are unaware of the overhead rates when they are making their decisions? That info is right there in the application. Second hand, but I have had a colleague claim to me that an IC Director stated directly that when it came to spending out the end-of-the-year budget, they were looking for applications with favorable indirect rates.
Posted by: Cashmoney | March 11, 2010 2:06 PM
Cashmoney - That's not my point, I know that reviewers know the indirect rates. What I'm saying is that if they limited the indirect cost rate to something more reasonable, say 50%, and slashed grants across the board by 15-20%, then there would be more money left to fund a larger number of grants and increase the payline. Also, in terms of your comment about spending out the budget, rather than looking for the most expensive grants, they could use this to pick up a few more ones which were in limbo. Or to provide supplements to those whose budget was slashed and claim that they really need the extra cash to complete the project. I haven a couple of Canadian colleagues, and they seem to be having no problems getting grants, because the grants are smaller and the paylines are higher. I also see no decrease in the quality of their science compared to people in the US, so the argument that lower budgets would lead to lower productivity and lower quality of science wouldn't hold, I think.
Posted by: Namnezia | March 11, 2010 2:15 PM
A key advantage of reforming the indirects structure is to reduce the incentive of institutions to create more positions. Supply & demand... if you just increase the supply ($) the demand (applicants) will increase just as much or more so paylines aren't any better off. There needs to be an effective way to cap the demand.
Posted by: whimple | March 11, 2010 2:39 PM
RE: Nobel-level genius and scoring a 1. (or a 7 in the antipodean system).
Warren and Marshall apparently couldn't get the Australian RO1-equivalent for their research which did end up winning a Nobel. So what's the point reserving the 1.0 for something you can't recognise ahead of time anyway?
Posted by: antipodean | March 11, 2010 4:27 PM
Namnezia,
I went to the Research Crossroads and did the search for "biomedical research" and "foundation" and just plain "research". Even though it only gives you a 10 hit return per search, there are quite a few non University research institutions getting 80-99% (boston biomedical!) indirect cost rates. A bunch more at the 60-70% level.
You are suggesting the NIH should put all of these out of business, correct? Politics aside, can you live with the scientific impact of that?
Posted by: Cashmoney | March 11, 2010 4:31 PM
Cashmoney,
No one would want to put any research or academic institution out of business. But excuse me, 60-70 % and 80-99 % indirect cost rates are exorbitant and unsustainable. How do you justify that ?.
Posted by: Cheapskate | March 11, 2010 4:43 PM
Cheapskate
I know of places with more than 100%. I don't know how they justify it.
Posted by: antipodean | March 11, 2010 5:46 PM
I think that a research institution that needs near 100% overhead probably has an unsustainable business model. I'm not suggesting they be put out of business, but maybe NIH should force them to revise their rates. Maybe such a high overhead rate is designed so that the institute can keep growing at a high pace, but this might not be the best idea. I don't know.
But say you compare some place like Cold Spring Harbor Lab (~65%) to Salk (~92%), both places are non-university institutes, of similar size and caliber. Why is one so much higher? Or is it that somehow CA charges very high taxes on non-profits and that is why overhead rates at Scripps and Salk are so high? Again, I don't know.
Posted by: Namnezia | March 11, 2010 5:53 PM
'Or is it that somehow CA charges very high taxes on non-profits and that is why overhead rates at Scripps and Salk are so high? Again, I don't know.'
I GUESS THAT FEDERAL AUDITORS SHOULD KNOW ABOUT THAT AND NIH HAS AN OBLIGATION TO DEAL WITH IT. THOSE BUSINESS MODELS ARE NOT SUSTAINABLE.
Posted by: cheapskate | March 11, 2010 7:00 PM
I have no idea what goes into the Federal negotiation but I'm pretty damn sure you don't either. So given that, 56% is as absurd as 62% is as absurd as 99%.
My point is that I've done some calculating for my own operation (which is by no means bizarre or atypical) and the direct costs don't even come close to covering the real costs. 55% overhead doesn't either! So somebody is footing the bill. That could be the state contribution to the University. It could be a hefty endowment income. Or it could be the Federal government through eye popping overhead rates.
Unsustainable business model? Sure. They all are if you propose taking away that extra contribution. Think it can't happen? How are those University of State budgets looking where you are these days? How about those investment streams at Harvard and Yale? What about Universities like Florida State (?) going down the path of folding up departments to do the unthinkable and lay off tenured faculty?
So, are you comfortable putting X type of NIH funded research institution out of business? If so, are you going to whine when the cream of the scientific talent descends on your type of institution and outcompetes you for jobs, tenure and local space resources? I don't want those mofos looking for jobs at my institution!
Posted by: Cashmoney | March 11, 2010 7:01 PM
Do it to Julia!, man. NIMBY!
Posted by: bikemonkey | March 11, 2010 7:04 PM
With sustained 10% paylines, all those research institutions are going out of business anyway.
Posted by: whimple | March 11, 2010 7:07 PM
No bikemonkey and cashmoney - that's not my point. As I said, if I was told that all grants are being cut by 20% across the NIH in order to be able to fund a larger number of proposals, I would not complain. Yes, I would miss that extra 20%, but my lab would adjust (as would everyone else), plus this policy would benefit me in the future when I try and renew my grants.
What's better, more funded labs with smaller grants, or fewer labs funded with bigger, fatter grants? You, yourself, or rather DM, has said that after a given point, increased funding does not scale with increased productivity.
Posted by: Namnezia | March 11, 2010 8:29 PM
Wait, Namnezia. The NIH already does apply across-the-board reductions. It varies from time to time and from IC to IC but 10% cuts, 1 module cuts etc come down all the time. And I've just been seeing reference to one year reductions being common for some IC or other recently.
What am I missing here?
Posted by: DrugMonkey | March 11, 2010 9:02 PM
(1) Anonymous: "Would it not be worthy and more productive right now to start a discussion proposing practical ways to NIH to increase the payline to 20%
That's easy. Cap the overheads at 25%.
Posted by: Gummibears | March 11, 2010 9:07 PM
DM - I guess what I am suggesting is that to make these cuts policy, say for example limit R01s to 4 modules. That way no one is surprised when cuts come, and it frees up some cash.
And yes, these cuts do come down all the time - my R01 was cut by a year. But I ain't complainin' (too much) since it's better than having no R01.
As wimple said, I think a 10% payline is not sustainable in the long run. Something's gotta give.
Posted by: Namnezia | March 11, 2010 9:15 PM
Cashmoney: "You are suggesting the NIH should put all of these out of business, correct?"
The Parkinson's Law says: The demand upon a resource tends to expand to match the supply of the resource.
Charging a 100% overhead does not mean that an institution would be put out of business without it. Such exuberant overheads are requested simply because NIH agrees to pay these high rates.
Posted by: Gummibears | March 11, 2010 9:28 PM
Gummibears
If NIH or equiv didn't pay it the institution will simply tax your directs to get it back
Posted by: antipodean | March 11, 2010 11:11 PM
Antipodean,
There is no "if" - the current practice of charging the overhead is nothing else than taxation of the direct costs, and the rates are shamelessly high at that. I see no problem with enforcing the cap - the institutions would be basically told "take it, or lose it". Alternatively, they should be allowed to charge services as direct costs, instead of accepting the fixed overhead. Direct costs, however, need to be justified and cannot be ridiculous.
Posted by: Gummibears | March 12, 2010 12:47 AM
Has Francis Collins said anything on this issue, or on other large structural issues?
Posted by: Neuro-conservative | March 12, 2010 1:33 AM
If Francis Collins has not said anything on the issue, we should ask him. He sounds like a very reasonable guy wanting the best for biomedical researchers and their institutions.
The best for biomedical researchers should not be in opposition to what is best for their institutions. After all the best assets of institutions are their people and their potential contribution to society.
Posted by: anonymous | March 12, 2010 2:50 AM
Anonymous: "The best for biomedical researchers should not be in opposition to what is best for their institutions."
Currently it is in opposition. The scientific community has long time ago devised the system of peer review as the means to secure the quality of research. Now, for every million dollars appropriated within the peer review system, another million dollars bypasses this system. This not only misappropriates the available funds, but also corrupts the peer review system itself (this is natural when the success rates are around 10%).
Posted by: Gummibears | March 12, 2010 9:45 AM
It's been clear for a long long time that the system is in fact 'broken'; by which I mean that it is being forced to do something that it is not really intended to do. As I imagine it, peer review is designed to weed out poor proposals. It's not really good at distinguishing between effectively equally good proposals - which is what happens when all three reviewers are scoring high, but incredibly slight differences make or break funding levels. The real problem is the complete disconnect between university hiring policies and total available federal funding. Unless this problem can be solved (which i suspect it cannot) there will typically be way too many people in the applicant pool, overwhelming the system to point where statistically insignificant differences in score are lethal. As long as enough young scientists are willing to accept this situation, it will continue. I think it excessively rewards tenacity over creativity myself. N.B. I am now a Canadian researcher, but CIHR suffers similar problems (the NIH equivalent up here).
Posted by: mark | March 12, 2010 2:09 PM