Effect Measure

In grantee parlance, the Study Section (also called a Scientific Review Group) is a committee of external scientists who review grant proposals and meet to discuss and grade them. There are many regular Study Sections, corresponding to particular scientific specialties in the 19 institutes that make-up the National Institutes of Health (for example, the "cancer institute," the "heart institute," the "infectious disease institute," each of which have longer official names). There are also one-off or ad hoc Study Sections that meet once for special purposes.

If you are insanely curious you can find a full alphabetical list of regular Study Sections, the areas they cover and their rosters, here. If you are over-the-top insanely curious, you can watch this long NIH video that goes through the mechanics in excruciating (but very realistic) detail, showing you what really happens in a Study Section meeting (note that the scoring system has just changed but the process hasn't). There are different kinds of NIH grants, including investigator-initiated research grants (called "RO1s") as well as different kinds of training grants and large program or center grants. I am working on a large program grant and used to sit on a Study Section that reviewed these kinds of grants as well as Center grants and Training grants (the grant I'm writing, though, wiill be reviewed by an ad hoc Study Section). To get the flavor of what a Study Section does and what it's like you only need to watch the video to about the half way point, where they complete the discussion of a typical RO1.

If you don't have the patience to watch it, here's a brief synopsis. Grant proposals are sent in advance to Study Section members and specific people are assigned to write written reviews and be prepared to lead a discussion of designated grants. There may be as many as 4 reviewers for a grant, but 2 or 3 is more typical (note that most Study Sections are much larger than shown on the video, usually 20 or 30 members). After advance review at his or her home base, each grant receives a preliminary score by the reviewer and when the grant comes up at the meeting, each of the reviewers announces his or her score. There is a primary reviewer who starts the discussion by summarizing the basis for the preliminary score, followed by a secondary, etc. At that point the discussion is opened to all participants (most of whom have not read the proposal but have listened to the summary) and there is often a lively back and forth. The video gives you a flavor for what happens. After 10 or 15 minutes the Chair will ask each of the reviewers to revisit their preliminary scores, which often change as the result of the discussion. At that point the remaining study section members (silently) mark scores of their own on a sheet and the grant's final score is the arithmetic average of the entire Study Section's judgment. The scores of the original reviewers don't count any more than any other score, but they obviously have a strong influence on everyone else. That means that if you are unlucky and draw a primary or secondary reviewer that is really off the wall, you can get screwed. Conversely, if you get someone who really loves your ass, you are in good shape. In that regard this is a bit like college admissions at a prestigious university. Everyone takes it very seriously, tries to do their best, but in the end it's a bit of a crapshoot. Not everyone who is worthy can get in and not every research proposal gets funded. In fact less than one if five will make it through.

The grant I am writing is complex and has 15 reviewable parts, each corresponding to a single RO1, but highly integrated around a theme. This is a competitive renewal and will likely be in excess of 600 pages in length (last time it was 900 pages but there are new page restrictions). It will have 30 or 40 different reviewers, each doing different parts. From the grantsmanship standpoint it is therefore a very big target, but the rewards are also large, running into many millions of dollars annually for 5 years, supporting scores of researchers, post docs, graduate students, technicians and staff at several academic institutions. The biggest reward is that it enables us to do what we love to do, science. So there is a lot of incentive to write the best proposal possible. We started on this one a full year in advance and now we are down to the last month.

We have competitors and they are scientific and academic heavy hitters. Some of them are coming back after just missing it the previous round and those are the ones I fear the most: they have reviews in hand and know what needs to be fixed. So I decided to get the same advantage and conduct my own Mock Study Section. Unlike a real Study Section, however, I asked all project leaders (all senior investigators) to act as reviewers of other projects and I also assembled outside reviewers from within and outside my institution. In the end there were 32 generous colleagues who produced written reviews, with 28 physically present for the all day meeting and 4 participating by telephone (this is also done in regular Study Sections). I Chaired, starting at 10 am. We finished just after 3 pm, with a half hour lunch break. I have chaired before, and I ran it just like the real thing. Younger faculty, post docs and advanced graduate students were offered the opportunity to write reviews and observe it as a training exercise and a number did so.

I wasn't sure how this would work out, but according to everyone who participated, and from my perspective as Program Director, it was a spectacular success. I had three goals and all were met in full. The first was the obvious one of getting substantive feedback to identify strengths and weaknesses of each component. Reviewers were conscientious, candid and constructive and helped us avoid some major problems. But I had two more objectives. One was to have all project leaders look at other proposals for this program through the eyes of a reviewer. This is one of the most valuable ways to learn how to write good grants. This turned out to be an "eye opening" experience in other ways, as well. I also knew that this kind of reading would suggest new kinds of collaborations that could be incorporated in the final overall product to enhance integration, one of the review criteria. The Mock Study Section was held on February 22, giving everyone a full 6 weeks to revise their projects, usually working together with other projects and reviewers on fixes for identified weaknesses. We harvested the details by looking at the internal scores of each component (there are subscores that are not announced) and locating the areas where we need to concentrate in the remaining time. I have also asked for written responses to the critiques, a request that produced initial grumbling but now is being seen as extremely valuable.

It is often the case that scientists have colleagues read drafts of their proposals before submitting them. That's always a good thing. But we are finding it is not the same as having your colleague write a structured review and laying it out for open discussion, as in a real Study Section. No one knows if this effort will be enough to be in the top 20% of a field that is probably already in the top 10% of the field (if you don't have the green stamps you won't apply for something of this magnitude). But there is no question this elaborate process will make our run at it much better.

It also produced a lot of work for all of us to do in the remaining time. If you wanted to know how I spend my days, nights and weekends, now you do. I hope I survive it.