April 16, 2008

Tell Us the 'Broader Impacts' of Your Science, NSF Says

The National Science Foundation says not enough grant applicants and peer reviewers are taking seriously — or even know about — an 11-year-old requirement that they evaluate the “broader impacts” of proposed studies.

So in a recent “Dear Colleague” letter, the agency reminds applicants and reviewers of the rule, which is meant to encourage researchers to pursue socially useful results. Those can include, for example, recruiting members of minority groups as scientists and improving the public’s understanding of science.

In 1997 the NSF simplified to two the agency’s principal criteria for the merit-based review of grant applications: intellectual merit and broader impacts. The NSF was reacting to pressure from Congress and the White House to show it was using federal tax dollars for more than just ivory-tower, fundamental research.

Ever since, many academic researchers have been perplexed and annoyed by the broader-impacts requirement, often called “Criterion 2.” Although the NSF has stressed that it gives equal weight to both criteria, some scientists have said that intellectual merit should count for more. They also point out that socially useful applications of fundamental research can be difficult to predict and take years to appear.

By some measures, compliance with Criterion 2 seems to be improving. Ninety-two percent of all external peer reviews commented on applications’ ideas for broader impacts in 2004, up from 84 percent in 2002. What’s more, the NSF has reported that in the 2005 fiscal year, the agency returned to applicants only 176 research-grant applications, out of more than 30,000, without reviewing them because their summary sections did not discuss broader impacts, as required.

More applicants ignored the NSF’s request for details in the separate project-description section of applications, but that is not grounds for rejection before review, said Luis Echegoyen, director of the agency’s division of chemistry, who wrote the letter on behalf of the entire agency. And even when grant applicants do elaborate about the broader impacts, some of the NSF’s external peer reviewers, who are drawn from the ranks of NSF grant recipients, remain silent on that part of the applications, he said.

The NSF has previously published examples of how, apart from future technological applications, scientists can describe the potential broader impacts of their work. The new letter pointedly calls for “rigorous” and “innovative” proposals. It says, for example, “a simple listing of outreach activities, or reference to inclusion of research personnel who are members of underrepresented groups, falls short of the rigor required to satisfactorily address this criterion.”

The NSF is expected to say more about Criterion 2 soon. Congress directed the agency last year to report by August on how it was carrying it out. —Jeffrey Brainard

Comments

1. My experience has been that many, perhaps most, researchers do take this criterion seriously, but just do not know how to create a significant broader impacts component for their proposals. This is a place where universities and colleges need to get on the stick and realize that they are going to start losing federal funding unless they provide the assistance and resources to help researchers be able to do this.

   — Bob M.    Apr 17, 06:52 AM    #

2. Perhaps it is time for NSF to look at this again. Is it really having any beneficial effect? Does Congress care this many years later? Why must experts in research and science, try to do this also? Is this the best way to advance science in society or should these two very different activities be done separately by experts in each field?

   — Alan P.    Apr 17, 08:43 AM    #

3. I read with great interest the “Tell Us the ‘Broader Impacts’ of Your Science” article and the posted comments. As a former member of one of the committees at NSF eleven years ago that implemented the then new criteria reducing the number of criteria from four to two, I am reminded of why the change was made. As I remember, the change was to made to make NSF’s expectations clearer for researchers to address and to assist peer proposal reviewers fairly weigh the importance of broader impact statements when evaluating proposals. I think it is working. Remember, NSF has always had a concern with broader impacts tax payer funded research has on the state of science and by extension, the quality for life for all citizens. The challenge is that many who are great researchers are not always as gifted when it comes to defining and implementing broader impacts of their work especially as it relates to recruiting and preparing a more diverse science, technology, engineering, and technology workforce. This criterion forces researchers to consider the broader impact of their work. NSF funds programs in the Education and Human Resource (EHR) Directorate (especially within the Division of Human Resource Development) that provide means for researchers to expand their research’s impact. NSF could help researchers with these criteria (1) by encouraging researchers to consider partnering or at least aligning their projects with appropriated EHR projects and (2) by providing a means for researchers to easily locate, identify, and partner with such broadening participation projects. This is the wrong time to consider lessening the weigh given to this criterion.

   For help in identifying projects on your campus or in your region that might help with broadening participation, you can visit http:ScienceDiversityCenter.org. This is a free resource that was developed with support from an NSF grant.

   — William McHenry    Apr 17, 09:18 AM    #

4. no matter how seriously we take the broader impact, 80-90% of us proposal writers won’t be getting the grant anyway.

   — ROM    Apr 17, 10:34 AM    #

5. To face the truth, the “broader impact” these days almost always implies outreach to minorities, without which your proposal’s chances for funding decrease dramatically.

   — Mark de Goz    Apr 17, 11:14 AM    #

6. Always include the phrase, “it will help the federal government….” Works like a charm.

   — first marci    Apr 17, 11:20 AM    #

7. Mr. Brainard’s post draws some much needed attention to an interesting issue: whether scientists and engineers can provide an account of the broader impacts of their research. Alan P.‘s comment also raises the issue of whether scientists and engineers SHOULD give such an account, or whether it is fair in the first place to ask them to do so. Shouldn’t this be left to the experts in science and technology’s broader impacts? Dr. Echegoyen’s letter seems to require as much, insofar as it suggests that broader impacts activities need to be of “the same caliber” as the proposed intellectual activities, as well as conducted with “appropriate expertise.”

   Unless one happens to be an expert both in a particular field of science or engineering AND in what one might call a broader impacts field — say ethics or education — it seems that the broader impacts criterion, taken seriously, requires interdisciplinary teamwork.

   Anyone who’s interested in what some “broader impacts experts” have to say about NSF’s broader impacts criterion should take a look at this recent NSF-sponsored workshop: http://www.ndsciencehumanitiespolicy.org/workshop/ . There, you will also find links to a host of other resources for dealing with broader impacts issues.

   — J. Britt Holbrook    May 2, 10:27 AM    #

Previous: Education Department Plans 'Emergency Survey' of Colleges' Student-Loan Plans
Next: Cuomo Aide Says Investigation Into Colleges' Credit-Card Deals Will Soon Be Public