Rank Beginner

Wednesday, January 10, at 2 p.m., U.S. Eastern time

Lawrence B. Martin, an anthropologist, has been graduate dean at the State University of New York at Stony Brook for 13 years and has spent more than a decade studying and writing about the productivity of faculty members. He served on the methodology committee for the National Research Council's rankings of doctoral programs, and is a member of a New York State Department of Education panel that focuses on assessment and institutional effectiveness.

A transcript of the chat follows.

Piper Fogg (Moderator):
    I'd like to welcome our guest today, Lawrence Martin, dean of the graduate school at SUNY Stony Brook. Thanks so much for joining us today. Let's get started.

Question from Karen Kemp, Dept. of Public Policy Studies, Duke University:
    We were very interested to see how highly our doctoral program was ranked in your new system and would like to learn more about your methodology. How are the names of faculty associated with a particular program identified, especially in a program as interdisciplinary as public policy, which draws on faculty from many departments such as economics, political science, law, business, medicine and so on?

Also, once the names are selected, what tool(s) are you using to identify their publications, citations, awards and other scholarly output? This has many implications for us, as it does for all the programs included in the survey, and we're trying to determine whether and how we ought to publicize this ranking. Many thanks. Karen Kemp

Lawrence B. Martin:
    Much of this information is available at the company's web site: AcademicAnalytics.com. We collect lists of faculty names for Ph.D. programs, rather than for Departments. This allows us to reflect groups of faculty based on their common intellectual interests rather than by who pays their salaries. We use journal publication and citation data from Scopus, which indexes more than 15,000 journals. We collect our own data from federal funding agencies and from book publishers and organizations giving honors and awards. Data are matched using custom algorithms within Dataflux (a SAS product). The measures that we use were developed as the result of many years of study of data from the 1995 NRC study. As long as you remember that we measure per capita faculty scholarly productivity and publicize based on those results then you will be on firm ground. It is tempting to interpret this measurement as meaning that program x is the best in the country. It may be, but that's not what we've measured. The program ranked number 1 is the most productive per capita with the variables that we've included and with the weighting scheme that we've adopted. Clients get data that they can weight according to their own preferences.

Question from Elyse Ashburn, The Chronicle of Higher Education:
    Many universities did not update their faculty lists for the index, which draws the reliability of the data into question. Are there ways you can get up-to-date faculty lists in the future, even if universities aren't willing to cooperate? What plans do you have to address this issue?

Lawrence B. Martin:
    We start with the assumption that universities present accurate information about their faculty members in the materials that they share with external constituents. Prospective graduate students make choices based on information on university web sites and in university publications so truth in advertizing requires that universities make sure that this information is accurate. Academic Analytics has a responsibility to reflect accurately the lists that universities themselves produce but the universities need to take responsibility for the information that they make available. Obviously, we will work hard to increase the number of schools that help us by updating or approving our lists and I expect that number to grow steadily.

Question from :
    Reputational rankings may be annoyingly qualitative, but is measuring the sheer number of publications generated by a department the best way to gauge its strength?

Lawrence B. Martin:
    No, but that's not what we do. John Lombardi often talks of the important distinction between measurements and indicators. We measure per capita faculty scholarly productivity. The extent to which you believe this is an indicator of the strength of a group of faculty will vary as it is an indicator of program quality rather than a measurement. I believe the the scholarly output of faculty sets important boundary conditions within which doctoral education takes place and we've worked hard to include metrics of quality and quantity.

Question from Steve Katsinas, University of Alabama:
    I am interested to know how you became interested in the subject of rankings.

Lawrence B. Martin:
    I became Dean of the Graduate School at Stony Brook in Fall 1993, shortly after the data were submitted for the 1995 NRC study (data collected F02 and S03). When the study appeared, our physics program had dropped compared with the 1982 study. On investigation, it turned out that the faculty list sent to reviewers lacked the name of our Nobel Laureate (and 14 distinguished colleagues in our Institute for Theoretical Physics). I started to look for other metrics related to quality and found the 1995 NRC study an invaluable source of discipline specific data. I developed a series of algorithms to estimate program quality based on publications, citations etc. Having got the system as good as I could I was frustrated (as were my audiences) to have to work with data covering 1987-1992. New data were needed, national and at the discipline level. And so it went...

Question from Scott Smallwood:
    How do you hope graduate deans and department chairs use this data? How would you use it in your own work?

Lawrence B. Martin:
    We have used it to engage in thoughtful conversations about who are peers for programs, how their performance in various areas compares to ours. We have talked about building the data into our external program review process for the selection of reviewers from places that are genuinely our peers but from whom we can clearly learn how to do better. I have used the data to advocate (successfully) for investment in programs of high quality.

Overall, I hope that the data will enable chairs, deans and other administrators to have a better sense of how their scholarly efforts stack up against national standards and peer schools.

At a typical research university we ask faculty to teach only about half the load of a colleagues in the same discipline at a four year college. We do this to enable them to pursue scholarly work. Another way to think about that is that half of your faculty payroll is deployed to enable scholarly work to be done. I think it is useful to see how that work compares with other programs on your own campus (across disciplines) and within your discipline around the nation.

I think that the data inform strategic planning, strategic decision making and program review and I'm sure that many of my colleagues will come up with other important uses from which we can all learn.

Question from Piper Fogg:
    I understand that some clients have asked your company to model programs to tell them if they are ready to move into the Ph.D. business. Do you see that as a big growth area?

Lawrence B. Martin:
    I don't think that it will be big as the scale is relatively small compared to the 7000+ programs that exist already. However, it's always a question as to when a program is ready to move up to the Ph.D. level and we provide a custom service to model this based on the faculty who would be proposed as members. This is also useful for universities who wish to see what a novel, interdisciplinary cluster of faculty might look like as a new program.

Question from Gerald Holder, University of Pittsburgh Engineering School:
    We received a top 10 ranking in environmental and health engineering. The Chronicle article indicated that we had 264 faculty in this area. Since we do not identify an area by this name, I was wondering how the faculty members for this program were identified.

Lawrence B. Martin:
    Each doctoral program for which we collect names is assigned to a CIP code (Department of Education, Commmon Instructional Program code) at the finest level available. These groups of faculty may then be aggregated up to a taxonomic category in the study so it may be a list drawn from more than one of your programs that was classified by us under this CIP category. We offer schools the chance to revise lists and classifications that we have made and we'll be happy to share with you the list for which the results were obtained.

Question from Robin WIlson, The Chronicle of Higher Education:
    How important is this index in guiding undergraduates who are trying to decide where to enroll in college? Does faculty productivity have a direct connection to the quality of teaching?

Lawrence B. Martin:
    Except for this limited release of data via the Chronicle we have targeted our reports within the higher ed. community as we see them as more useful in that arena. It's not obvious that picking a school for undergraduate study based on having faculty heavily invovled in research would automatically be a good thing. In some cases it would open up wonderful opportunities for UG's to get involved in research. In other cases it might mean that faculty don't spend much time with undergraduates. That is not what we measure so intepretations about UG education should be treated cautiously. It's interesting that some of the schools that have wonderful reputations for education also have very productive faculty but I do not think that their is a direct or guaranteed connection.

Question from Elyse Ashburn, The Chronicle of Higher Education:
    There seems to be significant interest in having the index include scholarly output for individual faculty members. Do you plan to include that information in the future? I believe you expressed concerns about misinterpretation and misuse. Are their ways you could safeguard against that and still give institutions information on individual faculty members?

Lawrence B. Martin:
    We believe that universities are in a better position to assess the contributions of individual faculty members using a complete set of information than would be the case if they focussed solely on FSP. If a program has a lower level than they would like to see it should be easy to take a look at faculty annual addenda, CV's etc. to make an assessment. Faculty may have many other responsibilities, such as teaching and service, that may impact their productivity. As a dean, I feel that it requires a holistic view to determine a faculty member's contribution to the University.

We fear that a table of productivity rankings could lead to simplistic decision making. Our goal is to provide universities with information that gives them a set of tools with which to understand how they compare. It is up to them to determine how to use and to act on the information within their local context.

Question from Scott Smallwood:
    What does "productivity" really mean? Is writing ten mediocre articles more "productive" than writing two really great ones? And while the citations part of the index addresses this to a degree, doesn't the index slant in favor of people who churn out a lot of work -- of whatever quality?

Lawrence B. Martin:
    Most of us know that we can get most of our work published and that is why we also use citations, grants and honors. People who publish a lot of minimal units tend to get low citation rates. Similarly, the grants process can also be seen in a vote of confidence that your work has been of a quality to warrant further investment. Likewise honors and awards. We looked at some of the bibliometric approaches that have been tried in, e.g., Economics, Pages per author etc. but on balance we decided to present information that was clearly defined and at the program level so that individual publication patterns would have less influence.

Question from william m epstein, UNLV:
    The SSCI covers some fields better than SCOPUS. Why not use both sources to indentify publications?

Lawrence B. Martin:
    Scopus has been expanding rapidly their coverage in the social sciences and humanities and we expect to see continued improvements in those areas. What's important is that we capture a representative sample of publishing, not all of it. I believe that ISI currently indexes about 8000 journals, while Scopus covers 15000 and is growing to broaden discipline coverage. The additional cost would also have been a factor.

Question from Elyse Ashburn, The Chronicle of Higher Education:
    You are quoted in this week's article saying that Academic Analytics was designed as a for-profit company because "somebody's got to pay the bills." While that's certainly true, there are plenty of non-profits that pay their bills and compensate their employees sufficiently -- and sometimes generously. Did you and your partners consider running Academic Analytics as a non-profit, and if so, why did you decide against it? Do you think being for-profit hurts its credibility?

Lawrence B. Martin:
    My partners invested the funds to build the database and to hire the staff. We have talked about the possibility of moving into a not-for-profit mode and would certainly be open to that. None of us went into this with the idea that we would make our fortune. At the time that we began, it was not obvious that anyone from the not-for-profit sector was interested in putting up the necessary funds to construct the database, although I'll admit that we didn't look hard at that. The company happened in its current form as the result of a chance meeting between me and Mark Shay (founder of Educational Directories Unlimited) at the 2004 CGS meeting in Washington. Mark was interested in the idea and offered to put up the money to build the database and to see if the higher education community would find this a useful set of information.

Question from Alan G. Kraut, Association for Psychological Science:
    I noticed Psychology is broken up: Clinical Psychology; Educational Psychology; Counseling Psychology. Where is the rest? Cognitive Science? Neuroscience? (Actually, the issue of separating applied psychology from the rest has always been an issue. Just wondering how you dealt with it.)

Lawrence B. Martin:
    Psychology appears at two levels in our taxonomy. Many schools have a single Psychology Ph.D. program so we have rankings at that level. We also cover some of the sub-fields for which large numbers of separately named programs exist and will likely add others in future years. Neuroscience is a separate field with its own table as is Cognitive Science. We welcome advise from scholarly societies as to the best way to organize the taxonomy of disciplines within their field.

Question from Scott Smallwood:
    Will the index still be useful once the NRC rankings finally come out? If so, how?

Lawrence B. Martin:
    I certainly think so. The NRC study is a much more comprehensive study that ours and covers graduate education in all its complexity, while we measure faculty scholarly productivity. The NRC study will include data on faculty journal articles and citations, honors and awards at the discipline level but may not include data on books published or research grants so our data should complement theirs. I have always maintained that there was a need for annual data on productivity that are produced quickly. Our goal is to report in the Fall what your faculty from the prior academic year did. We think those data will always be useful as they will be current and because trends will quickly be measurable.

It's one thing to know that you're ranked #20 but if last year you were #15 it's bad, while if last year you were #25 it's great. The NRC study has, at least until now, only made such data available every 13 years. Academic Analytics also has a broader disciplinary coverage than the current NRC study so for schools of Education, Business etc. there is that benefit also. I think that the two studies will be complementary and tremendously useful to all of us who wish to promote improvement in higher education.

Piper Fogg (Moderator):
    That wraps up our chat today. I'd like to thank Dean Martin for being here today and answering questions about the Faculty Scholarly Productivity Index.

Lawrence B. Martin:
    The FSP index and that data contained are designed to provide higher education with a national perspective on scholarly activity at the discipline level. It is a broad measurement containing information on books published, journal articles published, citations of journal articles, research grants awarded, fellowships, honors and awards won. The variables were selected for inclusion and were defined based on several years analysis of data from the 1995 NRC study, which provided the only modern set of data on productivity at the discipline level. It is our hope that the availability of clearly defined and transparently presented data on scholarly work will promote improvements in American universities that should result from better and more strategic decision making, informed by quantitative data.