The Chronicle: Colloquy Live Transcript

Working With Technology and Winning Tenure

Wednesday, February 20, at 1 p.m. U.S. Eastern time

Are departments and colleges giving professors enough credit, in tenure and promotion decisions, for their work with technology? How should the creation of scholarly Web sites and electronic teaching tools be evaluated?

Many faculty members feel that their work with technology does not help them win tenure or promotions. And many academic departments and colleges say they don't know how to evaluate such work. As a result, some professors say they lost their tenure bids because they were unable to make a case that their technological projects were as valuable as traditional teaching, research, and service contributions. Some scholarly groups are pushing for formal standards to evaluate scholarly and teaching contributions that make use of technology, and some colleges are adopting policies that specifically reward such work.

» Ever So Slowly, Colleges Start to Count Work With Technology in Tenure Decisions (2/22/2002)

Bruce Mason, an associate professor of physics at the University of Oklahoma, is leader of the physics review board of Merlot, a project that uses peer review to evaluate scholarly Web sites and electronic teaching tools. Mr. Mason, who believes that such contributions should count in tenure reviews, has been trying to persuade the American Institute of Physics to endorse Merlot's peer-review process. Mr. Mason will respond to questions and comments about these issues on Wednesday, February 20, at 1 p.m. U.S. Eastern time. Advance questions are encouraged and may be posted now.

A transcript of the chat follows.

Jeff Young (Moderator):
Hello and welcome to The Chronicle's live online discussion forum. I'm Jeff Young, a technology reporter here, and I'll be moderating today's chat.

Our topic is whether teaching with technology counts in tenure reviews. Many professors feel that their work with technology does not help them win tenure or promotions. Should their online creations get counted, and if so, how should they be evaluated?

Our guest today is Bruce Mason, who has been heavily involved in the Merlot project, a nationwide effort to develop a peer review system for online course materials. Thanks for being with us today, Bruce.

Bruce Mason:
Thank you, Jeff, for inviting me. I appreciate the chance to discuss some of the tools MERLOT is developing to support the Scholarship of Teaching.

Jeff Young (Moderator):
Let's get started with a question ...

Question from Nancy T. Farina, Nazareth College of Rochester:
Is there a difference among academic disciplines in acceptance of digital work as scholarship?

Bruce Mason:
What I have seen, Nancy, in working with the other disciplines in MERLOT is that the use and acceptance of digital materials for instruction seems to mirror the use in the discipline as a whole. Disciplines that have a history of using computer simulations and modeling, such as physics and math, tend to have more online and digital learning material available, and have people involved in research on the effective use of the material for learning. I don't think this translates into wide acceptance of the work developing digital learning materials as scholarship at the same level as more traditional research efforts, but it may be the first step.

Question from Jim Marshall, University of Colorado:
Should faculty members be given credit for making the effort to incorporate technology into teaching, even if it is not proven to be effective in facilitating student learning? In other words, should faculty be rewarded for their efforts, or for their results?

Bruce Mason:
Certainly we want to reward quality efforts, quality in this area is of course effectiveness in its ability to enhance student learning. I think that just as in research, quite often a lot of research gets done that is proven wrong, but it can be useful in the process of gaining new knowledge. The biggest credit should go to the most effective materials that are created, but the time spent developing this material is worth while also. In the Merlot project, we are not yet doing research on effectiveness of materials, we are trying to judge it trhough the experience of knowledgable instructors. The main reason is we want to create a large body of reviewed materials, and indicate as much as possible the quality of existing materials.

Question from Alethea Cook, Independence Community College:
Evaluation of these classes/instructors is quite a challenge. What is a procedure that is proven to work well?

Bruce Mason:
In reviewing of materials the process that we follow in Merlot is a three-dimensional review. The dimensions are: quality of the content, effectiveness as a teaching and learning tool, and ease of use for students and faculty. Quality of content is whether or not the material is correct, and its importance in the discipline. Effectiveness as a teaching and learning tool is the most difficult. Here we try to judge the best use of a material and its impact on student learning. Issues like active engagement with students are very important here. The usability dimension is simply does it work, is it easy to understand, are there instructions--those sorts of issues. The question you asked about judging the use of the material by instructors is another aspect that requires the observation of the use of the material. Even the best material can be used poorly, and poor material in the hands of a good instructor can become effective.

Question from Mills Kelly, George Mason University:
My own experience is that upper administrators (Dean, Provost, etc.) are already receptive to the idea of including electronic teaching tools and websites in tenure and promotion portfolios...but that at the department level there is much less of a concensus on this issue. What do you think will need to happen to change attitudes at the departmental level?

Bruce Mason:
At the departmental level, tenure and promotional decisions are often very personal among individual faculty. There's no magic potion to change these personal opinions. However, what we are trying to do is to build a review system similar to research work that will provide the foundation for people to recognize the development of educational materials as a scholarly pursuit.

Question from Dr. G. Jay Christensen, California State University, Northridge:
If a professor has written 3-400 pages on the Web for his classes, why is that not considered a scholarly publication? It takes hours and months to prepare these Web sites with all their links and text. It takes just as much effort to prepare a scholarly contribution to the Web for our students. Is the problem, trust, on the Web by other academic scholars?

Bruce Mason:
I think the problem we are facing is that anyone can publish anything on the Web, and there is no gate keeper indicating quality. I agree that a 400-page Web site that has reviews similar to a monograph or a text should have equal prestige.

Question from Dan, higher education association:
Has the Merlot effort helped faculty members receive more recognition for their work?

Bruce Mason:
I think in some cases it has helped, but Merlot is still in a developmental phase. As it is currently operating, Merlot has only been active for a little bit more than two years, and many disciplines are just now developing their collection of reviewed material and the review process. I see us moving forward by becoming more closely aligned with professional organizations and other groups that are recognized in the disciplines as indicators of quality. As our reputation grows, I see Merlot having an ever-greater impact.

Question from John Hollenbeck, Old Dominion University:
Isn't this an issue greater than technology? It seems all creative works, save research, are not counted highly towards tenure. Example: videos produced for educational use.

Bruce Mason:
Absolutely, and the process that we develop for Web-based material can just as easily be applied to other innovative methods for instruction. If you believe the theory of convergence, the medium is not the issue-- it's the content. We are trying to indicate quality of content, quality of content and quality of the learning process.

Question from Jeff Young:
Bruce, in looking through the peer reviews of teaching materials in Merlot, most of the reviews are positive -- lots of 4 and 5 stars out of 5. Some people could see that as suggesting that Merlot reviews are not as critical or rigorous as a traditional journal review of a scholarly article ... what is your response to that?

Bruce Mason:
Our goal in Merlot is to point users to the best existing material on the Web. We have collected a large number of links to material to help people find items, but the value we want to add is to help them find the best. Therefore, in the review process, the editorial panels are first screening the material for the best material available. These are the items that we review first. With a lot of effort and help from external reviewers, we hope to get to as much of the material as possible in the future.

Question from Deborah Lines Andersen, University at Albany:
How does an untenured professor deal with the discrepancy between upper level administration's desire for technology and departmental mandates of traditional scholarship for tenure, promotion, and review?

Bruce Mason:
Very, very carefully. My experience at the University of Oklahoma is that the professors who have sucessfully walked this line had support from as many in the administration as possible and got all expectations in writing from the very beginning. These issues vary from school to school and department to department. Untenured professors need to be very aware of the stated and unstated expectations that they face.

Comment from Kenneth Green, Director, The Campus Computing Project:
Data from the 2001 Campus Computing Survey again confirm that the majority of campuses support courseware development projects and other kinds of IT developmental efforts. And the majority of campuses also have IT support centers intended to assist faculty who want to find appropriate ways to make use of It resources in their courses.

The sad irony is that barely one campus in five has expanded the criteria used in the faculty review and promotion process to include technology in research, teaching, and scholarship. Moreover, this number has changed very little in the years that we have been tracking this particular issue.

We know that across all sectors, a growing number of departments and institutions are very eager to hire faculty with technology skills and interests. Yet the institutional message continues to be "go there - into technology - at your own risk." Indeed, depending on the department and the campus, to go "there" - into technology - may pose great risk for career prospects and advancement, especially for younger, untenured faculty.

Jeff Young (Moderator):
We're about half-way through today's chat. Keep those questions coming...

Question from Anthony Carpi, John Jay College, The City University of New York:
Clearly Administrators and Faculty do not consider technology-based materials to be on par with print materials during the tenure/promotion review process. From my own experience, I have created an online resource (http://www.visionlearning.com) that has been shown to improve science education and recently received funding from the National Science Foundation. Yet during a recent promotion review, I had to print these materials out and include a copy on CD for the committee only to be told that the physical width of the package was thin compared to a textbook. How can we provide electronic materials in a way that review committees can easily see the extent of work put into creating these materials?

Bruce Mason:
Perhaps we need to include a brick with the CD. More seriously, I think that portfolios that include both local descriptions of the effort involved in the development and independent respected reviews of the material will help change these biases. Certainly in your case, with funding from the NSF, I am somewhat surprised and chagrined that your resource did not get more respect.

Question from Charles Green, Saint Louis University:
Hi Bruce! In general, doesn't the criteria by which faculty are currently evaluated for tenure actually serve to discourage the use of technology and place those junior faculty members who devote large amounts of time to the development of non-traditional materials at some level of risk?

Bruce Mason:
I think that in some institutions and departments that's true, and in others expectations are changing. You are probably right that the historical evaluation standards do discourage this sort of development. I guess in Merlot, what we are trying to do is not necessarily start a revolution, but to supply a tool that will help change to occur.

Question from Carole Richardson, American U.:
There are those who argue that only those uses of technology which enhance teaching/learning should be rewarded. Since quality teaching is already considered as part of tenure review, there is no need to give special consideration to technology. How would you respond to such an argument?

Bruce Mason:
I think that the issues we are talking about here go beyond simply being an effective teacher. They involve creative work in developing new ways of stimulating learning. Development of effective multimedia and on-line learning materials is hugely time-consuming. This needs to be considered in the evaluation process. I think that currently writing a highly respected and reviewed textbook receives more attention than the in-class teaching activities of an instructor. The development work we are considering here is on par with this textbook writing.

Comment from Don Reed, San Jose State University:
This is more a comment from my experience than a question. I have successfully used my work in technology towards tenure and promotion to full professor by framing it in terms of the three traditional duties of faculty: teaching scholarship and service, The teaching side is self-evident, although faculty need to document how students and the university have been served well by this effort, for example, enhanced learning, but equally important, increased access to learning (have enrollments increased, have new communities of learners been served).

On the scholarship side, faculty should present papers at conferences on their efforts and publish papers in the peer-reviewed literature as well as obtain external funding (or internal) to develop and support these projects. Faculty can organize symposia at national conferences and become active in education committees of professional organizations and societies and in setting priorities on a national level, for example, by serving on panels of NSF or other such agencies. There are many possibilities if you approach it with the same vigor and commitment as we do our research efforts.

Finally, service...faculty should become involved in providing workshops on use of technology for other faculty, become members of department, college and university committees on use of technology, and provide outreach to K-12 educators. It is your choice, you can react to campus policies on tenure or play an active role and shape these policies.

These are just some of methods to use this work toward tenure promotion.

Question from George Guba,Broome Community College:
Are colleges that have a greater level of technological use on campus more willing to grant credit in tenure and promotion?

Bruce Mason:
This I'm really not sure about; certainly many of the participants we have in Merlot come from institutions where this work is highly valued. Because the institutions support the participation in Merlot, this indicates their willingness to put money where their goals are.

Question from Lee Williams, University of Oklahoma:
Are there any 'norms' or rules-of-thumb about how much additional effort is needed to develop a new web-based course, and the continuing effort to teach a web-based class. Put simply, how many 'course-equivalents' does it take to develop the class, and how much to maintain it? Can we use this in any way to develop norms for how much time relief (e.g reduced course load) we should offer, and would this help address the problem that the TIME and creativity put into electronic learning are not valued in promotion/tenure decisions?

Bruce Mason:
From my understanding of institutions that are leading these efforts, the development of a Web-based course is often given equal weight to the teaching of a course. This gives some indication of the effort and time needed to do it correctly. A large variable in this is the direct support the institution gives to faculty in terms of technological assistance from programers and designers. It's becoming much more common that these sort of support efforts exist on campuses, but they too are very expensive. One of the goals of Merlot is to reduce this expense through collaboration and sharing of effort. Merlot is a tool to allow faculty to find, rather than create, learning material.

Question from John Fritz, UMBC:
I work in the IT office and support faculty using technology in their teaching. But I worry that this issue of reward and recognition will not only stifle faculty, but rob us of the innovation, quality and best practices that can inspire other faculty to use technology--something our university wants. We find faculty learn best from other faculty, but if fewer people use it, it puts more of a strain on tech support. Do you think we're hitting the wall on scalability of technology enhanced learning if we don't get past the issue of technology being valued in tenure and promotion?

Bruce Mason:
Certainly the issues of reward are crucial, but another issue that's important is how easy the technology is to use. As you know, as someone in IT supporting faculty, if there's much of a learning curve many faculty will not attempt new approaches to teaching. My hope is that by collaborating and sharing both materials and uses of materials, the learning curve will be reduced to a point where faculty do not have to be whiz-bang developers to make effective use of technology in their classes.

Comment from Michelle, research I:
It seems to be that promotion and tenure - ie, credit - for working with technology sidesteps the major issue. Yes, it's hugely time consuming and the learning curve can be enormous. But isn't this an example of an area where we'd like to see more institutional support instead of increasing the burden on faculty? Faculty are content experts with real world experience. Why are we asking them to become html techs as well as instructional designers? Instead, shouldn't we provide support in the discipline by using people who combine a working knowledge of the discipline with their own content expertise in technology and instructional design? We use professional advisors, biomedical engineers, and professional business managers to help "run" a department - why not support teaching in the same way?

Question from Deborah Lines Andersen, University at Albany:
The American Association for History and Computing has developed guidelines for including digital scholarship in the tenure, promotion, and review process. In your work with Merlot have you seen indications of departments willing to work with national guidelines? How would Merlot fit into this process?

Bruce Mason:
I think Merlot is unique because of its multi-disciplinary nature. Merlot can play a central role in the communication and sharing of the sorts of guidelines that the association has developed. As for your question about departments repsonding to guidelines, there have been interesting changes within some of the institutions that are participating in Merlot. What needs to happen next is to expand the impact of these guidelines to as many institutions as possible.

Question from Alethea Cook, Independence Community College:
What we have considered implementing is having an administrator enrolled in the class to directly observe interaction, assignments, etc. as possible, accompanied by the use of student perception forms (class/instructor evaluations) to get the student-take. Quality is our main concern. How is the 3-dimensional review accomplished in your program?

Bruce Mason:
The review process you described sounds to be very ambitious. Within Merlot we are not evaluating classes, but learning objects that can be dropped into as wide a range of classes as possible. Our evaluations therefore are informed by best uses of materials, and not specific uses of the materials. Again, our effectiveness dimension in reviews is the potential effectiveness of material; the creativity of the instructor is at least as important as the material itself.

Jeff Young (Moderator):
That's all the time we have for today. Thanks to Bruce Mason for joining us, and thanks to all of you for the questions and comments.

Bruce Mason:
I'd like to thank Jeff and the Chronicle for this opportunity to discuss Merlot and its impact on tenure and promotion issues in academe. I encourage anyone who is interested in the Merlot project to visit the site at www.merlot.org. In particular, we are interested in those who would like to become involved in the review process. I'd also like to thank all the participants for their insightful and challenging questions and comments.