Funded by the PEW Charitable TrustSitemapContact UsSearch
Re-envisioning the Ph.D.
News and UpdatesAbout UsRe-envisioning Project ResourcesPromising PracticesPhD ResourcesNational/International Resources
Re-envisioning Project Resources What Concerns Do We Have?Recommendations from National Studies on Doctoral Education
2000 Conference MaterialsMetathemes from 2000 Conference
Selected Bibliography on Doctoral EducationPhD Career Resources
UW Doctoral Education Resources 

2000 Conference: Panel: National Surveys and Studies on Doctoral Education

Friday, April 14th, 8:00 a.m. - 9:30 p.m.;
repeat session 10:00 a.m. - 11:30 a.m.

Don Wulff, Moderator


Maresi Nerad, Ph.D.s -- Ten Years Later
Chris Golde, The Ph.D. Survey
Ann Austin, The Development of Graduate Students as Prospective Teaching Scholars
Geoff Davis, The Graduate School Survey
Charlotte Kuh, Reassessing the Assessment of Research Doctoral Programs
Jody Nyquist, Re-envisioning the Ph.D.: What are Our Concerns?


Don Wulff, Associate Director, Center for Instructional Development and Research, University of Washington

I am Don Wulff from the University of Washington and would like to welcome you to Seattle. In the opening session this morning, we are going to be reviewing some of the work that has been done in several national studies and surveys to help us understand the status of graduate student education in the United States. Each panelist will present for about 8-10 minutes.

Maresi Nerad is Dean in Residence at the Council of Graduate Schools, so she has been away from her normal work at UC Berkeley, where she is the Director of Graduate Research. She has been conducting research on Ph.D.s 10 years after they received their doctorates, which she will tell us more about today. Next to Maresi is Chris Golde, who is an Assistant Professor of Educational Administration at the University of Wisonsin. Along with Tim Dore, her colleague and a postdoc at UC San Diego, she has conducted a survey of graduate student careers and graduate student perceptions about their experience. After Chris we'll hear from Ann Austin, Associate Professor of Higher, Adult and Life Long Learning Education from Michigan State University. Ann and I have been working as part of a team for the last four years conducting research interviews on the developmental aspects of the graduate student experience. She'll tell you more about that work, which is primarily focused on students who aspire to go into the professoriate.

Geoff Davis, a researcher at Microsoft and former professor of Mathematics at Dartmouth College, will then discuss a survey that he and his colleague, Peter Fiske of Lawrence Livermoore National Laboratories, have conducted, a very innovative, on-line survey of graduate programs. Then we'll hear from Charlotte Kuh, who is the Executive Director, Office of Scientific and Engineering Personnel from the National Research Council. She has been doing a lot of work recently on program assessment, which she will share with us today. Finally, we'll hear from Jody Nyquist, Director of the Center for Instructional Development and Research at the University of Washington. Jody has been traveling around the United States for the last two years talking to hundreds of people at all levels in all of the sectors from business and industry to K-12, getting their perceptions and trying to help us think about the various concerns people have about graduate education.

With that series of introductions I am going to turn the floor over to our first speaker, Maresi Nerad.

Ph.D.s -- Ten Years Later

Maresi Nerad, Dean in Residence, Council of Graduate Schools; Director of Graduate Research, UC Berkeley

This talk focuses on selected highlights from the Ph.D.s--Ten Years Later study. Numerous questions and issues pepper the contemporary debate surrounding graduate education. Presently there is have no mechanism that both systematically develops an understanding of the outcomes of doctoral education in terms of employment and provides feedback to doctoral programs regarding the appropriateness of education and training offered in relationship to the various employment sectors. Graduate schools do not know how long it takes their students to find professional employment, or when the students settle into more permanent positions. They do not have information on employment patterns by field, by type of program, gender, ethnicity, first generation college-goers, or multi-year fellowship recipients. In addition, there exists no retrospective analysis of doctoral education in light of graduates' subsequent employment history. Until very recently, there has never been a national survey requesting doctoral graduates' opinions about their doctoral education. Furthermore, no detailed information currently exists on postdoctoral appointments and their role in continued training and career outcomes. Also lacking is empirical evidence from the postdocs themselves regarding their reasons for choosing to accept one or more postdoctoral appointments and the choice of a career path subsequent to completing the postdoc(s).

Concerns arise outside academia as well. Industry representatives claim that highly trained doctoral graduates lack essential skills, such as communication and management skills and the ability to work in teams. The 1995 COSEPUP report of a National Academy study, "Reshaping the Graduate Education of Scientists and Engineers," examined these issues. This, coupled with concern about the poor academic job market for Ph.D.s in certain fields, has fueled debates about the merit of doctoral education and the cost to society of educating so many Ph.D.s. Currently, some critiques call for reduction in graduate enrollment as a solution to the perceived problem of the overproduction of Ph.D.s.

Also information is lacking with respect to the various graduate student populations. To date, there is little known about the experience of women after degree completion. Especially little is known about how the desire to combine family and career affects employment choices for both men and women, particularly in fields that require laboratory-based research.

Ph.D.'s--Ten Years Later, a national study of Ph.D. outcomes, surveyed graduates for career path information and evaluations of their doctoral programs in order to address the issues and concerns surrounding doctoral education; to provide current detailed information on the career paths of Ph.D.s; and to offer an assessment of the usefulness of a doctoral education from the perspective of the doctoral recipients, ten years after degree completion. The study was a collaborative undertaking by the Vice Chancellor of Research and Dean of UC Berkeley's Graduate Division, Joseph Cerny, and me, Dean in Residence, Council of Graduate Schools and Director of Graduate Research, UC Berkeley. The study was funded by the Mellon Foundation, with additional funding from the National Science Foundation.

Ph.D.'s--Ten Years Later surveyed nearly 6,000 Ph.D.s in 61 US universities and in six disciplines: biochemistry, computer science, electrical engineering, English, mathematics, and political science. The individuals surveyed received their doctoral degrees during the 1983, 1984 and 1985 academic years. This population accounts for 57% of the total number of Ph.D.s awarded in these six fields during the three years. Our response rate was fairly high: 66% of the domestic Ph.D.s and 52% of the international Ph.D.s (those in the U.S. on a temporary visa at the time the Ph.D. was awarded) responded.

The study consisted of a 22-page survey instrument supplemented by 70 in-depth interviews, which provided additional information about the context within which career decisions were made. The survey comprised six parts. Partone focused on the employment history of participating doctoral recipients, from the time they received the doctorate until 1996. It included questions about postdoctoral positions, if held. The second part of the survey focused on the job search process and inquired about how respondents went about the job search; what help they received from their department and advisor, the usefulness of that assistance, and factors important in the decision to accept the first and current positions. A third section asked respondents to make a retrospective evaluation of the doctoral program, after having been in the work force for 10-13 years. What did they think about their program and the usefulness of the doctoral degree? The fourth section requested demographic information, especially concerning the spouse, because we are aware that the dual-career couple phenomenon is wide spread among Ph.D.s. In this study, 61% of the women doctorates have a spouse who also has a Ph.D. or MD or JD, while 27% of the men have a spouse with a Ph.D., MD, or JD.

The first highlight concerns career outcomes. Since this national study was developed in order to collect information on the career paths of doctorates and to examine the feasibility of assessing doctoral programs in terms of the career outcomes of their graduates, we wanted to know where these Ph.D.s were employed 10 to 13 years later. In 1995, in three fields, biochemistry, computer science, and electrical engineering, half or more were working in the business/government/non-profit sectors (BGN sectors). In English, mathematics and political science, nearly 80% of our respondents worked in the academic sector. But that doesn't mean that they are all professors. In English, for example, in 1995, ten years after Ph.D. completion, 65% of the women working in the academic sector were tenured professors; ten percent of the women were assistant professors, and nearly 25% non-tenure track faculty. Seventy-nine percent of the men were tenured professors, 5% were assistant professors, and 17 % non-tenure track faculty.

When comparing career outcomes for men and women in the sciences and engineering, a higher proportion of women (except in mathematics) worked in the academic sector than in the business/government/non-profit sectors. More women mathematicians were employed in the BGN sectors in 1995.

We also inquired about overall job satisfaction relative to 24 different factors, including salary, autonomy at work, intellectual challenge, and opportunities for advancement. This inquiry revealed that over 80% of Ph.D.s were satisfied with their present work. Those working in the BGN sector were slightly more satisfied than those in the academic sector (includes responses indicating "very" and "fairly" satisfied), with one curious exception: Ph.D.s' in computer science. Computer scientists working in academia were actually more satisfied overall than those in industry, government, and non-profit sectors.

We also included in the survey questions about "quality of life" aspects. We compared the results of responses to these questions from Ph.D.s in the business/government/non-profit sector with those in the academic sector. On quality of life indicators, such as geographic location, support for women of color, or people of color, support for women and location for families, flexibility of work, and time and leisure, a higher proportion of those working in the BGN sectors were more satisfied, especially women.

A second highlight regards the usefulness of the Ph.D. We asked respondents this question: Knowing what you know now, if you could do the Ph.D. again, would you? Would you complete a Ph.D. in the same field? At the same university? Would you obtain a professional Degree, JD, MD, or MBA instead? Or would you choose not to acquire a graduate degree at all? With one exception, about 90% of respondents said they would complete a Ph.D. program again, and even about 80% would choose the same field including English. However, only 68% of biochemists said they would complete the Ph.D. again in the same field; that is, of the remaining 32%, over half (16%) would rather have studied for the MD degree.

Since so many of our survey respondents reported that they would do the Ph.D. again, given the opportunity, we asked about the usefulness of the PhD experience. Among the ten available answers, we present four that reflect different aspects of doctoral education and reveal interesting similarities and differences between the different demographic groups. Our survey population included African Americans, Chicanos, Latinos, and Native Americans, primarily in English and political sciences; and Asian Americans, primarily in science and engineering fields. Individuals from all ethnic groups reported high levels of intellectual satisfaction from accomplishing the Ph.D. Most indicated that achieving the doctorate increased their self-confidence, with Asian Americans reporting this more often than other groups. Chicano/Latino and African American Ph.D.s particularly appreciated the professional contacts their doctoral studies provided. That the Ph.D. functioned, as a "union card," granting access to a variety of jobs, was very much appreciated by African Americans, Chicanos/Latinos, and by Caucasians.

A third highlight concerns advice for improving graduate education. What suggestions did these Ph.D.s have for improving graduate education, and what advice are they passing on to future generations of graduate students? Despite the fact that we surveyed doctorates from six very different fields, the responses were fairly homogeneous. First, they recommended that doctoral programs offer a curriculum that provides breadth and interdisciplinary exposure. The curriculum should be consistently updated for currency, so that students complete the doctorate with marketable skills. For those three fields with graduates employed primarily in academia in 1995, respondents recommended that graduate programs take seriously the task of teaching graduate students how to teach. In biochemistry and English, two fields that were facing a limited academic job market (remember, in the mid-80's, biochemistry students were not being prepared for jobs in the country's biotechindustry; they were being prepared for academia), respondents recommended downsizing doctoral programs. Respondents in engineering fields recommended incorporating more hands-on experience into doctoral studies, preferably some form of internship. Respondents also recommended that doctoral students be offered more direct contact with professionals in the workplace over the course of their doctoral studies. Biochemists recommended repeatedly that doctoral programs provide students with information about business, government, and other employment sectors outsideacademia.

The fourth and final highlight, what advice to graduate students just beginning the doctoral program did respondents offer? Again, the responses among the six disciplines were relatively homogeneous. The number one ranked recommendation was to maintain an interdisciplinary focus, to go for breadth. The second ranked recommendation was to achieve focus early on in graduate study and to clearly define one's goals. It is interesting, however, that neither English nor electrical engineering doctorates thought that defining goals early was particularly relevant. Furthermore, graduates in fields faced with a tight job market, biochemistry, English and mathematics recommended pursuing fields you love. The two fields with a higher proportion of graduates who held professorships in academia, English and political science, recommended that students publish and present papers at national conferences while in graduate school.

To conclude, overall we found high employment rates (less than 2% were unemployed), high satisfaction with the value of the Ph.D. and the opportunities, prestige, and esteem it afforded, and high satisfaction with current employment, particularly in business, industry, government and the non-profit sectors. Although not presented here, the results of the study provides us with a clearer picture of the challenges women Ph.D.s and dual career couples face, and we know more about factors influencing the career paths on international Ph.D.s. The study also collected information with which to benchmark program effectiveness. But we also learned that there are significant challenges, and serious needs for improvement. We must spend more time, money, and effort on career planning and placement activities for our doctoral students. In addition, we need to work with faculty to broaden the acceptance of BGN careers as legitimate and desirable Ph.D. outcomes. We must continue to support all students to complete their degrees. We must also continue to address the conditions for our postdoctoral appointees. We further suggest improving assessment of program evaluation by students. Finally, we must develop creative solutions for dual career couples within the university.

We hope that the results and recommendations from the PhD.s--Ten Years Later study will assist the various groups assembled at the "Re-envisioning the Ph.D." conference in making the kinds of changes and augmentations to current practices that will address the need for improvement in graduate education and doctoral program outcomes assessment.

The Ph.D. Survey

Chris Golde, Assistant Professor, Educational Administration, U. Wisconsin.

Good morning, it's really a delight to be here. My colleague, Tim Dore and I worked together on a survey on doctoral education with the goal of entering the debate about how to improve the preparation of graduate students. The goal of this project is to provide a snapshot of the lived experience of today's doctoral students. My presentation today focuses on some of the survey's findings. The data I am presenting are preliminary. Please visit the web-site of the project for up-to-date data and a copy of the project report.

We asked doctoral students to tell us the details of their experience in their doctoral programs, their career plans, and in particular their preparation for faculty careers. We offered the survey in two formats, both web-based and paper-based, and had about 50% of our respondents using each of those two media.

We administered the survey in the summer and fall of 1999 to a sample of currently enrolled doctoral students in their third year and above. Our effort was to get students who were somewhat more socialized into their discipline, whom we hoped would have thought about their careers a little bit more thoroughly, and, since we know from some of Maresi Nerad's work that much attrition happens in the early years, we wanted to really select the students who probably were going to complete. We worked with 27 universities and one cross-institutional program (the Compact for Faculty Diversity), and we selected 11 arts and sciences disciplines. Overall, we had over 4,000 respondents, for a 42% response rate.

I am going to talk to you about findings in three areas. The first area is the purpose of doctoral education, the second is the content of doctoral programs, and third is the mechanisms and processes. For each area, I'll give a bit of an overview and then talk a little bit about implications for the future of doctoral education.

Our first main finding is that what doctoral students are trained for is not what they want, nor does the training prepare them for the jobs that they take.

Most of today's doctoral students, despite a decade of media attention, reports on faculty life, and activism on the part of students and professional associations, are still aiming at careers in faculty life. Those aspirations differ by field. The highest percentage of students aiming for a faculty career was 88% among the philosophers and the lowest was 37% among the chemists. Those are our two end points.

We also found, however, that students reported a decline in their interest in faculty careers from the time they had started their programs. Thirty-five percent of respondents said they were less interested now in a faculty career than they were when they started their program. Conversely, there was an increased interest in all kinds of other career possibilities, in particular those that involved research, both in the profit and non-profit sector. Finally, we found that women reported being less interested in faculty careers than their male colleagues. Moreover, those women who were interested in faculty careers were more interested in working in community colleges, liberal arts colleges, and comprehensive universities, than their male counterparts. Whereas the men were more interested than the women in working in large universities.

We also found that what interested students about faculty jobs, and what they were prepared for, really didn't match. Students are interested in performing a wide range of roles in teaching, research, advising and even service, and yet they only feel prepared to do research, publish, and lead discussion sections. This is not surprising, because these are the tasks that they do as doctoral students. It was also encouraging to find that students are very interested in what I think of as the civic role of academic life, that is to say, working with undergraduates outside of the classroom, serving on academic senates, working on service to the community, etc. But, again, they report that they are not prepared by their doctoral programs to perform these tasks.

We were also interested in knowing what aspects attract people to a faculty career, and what aspects repel them away from a faculty career. It is an idealistic vision of a faculty life, a faculty lifestyle of teaching and research and service, in combination, that attracts people to a faculty career. But it's the conditions of the work, the perceived inability to raise a family and lead a balanced life, the presumption that the workload is extremely difficult, that the tenure process is problematic and onerous, and that the job market is very constrained, that are leading people to think about other career options. The implications of these findings about the purpose of doctoral programs, are that:

  • Efforts to diversify the perceptions of different career paths need to continue. That work is not done.
  • Future faculty need a broader understanding of the range of roles that faculties undertake, and they need to be prepared for them.
  • Future faculty need to be prepared for life in the kinds of institutions in which they'll actually take jobs, which are not research universities. Many of the perceptions about faculty life comes from seeing life on research university campuses.
  • To ignore all these civic roles that faculty members historically played on campuses is a very dangerous thing for the future of our universities.

We turn now to the second set of findings. We asked people about their experiences and the types of structures in their doctoral programs. What we found, not surprisingly, is that doctoral education remains primarily and predominantly training in research. It is, if you will, an intensive training rather than a rich one.

Students are calling quite clearly for additional breadth. Over a quarter of the students said they wished they could take more classes outside of their departments. In particular, we asked them what kind of courses, and they said: business, computer sciences and the humanities. We heard interest in the humanities from students in the sciences as well.

We also asked about professional development opportunities. Respondents reported that most available was the opportunity to present research at conferences, and least available were the opportunity to take trips to other kinds of college campuses or to take workshops on how higher education works. In addition, we also discovered that students at the same campus have very different, sometimes non-congruent perceptions of the kinds of professional development opportunities that are available. There is no consensus on what exists on a given university campus. But, if students do perceive an opportunity to be available, they take advantage of it at fairly high rates. Two-thirds of students have participated in workshops on the academic career, on ethics, and on how higher education works. Another third has taken advantage of trips to other campuses, internships, and the opportunity to participate in campus government. Finally, students who did participate in professional development opportunities reported themselves being better prepared to be faculty members. The implications are that: This kind of program flexibility, and opportunities for a range of experiences, needs to continue to be encouraged. These opportunities need to be made available and widely publicized; people simply do not know that they exist.

We turn now to the third category, which is about the process of doctoral education, by which I mean the mechanics of doctoral education administration. The main issue is that many students do not understand what the process is, even when they have been enrolled for three or more years. There appears to be a mismatch between people's expectations and the reality of doctoral student life that is the source of a good deal of the tension and unhappiness that we hear from our students.

Over a third of the students expressed dissatisfaction with various aspects of their doctoral program. A third said that their course work had not laid a good foundation for doing research, and had not imparted broad knowledge in their field and related sub-fields. A third said that their program was not flexible.

I want to focus on the finding about whether students have been annually reviewed for a moment. Many of the recent reports have expressed a consensus emphasizing that good practice in doctoral education includes reviewing students annually. Yet half of the students in our 1999 sample reported that they were not reviewed on an annual basis. Intriguingly this really varies by discipline. Over 80% of the psychologists were reviewed annually, but only about a third of those in chemistry and English were reviewed on an annual basis. It's also very clear that there are some institutional policy factors at work. At the University of California, San Diego, for example, it is a matter of institutional policy that all students be reviewed, and over 90% of the students there reported having a review.

Finally, less than half the respondents reported having a clear understanding of the core, basic mechanisms of doctoral education. They said they were not sure how long their degree program would last; they were not clear on what the criteria would be used to determine when they were ready to graduate; they were not clear on how much time they would be expected to spend with their advisor, and they did not know where the funding for their research would come from.

The implications for practice are pretty clear.

  • Predoctoral students need better information to develop realistic goals about graduate school. They are working in an information vacuum. I think the responsibility for information lies both with the bachelor's-granting institutions, the sending institutions if you will, and the institutions that enroll them, the receiving institutions. No doubt there is also room for professional associations and other organizations to weigh in on that.
  • We also have some intriguing pointers about change and that change will result both from changing the disciplinary norms as well as from well-developed and well-implemented institutional policy.
  • I hope that this overview of our findings, and thinking about the purpose, the content, and the process of doctoral education, stimulates ideas about changing practices that would really help improve the nature of the enterprise.

    Thank you.

    The Development of Graduate Students as Prospective Teaching Scholars

    Ann Austin, Associate Professor, Higher, Adult and Lifelong Education, Michigan State University

    Good morning. I'm going to be talking about a project called "The Development of Graduate Students as Prospective Teaching," a longitudinal study which has been underway for the past four years. We are grateful for the support that has been provided by The Pew Charitable Trusts and by the Spencer Foundation. My colleagues on this project are all in this room: Jody Nyquist of the University of Washington, our host and coordinator for this very interesting conference, Don Wulff of the University of Washington, and Jo Sprague of San Jose State University. We were fortunate also to work closely with a group of wonderfully able graduate assistants, one of whom is Bettina Woodford, who is now very involved as the Project Manager for the Re-envisioning Project.

    The research questions we were addressing focused on the graduate student experience. How do prospective teaching scholars experience graduate education? How does their understanding of faculty work change during the graduate experience? What factors in graduate school contribute to their understandings about faculty work? We were interested in learning whether there are developmental patterns in their experiences as they go through graduate school. Our study used a four-year longitudinal, qualitative design, and we focused on graduate students at three institutions: two research-intensive institutions and one comprehensive university. We began with almost 100 participants, and we have followed 55 of the original 100 students throughout the four years of the project.

    Our data collection consisted primarily of intensive interviews. The voices of graduate students that emerged through this study constitute one of the special contributions of this project. Because we've been talking with these students for four years, we know them very well, and many of them actually say that we are among their greatest confidantes about their graduate experience.

    In this presentation, I will highlight some of the findings from the study that may be of particular interest at this conference.

    We've come to understand, in contrast to our initial working hypotheses, that graduate student development is neither linear nor are its stages fully predictable. Rather, we are discovering that graduate student development is very much shaped by a number of factors. Factors that differ by individual affect how a person experiences and develops in graduate school. Such factors include age, educational background (for example, whether one attended a liberal arts college or not), family situation (whether there are teachers in one's family, or whether one is married or partnered, or has children or not), and previous employment (whether, for example, one has prior teaching experience). Other key variables that affect graduate student development are the extent to which a student's locus of control is external or internal (that is, the extent to which a person perceives that he or she has the power to make decisions and manage the graduate experience); the extent of a student's sense of self-efficacy (that is, the extent to which the student perceives that he or she has the ability to do what is expected); and the extent of a student's ability to make connections with people and opportunities.

    We've also found that graduate students' experiences are shaped by the explicit and implicit messages that they receive and interpret as they go through graduate education--messages from ongoing interactions with peers, with faculty, and with friends and family. And certainly, specific disciplinary and institutional contexts are very important shaping experiences as well.

    More specifically, I want to highlight three particular themes that we have identified in the graduate students' voices. I will illustrate my comments concerning these themes with the metaphors and pictures that we asked participants to draw in the middle and at the end of the study to capture their experiences. These pictures will give you a sense of their experiences from their own perspective. I also want to emphasize that we were talking with graduate students from a range of disciplines in the humanities, social sciences, and sciences. The particular pictures and graduate experiences to which I will refer have been chosen from a range of those disciplines.

    The first theme that I will highlight is the challenge graduate students face to balance the individual passions that they bring with them into graduate school, with the interests and expectations found within their disciplines, departments, and institutions. We have come to realize that students often enter graduate school with particular driving passions and questions, and sometimes they have a hard time linking those passions with the interests of their faculty or their departments. In fact, participants in our study have commented that conversations typically are infrequent about how their own passions may or may not link with the emphases of their departments, disciplines, or institutions.

    This issue of "fit" raises questions, we believe, for all of us to be thinking about. How can we help students balance their own passions and interests with departmental and disciplinary emphases. How can we help students to specialize, as graduate study requires, while also encouraging them to connect with the bigger picture in their disciplines? One of the student's depictions of his journey through graduate school illustrates this problem from the point of view of a graduate student. This picture was drawn by a male doctoral student in mathematics, and the components of the drawing say a lot. It starts out with a tree. The three panels of the picture represent the student's journey from the masters through the rest of his graduate education. By the third panel, the tree is blooming quite nicely. As the student progresses through his doctoral program, the tree begins to appear more and more pruned back, and it starts to look a little pathetic and bare. When the student projects his journey to graduation, represented by a graduation cap, the tree is looking rather sad. Thedoctoral student expects that, by the time he reaches this point in his graduate school journey, the specialization required of him will have withered his passions or interests.

    When Don Wulff was interviewing this individual, he was thinking that, in the student's mind, the picture meant "The tree eventually shrivels up and dies...That's what will happen to me." The student, however, did not actually expect such a dire outcome. He anticipated that, once out of graduate school and into a faculty position, he would bloom again--and if the picture were to go into the future, the tree would be blooming again in the sixth panel. This depiction communicates the tension, as experienced by a doctoral student, between doctoral-level specialization and the student's passions as these are constrained by the graduate experience.

    The second theme I want to highlight concerns the lack of systematic, developmental preparation of graduate students for both academic and other careers. In our data, graduate students report not feeling particularly knowledgeable about what a faculty career involves, and even less knowledgeable in many cases about careers outside of academia, a theme that is also being mentioned at this conference. This lack of knowledge raises important questions: How can universities provide opportunities for graduate students so that they really understand both the faculty career, with its multiple components, and other career paths that might be options? A picture, drawn by a female doctoral student in English, underscores this theme. In this picture, she depicts her thoughts about her possibilities for the future. In the far left of the picture, she has drawn herself saying, "Yikes!" The rest of the picture depicts her realizing that "Gosh, I could be a teacher." But, she has also scribbled, "I could be a scholar" as well as "I'm a student," "I need to develop as a person," and "Now what is this Academy all about?" This doctoral student explained that the pieces of the picture are all disjointed, and no one in her graduate program is talking with her about how to navigate her own development and professional goals.

    The third theme I want to highlight concerns reflection. We learned in these extensive interviews that graduate students truly value the opportunity to be reflective about such issues as their graduate experience, their goals, their options, the meaning of faculty and other kinds of work. On a number of occasions, students told us that our regular interviews with them over a four-year period were actually the only occasions in which anyone in the academy was engaged in serious conversation with themabout their goals, about how they thought about careers, or about how they were developing as teachers, as researchers, as prospective scholars. Learning about the value of reflection--and how infrequently reflective conversations with faculty apparently occur for at least some graduate students--- leads us to ask some probing questions: How can we incorporate those kinds of reflective experiences into the doctoral program? What should be the role of faculty in encouraging that sort of reflection? What might be the departmental and graduate school roles in fostering such opportunities for reflection? What responsibilities do students themselves have for engaging in such reflection? The picture that illustrates a positive view of this issue was drawn by a doctoral student in English. Her picture of doctoral education shows her progressing down a path with various twists and turns, including places where she has taken time to take stock and reflect on her progress and next steps. In the picture, she uses flowers to show some of these important times of reflection.

    Finally, an overall observation. The experiences of participants in our study lead us to believe that current approaches to graduate education often are not systematic or developmental. Students often do not receive systematic feedback and mentoring. I want to conclude with a final picture, drawn by an engineering doctoral student who has been fairly successful. In the picture, he has portrayed himself running a marathon. The path displays signposts, but he explained that the signposts are not accurate. He gets to various intersections and doesn't know where he is supposed to go, and unfortunately, nobody has given him good directions. Then some geraniums up on a building ledge start falling down on him, a few logs obstruct his path, and a car is about to hit him. Fortunately, he was able to explain, "You know, I've been pretty successful despite these things," but he also carefully emphasized that so often unexpected and unnecessary barriers occur in the graduate student experience.

    Based on the themes that have emerged in the voices of the doctoral students we have studied, we believe that faculty advisors, department chairs, and graduate school deans should consider specific ways to strengthen the graduate experience. The discussions in this conference are bringing to light a number of useful strategies. My colleagues and I particularly urge faculty members and administrators to ensure more opportunities for graduate students to be supported as they reflect on their goals and plan their careers, more regular and systematic feedback about graduate students' development as researchers, teachers, prospective faculty, and disciplinary members, and more developmentally-oriented experiences as they move through their graduate education. The voices of the graduate students whom we have interviewed speak of their commitment, energy, and ideals but also of their need for more support and guidance from the faculty and administrators at the institutions where they study.

    The Graduate School Survey

    Geoff Davis, Researcher, Microsoft Research.

    Good morning. I'm here from Microsoft Research. Before beginning, I just want to point out that my discussion is based on research that I am conducting on the side with my colleague Peter Fiske, a scientist at Lawrence Livermore National Laboratory, and not something that is an official Microsoft project.

    The motivation for the Graduate School Survey was a set of recent reports calling for some fairly significant, systematic changes in graduate education. The COSEPUP Report, which Philip Griffiths will be presenting on during this conference, reports from the Association of American Universities, the National Research Council, the National Academy, and many other reports that have been issued, all called for similar kinds of reform. The question that Peter and I had was, "To what extent are these various best practices or recommendations being implemented in various programs?"

    The intent of our survey was to assess the levels of implementation of these various best practices in individual departments and programs by asking the graduate students themselves. The plan was to conduct our survey on-line, to break the results down by department, and to present the results on-line, on the web, at the level of individual program. Our goals in doing so were to spark a national discussion on graduate education, to challenge institutions to examine their own educational practices to see if they were doing all that they could possibly be doing to promote excellence in their education, and to catalyze implementation of these best practices.

    The survey that we conducted consisted of 46 questions divided into eight different sections reflecting areas of concern that were raised in the earlier reports. We asked about information that was available for prospective students at the time of application to the program. We asked about the breadth and the flexibility of the curriculum in the students' programs, and we asked about the level of preparation of students for their teaching experiences and the kinds of follow-up that they received. We asked about course work in professional ethics and professionalism in general. We asked about the career guidance and placement services that were available to students. We asked about the time to degree and the mechanisms that exist within departments to control time to degree. We asked about the mentoring that students receive, and about the climate of the department, including such things as whether women and underrepresented minority students felt that their environment was a supportive one. And finally, we asked questions about overall satisfaction. Again, these questions were motivated primarily by specific recommendations already called for in the various reports.

    The survey went on-line on April 25, 1999 for a period of about 10.5 weeks, through July 8, 1999. The survey was publicized in print and on-line. An article appeared in the Chronicle of Higher Education (and also in the web-based edition), and a blurb appeared in Science. Additionally, articles appeared in Science's Next Wave, an AAAS on-line publication targeted at young scientists, and in the HMS Beagle, an on-line magazine for life-sciences students and faculty. We also did "viral marketing," in which all survey participants were asked to send an announcement to their colleagues, and we provided a web-based tool that would enable them to do that easily.

    Over the course of ten weeks, we gathered some 6,500 responses. We got about 100 responses per day, representing altogether 1,800 different departments in some 250 institutions.

    As someone who has taught statistics for many years, I should emphasize that the survey is not a controlled experiment; it's an observational study. The respondents were self-selected, not a simple random sample, and as such there may be some kind of self-selection bias. So, the results may not be representative of the graduate population as a whole. However, we believe that these reports are important and we have compelling evidence to suggest that the results do, in fact, reflect genuine concerns of the overall graduate student population.

    First, the responses we obtained came from a broad cross-section of the graduate student population. If you break down our results by demographics, that is, by gender, ethnicity, discipline‹ all the different types of variables, you see that the participation rates were roughly proportional. That is, they were comparable to the percentages of these different types of categories in recent cohorts of Ph.D.s as measured by the National Science Foundation's annual survey.

    Second, many people have raised the concern that our survey would become a magnet for disgruntled students, and that we would merely be providing a voice for people who were simply dissatisfied and wanted to vent. But in fact, on the contrary, we found that the vast majority of respondents expressed overall satisfaction with their graduate education and with their advisors. In fact, we borrowed a number of questions from Professor Chris Golde's survey. In cross-validating, or comparing, our results with hers we found that, in our survey, respondents on comparable questions were almost uniformly more positive in their assessments of their own graduate educational experiences than they were in Chris's survey, which was conducted with more traditional methods and with a much higher response rate. So, what we see is that our respondents appeared to be energetic, overall positive students who were more or less satisfied with their graduate program but who nevertheless had some serious concerns about it. I want to emphasize this point because students' responses do raise a number of serious concerns. And, again, these graduate students appear to be the happy ones.

    But, what's important about the survey is not so much that it's a particular set of numbers describing a snapshot of the present. What's important about the survey is that it is an ongoing process, and as a process, it is a fundamentally new form of feedback that is now available to the Academy. It can enable programs to get a better sense of the true state of graduate education at their institutions and increases the transparency of the educational process. What's really significant about the survey is that it offers graduate students a way to engage collectively, in a responsible and constructive fashion, in the process of improving their own education. This is really the first time that the student voice, in aggregate, has been available in the discussion.

    So, I am going to just give a brief overview of the results. The good news is that, as I mentioned earlier, the majority of the students who responded expressed overall satisfaction with their educational experiences and they had good things to say about their advisors. It's important to keep this in mind when reading about the bad things. The bad news is that, actually the results can be quite sobering. The concerns the students raise echo all those mentioned by the previous speakers on today's panel. The students say that:

    • The information available to them at the time of their enrollment was inadequate.
    • The training and supervision of teaching assistants is not up to par.
    • Their curricula are too narrowly specialized.
    • The career guidance and placement services that are available to them are not effective.
    • Safeguards against adverse advisor/student relations are lacking; there is no office or faculty member to whom they can turn if they have a problem with their advisors.
    • And, finally, graduate program environments are not always supportive of women and underrepresented minority students.

    As I mentioned earlier, this survey is an ongoing process. We have just launched a follow-up survey a couple of weeks ago, funded by a generous grant from the Sloan Foundation, and the response has been quite dramatic. Students really want to have their voices heard. In the first two weeks in April we received responses at a rate of 500 or more a day. So, already we have exceeded the total number of responses of our 10-week survey last time.

    The follow-up survey was expanded to cover humanities and social sciences, and to reach Canadian universities as well. We have made numerous design and methodological improvements, and we have had all the questions reviewed by a number of researchers. We have upgraded the privacy protections that are available and the verification of identity. The current survey is being administered by the National Association of Graduate and Professional Students.

    I invite you to visit the survey at The site enables you to easily invite your own students and those in other departments to take part. There are resources describing the survey's background, sample announcements, banner ads that you can put up on your web site, and links to the results of last year.

    In conclusion I would like to emphasize one of the consistent themes in all of the reports that I have cited, and that is the importance of publicizing outcome data. Right now, one of the most important sets of data points that students use in selecting a program are reputational measures from such organizations as the National Research Council and US News. As the adage says, "We treasure what we measure." Unless we look at measures of educational excellence aside from faculty reputation, other important things will continue to get lost. Public outcome data provides a second axis along which students can compareprograms.

    One of the most important things about outcome data is that it helps better align departmental interests with student interests. Students want good outcomes from their educations, and publicizing outcome data gives departments a strong interest in working to ensure that their students have good educational outcomes. Ongoing surveys like ours can provide the stimulus for some really healthy competition between departments based on the implementation of educational best practices and innovations. This is the kind of competition that is good for students, good for the departments, and good for the process of academic inquiry. Thank you.

    Reassessing the Assessment of Research Doctoral Programs

    Charlotte Kuh, Executive Director, Office of Scientific and Engineering Personnel, National Research Council

    What I would like to discuss today is how to bring some measures of graduate education into the National Research Council assessment of research doctoral programs, often called the NRC rankings. We are developing a new NRC assessment of research doctoral programs and, to that end, we plan to spend a couple of years looking at the methodology of the last survey and determining how we should change it. Then, we'll spend the following three years conducting an assessment study, and then analyze the new data collected in the course of that study.

    Why are we spending so much time on methodology? Because there are a number of problems with the current assessment measures. I have been spending this year talking to people about this and I've been asked lots of questions at all different levels and parts of academia. The first issue is that direct reputational rankings have problems. The second issue is that the taxonomy needs to be reexamined. I recently spoke with faculty at the University of California-Davis. It turns out that over 60% of their graduate programs fall outside of the taxonomy used by the NRC in the last study. We also had serious problems finding an appropriate taxonomy for the biological sciences. Third, we need better measures of scholarship, especially in the humanities. Fourth, and finally, we want to be a resource to understand change in research training, and although our last study provided a lot of numbers, it really didn't provide a lot of guidance as to the use or interpretation of the numbers.

    I'd like to discuss the dual purpose of this study because it is really quite important. The purpose is primarily to provide the Academy, that is, university administrators and faculty, with a set of common measures, both quantitative, and reputational. These measures will permit a comparison of programs that address similar fields of knowledge. Note that I didn't say departments. The study will also assist potential graduate students by providing common measures of graduate education, and it is this objective that I need your help with.

    Second, the achievement of these purposes will assure the external constituencies of research in doctoral education that the Academy defines, monitors, and seeks to improve the quality of research education. We don't have good measures of the quality of graduate education, but it is something that we are looking at. We want to assess ourselves using our values in a way that is methodologically acceptable. Within the Academy, the study will be useful in program review, assessment, and comparison of similar programs in other institutions.

    There are many changes in doctoral education that this study needs to reflect. The first change is the mushrooming of multidisciplinary programs. The second change is the growth in business/government/non-profit (BGN) careers outside of academia. The third change is that there is much more university-to-university collaboration, especially in the sciences. Fourth, there has been a significant increase in research doctoral programs in the professional schools, and in the past our study has looked primarily at arts and sciences faculty. Finally, there were a few measures relevant to diversity in the last study, but they were not discussed, interpreted, or highlighted. With the ever-changing student body, we really need to get a much better handle on the changes in diversity.

    What, then, should we measure to encourage re-envisioning of the Ph.D.? First, we need to understand things in context, to understand the environment for doctoral education. Second, we need to understand better the kinds of outcomes, the kinds of placements, for doctoral students. We can learn some of that from the NSF data, but we need to know more. We also need to know more about employer satisfaction with the Ph.D.s that they hire and the satisfaction of the graduate students themselves with the process of achieving doctorates. We also need to take into account that there are different kinds of missions in different programs. Not all programs aim for national prominence. Many serve the research needs of local employers. It would also be helpful if we could find some measures of effectiveness of use and exposure to information technology.

    In the process of developing the methodology, we must admit to ourselves that we really don't understand the process of doctoral education very well but also that we are not going to be able to wait for a great body of literature about the process of doctoral education. But we do need to know what the good practices and good outcomes are, and how they can be compared across many, many doctoral programs. We need not only to engage in good measurement of the process with comparable data, but we have to relate outcomes to the process in order to develop a theory. Is the process just individual growth, or is it actually something that results from the process of graduate education?

    Finally, assessment itself can encourage change. Although people often think of assessment as something to contend with, in fact, assessment provides useful information to institutions and to the public in a public way. What is being measured tells the world what we think is important as an indicator of the value of doctoral education. It provides incentives to change and it also provides measures of it. I think the new NRC rankings will provide that encouragement for change. Thank you.

    Re-envisioning the Ph.D.: What are Our Concerns?

    Jody Nyquist, Director, Center for Instructional Development and Research, University of Washington

    This has been a very exciting panel because it is clear that there is a significant level of agreement among our findings, that is, similar issues have been articulated across the student voices and across the differing experiences, disciplines and ways of surveying them. It is also exciting that the work Charlotte Kuh is doing has a real chance of really effecting change, much like some of the other strategies that we are trying to investigate through our collective work and collaborations at this conference.

    Let me tell you just briefly about our project. We set out to conduct an environmental scan, to sample opinions about the Ph.D. from each of numerous members of the nine sectors of stakeholders in doctoral education. The relationships among these sectors can be characterized as a loose, interdependent set of partnerships that comprise doctoral education: research institutions who prepare Ph.D.s, teaching-intensive institutions who hire Ph.D.s, business and industry who hire Ph.D.s, K-12 education who also increasingly hire Ph.D.s, government agencies and foundations who fund and also hire Ph.D.s, doctoral students themselves who obtain Ph.D.s, and disciplinary societies and educational associations who influence the Ph.D. process. We also interviewed some members of governing boards and accrediting agencies who approve Ph.D. programs. Altogether, we conducted over 375 interviews, primarily with people who held the Ph.D., or people who held the same positions as people who have Ph.D.s. Our sample was based on availability in the sectors -- these are people who agreed to talk to us. Our communications were held largely in situ, that is, we often visited with the interviewees' in their own environments &endash; campuses, businesses, non-profit offices, government offices, labs, and conferences -- to get a sense of the contexts for concerns about Ph.D. education. We also conducted several email surveys and numerous phone interviews. We found that there was a great deal of difference in perception about what was going on in doctoral education and what needed to be done. One of the key observations that most everybody was concerned with, however, was that we were preparing doctoral students for which there seemed to be no career. This observation speaks to the issue of over-supply.

    We tried to ask then, "How did this happen?" I believe that it is the unintended consequence of some very important needs that we as a society, and as educators, were trying to meet. First of all, many public institutions have been facing burgeoning undergraduate student populations and having to respond to this growth with stable or even diminishing funding. That meant having more graduate teaching assistants to teach more courses. So, meeting a really important societal goal also ended up contributing to the production of more Ph.D.s than there are positions available in academia. Second, society's thirst for scientific advancement, including large scale labs, over the last decade or two created a greater demand for research assistants and, in some fields, created an oversupply of scientists. Third, to echo what Charlotte Kuh has discussed, higher education's commitment to self-improvement has been based upon the Carnegie Classification System and the National Research Council Rankings. That basis meant privileging the research model, thus driving many institutions to improve their status based primarily on research production, resulting in an rise in the number of Ph.D. programs across the country.

    Thus, we ended up with more Ph.D.s than can be accommodated in higher education. Now, is that a bad thing or is that a good thing? I think that's what we are all trying to sort out at the moment. Some would say we need to reduce the number of Ph.D.s awarded. Others argue that we have these Ph.D.s and that society very badly needs them for their problem-solving abilities and creativity. What we need to do is re-think the state of doctoral education, and that's precisely what the oversupply issue has enabled and prompted us to do: to rethink the purpose of doctoral education, and the kinds of training that Ph.D.s undergo.

    The members of these sectors use somewhat different language even when talking about the same issues. Business and industry, for example, will talk about Ph.D.s coming out of their programs with a "disconnected specialization." In other words, they cannot connect what they have been studying to the larger concepts or the larger operational issues with which a business deals. Institutions other than research intensives echo that same theme. The faculty they hire are asked to teach across disciplinary boundaries because many teaching intensive institutions do not have discipline-specific departments like research intensive universities do. That kind of interdisciplinary, pedagogical preparation is not happening enough in doctoral education as far as the teaching-intensive constituency is concerned. But even research intensive universities are likewise saying that the ability to make innovative discoveries is crucial to the kinds of research questions being asked in the complex, 21st Century world, and that ability demands a multi-disciplinary approach. So, although the language to articulate a problem may change among the sectors, the different stakeholders really are voicing a high level of agreement over similar concerns.

    But, there are also areas of strong disagreement over doctoral education, articulated with much passion and conviction in the comments heard:

    • whether the Ph.D. is research certification only or whether it is broader professional preparation;
    • whether it is to prepare students as academics or to prepare students for a variety of career options; 
    • whether Ph.D. programs should be limited in size, admitting very highly selected students only, or whether anyone who is eligible to do a Ph.D. program should have the opportunity;
    • whether there is a need to decrease the number of Ph.D. programs currently in existence, or whether we need to increase the number of Ph.D.s because that deep analytical capacity could be applied to so many societal needs;
    • whether US students ought to be privileged over international students for admission, particularly in the sciences and engineering, or whether international students should be freely encouraged to apply because of our society's need to operate in a global context;
    • whether funding practices work well now, or whether funding practices need to change;
    • whether the best model of doctoral training is the apprenticeship model or whether some other model, one that includes multiple mentors and a less campus-anchored approach, is needed;
    • whether the current model attracts the best and the brightest to the Academy, or whether the current model actually discourages the best and the brightest from choosing the Academy.

    So those are our areas of disagreement.

    We considered it terribly important, however, to try to map out areas where we did find agreement so that we might think about ways to change doctoral education that stakeholders from the different sectors would really be able to support. This whole effort has been based on the fact that, in our belief, the opinions of decision-makers involved in preparing doctoral students, of those who hire them, and of the doctoral students themselves, have a great deal to do with the reshaping of doctoral education. Those opinions are what counts when foundations fund initiatives. Those opinions are what counts when students decide whether they are going to go on for a Ph.D.

    In our analysis, six areas of overlapping agreement emerged which held particular promise for change: 1) Shortening time to degree and determining the essence of the Ph.D.; 2) developing more diversity among the recipients of Ph.D.s (we say "recipients" because diversity raises not only issues of pipeline but issues with completion rates, and that's an important distinction); 3) increasing doctoral students' exposure to technology, especially instructional technology; 4) preparing doctoral students for a wider variety of professional options; 5) making interdisciplinary work a more integral part of doctoral education; and, finally, 6) incorporating an understanding of the global economy and the environment. As you know, our Advisory Board selected three of those issues to focus on specifically in this two-day conference, although re-envisioning the Ph.D. in its broader sense involves making change in all six areas.

    I would like to conclude by emphasizing that despite differing vantage points in the doctoral education process, there is a very obvious, shared set of concerns to be dealt with in re-envisioning the Ph.D. We wouldn't have this conference if we didn't think real progress could be made by collaborating together to effect change in doctoral education. I look forward to using this venue as fertile ground for the ideas and energy that everyone has brought here. Thank you.

  • Funded by the PEW Charitable TrustSitemapContact UsSearch