PACE Method Overview
In conjunction with the substantive goals of the project, the PACE research team also is committed to a set of scientific standards whereby the data are valid, reliable and ethical reflections of students' experiences. In order to meet these methodological goals, the team implemented a mixed-mode research protocol in which online surveys are complemented with in-depth, one-on-one semi-structured interviews and focus groups.
Sampling Strategy
Institutions: To reduce variation by site, the PACE research team restricted the PACE project to those undergraduate engineering programs defined as one-tiered. That is, each of the programs either enrolled its students directly from high school into the College of Engineering and/or provided an engineering advisor during the first year to students who indicated an interest in engineering on their college application form.
Individuals: Undergraduate engineering students were invited to participate in PACE based on a stratified random sample rather than a voluntary sample, thereby increasing generalizability to those students at similar institutions not included in PACE. The research team intentionally oversampled women and under-represented minority students as defined by the National Science Foundation (i.e. African Americans, Hispanics, Native Americans and Native Hawaiian/Pacific Islanders) to ensure that these groups would be sufficiently represented among those who completed the survey.
Online Climate Survey
The PACE survey, which was designed and pre-tested to accurately measure undergraduate engineering climate (culture), was administered online in 2008 to students at all 22 PACE institutions and again in 2012 at 16 of the PACE institutions. The new 2012 survey results enable assessment of the change in climate at the PACE schools. Students from all years of matriculation were eligible to participate and the 2008 and 2012 surveys each had over 10,000 student respondents. The surveys covered the following areas:
- Interaction with professors.
- Interaction with Teaching Assistants (TAs, GTAs, GSIs).
- Lab experiences (if applicable).
- Resources offered by the College of Engineering.
- Interaction with other students.
- Involvement with extracurricular activities, organizations and programs.
- Perceptions of engineering career.
- Perceptions of engineering major.
- Personal experiences.
- Transfer student experiences.
- Demographics.
Climate and Leaver Interviews
We completed 179 interviews with students on-site at 16 PACE institutions in 2008 and 2009. Of those, 124 (116 audio-recorded) were with current engineering students and 55 (50 audio-recorded) were with students who had either left engineering for another major or were in the process of leaving. All students were eligible for the interviews and we recruited a diverse sample.
The interviews with current students, ("climate interviews") addressed issues included in the Online Climate Survey and provided students the opportunity to describe, in detail, the environment for engineering undergraduates at their institution. In addition, students were asked to describe the areas in which their department excelled and ways in which it could improve. Finally, students were asked specific questions about their perceptions of female and under-represented minority students and faculty in their program.
Former engineering students, "leavers," were asked similar questions but also asked to describe how they came to the decision to switch majors, how this change affected their career plans, peer and family relationships and college experiences more broadly.
Focus Groups (2012 - 2013)
Focus groups are being conducted with undergraduate students at participating schools.
Focus groups explore student perspectives on overall climate, expected retention and career expectations, unanswered questions from interview transcript analysis, and intervention impacts, providing key understanding of the “why and how” of each.
Focus groups with engineering undergraduates were conducted at five PACE schools in spring 2013. Each school chose an intervention of particular interest, and focus groups were divided between intervention students and non-intervention students. So far, a total of 29 focus groups were conducted, reaching 118 students. Focus groups are currently being scheduled with ten additional PACE schools, and these will be completed by the end of 2013.