Keywords: mixed-methods, design, practice
Research Design Becomes Research Reality: Colorado School of Mines Implements Research
Methodology for the Center for the Advancement of Engineering Education
The Center for the Advancement of Engineering Education was founded in 2003 with five collaborating institutions.
A multi-institutional, multi-year grant offers many opportunities for the demands of reality to interfere with
design goals. In particular, at Colorado School of Mines (CSM) student demographics required adjustment of the
original APS research design. The following paper describes the challenges and solutions of the recruitment process
at CSM.
Method and Background
The Academic Pathways Study (APS) element of the CAEE involves surveys, structured interviews, ethnographic interviews,
and a performance task. The following describes the campus-specific implementation of this collaborative study and how
research design implementation must conform to the reality of a specific target campus and its representative population.
To enact fundamental change in engineering education, cross-campus and multi-method research like CAEE
will be ever more common. Managing the transition from research design into practice is critical for success.
|
In the first year of the APS, primary tasks were selecting the participant pool and collecting data through ethnographic
observations, on-line surveys, and interviews. Most CAEE communication takes place by e-mail and telephone conferences
due to the great distances between campuses and the large number of researchers involved. In the first year, conference
calls were used to clarify, refine, and implement the research, and two face-to-face meetings of research team members
were held for making decisions about research design and policy.
In the first year of the study (academic year 2003-2004), CSM had 750 incoming, first-year students. To be eligible
for the APS study, CSM participants had to be May/June 2003 high school graduates, eighteen or older by October 1, 2003,
and a US citizen or permanent resident. CSM also required participants to have enrolled in or have the intention to
enroll in an ABET-accredited engineering major (3 CSM majors were excluded: chemistry, math and computer science,
and economics and business).
The CSM sample design was fifty percent female, fifty percent male for both study and control groups. Fifteen
participants would be White/Asian, five would be Latino/a, Native American, or African American. CSM’s total APS
population would be eighty students: forty enrolled in the control group, thirty two participating in interviews
and surveys, and 8 participating in all APS methods. The control group was designed to verify
whether or not participation in APS would affect the success of study/ethnographic participants. In year two, the
control groups were disbanded at all institutions in recognition of the impossibility that participation would have no effect, given the
frequent contact participants have with the researchers.
What We Found
CSM's size presented additional challenges to fulfillment of the study design. First, there were personnel constraints since
CAEE was integrated into CSM's one-person Center for Engineering Education (CEE) at CSM. Second, the primary researcher left
temporarily for maternity leave in the first year and was replaced by two other researchers. Third, REU students were
employed as part of the multi-method team. However, CSM's small size required careful attention to detail to protect the
confidentiality of APS participants who might share classrooms, living accommodations, or dining facilities with undergraduate
employees of CAEE.
Participants were recruited at nine events in the 2003-2004 academic year. Researchers received 342 signatures indicating
willingness to participate – 98 were women and 244 were men. Forty-four signatures were from ethnic minorities.
These students were invited to information sessions where interested and eligible students signed consent forms. After
these sessions, the team still lacked enough females to complete the sampling plan and had no African-American candidates.
Two more recruiting sessions for female students were scheduled and the primary researcher attended a meeting of the National
Society of Black Engineers (NSBE) in an attempt to recruit African Americans. Ultimately, team members chose not to pursue
these students further and were thus unable to include any African-American students in the study sample. The team also
had to revise downward the sample of women.
At the end of the recruiting sessions the team had received 112 consent signatures. Forty of these were women, one from
an ineligible major. CSM lacked enough non-Caucasian females to meet its designated sample and instead over-sampled
for ethnic-minority males (Latinos and Asian American/Pacific-Islanders). At CSM, broad representation is a relative term
as it has about 77 percent in-state enrollment, 77 percent males, and 86 percent Whites. The team placed participants
according to majors, hometowns, ethnicity, sex, and the answers to questions on a recruiting questionnaire. Students
who responded “money” as their primary motivation or who selected "academic records only” as their principal level of interest on
the questionnaire were eliminated.
Furthermore, there were ongoing challenges associated with scheduling interviews and
observations that resulted in time pressures. Researchers needed to be mindful of the participants’ value however,
remaining accommodating and tactful even when participants required multiple prompts to complete tasks. Human subjects
approval was also a challenge as CSM lacks an Institutional Review Board (IRB). Howard University sponsored
CSM’s informed consent protocols. As the APS methodology was refined and updated, the informed consent protocols also
required updating and new signatures from participants.
Because of the need to have protocols in place before data collection could begin, and the research imperative to collect
data at the same time on all the campuses, CSM could not begin data collection before subjects were in place at the
collaborating institutions. Thus, students at CSM were forced to wait several weeks before their participation actually began.
Variations among institutional calendars can hinder the effectiveness of cross-institutional research. Anticipating
and planning for this complication could make such multiple-institution research run more smoothly.
Implications of Findings
To enact fundamental change in engineering education, cross-campus and multi-method research like CAEE will be ever
more common. Managing the transition from research design into practice is critical for success. Anticipating
scheduling challenges and coordination of different academic calendars is critical, as is maintaining team-oriented
communication alongside task-based functions. Flexibility is important as research design changes to meet the
pragmatic realities of each campus.
Author: Heidi Loshbaugh, Ruth Streveler, Kimberley Breaux
Source: Proceedings of the 2005 American Society for Engineering Education Conference
The full paper, including references, is available via ASEE proceedings search.
For a printable pdf of this research brief, click here.
Brief created June 2007
Back to Research Briefs