Keywords: survey, recruitment, undergraduates
Targeting Undergraduate Students for Surveys: Lessons from the Academic Pathways of
People Learning Engineering Survey (APPLES)
The Academic Pathways of People Learning Engineering Survey (APPLES or APPLE survey) is a component of the
Academic Pathways Study (APS) of the Center for the Advancement of Engineering Education (CAEE). The APS aims
to provide a comprehensive account of how people become engineers by exploring key questions around the
engineering learning experience. The goal of the APPLE survey is to validate the longitudinal data from the
APS Persistence in Engineering (PIE) survey, as well as findings from other APS data collection methods.
| ...in order to validate longitudinal data in one survey administration, the target sample needed
to be cross-sectional and include respondents from all stages of the undergraduate academic degree.
|
Lessons Learned
Despite the Team’s experience with the PIE survey deployment during 2004-2007 and comprehensive planning, many
lessons were learned in the design and administration of the APPLE survey:
- Start the Internal Review Board (IRB) process at least four months prior to the planned deployment.
This is especially important for multi-site studies where multiple IRB approvals are required.
- Consider the different stakeholders when determining the incentive for participation. In addition
to the potential student respondents, the team recognized that incentives or other compensation may be needed for
institutions’ and their liaisons’ participation.
- Allocate resources for comprehensive piloting of the survey. Measuring the time-to-take of the survey
instrument during piloting was particularly valuable in guiding difficult decisions about which items and constructs
to keep or delete.
- Emphasize sharing and learning from other recruitment plans across institutions. Recruitment plans
were shared across institutions with limited success because liaisons reported that the framework was helpful in
developing their own campus recruitment plans, but they did not fully appreciate the other institutions’ plans
until after the deployment.
- Add survey items to weed out ineligible subjects. The team used responses in the demographic
questions to easily identify ineligible subjects (such as graduate students and students from other universities)
and eliminate them from the analyses.
- Allow multiple submissions from a single IP address. The decision to allow more than one submission
per IP address proved to be a good one as 22% of one school’s submissions came from a single IP address, presumably
due to the firewall system used there.
Apple Survey Deployment and Recruitment of Participants
The APPLE survey will be deployed twice (APPLES1 and APPLES2), to two different populations of American undergraduate
engineering students. APPLES1 was administered in April 2007 and targeted students from the same four schools as
APS' Cohort 1 (Coleman University, Mountain Technical Institute, Oliver University, and University of West State;
pseudonyms).
Due to APS’ focus on persistence in engineering education, APPLES recruitment targeted three groups of undergraduate
students: (1) engineering students: those who have declared an engineering major or have already committed to
engineering programs; (2) pre-engineering students: those who intend to declare an engineering major; and (3)
non-persister students: those who were initially interested in majoring in engineering, but have since decided to
pursue a non-engineering major.
The engineering program structures at the four APS institutions differ significantly. Recruitment methods that
are successful at one institution may not be appropriate or even possible at another institution. Thus, the team
was under certain constraints in devising a sampling plan. The team had to use a convenience sampling method
whereby targeted respondents were requested to voluntarily take the APPLE survey as compared to a random sampling
where the team could choose the subjects randomly. Also, in order to validate longitudinal data in one survey
administration, the target sample needed to be cross-sectional and include respondents from all stages of the
undergraduate academic degree (see full paper at the link below for recruitment method details).
In designing the APPLES administrations, the APPLES1 deployment was intended to be a model for the larger-scale
APPLES2. Campus liaisons worked with the team at each of the four institutions. These campus liaisons assisted
in securing IRB human subjects approvals, planning campus-specific recruitment, and coordinating local deployment.
The liaisons also provided valuable insights into their institutional and student cultures, campus infrastructure,
and student preferences during the development of recruitment materials and plans. None of the institutions reached
their targets for all strata (pre-sampling grouping of the target population, e.g., persistence and academic class).
Deployment dates at various institutions were constrained by spring break schedules, final administration of the
Cohort 1 PIE survey, the end of the academic year at semester institutions, and response rates.
IRB approval was required for each of the four institutions because all had researchers who were involved in the
APS. Universal IRB requirements, such as not requiring the subject to answer one or any of the survey questions,
significantly guided the PIE and APPLE survey designs. Institution-specific IRB requirements did vary.
Information consultations with other survey researchers suggested that recruiting engineering undergraduate
students was more difficult than recruiting other students and that they should be offered an incentive for
participation. A $4 electronic payment was chosen and offered to every individual who took the survey through
a commonly-used financial transaction company.
Patterns of responses from the Cohort 1 respondents, who the team believes to perceive the survey with moderately
high levels of importance and interest, encouraged the team to expand recruitment efforts to include basic
recruitment methods aimed at a larger audience: poster, emails to student distribution lists, and an ad in the
student newspaper – all of which carried consistent branding of a red apple logo. The IRBs required the survey
be open to all students. However, posters were designed to focus on the target respondents: pre-engineering,
engineering, and non-persister students. Because the four institutions differed in terms of student demographics
and when students were required to declare their majors, recruitment targets were set on a per school basis and
campus liaisons received daily reports on responses for each recruitment strata to help direct recruitment
efforts.
The APPLE survey underwent two rounds of piloting: a first round primarily to refine the survey questions for
clarity and identify questions for elimination, and a second to streamline the final product. The length of the
survey was a primary concern given the limited compensation of respondents. Using timing data that recorded how
long respondents spent answering individual survey questions during the first pilot, the team was able to pare
down the survey instrument to the target time of approximately 10 minutes.
There were over 900 submissions for the first APPLES administration. The average response rate was 17 percent
across the four institutions. APPLES2 will be deployed in early 2008 at 22 institutions in the US.
Authors: Krista M. Donaldson, Helen L. Chen, George Toye, and Sheri D. Sheppard
Source: Proceedings of the 37th Frontiers in Engineering Conference 2007
For a printable pdf of this research brief, click here.
Brief created October 2007
Back to Research Briefs