What purposes does the Pulse program serve?
Pulse data allow us to better understand the needs and experiences of Penn State students. Such knowledge allows for data-driven decision making about Penn State's Student Affairs programs and services. More specifically, the surveys provide us with the ability to measure:
- Student characteristics
- Students' needs and perceptions
- Students' awareness and understanding
- Use of and satisfaction with facilities, programs, and services
- Differences between subgroups of students
- Changes over time
- Effectiveness of programs and services
- Potential learning outcomes
In most cases, the Pulse survey results are one component of our assessment plan and are coupled with other measures to more fully understand the effectiveness of our services.
In addition, Pulse serves as a vehicle for informing and educating our students. Each project has an element of the "teachable moment" whereby participants learn about a service or topic of concern that may be of interest to them or someone close to them. For example, during the Newspaper Readership Survey, students commented that they had not been familiar with the program prior to the survey but were now interested in taking advantage of the service.
How are Pulse surveys administered?
Most Pulse surveys are conducted by both the phone and over the Web – what is known as a “bimodal design” in survey lingo. This approach is used in order to maximize the response rate, and thus increase the confidence of generalizing findings to the population.
Approximately one-third to two-fifths of our UP undergraduate students do not currently provide local phone information to the University Registrar. When examining the characteristics of those students, we have found that they are more likely to be male, over the age of 21, and to have upper-class (junior/senior) status. In addition, students who live off campus and indicate non-minority status are also over-represented among students who do not provide local phone information, as are students with higher grade point averages. Hence, in order to reduce the sampling error, a bimodal design is used.
For a bimodal design, one sample is extracted regardless of whether or not the student provided a local phone number to the Registrar. In the first phase of data collection, Pulse staff attempt to contact potential respondents by phone. At the end of the first phase of data collection, potential respondents who do not wish to complete the survey over the phone and those who could not be contacted by phone are sent an email invitation to participate in the second phase of data collection by completing the survey on the Web.
Occasionally, a Pulse survey is administered only via the Web. Like the bimodal design, one sample of students is extracted. Rather than a two-phase data collection, all potential respondents are only sent an email invitation to complete the survey over the Web.
Who receives Pulse surveys?
Pulse surveys are typically administered to a random sample of Penn State students. For example, the Student Drinking Survey is administered to a random sample of full-time, undergraduate students at University Park. The defined population varies depending on the topic (e.g., just students who live on campus or adult learners). Some surveys include a sample of graduate students and/or students from other Penn State Commonwealth campuses.
How are students selected?
In order to determine the sample used, all students who meet the defined criteria are 'pulled' from the University's Data Warehouse. Those students are then separated into two sets - students for whom local phone information is available and students who have not provided local phone information. A random sample is drawn from each file. The size of the sample is determined based upon the survey topic and anticipated response rate.
What is a typical response rate?
Based on the surveys conducted between the fall of 2010 and the spring of 2011, our phone surveys elicited a response rate between 19% and 35%, with an average of 27%. When administering the survey via the web, the response rate was between 15% and 24%, with an average of 18%. Overall response rates were between 24% and 31%, with an average of 28%.
What are the limitations of Pulse surveys?
All surveys are subject to potential sources of imprecision and bias, which may be associated with the wording and/or ordering of questions, the response rate, and the quality of the interviewers.
Typically, among our survey respondents, we find a higher proportion of women and on-campus residents than what is found in the population.
Other possible limitations of Pulse surveys may stem from the use of the bimodal design. As discussed above, the bimodal design is used to reduce the sampling error that results from the increasing number of students who do not provide local phone information. It is our belief that minimizing the sampling error outweighs potential differences in responses given by phone compared to responses obtained online.
Pulse surveys rely on the accuracy of self-reported behavior, experiences, and learning outcomes. Although research suggests there can be statistical validity in self-reported data, this needs to be considered when examining findings. In addition, Pulse provides a snapshot of our students based on one point in time, which may influence results.
How are Pulse topics chosen?
Topics are reviewed by Student Affairs staff based on the following factors: the ability to better understand the experiences of students; the level of fit with the priorities of the Division and the University; the appropriateness of the topic and questions to the Pulse format; the usefulness of the findings to improve programs and services for students; as well as resource and scheduling realities.
How long does it take to conduct a Pulse survey?
A project typically takes approximately four months to complete.
In general, it takes two to three months to develop a Pulse survey - to research the topic, consult with various stakeholders, compose the survey, pilot the draft with the Pulse workers, and obtain approval from the Office of Research Protections.
Data collection typically takes two to three weeks. For the phone administration, trained student workers make phone calls on Sunday and Wednesday evenings utilizing survey research software. Several attempts are made to reach students at a convenient time. Students who are selected for the Web survey receive an email invitation to participate containing the URL for the survey. At least one reminder is sent asking students to complete the survey online.
Analysis of the data and reporting of the findings take a month, on average. Analysis includes an executive summary, basic frequencies, selected comparisons between student groups, and coding of open-ended data.
What is Research Protections and why is it important?
Research-related projects that involve human participants require the approval of the Office for Research Protections. The Office for Research Protections ensures compliance with Federal, state, and local regulations as well as institutional policies and procedures regarding the use of human participants in research. The policy ensures that participants are fully informed about the project and consent to participate. Aspects of the process include providing information regarding the purpose of the study, the amount of time it will take, the potential risks to participation, the manner in which confidentiality will be maintained, and the name and phone number of the primary contact for the project. For more information visit the Office for Research Protections Web site.
For the Web survey administration, students' Penn State access IDs are used to send email invitations and reminders, and to enter students that complete the survey into a drawing for incentives when they are offered. Telephone survey administration replicates that of the Web survey, but also employs the use of student names and telephone numbers.
Students’ identifying information is used only for these purposes and is deleted following the selection of incentive winners. Responses are confidential and are only reported as summarized data. In the event of any publication or presentation resulting from the research, no personally identifiable information is linked to responses.
Based on this process and the mission of the Pulse program, students are able to respond to a Pulse survey with the understanding that confidentiality will be strictly maintained.
How are data used?
Findings from Pulse surveys are used in many ways both within the Student Affairs and the broader Penn State University community. Findings are used to:
- Educate students
- Inform marketing campaigns
- Communicate to students and others the effects of the out-of-class experience
- Inform policy decisions and funding initiatives (including grants)
- Create and refine programs and services
- Assist staff in serving as advocates on students' behalf and in providing expertise on the student experience
In addition, Penn State staff sometimes present Pulse survey findings at regional and national conferences. Other colleges and universities often craft their own assessments based upon Pulse prjects and use Pulse findings to inform programs they are initiating.