Student Truth-Telling on Student Surveys

The honesty of survey respondents is always a concern in survey research. The Bismarck-Burleigh Public Health Student Survey uses a variety of procedures to encourage students to respond honestly. Based on these procedures as well as other evidence explained below, it is expected that the vast majority of students will answer the Bismarck-Burleigh Public Health Student Survey openly and honestly. As a result, we are confident that the data from the survey can be relied upon as an accurate picture of Bismarck’s students.

Procedures to Encourage Honesty

Survey environment. The survey administration procedures are designed to foster an environment in which students feel that they can respond honestly. The procedures promote the anonymity of students’ responses and the confidentiality of the schools’ data. The instructions given to the students encourage open and honest responses. The procedures include:

• Assigning the students to sit as far apart as possible throughout the classroom.
• Restricting the teachers and/or survey administrators to remain in a location in the classroom from which they cannot observe the students’ responses.
• Telling the students about the importance of providing honest answers. Teachers will tell students that their answers will help improve programs and policies for students.

Questionnaire design and content. The Bismarck-Burleigh Public Health Student Survey questionnaire is designed to protect the anonymity of the students. Each question is also designed to be easily understood and answered.

• No names or other types of personally identifying information are ever requested. Students are instructed not to put any identifying marks on the questionnaire.
• No skip patterns are used on the questionnaire (e.g., “if you answer ‘yes’ to question 7, skip to question 105”). Every student is asked to answer all the questions. This helps to ensure that all the students complete the questionnaire in approximately the same amount of time.
• To help students accurately understand and respond to the survey questions, the questionnaire is written for appropriate reading levels. Questions are also written in a straightforward and direct manner and almost never require the student to choose more than one response from the answers provided.
• The survey has been tested to make sure that it can be completed a short time of no more than 15 minutes.

Edit Checks. The Bismarck-Burleigh Public Health Student Survey is based on other student surveys, which include a series of dishonesty criteria to identify surveys where the student’s responses appear to be highly exaggerated or where the student admits to being completely dishonest. On other surveys, typically seven percent or less of the returned surveys were flagged for potential dishonest. Because of the shortness of the Bismarck-Burleigh Public Health Student Survey, the researchers are unable to include the same honesty checks; however since the Bismarck-Burleigh Public Health Student Survey uses a similar methodology as other student surveys, it is expected that it will have the same low level of dishonesty.

Evidence for Honest Responding

Logic within groups of questions. Questions with similar topics often have a logical relationship that would be expected if the students were responding honestly to the survey. For example, more students would be expected to admit to having tried cigarettes during their lifetime than the number of students who have used cigarettes during the past thirty days, and even fewer students would be expected to admit having smoked more than half a pack of cigarettes per day during the last 30 days. These logical patterns will be validated during the Bismarck-Burleigh Public Health Student Survey administrations. Students with a large number of logically inconsistent responses will be excluded from the analysis.

Comparison of Bismarck-Burleigh Public Health Student Survey results with results from other surveys. The evaluation team in charge of the Bismarck-Burleigh Public Health Student Survey will compare the results with other national, state, and local surveys on the same topics. Based on past, experience with similar surveys, the results are expected to be quite similar.

Psychometric studies. In 2002 a set of researchers from the University of Washington published a scientific paper explaining the development and psychometric testing of the core survey questions used in the survey (Arthur, Hawkins, Pollard, Catalano & Baglioni, 2002). The development included using and modifying established survey measures for key attitudes, beliefs, and behaviors that have been shown in scientific literature to increase or decrease the odds that youth will use alcohol, tobacco, and other drugs or exhibit other problem behaviors. The survey questions demonstrated adequate reliability and validity with regard to predicting youth substance use. Further studies (Glaser, Van Horn, Arthur, Hawkins & Catalano, 2005) have found that the core survey questions seem to measure similar attitudinal and behavioral constructs across five different race and ethnic groups and for both genders. Overall, these studies indicate that students respond to the survey with enough honesty that it does not generally threaten the reliability and validity of the results.

References