Читать книгу The Research Experience - Ann Sloan Devlin - Страница 145

If you are doing an experiment and are manipulating one or more variables, you are naturally concerned with whether the manipulation worked, that is, with whether it produced its desired effect. To avoid being disappointed when the null hypothesis cannot be rejected, it is wise first to make sure that the manipulation works (through a pilot test). To assess the effectiveness of the intervention in the study itself, you use a manipulation check at the end of the study. In a pilot test, you run your study, often with a just a few fellow students, to test your conditions. For example, if you were doing a problem-solving test and your review of the literature had not specified the difficulty or number of items to ask people to solve, you might need to test out your protocol. You would want to make sure that you weren’t making it too easy (and hence have a ceiling effect where everyone essentially correctly answered all of the items and the scores cluster at the top of the distribution) or conversely making it too hard (and hence have a floor effect where essentially everyone answers none or very few of the items and all the scores cluster at the bottom of the distribution).

Ceiling effect: Outcome in which scores cluster at the top of the maximum value; creates difficulty in evaluating group differences.

Floor effect: Clustering of scores on a measure at the low end of the possible scale values; typically linked to the difficulty of the assessment; creates difficulty in evaluating group differences.

Related to the pilot test is the notion of a manipulation check. Imagine you were conducting research on the effect of hair color on hirability and you used an application that took a photo and changed the hair color of that person. Apple, Inc. (Cupertino, California) has such an application that a student used in a research project (https://itunes.apple.com/us/app/hair-color/id485420312). If your hypothesis is that blondes are rated differently than are brunettes or redheads, you need to be sure that people can discern the hair color and pay attention to the hair color they see. If not, your manipulation is unlikely to be effective and you haven’t adequately tested your hypothesis. At the end of your study, you would include a question asking the participants to indicate the hair color of the person they rated.

If a manipulation check does not reflect the expected differences between conditions, you have no way of knowing whether the targeted information in your conditions was not attended to by participants when they made their responses or if it was attended to but did not influence respondents significantly differently. One recommendation to avoid this situation is to run a pilot test and analyze the manipulation check question(s) at that time. If you have already started running the study, you could analyze the manipulation check question(s) fairly early on when about five to seven participants per condition have completed the study. You should be able to get a sense of whether the manipulation check reveals the expected group differences. For example, if you varied hair color in a study of the effects of hair color (blond, brunette) on hirability, your manipulation check question might be, “What was the hair color of the woman in the photo?” If most respondents say something that is on track (e.g., for the brunette condition, acceptable responses might be brown, dark hair, or brunette), you can assume that they perceived the hair color.