Chapter 9 Evaluating Literature

9.1 There is no perfect study

This week, we are mostly discussing just one new research methods concept (sampling). Instead of talking more about how studies are conducted, we are going to start applying what you have learned so far. You now know the fundamental building blocks of studies. They can be qualitative or quantitative; they can use one of many different methods to address their research questions. Studies provide empirical data about constructs of interest through their representation in study operations. Experiments are our strongest tool for generating casual relationships.

We have also seen that we can critique the claims made about research. We don’t need to critique researchers themselves (called an ad hominem argument), and we don’t need to label studies as “good” or “bad.” There is no such thing as a valid study. All research involves compromises. Take a non-experimental study about smoking and lung cancer, for example. It is impossible for researchers to ethically or practically conduct an experiment about the effects of smoking cigarettes. Does this mean the effects of cigarette smoking is a non-scientific question? No, of course not. It simply means that a single non-experimental study cannot demonstrate a causal relationship. When combined with other studies, however, we can obtain a clearer picture of the constructs we study. Thus, with decades of research, science has firmly established the negative effects of cigarette smoking, despite not being able to run any experiments.

Because all research involves compromises, there is no such thing as a perfect study. Because of the limitations of samples, constructs, and statistics, it’s almost never possible for a single study to provide a definitive, conclusive answer to the research question. No matter the study, there are always ways of critiquing the claims made by the researchers. You already know how to do this; we call them threats to validity. A threat to validity is a reason why a claim made about a research study is false. If you can identify threats to validity, you can critique the research of others.

9.2 Science happens in public

The textbook discusses how science is something that takes place in public. That is, we make our research results widely available so that other researchers can critique them. This process of publishing research, listening to others’ ideas about the limits of the research, and conducting follow-up studies is what leads to good science. As scientists, we make our ideas available for critique. Scientists understand that this is about the quality of the ideas, not the quality of researchers.

In this section, we will learn how to evaluate published research (the literature), the ethical standards associated with publishing research, and introduce one new concept that is important to external validity: sampling.

9.3 Reading, citing, and evaluating journal articles

In the activity, you will practice reading a journal article, citing it, and summarizing its meaning.

Journal articles are challenging reading because they are written by scientists for other scientists in the same field. You can still make sense of them, even when you don’t know everything about the topic. The key is to understand what each section of a journal article provides. One of this week’s videos explains how.

Usually, you’ll be reading a journal article because you are discussing it in your own paper or research report. When you discuss the ideas of others, you need to provide an in-text citation. A citation is a reference to the work, like this (Schuster, 2020). A citation says which paper is being cited, but it’s not enough to find the paper. To do that, you need to provide a reference. The reference comes at the end of your paper and provides the authors, title, year of publication, name of the journal, and other information so that your readers can find the work. In Psychology and this class, we use APA style citations. A handy PDF explaining how to cite in APA style is included in the course materials. Use it as a template.

When reading a journal article, it’s important to not take everything said at face value. Most of the sentences in the article provide some type of claim. These claims can be about:

Constructs. Do you agree with the way the authors represented constructs in their study? This is construct validity.
Causality. Did the research establish a causal relationship between the IV and the DV? If not, what are other possible explanations? This is internal validity.
How the study findings apply as things change. Do you think the results apply across the entire population of interest? Do the results apply in every possible setting (they rarely do)? If there are limits on the generalizability of the study findings, what are they? This is external validity.
Statistical conclusions. Did the authors use the proper statistical techniques? Were their conclusions about statistics appropriate? For example, authors will sometimes claim that a non-significant finding means the two conditions were the same, but this is a logical fallacy that we call affirming the null hypothesis. Or, authors might claim that results are “approaching significance,” but this is another fallacy; NHST leads to decisions to reject or retain. There is no such thing as approaching significance.

Thinking in terms of the types of validity can help you list ways in which you agree and disagree with the claims made by the authors. As you read, highlight or take notes with your questions or critiques about the research.

9.4 Ethical Standards in Publishing

We discussed research ethics as they pertain to our human participants. Our field has established principles to protect the people who provide data. This is necessary because history has many examples of psychologists failing to protect participants, or worse, harming them directly. And, these examples continue to persist to present time. We must understand what the ethics are and be vigilant about following and enforcing them.

Also important are ethics relating to publishing research. Research publication requires a lot of trust. We trust that scientists are not lying or making up their data. If they did, it would threaten trust in all of science. Imagine if you read an article and there was a chance that all the numbers were made up. How could you believe anything the authors said?

Here is a summary of ethical principles in publishing, adapted from the APA manual. Ethical researchers:

Obtain institutional review board (IRB) approval for any human-subjects study that they conduct before they begin.
Accurately report their research results, including parts of their research that do not support their argument. They report their complete data and do not edit out parts of it.
Make corrections when they discover their reports were wrong. If the problems in the research report are major, ethical researchers retract their article, which means that it is removed from the scientific literature.
Retain the data from their published studies and share it with other researchers for the purposes of auditing, review, and meta analyses, subject to IRB approval.
Never publish the same results in more than one outlet (for example, reporting the same data to two different conference papers, even if the papers are different). This is called duplicate publication and is a major no-no. The problem with duplicate publication is that other scientists will see this as two different samples. A study becomes over-represented and its findings have too much weight if double published. Whenever conference or journal papers are published, researchers agree that their results have never been published elsewhere.
Never break up a study into small bits in an effort to have more publications. This is called piecemeal publication and is bad because the results become misleading when researchers present them as several studies. There are some exceptions for this in the case of large studies, but researchers should at least disclose the existence of the other research report to the journal editors.
Avoid plagiarism and respect intellectual property. Plagiarism is presenting someone else’s words or ideas as your own. Self-plagiarism is the presentation of your prior published work as new work. Intellectual property means respecting the copyright and authorship of work done by others.
Avoid conflicts of interest. APA defines a conflict of interest as “personal, scientific, professional, legal, financial, or other interests of relationships that could negatively affect professional conduct or cause harm to persons with whom a professional interacts” (APA Ethics Code as cited in APA manual, 7th ed., p. 23). An example of a conflict of interest would be a journal reviewer reviewing work done by their student. Their duty to impartially review the article and their collaboration with their student are in conflict; they cannot do both well.

9.5 Sampling

A separate but relevant topic is sampling. Sampling is the method of obtaining a sample. We have previously discussed random sampling, which is the best way to represent a population of interest. What happens if your population of interest is “schoolchildren in the United States?” There is no realistic way to obtain a random sample from such a population. Therefore, you should consider which type of sampling is possible and the limitations of non-random sampling (i.e., it weakens your claims about your sample being a good representation of your population and the sample may be biased). Both representation and bias are discussed in the video.

Simple random sampling: All members of the population have an equal chance of being selected. Requires that you can list every member of the population.
Convenience sampling: Choosing participants based on who is available and willing to participate in the study. This is the easiest method but most likely to be biased.
Systematic sampling: Starting from a random point, select every Nth participant.
Cluster sampling: Divide population into clusters or units (such as schools), take a random sample of the clusters (i.e., randomly select a school) and then measure all the participants within the cluster (i.e., measure every student in the school).
Stratified sampling: Divide population into strata based on some characteristics (e.g., level of education) and then randomly sample from each stratum so that you have equal representation of the characteristic.

Note that experiments are defined by random assignment, which is how participants are put into conditions (levels of the IV). Random sampling, discussed here, is how participants are selected to participate in the study.