Category Archives: evidence-based policy

This is a #TBT post, sharing a research guide I wrote in 2001 for the National Association of Child Advocates, which became Voices for America’s Children. As the push for evidence-based policy grows, these points remain relevant today. The original four-page brief (PDF) is also available here.

Research allows us to assess the effectiveness of policies and programs affecting the lives of children and families. Having research evidence to recommend or refute specific policy choices is especially relevant in this era of increased demand for accountability in human services and government.

But how can you tell if a given research study is one you can trust? Below are tips to evaluate the research you encounter.

1. Consider the source. Evaluate the credibility of the individual(s) and the organization that produced the research. Research produced by respected researchers and institutions is more likely to be trustworthy. Also, research produced or funded by groups with a strong political or commercial agenda is less trustworthy, since these groups have a vested interest in the study’s findings supporting their viewpoint.
2. Media is also a source to be evaluated. Media coverage may not fully or accurately summarize the original research. Because research can be technical and complex, and because media coverage often seeks to be attention grabbing and succinct, media reporting of research sometimes oversimplifies the research, leading to misinterpretation.
3. Has the research been published, and where? Research published in peer-reviewed research journals is more trustworthy because it has been scrutinized by other researchers before being published. Unpublished research, or research published in publications that don’t critically evaluate it, has not gone through such scrutiny. However, even good research starts out unpublished, so just because a study is unpublished does not mean that it is poor quality.
4. Research results are really about the topic as measured, not as we may think of it. Look closely at how the topic in a study was measured. Since a research topic, such as aggression, could mean different things to different people, researchers always come up with a more specific definition of the topic they are studying. The results from a study are really about the precise definition, rather than the larger topic.
5. Different types of research have different strengths. Another indicator of the quality of a research study, and the claims that can be made based on it, is the study’s research design. Experimental design studies offer the strongest evidence about the impact of a program. Quasi-experimental studies are especially useful for studying complex systems as they exist naturally in the community. Qualitative studies often provide descriptive, story-like accounts of people’s experiences in a program or in a community.
6. Sampling is more important than sample size. While a study’s sample size is important, even more important is the way the sample was collected. Quantitative research is based on the assumption that the findings for a sample of people can be generalized to the larger population. If the procedures to select the study’s sample are not done well, then we cannot assume that the findings for the sample generalize to the population, and the study’s findings would not be valid.
7. Statistical significance explained. One of the things advocates value most about research is getting “hard data,” i.e., numbers, about the effects of a policy on children. A study reports a statistically significant difference between those who received a program and those that did not. But what does statistical significance mean, and what can we conclude from it? A statistically significant result is one that is unlikely to be due to chance. Researchers use statistics to test whether the results they found are likely to be due to the effect of the program being studied and not to other unrelated factors. Statistical significance is different than the substantive significance, or meaningfulness, of a finding. A result may be statistically significant but unimportant. Conversely, a result may not be statistically significant, but it may be meaningful.
8. Research findings are about groups. Research results are usually based on comparisons between groups of people. This makes research findings particularly relevant for policy decisions since policies affect groups of people, but less relevant for individual case decisions.
9. All research is not created equal. When comparing the results from different studies with conflicting findings, higher-quality studies should be given more weight. Better studies can refute poorer studies; there is not a one-to-one comparison.
10. Any one study is not the whole story. Although we usually come across research one study at a time, research is most valuable when many specific studies are taken together to tell the whole story of what we know on a given topic. Any single study, no matter how good, needs to be viewed in the context of other research on the topic.

 These research tips were also presented in my 2002 Evaluation Exchange article published by the Harvard Family Research Project.  That issue was devoted to public education campaigns and evaluation, and provides additonal good resources and examples.  

Image: Shutterstock, via leungchopan.