Category Archives: research for policy

June 12, 2014 · 4:14 pm

Understanding Research: Top Ten Tips

This is a #TBT post, sharing a research guide I wrote in 2001 for the National Association of Child Advocates, which became Voices for America’s Children. As the push for evidence-based policy grows, these points remain relevant today. The original four-page brief (PDF) is also available here.

data chart imageResearch allows us to assess the effectiveness of policies and programs affecting the lives of children and families. Having research evidence to recommend or refute specific policy choices is especially relevant in this era of increased demand for accountability in human services and government.

But how can you tell if a given research study is one you can trust? Below are tips to evaluate the research you encounter.

  1. Consider the source. Evaluate the credibility of the individual(s) and the organization that produced the research. Research produced by respected researchers and institutions is more likely to be trustworthy. Also, research produced or funded by groups with a strong political or commercial agenda is less trustworthy, since these groups have a vested interest in the study’s findings supporting their viewpoint.
  2. Media is also a source to be evaluated. Media coverage may not fully or accurately summarize the original research. Because research can be technical and complex, and because media coverage often seeks to be attention grabbing and succinct, media reporting of research sometimes oversimplifies the research, leading to misinterpretation.
  3. Has the research been published, and where? Research published in peer-reviewed research journals is more trustworthy because it has been scrutinized by other researchers before being published. Unpublished research, or research published in publications that don’t critically evaluate it, has not gone through such scrutiny. However, even good research starts out unpublished, so just because a study is unpublished does not mean that it is poor quality.
  4. Research results are really about the topic as measured, not as we may think of it. Look closely at how the topic in a study was measured. Since a research topic, such as aggression, could mean different things to different people, researchers always come up with a more specific definition of the topic they are studying. The results from a study are really about the precise definition, rather than the larger topic.
  5. Different types of research have different strengths. Another indicator of the quality of a research study, and the claims that can be made based on it, is the study’s research design. Experimental design studies offer the strongest evidence about the impact of a program. Quasi-experimental studies are especially useful for studying complex systems as they exist naturally in the community. Qualitative studies often provide descriptive, story-like accounts of people’s experiences in a program or in a community.
  6. Sampling is more important than sample size. While a study’s sample size is important, even more important is the way the sample was collected. Quantitative research is based on the assumption that the findings for a sample of people can be generalized to the larger population. If the procedures to select the study’s sample are not done well, then we cannot assume that the findings for the sample generalize to the population, and the study’s findings would not be valid.
  7. Statistical significance explained. One of the things advocates value most about research is getting “hard data,” i.e., numbers, about the effects of a policy on children. A study reports a statistically significant difference between those who received a program and those that did not. But what does statistical significance mean, and what can we conclude from it? A statistically significant result is one that is unlikely to be due to chance. Researchers use statistics to test whether the results they found are likely to be due to the effect of the program being studied and not to other unrelated factors. Statistical significance is different than the substantive significance, or meaningfulness, of a finding. A result may be statistically significant but unimportant. Conversely, a result may not be statistically significant, but it may be meaningful.
  8. Research findings are about groups. Research results are usually based on comparisons between groups of people. This makes research findings particularly relevant for policy decisions since policies affect groups of people, but less relevant for individual case decisions.
  9. All research is not created equal. When comparing the results from different studies with conflicting findings, higher-quality studies should be given more weight. Better studies can refute poorer studies; there is not a one-to-one comparison.
  10. Any one study is not the whole story. Although we usually come across research one study at a time, research is most valuable when many specific studies are taken together to tell the whole story of what we know on a given topic. Any single study, no matter how good, needs to be viewed in the context of other research on the topic.

 These research tips were also presented in my 2002 Evaluation Exchange article published by the Harvard Family Research Project.  That issue was devoted to public education campaigns and evaluation, and provides additonal good resources and examples.  

Image: Shutterstock, via leungchopan.

1 Comment

Filed under child policy, evaluation, evidence-based policy, research, research for policy, statistics

Tagged as , , , , , ,

January 25, 2014 · 1:29 pm

Study of North Carolina Pre-K and Child Care Explores Community Effects, Finds Improved Reading and Math Scores

Recently published research on the community-level effects of pre-K (More at Four) and child care (Smart Start) in North Carolina finds that both programs led to increases in reading and math scores in third grade.

  • North Carolina third graders had higher reading and math scores in counties that had received more funding for Smart Start and More at Four when the children were younger.
  • The estimated effects of Smart Start and More at Four investments were equal to test score gains of about four months of third grade reading instruction and two months of math instruction.  Notably, these results are for all children in the community, not just children who participated in either early childhood program.
  • While the authors did not do a full benefit-cost analysis, they note that the benefits in math and reading gains outweighed the program costs.  The expected savings from reduced instructional costs for children whose community received Smart Start or More at Four funding was at least equal to the cost of those programs.

Community-level effects – the concept of spillover:  One really interesting aspect of this study was that the researchers assessed the outcomes of children at the county level, including children who participated in the preschool or the child care initiative as well as children who did not.  The authors note that this approach allows them to measure the potential spillover effects of the program; that is, children living in the county but not participating in the program may also be affected indirectly, by other students’ participation in the program.  In theory, spillover could be positive or negative.  Positive spillover effects could occur in the elementary school classroom, for example, with a larger proportion of classmates arriving at elementary school ready to learn, enhancing the learning environment in that classroom.

More study details:  The article by Duke University researchers Helen Ladd, Clara Muschkin, and Kenneth Dodge and published in the Winter 2014 issue of the Journal of Policy Analysis and Management, examined the effects of two North Carolina early education initiatives:  Smart Start, the comprehensive child care and community services initiative, and More at Four, the state-funded preschool initiative.  The study used county-level program spending data matched with children’s birth records for children born between 1988 and 2000 and their third grade test scores to look at changes in test scores that took place after the introduction of More At Four and Smart Start in those counties.  The differences in when counties invested in these initiatives across the state allowed the researchers to assess the impact of More At Four and Smart Start on children’s math and reading performance.  To address other potential differences over time in counties which could also affect children’s math and reading scores, the researchers included statistical controls for related variables, a common and appropriate technique.

The numbers behind the basic benefit-cost results reported in the study were:  math and reading test score gains of two to four months translate into cost savings of $1,700 and $3,400 per child in the community (based on per-month costs out of a 10-month school year, which was $8,500 per child in North Carolina).  This was compared to program costs for More at Four and Smart Start of $1,100 per child in the community for each program.

Another benefit of measuring community-wide effects is that the authors were able to avoid a common potential problem in research studies that do not randomly assign people:  selection bias.  Selection bias is the potential bias that is created in a study based on how participants are chosen, such as when people self-select into the “treatment” being studied.  In this case, the treatment is preschool or child care, and the children whose parents sent them to public preschool may be different in meaningful ways (such as child health, family income, or parent education level, to name a few) than the children whose parents did not send them to preschool.  The differences observed between kids in preschool and those not in preschool may be due to preexisting differences between those groups of kids, rather than due to the experience of preschool itself.  Since this study examines outcomes for all children in the county, those participating and not participating in the preschool and child care initiatives studied, the researchers avoided this self-selection problem.

Leave a comment

Filed under early childhood education, research for policy, state policy

Tagged as , , , , , , , ,