Category Archives: evaluation

June 12, 2014 · 4:14 pm

Understanding Research: Top Ten Tips

This is a #TBT post, sharing a research guide I wrote in 2001 for the National Association of Child Advocates, which became Voices for America’s Children. As the push for evidence-based policy grows, these points remain relevant today. The original four-page brief (PDF) is also available here.

data chart imageResearch allows us to assess the effectiveness of policies and programs affecting the lives of children and families. Having research evidence to recommend or refute specific policy choices is especially relevant in this era of increased demand for accountability in human services and government.

But how can you tell if a given research study is one you can trust? Below are tips to evaluate the research you encounter.

  1. Consider the source. Evaluate the credibility of the individual(s) and the organization that produced the research. Research produced by respected researchers and institutions is more likely to be trustworthy. Also, research produced or funded by groups with a strong political or commercial agenda is less trustworthy, since these groups have a vested interest in the study’s findings supporting their viewpoint.
  2. Media is also a source to be evaluated. Media coverage may not fully or accurately summarize the original research. Because research can be technical and complex, and because media coverage often seeks to be attention grabbing and succinct, media reporting of research sometimes oversimplifies the research, leading to misinterpretation.
  3. Has the research been published, and where? Research published in peer-reviewed research journals is more trustworthy because it has been scrutinized by other researchers before being published. Unpublished research, or research published in publications that don’t critically evaluate it, has not gone through such scrutiny. However, even good research starts out unpublished, so just because a study is unpublished does not mean that it is poor quality.
  4. Research results are really about the topic as measured, not as we may think of it. Look closely at how the topic in a study was measured. Since a research topic, such as aggression, could mean different things to different people, researchers always come up with a more specific definition of the topic they are studying. The results from a study are really about the precise definition, rather than the larger topic.
  5. Different types of research have different strengths. Another indicator of the quality of a research study, and the claims that can be made based on it, is the study’s research design. Experimental design studies offer the strongest evidence about the impact of a program. Quasi-experimental studies are especially useful for studying complex systems as they exist naturally in the community. Qualitative studies often provide descriptive, story-like accounts of people’s experiences in a program or in a community.
  6. Sampling is more important than sample size. While a study’s sample size is important, even more important is the way the sample was collected. Quantitative research is based on the assumption that the findings for a sample of people can be generalized to the larger population. If the procedures to select the study’s sample are not done well, then we cannot assume that the findings for the sample generalize to the population, and the study’s findings would not be valid.
  7. Statistical significance explained. One of the things advocates value most about research is getting “hard data,” i.e., numbers, about the effects of a policy on children. A study reports a statistically significant difference between those who received a program and those that did not. But what does statistical significance mean, and what can we conclude from it? A statistically significant result is one that is unlikely to be due to chance. Researchers use statistics to test whether the results they found are likely to be due to the effect of the program being studied and not to other unrelated factors. Statistical significance is different than the substantive significance, or meaningfulness, of a finding. A result may be statistically significant but unimportant. Conversely, a result may not be statistically significant, but it may be meaningful.
  8. Research findings are about groups. Research results are usually based on comparisons between groups of people. This makes research findings particularly relevant for policy decisions since policies affect groups of people, but less relevant for individual case decisions.
  9. All research is not created equal. When comparing the results from different studies with conflicting findings, higher-quality studies should be given more weight. Better studies can refute poorer studies; there is not a one-to-one comparison.
  10. Any one study is not the whole story. Although we usually come across research one study at a time, research is most valuable when many specific studies are taken together to tell the whole story of what we know on a given topic. Any single study, no matter how good, needs to be viewed in the context of other research on the topic.

 These research tips were also presented in my 2002 Evaluation Exchange article published by the Harvard Family Research Project.  That issue was devoted to public education campaigns and evaluation, and provides additonal good resources and examples.  

Image: Shutterstock, via leungchopan.

1 Comment

Filed under child policy, evaluation, evidence-based policy, research, research for policy, statistics

Tagged as , , , , , ,

June 7, 2013 · 8:55 pm

Two-Generation Approaches: Supporting Parents and Children

In recent years, there has been increased attention to addressing the needs of at-risk families via two-generation approaches, with programs that focus on parents and children.

Last week, at the 16th annual Welfare Research and Evaluation Conference, hosted by the Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families, researchers, practitioners and policymakers gathered to discuss new findings and approaches in providing services and supports to low-income families. The conference included a session on two-generation strategies to provide services that support parents and their children.  The session provided useful information and food for thought on what we are learning about how to do this effectively, and what we still need to learn.

CC Image courtesy of PJR on Flickr

Lisa Gennetian of the National Bureau of Economic Research moderated the session, titled “Two Generation Strategies to Support Family Well-Being and Stability.” Gennetian recapped past approaches which focused on teens and young mothers, including New Chance and various home visiting models, noting that for many of these models, the results have not been as strong as would have been hoped.  The field currently is focused more on economic self-sufficiency approaches in addition to parenting, with more optimism.  Holistic approaches in two-generation programming were noted, including the Jeremiah Program in Minnesota. Also mentioned was the Ascend initiative at the Aspen Institute, a leading player in the two-generation approaches space.

Shelley Waters Boots, Senior Consultant with the Annie E. Casey Foundation, spoke about the Foundation’s work on two-generation approaches.  In the Casey Foundation’s framework, there are three major components in two-generation interventions:

  1. Family economic success strategies
  2. Capacity building for parents and caregivers
  3. Early care and education and early grades

The expectation is that if families receive all three components simultaneously, it will lead to stronger outcomes than would be achieved with just one of the pieces.  The Foundation is supporting efforts employing this two-generation approach via its Family-Centered Community Change initiative, with community-based partners delivering these multi-level services in Buffalo, Columbus, and San Antonio.  Casey seeks to test new ways to deliver two-generation programs that can be both cost-effective and scalable.  This is an encouraging approach for the field, since developing programs likely to be effective and replicable in the real world, rather than boutique programs, is essential to making a real difference for at-risk children and families.

Susan Popkin, Senior Fellow at the Urban Institute, discussed lessons learned from the Housing Opportunities and Services Together Demonstration, which coordinates public housing and human services in four sites, in Chicago, IL,  Portland, OR, Washington, DC, and Brooklyn, NY.  In each of these public housing sites, the HOST demonstration program also provided services for families and for children and youth specifically, from afterschool programming to sexual health services.  These ambitious demonstration programs are still underway, and they have had some implementation successes and faced several challenges.  Among the lessons learned were: figuring out how to serve lots of kids; determining how to target kids most at risk; building trust within the community in the broader context of a long history of distrust and isolation; and the need to choose project partners carefully, ensuring that community and service partners are willing to work collaboratively toward shared goals in order to be effective partners.

JoAnn Hsueh, Senior Associate at MDRC, presented findings on OPRE’s Enhanced Services for the Hard-to-Employ Demonstration and Evaluation Project.  This demonstration project provided Early Head Start (EHS) services enhanced by incorporating employment and education services for parents.  The project, which included a longitudinal experimental design with an implementation study, has been completed.  The study, which examined impacts on employment, earnings, and child care at 18-month and 42-month follow-ups, found only a few limited significant impacts, but the project has yielded some key lessons learned that provide insight in to the challenges of delivering these services effectively.

The program faced implementation challenges in delivering the full program model with fidelity.  Most interesting among these challenges was the problem of adding self-sufficiency services for parents on to an Early Head Start program; EHS staff, accustomed to providing early childhood services, did not feel they had expertise to help parents with job and education goals.  Further, in some cases the EHS staff were not in agreement with the demonstration program’s employment goals for parents, feeling that employment was not in the best interest of these parents of babies and toddlers. On a more technical note, the evaluation was also unlikely to detect differences between treatment and control groups given that the difference in the amount of services received between the two groups was small — this means that both groups received fairly similar services, so understandably there was not a great difference between groups in results.

Findings from Popkin and Hsueh both illustrate the many real-world challenges of trying to meet the needs of low-income families by working across service areas that have traditionally operated in silos, being delivered in separate places by separate agencies.  Two-generation programs are a promising area with more work to be done to continue efforts to transcend traditional program boundaries and address the needs of at-risk parents and their children.

2 Comments

Filed under child policy, demonstration program, early childhood education, economic self-sufficiency, evaluation, home visiting, parent education, program models

Tagged as , , , , , , , , , , , , , , , , , ,