Building the Evidence Base for EDGE
By Lily Alpert, Fred Wulczyn, Molly Van Drunen, Scott Huhr, and Kristen Hislop
Several years ago, the W.T. Grant Foundation supported the Center for State Child Welfare Data to conduct a randomized control trial of our research evidence use (REU) training program, EDGE: Evidence-Driven Growth & Excellence. The project falls under the Foundation’s interest in interventions that advance REU among child and family serving organizations. In short, if REU is associated with improved outcomes for children and families, as emerging research suggests it is,[i],[ii] then agencies will be wise to invest in professional development programs that improve REU among their staff. To that end, EDGE aims to improve REU among leaders and managers of child welfare agencies.
Our study took place in a state where counties contract with private agencies to provide care and placement to children in foster care. We delivered EDGE to representatives from twelve such agencies—seven in the treatment group (Wave 1) and five in the control group (Wave 2). This brief summarizes our findings to date regarding EDGE’s effectiveness.[iii]
Child welfare agencies have room to improve when it comes to REU.
Prior to beginning the course, instructors ask EDGE participants to submit examples of documents that reflect their agencies’ use of research evidence to shape policy or practice. These can include annual reports, strategic plans, or any kind of analysis the organization produces to monitor its performance. Instructors review those documents and provide feedback on ways in which they align with the principles taught in class and ways in which the agency might improve its REU practice. In this way, document review offers a baseline understanding of an agency’s REU prior to EDGE.
Across agencies, several themes emerged, but perhaps the most prominent was the habit of “selecting on the dependent variable” (typically, but not always, by analyzing an exit cohort) to make generalizations about the speed and likelihood of outcomes for children in care. As we teach in class, this method creates unrepresentative information about agency outcomes (i.e., it does not provide evidence) because it examines only children who achieve the outcome of interest rather than all children at risk of experiencing the outcome of interest. Specifying the correct analytic population—i.e., ensuring your measure includes all the people it is supposed to—is perhaps the most important skill taught in the EDGE curriculum.
EDGE improves participants’ REU capability.
EDGE’s primary educational goal is to improve participants’ REU capability, particularly when it comes to generating evidence about child and family outcomes at the agency level. Therefore, one of our core research questions was whether EDGE has the effect of improving participants’ REU knowledge and skills. We found that it does. Members of the treatment group showed significantly more growth in REU capability at posttest than members of the control group. In other words, EDGE is effective at improving leaders’ and managers’ REU capability.
EDGE has shaped participants’ work in the field.
Of course, one of EDGE’s long-term goals is for participants to translate their improved capability into new ways of doing business. We were pleased to see this unfold in two ways. First, we saw change among members of the state’s trade organization for child- and family-serving organizations. A number of EDGE participants, who were also members of the organization, used their new skills to improve the way they were generating, interpreting, and using evidence to build the narrative about their individual and collective performance. Second, one participating agency put their new skills into action by revamping their approach to measuring the use of restraints and examining permanency outcomes for children in their care. Both of these examples illustrate the effect EDGE can have on the way agencies develop and use evidence to shape practice in their ongoing efforts to improve child and family outcomes.
[i] Chamberlain, P., Feldman, S. W., Wulczyn, F., Saldana, L., & Forgatch, M. (2016). Implementation and evaluation of linked parenting models in a large urban child welfare system. Child Abuse & Neglect, 53, 27-39.
[ii] Wulczyn, F., Alpert, L., Monahan-Price, K., Huhr, S., Palinkas, L., & Pinsoneault, L. (2016). Research evidence use in the child welfare system. Child Welfare, 94(2), 141–165.
[iii] To read the full study, see Wulczyn, F., Alpert, L., Huhr, S., & Van Drunen, M. (in press). Research Evidence and Research Evidence Use. In Elizabeth Fernandez, Penelope Welbourne, Bethany Lee, & Joyce Ma (Eds.), Routledge Handbook of Child and Family Social Work Research: Knowledge Building, Application, and Impact.