Recipe: Establishing baseline performance

Child welfare systems are constantly implementing new policies and practices as they endeavor to improve outcomes for the children and families they serve. But how do you know if your policy and practice interventions have the effects that you intend?

To get to the answer, you must start by establishing your system’s baseline performance on the outcomes that you expect your intervention will improve. In establishing baseline outcomes, you answer to the question, “Based on historical performance, how should I expect my system perform on this outcome if we do not intervene, if business proceeds as usual?” Put differently, baseline outcomes are your “before” outcomes. They provide a reference point on which you expect to improve.

In this Recipe I’ll use the web tool’s Baseline Exits and Care Day Summary report to show you how to generate baseline performance on two permanency indicators: (1) proportion of children exiting to permanency and (2) care day use (i.e., length of stay).

This Recipe can be completed in just a few clicks.

Questions: What proportion of children in foster care will exit to permanency in the next two years if my system does not do anything differently? How many foster care days will children in care use during that period?

1. On the left-hand sidebar menu, click Benchmarks. Note that Benchmark reports are only available on the Multistate section of the website.
2. On the next screen, select Baseline Exits and Care Day Summary.
3. On the next screen, select the geographic area you want to analyze.

Your results will open up in a new tab. Below are the results from the jurisdiction I selected. (Click on the image to see a larger version.)

In the previous post, you learned that the impact of interventions must be measured over the course of a reform period, or “window” of time. The Baseline Exits and Care Day Summary (BECDS) report provides a template for measuring change by estimating baseline outcomes for the two-year window from 1/1/2012 to 12/31/2013.

During any reform window, your system serves two distinct populations of children: those who are in care at the beginning of the window, and those who enter care during the window. Because those two groups have different potential for achieving permanency outcomes of interest, the BECDS starts off by separating them. In the top half of the report, you see calculations for children who will be in care at the beginning of the window (1/1/2012); in the bottom half, you see calculations for children who will enter care during the window (between 1/1/2012 and 12/31/2013).

Each of these groups is stratified based on age group (and, in the case of the in-care population, by length of stay to date). Then, for each subgroup, the BECDS uses historical information about permanency and care day utilization to estimate the outcomes you would expect to see for that subgroup at the end of the two-year window (12/31/2013) if the system were to keep performing as it has been. For example, let’s take this excerpt from the in-care population estimates:

 Starting on the far left:

• The BECDS estimates that on 1/1/2012, this jurisdiction will have 690 children in care who began their spells as infants.
• Of these 690 children, 86% are expected to exit to permanent exits by 12/31/2013; 1% are expected to exit to non-permanent destinations, and 13% are expected to still be in care.
• The exits of the 86% of children exiting to permanency are expected to break down as follows: 21% reunified, 9% discharged to relatives, and 56% adopted.
• These 690 children are expected to use a total of 238,050 foster care days during the two-year period—345 days on average.

These figures are generated for each of the strata and then for the whole in-care population, constituting the baseline performance of each.

Then, using the baseline performance for the entire in-care population, the BECDS calculates what would happen if the system experienced a 10% increase in permanent exits during the two-year window. In this case, instead of 71% of the in-care population exiting to permanency, 78% would exit to permanency. Moving over to the right, the report calculates what would happen if the in-care population used 5% fewer foster care days. In this case, they would use 89,985 fewer days (18 fewer days on average).

In the BECDS, a 10% increase in permanent exits and a 5% decrease in care day use are provided as examples of performance targets—outcomes that you hope that you hope your intervention will help you achieve over the reform period. Establishing baseline and target performance prior to implementing your intervention enables you to return to those figures at the end of that period and measure your system’s actual performance against those benchmarks.

Regardless of the type, size, or scope of the intervention you’re planning to introduce, your agency will need a template similar to the BECDS in order to set performance goals and measure whether or not they are met. Currently, the web tool’s BECDS is only “dynamic” insofar as you can alter the geographic area you wish to analyze—the dates of the performance window are fixed, as are the target calculations. To apply this kind of template to an intervention you plan to implement in the future, you’ll need to generate something similar for the performance window during which you expect your intervention to have an effect. Also, you’ll want to set targets that make sense given the impact you expect your specific intervention to have; a 10% increase in permanent exits or a 5% decrease in care day use may be too high or too low, or your intervention may focus on some other outcome altogether. Data Center staff can work with you to develop customized templates for establishing baseline and target outcomes as you embark on reform. For more information, contact me at lalpert@chapinhall.org.