What's a Scorecard, and How Do We Win?

A podcast that unpacks why Commit maintains a county scorecard, and what it measures.

1 March 2018

If you come up to Commit’s office this week, you’ll see charts and graphs plastered on every available square inch of their glass-walls and whiteboards.  It’s not too out of the ordinary for an organization that has made “measuring what matters” a cornerstone of their mission, but at this time of year, the powerpoints and PDFs are inescapable.

And that’s because for months now, the team here has been gearing up for the 2017 Dallas County Community Scorecard, an annual presentation of a year’s worth of data analysis, measuring the health of the overall education system here in Dallas County.  I wasn’t quite sure what that all meant, so I somehow convinced three very busy staff members to sit down for a conversation.

I’m Ashwina Kirpalani, and I lead the Analytics team.

I’m Chris Hudgens, I’m the Senior Manager of Regional Analytics. I work on Ashwina’s team.

And I'm Rob Shearer, I am the Director of Communications and Marketing for the Commit Partnership.

As it turns out, the Scorecard is a lot more that just data sets.  It’s a way of taking a head-on look at some of the biggest challenges facing our region, and finding the solutions that can lead to the biggest change.  

Ashwina: We’ve got about 2.5 million people living in Dallas County of which approximately half a million are in the Pre-K 12 public school system.

Rob: Of that two and half million 18% are considered in poverty which is a family of four living in a household with less than $25,000 a year of income.  That number is even greater for children living in poverty. 28% of the children in Dallas County are living in poverty.

Chris: So when we look at our public education system, kindergarten through 12th grade, we've got 500,000 kids attending Dallas County schools. 72% of those kids are considered economically disadvantaged. When we say economically disadvantaged what we mean technically is that they qualify for free and reduced price lunch, which is a threshold that is slightly above poverty level so that's why there's more students who are economically disadvantaged then we would see in the child poverty rate. 31% of them are considered English-language learners, and so when we look at a lot of the performance gaps or achievement gaps a lot of those hit economically disadvantaged kids harder than non-economically disadvantaged.  A lot of those hit English language learners harder than kids who speak English at home.  And so we’ve gotta think about how we can serve those kids in ways that would give them an equitable education.

Rob: I think it's also worth noting, one of the reasons our percentage of economically disadvantaged students is so high is a lot of families with means take themselves out of the public school system. And so particularly in Dallas County you’ve got a high percentage of middle and upper income families who have chosen private school or taking an alternative path other than the public school system and in a way there is a almost a self-fulfilling prophecy to that. As we further concentrate and isolate our schools to be low income students, it makes it much, much harder for one highly qualified teacher to bridge the gap that the students need. Students come in with more needs than simply filling their brain with the academic knowledge that were hoping they get. Students are coming into the classroom hungry, tired, with a whole lot of stress in terms of things in their life that are happening outside of the classroom and are absolutely outside of those children's control.  And so it’s a really really challenging situation when you have a teacher who has 90 or a hundred percent of her students in situations like that, for her to be able to, or him to be able to accomplish everything that needs to happen in terms of preparing the child for standardized tests, or just success in life.

Chris: So in this community achievement scorecard, we want to do is really give a sense of how well is our system working from cradle to career, when a kid enters the education system from the beginning of Pre-K all the way through to when they finish college.  We wanted to be able to measure how does that pipeline, if you will, how healthy is it? And so what Commit has done is figured out 11 different indicators of how healthy that pipeline is that cover all with in the beginning of Pre-K all the way through to college completion.  So these indicators are really broken up into three big areas we've got a few indicators touching on early childhood education, which is specifically around pre-k enrollment, around kindergarten readiness, and then third grade reading.  We’ve got another batch of indicators that focus on how students are doing in the middle grades, so we’ve got fourth grade math, eighth grade science, algebra one. And then we’ve got our final group of indicators that ask “How well are kids getting prepared for college?”  And then “how well are they succeeding in college?” And so we've got a metric around college readiness, around high school graduation and then we’ve got three final indicators that touch on college enrollment, college persistence, and then college completion. All in all those are the 11 indicators that really measure the health of our education pipeline and try to get a read on how many of our kids are passing through this pipeline and coming out the other end with a college diploma in hand ready to succeed in life.

Ashwina: And we’re very cognizant that these 11 indicators are not the only things that are important in a child's ability to succeed, or measuring if a child is succeeding. There are things like social emotional health that we have difficulty measuring, but if we do have access to data we would love to work with it. Teacher quality. Those are both really important pieces to this whole puzzle that unfortunately right now we just can't get our hands on data that we can use to understand how well students are working within the system and how well the system is working for students.

Rob: But the point of the scorecard has always been to try and align adult behavior to benefit student achievement. So our hope is that the tracking of these 11 indicators helps to make sure that everybody involved in education, whether it is independent school districts or early childhood centers or nonprofits who are providing you know reading programs in the schools or after school care are all focused on how can we make sure we’re moving these numbers up as well as other metrics that were not measuring that just indicate every student getting excellent education.

Ashwina: I think we chose a couple of the first indicators so that’s pre-K enrollment and Kindergarten readiness, because so many of the indicators or the metrics that public schools use don’t focus on the K2 space. So the indicators usually start in third grade forward with No Child Left Behind.  We thought that we just really had to focus or draw some attention to those early grades, K2.  And so Pre-K enrollment is at the definition level is what percentage of our students are enrolled in Pre-K who need to be enrolled in Pre-K, who are eligible to be enrolled in Pre-K.

Rob: Who qualify.  It's a work in progress.  And it’s, I will say that we are challenged by funding in this situation. The state does not fund full day Pre-K, and the reality is that for parents who qualify, half day is not got helpful. If you’ve got, you know, both parents or a single parent who you are working to put food on the table and keep a roof over their head, stopping work at noon to pick up your child from half-day Pre-K doesn't make a lot of sense. We also know that 90% of the brain is developed by the age of five and so we know that that opportunity at that three or four-year-old space is critical to have them in a high quality early learning environment where they are learning all day long. And it's not, I think some people get freaked out by the idea of school for three and four-year-old. It is age-appropriate, and it's fun, and their painting and their playing and they're getting a lot of social emotional learning opportunities. It's not worksheets and reading drills. I think it's a really good time for their brains to develop and we’ve seen huge progress in terms of getting more families aware that it’s beneficial and then you can absolutely track the impact it's having on how many more kids are kindergarten ready and what that kindergarten readiness does in terms of leading to third grade reading scores.

Ashwina: And the other piece that I really want to share and let the community be aware of is that those districts that offer full day Pre-K have definitely seen a larger uptick in enrollment than the school districts that offer half-day Pre-K.  So parents are taking advantage of it when it is offered and enrollment is higher.  

Rob: Yeah, the districts that that offer full day Pre-K, they have to do so out of their own pockets. So a district like Dallas ISD which has a huge percentage of students that qualify for Pre-K, they have to go over and above what other districts would to extend the Pre-K day to a full day. It's a little tricky to know exactly how many eligible students there are so we use a formula that's based on-

Ashwina: The Texas education agency providing some information to us looking at the eligible students and actual enrollment for three and four year olds.

Chris: And students are typically eligible for free pre-k if they are considered economically disadvantaged, if they are in foster care, there's a few different categories that are used.

Ashwina: English language learners.

Rob: Military families.

Chris: Military children. All of these children are considered eligible for free Pre-K.  And so what we’re doing is saying “How many of those children who are 3 and 4 years old were eligible are enrolled in pre-k?”

Rob: Now Ashwina, in years past I think we’ve only tracked four-year-olds and this is I think the first scorecard that we have included both three and four-year-olds as part of our metric, right?

Ashwina: Yeah, thankfully TEA did a great job of producing a report this past year that allowed us to cut the data in that way, which had not been available before, there was no pre-K enrollment report at all and TEA made that available, and we’re thankful to them.

Rob: So kindergarten readiness is an interesting conversation because currently TEA does not mandate a kindergarten readiness standard across school districts so they they effectively let each school district decide how to measure kindergarten readiness.

Ashwina: With a subset of about ten assessments that they can choose from.

Rob: And so, for example Dallas ISD primarily looks at literacy, or exclusively looks at literacy for their kindergarten readiness number. We’d really love to see a more holistic kindergarten readiness assessment created that would that would be more whole child focused. So let's let's look at, you know, social emotional preparedness, let's look at literacy, let's look at some other factors that probably give you a better sense of how ready a child is to be in a learning environment other than simply a literacy metric.

Ashwina: And also the other piece is that the reason DISD, or most school districts in our area, not just DISD, measure literacy is because that's what’s mandated from TEA.  And they get their direction from the legislature. Now if TEA is mandated to collect more than just early reading or early literacy, then school districts would have to do that. Definitely have to keep that in mind. They do what they're told to do. So for example Istation may been on the list for 5, 10 years but some of the more holistic assessments are more recent.

Rob: And not necessarily mandated by the legislature.

Ashwina: Correct, so you can choose which, you know, each district can choose whether to do a holistic assessment which could be potentially more expensive or a more computer-based click click click type assessment which is much cheaper.

Rob: And what the majority of districts in the state use.

Ashwina: Correct, and definitely Dallas County as well.

Rob: So 3rd Grade Reading is the next indicator which is the first kind of standardized test or STAR indicator on the chart. 3rd Grade Reading is interesting because it's such an indicator of future success, it really correlates strongly to high school graduation, it really correlates strongly to success in postsecondary. There is a lot of correlations with 3rd Grade Reading and health and wellness in general in the rest of your life. It’s a huge factor in terms of how someone is going to do, often, in their life. And it is also one that is particularly tied to income. We see pretty strong correlations between household income and performance on third grade reading which further complicates it as a measure so Chris has done some really interesting analysis recently on 3rd Grade Reading, Chris if you want to talk a little bit about what you looked at and what you found.

Chris: Yes when we look at Dallas County and 3rd Grade Reading rates you know it's really clear that there is a substantial gap based on race, based on economics, based on the language spoken at home and in third grade reading proficiency. I think it that the size of this gap really is kind of a call to action for the community to ask how we can better serve all students rather than just serving a handful of students and so for example with the most recent data from 2017, 62% of white students are reading on grade level in third grade, and compare that 62% to 28% for black students reading on grade level.  And so that's a substantial gap and I think it speaks to just the fact that the systems that the adults have created to educate children are not educating kids with the same level of resources, the same level of quality, and when you look at it at the campus level we can always ask like hey, is an individual campus on track to close that gap or are they actually widening the gap, and what we see is it up to I think it's a full two thirds of black students attend schools that are actually widening the gap between white students and black students and I think there's just a small share of black students that are attending campuses where the gap is actually being narrowed. So in short when we look at 3rd Grade Reading, the county is not on track to close the gap between white students and black students, and white students and Hispanic students, or wealthy students and poor students. That we’re actually adding to the widening of that gap.

Ashwina: So when we think about reading on grade level we’re talking about approximately 75% of the questions correct on the assessment, that is the meets standard. And we in our work use the “Meets” standard because we think it's a better measure of achievement. 75% of the questions is the middle level and in the level below that, at the “approaches” level is about 50% of the questions correct and we don't want to hold our students to a 50% measure especially if that is not what we would want for our own children. We don't believe that 50% of the questions correct is something we would ever want for our own kids. And so we hold the entire system accountable to 75% of the questions correct.

Rob: We have three more STAAR-related indicators: fourth grade math, eighth grade science, and algebra one are the next three indicators in order. We’re seeing growth almost across all of those in the past year, so 7% growth in fourth grade math, 2% growth in algebra one. Eighth grade science is flat but we’ve seen an 11% growth in science since 2012 since we started tracking so those indicators are fairly strong.

Chris: In 3rd Grade Reading we know that the gap is widening between white affluent students and non-white, non-affluent students. I think if we were to do a similar analysis for the other standardized test measures you’d see similar patterns. I don't think that we are meaningfully closing gaps in the county at the current rate. There are pockets of it. And I think that there are definitely schools that are doing a really good job and working really hard to make sure that black students, Hispanic students, economically disadvantaged students achieve an equal level, an equal quality of education, but those are exceptions to the rule and I think the larger trend here is that we’re not on the right path.

Ashwina: Another theory we’re exploring is that because prior to third grade there isn’t much emphasis on achievement in general students may not be prepared by the time they reach third grade which is why we focus on that Pre-K2 space because there just hasn’t been much focus on it prior.

Rob: And we’ve seen some evidence that campuses will move their most effective teachers into testing grades so if you have a great kindergarten or first grade teacher who's really standing out pretty quickly they get moved to third or fourth or fifth grade so that they can impact STAAR scores because STAAR scores end up driving campus ratings and other indicators.

Ashwina: And teachers who are teaching in early grades may not be trained to teach in those grades but they can pass a certification assessment and teach in those grades.

Rob: Then we move to college readiness, which I think we would admit that college readiness the way this indicator is measured again is probably not holistic but currently it is the way that we've measured it for the past five years.

Ashwina: Definitely imperfect, so it relies solely on SAT and ACT at a very high level, so it’s 1110 or 24, 1110 on the SAT or 24 on the ACT. It is a high bar to ask kids to meet.

Rob: Only 13% of Dallas County seniors are at the college-ready level on SAT and ACT, so obviously that is close to one in five of our high school seniors, and that’s a problem.

Ashwina: 1110 and 24 are set by TEA, so that's what they provide to us to look at because we don't have student level data or student level SAT or ACT scores. And that's is what they provide.  I think it’s also because it correlates strongly, an 1110 or 24 correlate strongly to college achievement.

Rob: Now I will say the good news is that we have are more students completing college then we then we do who are measuring a college ready level, so 28% of Dallas County students completed a postsecondary degree within six years of high school graduation compared to the 13% that were deemed ready for that, so we take that as a good sign that a) more kids are ready, and we need to provide more support to students to bridge that gap from high school to college, and that’s been a huge part of our work with the Dallas County Promise, creating more pathways for students to enter college, have support systems in college, and hopefully then succeed, whether that's a two-year certificate at a community college or a four year degree from a University. High school graduation numbers are really at an all-time high across the country, this is one of those things where you’ve seen a lot of pressure placed on adults to graduate as many kids as possible and so, you know across the state rather 90% level, locally we’ve at an 86% level. We really don't think measuring our high school graduation rate is the is the most important indicator of success for these students, we think of were really giving every child an excellent and equitable education, we’re preparing them post-secondary, and success at that level, that’s ultimately how we’re going to shift some of the concentrated poverty that we see in Dallas County.

Chris: When we look at the final three indicators of the pipeline, as far as enrollment goes, in this year 61% of high school graduates in Dallas County enrolled in a two or four year college. When we look at persistence we see that 50% of high school grads will return after their first year in college, and then only 28% of high school grads who enrolled in college complete a degree.

Ashwina: Our district partners, who we are very grateful to, provide us with reports on how their graduating classes are doing. So we request National Student Clearinghouse reports from each of our partner school districts and they share that with us very graciously.

Rob: Now is this tracking students who go outside the state as well for college?

Yes, through the National Student Clearinghouse reports we are able to look at students within Texas as well as outside of Texas.

Rob: And those three indicators are fairly steady this year, postsecondary enrollment and postsecondary completion stayed the same as last year, we did see a 2% increase in postsecondary persistence, which we were surprised and happy to see. Typically as the economy heats up, as wages grow, you see some downward pressure on postsecondary persistence. It often creates more short-term financial incentive for students to drop out and get a job then to stay in school and complete. So we were pleasantly surprised to see a 2% growth there and we’re hopeful that this is an area where in particular through the Dallas County Promise we see some pretty significant shifts over the next several years in these three indicators.

Ashwina: And there is also less a than 1% increase in enrollment which we were not expecting, so that was a good thing.

Rob: So we've got three coalitions that focus on big levers that we think drive improvement across the cradle-to-career pipeline. Early Matters Dallas is a coalition that we formed to focus specifically on really birth through third grade, understanding the so much of the child's brain is developed before often they even hit traditional schools, so knowing that we really need to impact how parents are seeing themselves as their child's first educator. We've got a coalition that is focused primarily on teacher retention, recruitment, and support, so that's our Best in Class coalition in partnership with Communities Foundation of Texas.  And that really impacts the entire pipeline, that is a pre-K-12 effort that that's really trying to understand how we make sure our teachers are getting the training they need to be as successful possible and to make sure that they’re in the classrooms where their needed the most.  And then Dallas County Promise is our newest coalition that just launched this year, and it is focused exclusively on how do we bridge the gap for high school seniors and help them transition through pathways to careers that can help break the cycle of poverty. It’s worth noting that the average earnings of college graduates are twice that of workers with only a high school diploma and so we really believe that if we can drastically increase the number of students who attend college, even if it's just for a two year certificate, it can have a significant impact on the poverty problem that we face as a county.

Chris: So if you earn a high school diploma in Texas, over the course of your career, you can expect to have earned about $1,500,000. Now compare that to somebody who’s graduated college. Over the course of their career, they can expect to earn $2,500,000. And so that’s a $1,000,000 gap between folks that only earn a high school diploma, and folks that graduate from college, whether it's two, four-year, or a certificate for college. It's a $1,000,000 gap.

What this means, for every high school graduating class, the entire economic system is losing about, we’re missing out on about $20,000,000,000.

Rob: Of additional earning potential.

Chris Over a lifetime or over a career.

Rob: Yeah, we see a lot of this as absolutely nonpartisan. The reason we think this work is important is we think every child deserves an excellent education, but it's very easy to make the case that this is an economic issue, this will drive GDP in the county in a significant way if we can get more of our K-12 students into college and successfully completing college. It's good for everybody for this to be a focus.

Chris: I think the big message is the importance of early education, especially when you think of the relationship between strong early academic performance and later outcomes in life, and so when you look at these metrics like Pre-K enrollment, getting kids ready for school early. When you look at metrics like third grade reading and fourth grade math, getting these really core skills early on, that's going to impact college enrollment rates college success rates. And so I think that's just a real important message there, is that the better we can educate our kids when they're young, the better they're gonna do later on in life.

Ashwina: And I think for me it's a little different. I completely agree with pipeline. For me it’s, I'm hoping that people use this information and go ask legislators what they are doing to change these numbers and improve these numbers. So how can we be better informed public and ask a legislator, “do you think that 40% of our students reading on grade level, meaning four out of ten kids reading on grade level, is that okay?” And if not, “what are you doing?”

Rob: Peter Drucker said, and I'm paraphrasing, “You can’t improve what you don't measure,” and so that's the other thing I think is really important about the Scorecard, and the 11 indicators, is it’s our attempt, as imperfect as it is, to add some measurement so that we can make sure we’re all focused on improving outcomes for students. That is our number one priority as an organization.

Joshua: The Miseducation of Dallas County is powered by the Commit Partnership and produced by me, Joshua Kumler.  It was executive produced by me, along with John Hill and Rob Shearer.  Mixed and mastered by Will Short.  Music by Trevor Yokochi. Special thanks to Ashwina, Chris, Rob, and everyone else at Commit.  You can read a full transcript of this conversation and see even more Scorecard info at our website, commit2dallas.org.  This podcast is dedicated to educators everywhere.  The future is in your hands.  We’ll be back in two weeks with more Miseducation.