Data Based Decisions? I doubt it.

Join me as I discuss what I see in the world related to how we abuse the phrase "data-based decisions". It'll be fun. 

Transcript

Copy Transcript Copied

0:01

All right. Welcome back to Scratching the Surface of Workforce Development with Courtney Taylor, and I am Courtney Taylor. And my allergies are with me today, so may be a little bit scratchy, It is July in Mississippi, which means most of us are dealing with the heat, and I obviously have chosen to live a life where I have to be outside much of the day when I'm not at work. And I'm just trying to survive. I hope you are, too. And if you're in one of the cooler climates, I don't wanna hear from you. But today, I am going to tackle something that I have tackled in some ways probably a little bit more passively m- than I ever really intended. But now I'm ready. So let's start by asking you a question, and it's a pretty simple question, Have you ever noticed that lots of people claim to make, quote, "data-based or data-driven decisions"? But have you also noticed that almost nobody is actually asking whether the data can answer the question and what the real question is? And so to start, I will tell you that my known bias in this conversation is that we can do better with data in this country. We're just choosing not to. We are limited by outdated technology control mechanisms at at every state. This is... The more states I talk to, the more I realize that we still have a 1990s and maybe even an '80s view of how to manage technology, especially in, in my case, the only technology I really care about is the technology That drives our ability to access quality data points. So most people in most states, they just end up with workarounds. We don't really wanna make the hard decision as a whole because, then #politics or somebody doesn't want to be the bad guy, which is the most toxic thing in education and government is nobody wants to be pinned for it. And I think, as I think about it and look at it, I think there's a willingness. I think people want to know. I think it's aspirational in most cases to know what's actually happening. And I think it's just a buzzword and it becomes a defense mechanism. "Oh, we make data-based decisions." Okay, with what data? Let's define the data that you collect directly, okay? What it ends up usually being, because of the way the system is set up, is more of a, an intuition, or we get a couple of people who respond to our request for information, and so then we know a couple of people are doing better, or a company tells us that people are doing better because they certainly won't give us the information because they got some policy that says they can't. So we have this system that I honestly I'm not... I have a mentor who tells me I'm not supposed to think this way, but here we go. I do genuinely believe it's intentional. If we can keep the wool over everybody's eyes just enough to give them enough information so they stop asking questions, but not enough where we can actually look at the hard data Then we'll all survive and we'll be okay. And I'm not mad about it. It just, it is. And so I think we have a lot of good people who are trying really hard to do things, but they run into roadblocks, and they quit, and they go do something else. This is one of the issues, at least where I think my organization is concerned, is because when we run into roadblocks, we double down, and that becomes a problem for everybody. So it just is what it is. And we have the luxury of not having to be the one dealing with the program. So this isn't a criticism of the people who give up and go do their actual job. This is our job. So that's the difference. Anyway, I sit in rooms with people and and they just... th- this buzzword. And then another person will repeat it, and then another person will repeat it. And oh, it's like, okay, where's the da- data coming from? Oh, we surveyed them. Okay, great. What, what did you... what questions did you ask them? And these usually end up being some kind of satisfaction questions. Are we doing a good job? And so when I'm talking about workforce, I think we, I think we have kind of a weak expectation of what we are producing because we, across the board, I think maybe, lack leadership, lack discipline, direction. I don't really, I don't really know. So as I think about it what I have tried to do is be very transparent and direct with what is working and what isn't working. And as we've worked through this, and we have a state longitudinal data system in Mississippi, which is fabulous for people like me who just want to know did this thing make a difference to the people who went through it? And we can ask that question, and we can answer that question. The challenge becomes the timeline of the answer. So a lot of the times what my brain and what I try to work through really does put me at odds with the timelines of my job. And that is where I think kind of some of the fun, frankly that I have is trying to figure out the puzzle piece. And I have really good teammates who can dive deeper into it when I'm like, "Hey, this thing." And they're like, "Oh, hey, this other thing." And so we work through it. But I think I think at the end of the day, what we fail to own is the fact that the person made the decision, which is a judgment call And ultimately, that's pretty much what we have is we have to make rapid successions of judgment calls in workforce, and we need to do it faster. Which means we need data that is faster to get to with insights that are easier to get to. And with the proliferation of AI, this ought to come pretty quickly. But people are still protecting the data. We have it. We know we have it. Why can't we use it? Because we typically only really use data to hurt somebody or to make a change, not to support or even frankly, to understand, and that's something that I have been working with w- on my side of the house with the legislators that we work with is l- let's stop, let's pause, let's understand things. And hoping that we can present the data in a way that doesn't skew their opinion, meaning we only produce the data that we wanna see, 'cause that is an absolute thing that, that sometimes happens in this world. But that we produce the data that just is what it is, and then we pause, and we work on the insights of that. We have a whole division in our office called Data and Insights because if all I do is produce data, I am no better than every other organization that exists in workforce who's just producing large reams of data that is requested by legislators, by Congress to show us the data, and the data can lie to you every step. What you need is the insights, and what you need is context, and that is something that I think we are really committed to, my board is really committed to. That is not easy, and so we have worked with some national organizations. I'm really proud of the work they're helping us do to try to help us get to insights faster and to set systems up where we can pull those insights really quickly long term because just producing a report and putting some words on the paper about what was happening in this moment in time is not an insight. It's a summary And that's what we're trying to work through. And, sometimes I just remind myself, I could have chosen the easy way, and I didn't. So as we work through this, the data should inform decisions. I completely agree with that. I do think we need to have data-based decisions or evidence-based decisions. The challenge though is we often pick the data or the evidence that suits our need one way or the other, whether it's positive or negative. And the reality is probably that it's a lot more, the results are probably a lot more even keel than really good or really bad. Now, I say that knowing that, man I wanna break away from this for just a second because we had a committee meeting a couple weeks ago. And Senator Sparks is our chairman from, for Workforce and Economic Development. And we were talking through data. We were trying to identify which, what post-secondary attainment, credentials need to be. And I am not a believer in blind credential attainment goals because I have watched lots of states, lots of places produce these things, and you end up producing a lot of people that have a credential, but it's not necessarily one that's driving your economy. So you're not really doing anything in my opinion. That's a negative opinion, I get it. Welcome to the party. But I prefer to just focus on... these other things can still happen. They're not bad, right? Just go do those other things. We're just not gonna put any time into it. And so he was talking and it caught me, and I'm not sure a lot of people heard what he was saying, but it caught me because he basically flipped the conversation and said so- some extent of have we educated or trained the person? Are we focusing on the programs and whatnot that allow us to disqualify people from government benefits And I said, "Holy cow." I wrote it down. I literally in the middle of all this, I have no idea what was said after that because I said, "That is really super powerful." And depending on your politics and where you sit in the world I don't know what everybody believes or why they believe it, but in my world, if human capital invests in itself and takes money, takes time, all these really precious resources away from maybe working right now or doing other things, it invests in itself and we're giving it state funding, federal funding, whatever. We're giving the individual, the human capital in this case And we're not even doing enough to get them off of benefits or disqualify them for benefits? Holy cow. We often talk about education like, "Oh, did you earn more than 30,000 with the new earnings premium test?" Did the degree produce somebody earning $30,000? Now, understand that it's not up to the human capital to build jobs and create jobs. That's up to private industry. And so this is a really complicated reality.. I have a friend, Emily, who one day made this other statement and I'll s- it was some, some version of, "If someone can be poor without working, but they're going to still be poor when they're working, should they work?" And that gets into a lot of personal beliefs and politics and all of this kind of thing. I will tell you that I think simple economics applies here, and it should apply. And so when we're talking about are we training people to get them off of benefits, full stop, or are we training them and they're staying on benefits? I think we have this culture where we just assume, the, the welfare state is oh, the, the people just want to stay poor. And I'm not saying there's not people that want to live the way they're living, and that's their problem. That's their life. That's none of my business. It's certainly none of the government's business. But if we are putting government resources, local, federal, state, whatever, or even private resources into things that we know are gonna keep people poor, what the hell are we doing? If we know going into it that they're going to go through this program and they are gonna remain eligible for federal benefits, what are we doing? Okay? My governor has a very simple request of me, which is work on things that get more people working, making more money, so if I wanted to be a typical bureaucrat or whatever, maybe I could say if they were making $0 and now they're making $14, they're better off. I don't believe that personally, professionally, not at all. The data in that case says, yeah, okay, cool, they're making 14, $14 an hour, and in this economy, that, that is nothing, okay? In, the wor- numbers we use is $20.06 an hour. I think it's six cents an hour. 20.06 an hour is what you need to make to have a living wage as a single individual in Mississippi. Now, I don't get into what you need to make as a parent with two kids or a single parent with a ki- I don't get into all that. A single individual needs to make $20.06 an, an hour. So if you're not paying that amount of money, they're not gonna earn a living wage. And so when I look at it, what I care about is these higher wage things. These lower wage things are gonna take care of themselves. Society will take care of people. We'll figure it out. But if you're being very selective as an employer and you're not paying a living wage, you have a, quote, "workforce problem," what you really have is a pay and alignment problem. So data can be very helpful, but it does not tell the whole story. Now, I will also put a plug here for enhanced wage records 'cause this is something that we've talked a lot about in our state. We've got some really smart people who have worked on this who really want it and, we are trying to, or at least I am, navigate through this world of How do we get this information without it being Big Brother situation, without it being used against industry? But we're using the same data, poor data, against individuals, so maybe it's all fair. I don't really know. I don't really know how I feel about it, and I'm trying to get through it. I know from a tactical standpoint that if I could show that Courtney went through this training program to be a production operator, and it was a 96-hour course. She went through it. She learned all the things she needed to learn, blah, blah, blah, blah, blah. She is now working as a production operator. If I could just very simply get that report on demand, right? Like just, boom, give me the data. Let's go. I could move so much faster as an organization, as a state, as an industry, and it's possible. It's there. We've got some, some people doing... Some states are doing enhanced wage data. Most aren't. E- everything is there. The system has been set up. Our, in Mississippi, our Department of Employment Security, they already set the system up. Why can't we do it? It would allow me to make a lot more decisions a lot faster and with better judgment calls because right now, the data I am using to make my decision is based off of, two quarters after they exit a program, so that means six months before I can tell you anything. But it's based on the simple fact that they worked, they earned an income through a W-2 employer, frankly. They worked some amount, they worked some amount of time during the quarter, say, okay? So during the three months, they worked some amount of time and earned some amount of money. I have no way of knowing in what actual job. We can extrapolate. We can make estimations, I have no way of knowing, How many dollars per hour this person was making. So if my system, if our longitudinal data system comes back and reports out that the person earned $13,000 in the quarter, was that for three months or one month or, whatever? No, they have some very complex, nerd things, I love you guys some nerd things that they do to get to, a reasonable amount, but it's all still estimates. And they're very critical estimates in this en- environment that we're in right now. And so it is a, it is still very much a lose-lose game if we're not really thinking about this. And so anyway, at some level in the next couple of years, we've gotta get to and this is law in, in the country. Did they go through the program? What are they earning per year? And then I don't even like the per year number because I wanna take out overtime. Overtime is fantastic for individuals and they're in hourly roles, like love it, but I don't want to make decisions on programs knowing that a person is gonna have to work 80 hours a week to earn a living wage at the end of the day, a living salary. They should be able to do it on a 40-hour week, and everything above that should be what they use to go buy luxuries, right? So anyway we have some flaws as a nation in our data, and it really comes down to politics because it re- and it should. It comes down to what do we believe, why do we believe it and what do we wanna do about it? And I sit in this, this gray world of I have a luxury, as I have been told in the past, is I don't have to worry about necessarily those things, and I do. I cannot be I cannot be dumb about it, but I do have to ask the question, and I do have to consider it as an organization who is trying to make genuine database decisions, not knee-jerk reactions, because the initial data is usually always going to be worse than one year, three year, five year, and I'll talk about that in a minute. But as we, as you kinda think through the continuum, it's you've gotta get the data, and it's gotta be reasonable data. But just having it in some data lake somewhere does you nothing. It has got to be in a form where you can glean insights from it and we have to have really good data dictionaries. We have to have really good data or research agendas. We need to have quality research research questions, research agendas. We just have to be a lot more intentional about this State governments are genuinely not gonna be the place to do this generally speaking because they don't pay enough to get that quality talent in. You're not gonna pay somebody a limited wage that's very good at this because a research entity is going to scoop them up, a university, somebody like that. So government by design, based on the way I see it right now, the system itself is literally by design staying hopeless in some situations because it wants to. This is the system that we have. We hire people at low wages, but they get a good retirement, so if you're better off in retirement than you are while you were being productive, that's a backwards-ass system. I don't care who you are. What if, hear me out, what if we just paid people what they're worth today? We hired the talent that we need, and when we no longer need it, we send it down the road to go do something different. It's shocking. So in the same way that we control technology, we control human capital and personnel through the control mechanisms that every state has And I know I'm supposed to be talking about database decisions, but all of these things impact it. So you get through all of this you lack data, you lack this, so you have to use a ton of judgment calls, and you have to be very comfortable with that. And that is where the system really breaks down further, is because most people aren't, 'cause it's scary. You are talking about making judgment calls that could potentially negatively impact people, that could negatively impact politics, that could do whatever. You still have to be willing to make the decision, or in my opinion, you need to go home and go do something different. So as, as we get into this data conversation, and I'm talking around the world, we have to look at the simple reality of causation versus correlation. And this is where I think WIOA is just... Oh, man. WIOA, m- my funding, your state funding, whatever is, they went through a program, they're better off, therefore the program caused that, okay? Y- community colleges, universities, "Oh, our economic impact." We all live this way, right? There's no way to really look at extraneous factors. There's there is a way, but that would require real data with real research and paying a lot more people to do that kind of work. But this is where I think we all get into trouble is we don't take certain things into account. The person went through the program. That, I have programs right now that we're evaluating where we started with more people than we ended with and there were more people working before they took the program and on the back end than, than there are on the back end. So we start with 100 people on the front end working, and you get to the back end, there's only 90 working. Are they not working, or did they get a job with an employer that reports to another state? Again, a problem we already have the data for this. The feds know it. I can't know it, though, unless it's attached to a federal program. What? We know it. It's knowable, but I can't access it because of some data sharing rule My tax dollars pay for that person to go to a training program. My federal tax dollars pay for this entire infrastructure for data. But because I didn't use federal resources, I can't report out on that person. My answer is that they are not working. Now, I can solve that by getting into a data collaborative with another state, but I'm still not gonna solve it all if they're going to work for an entity that's working in my state, maybe building a data center and they're from Seattle and the person is living in Mississippi, they're working in Mississippi, they're paying taxes in Mississippi, but the reporting comes from another state. I can't help that. So correlation does not equal causation. Yet I have to make a decision and a judgment call on whether or not I'm gonna fund that program again And I'm gonna have to make it. And I have to be comfortable with being wrong 'cause I might be wrong. I might be right. I don't know. But it's knowable. And so it's like, what are we doing, y'all? And why aren't we, why aren't more people mad about this? Why are we just continuing to accept this and being good little boys and girls because we don't wanna hurt somebody's feelings, or we don't wanna frustrate the feds, or we don't wanna frustrate our industries? Give me the data, or figure it out yourself. That's really where we're gonna have to get to, is if you're asking the state and the feds to spend someone's hard-earned money on these things, you probably should consider giving them the data so that they know if it's doing good or not good and stop protecting it. That goes for everybody. I think I think the world is messy. I think our economies are shifting, and they're going to continue to shift faster, and government solution is, "You know what we need to do? We need to control this process more because we've had some bad actors. We just need to control it more." Punish the people. Get better on your audits and your monitoring, and let the rest of us do work Knee-jerk decisions So as we've worked through the data, and I usually giggle honestly, internally, and sometimes I will laugh in your face. I am that person. We- we made database decisions. It's like that's funny because I can't even make a database decision on X program because I haven't been given the data. So how are you making a database decision or a data informed decision? You're doing what feels good because someone had one positive result. I've heard welding instructors use a kid going like to Puerto Rico or somewhere making $200,000 in a year as "Oh, that's what you can do." Yeah, if you're willing to give up your whole life and move to Puerto Rico for a year, you absolutely can. But on average, you're not gonna do that. So knock it off, so we use data and extreme data on either side and it sells. I talk about this and it depends on the kind of the group I'm with when I'm talking about it, how it comes off. But why is a bottle of water a unit of measurement suddenly? All of a sudden when we wanna whine about data centers and we wanna attack them, it's, "Oh, you use a bottle of water every time you use it." Where in the world is that an actual unit of measurement? It is not. It is some data point, some nebulous thing that someone pulled out because it sells politically, and then they're using it against the whole system. And nobody's questioning that. Like, why am I the one? 'Cause I've questioned it. People are like, "Oh, that's a good question." I'm like, "That's a simple question." Why is it a bottle of water? Why isn't it eight ounces? Why is it a bottle of water? And what size bottle are we talking about? A gallon jug of water? Are we talking about like a little 16, 12 ounce? What are we talking? A little mini? What is it? The context matters, and so data without context is either useless or painful or both, because you're just gonna make a knee-jerk decision without any insight. So are our people going into programs where they can be disqualified from federal benefits because the outcomes are so good? Holy crap, that's a question. And that's a good one. And I shouldn't even be putting it on here. I should wait until I roll out something I've come up with based on that thing. But I think you all need to think about it, so I'm gonna tell it to you. And it ain't my idea. It is Senator Daniel Sparks. And I appreciate that concept that he's thinking critically about these things, 'cause that's what we need more of. So as we work through this, as we think about data as, just as we go through there, I think the next phase of this is obviously, I've talked a lot about like the front-end data collection. We've gotta get better at that. We've gotta go fast. We need to get mad access to systems and technology that will allow us to do this faster and on the fly, okay? You shouldn't have to create some of the stuff we have to create right now and let me be plain, like the people we're working with in my state specifically on this, they want that too. But we have technology and discipline issues to get there. So now as we think about these different interventions and evaluating them, 'cause that's what I'm ultimately talking about when I talk about making database decisions, is can we evaluate an intervention in a person's life? And usually the data is almost always going to be single dimensional, and I just really dislike single dimensional things. Mostly because in my my experience, I have had to make a lot of decisions on single dimensional data and information, and it never tells the whole story, and you're usually being manipulated if it's single dimensional. Once you start getting into multidimensional data where the points start really producing a story, it kind of hearkens back to when it kind of hearkens back to when I was working on my dissertation and you learned that you would get to when you started seeing the same thing over and over again, right? And so that's when you're like, "Okay, enough down that rabbit hole." That's how I'm thinking about data. But, have we designed in workforce the appropriate evaluation windows for whether or not a- the program is successful? Okay? Whether a resume writing w- workshop was successful, you ought to be able to tell that pretty quick if you're teaching it to people who are looking for a job. Did they get a job in the last couple of weeks? Okay? You could probably tell that pretty quickly. It shouldn't take you six months to figure out if your resume writing workshop worked. A CDL program, we're gonna need two quarters right now, right? You gotta get them into the workforce. You gotta get a quarter data behind them, two quarters so that you could potentially have a full quarter of data. A nursing degree, you might have to go, what, three or five years afterwards. We can look at initial impacts, right? Did you get a job and how much did you make? What was your starting wage? And then where are you a few years from now? I'll use a fourth grade reading gate, a fourth grade reading intervention that you pass, over a decade ago in Mississippi. You're just now starting to see the impacts of it And they're not in the workforce yet. So it is, it... Some of these things are just really long-term, and that doesn't fit a political window. That's just where it is. The data points have to exist before you can evaluate them and that takes time. I think we use by and large in government the same evaluation timelines for everything. And I think that's painful. And so that's some of the stuff we're thinking through is, when do we evaluate what, and what does success look like at that point? When we talk about these different... If we're looking back right now on 10 years, if you're six years out from a person starting a program, that's the typical completion date, and you wanna go back 10 years, that's 16 years you gotta go back to get a full 10-year data on a degree, a four-year degree program. Okay? You're looking at 13 to 14 years on a two-year degree program to look at the long-term impacts of that program. This is not easy, and I'm not suggesting it is. But I also think it's something that we need to do, and we need to be really disciplined about it, is what is the evaluation timeline? What were the judgment calls? What were the leading indicators and the lagging indicators? So I think as a manager I gotta have some leading indicators. It's what do we have to fill the funnel with to get what we need done on the back end? If, 30% of all the students who enter higher education are completing it, and I need more of, If I just wildly, which no one does, but if I just wildly need more bachelor's degree holders how much bigger does the funnel have to get before I can complete them? Or are there interventions along the way that improve retention and completion without sacrificing quality? And that's a whole nother conversation. But researchers they are gonna need some lagging indicators. And so let's think about think about it... I'll use a, a community college as an example, but I'll also, harken it back to my world. But if I'm a dean at a community college, I don't have five years to wait. Okay? As an executive director of an agency, I don't have five years to wait to say, "Did that program work?" I have to go right now. Are we gonna stand on its successes, or are we gonna say, "Nope, we're not doing that anymore"? I gotta know, did the people actually attend the class? Did they complete it? Did they get placed into a job or a clinical or whatever the next step is? Where are the employers? Are they earning what they said they would? And I've got an example right now of a program and a company that I have just been so impressed with, and that's The Industrial Company, TIC. It's a subsidiary, I think, of Kiewit and the man's name is Ron Duce. And Ron has worked in the Delta specifically, this is the program we'll talk about to produce a a training program. It's very small. At the end, we're talking about, eight to 12 people a class. You can only put so many very green people on a construction site. The company has... they monitor attendance. They've been there the whole time. The local workforce development area Mitzi Woods, her group, Jacqueline, Ms. Jacqueline is amazing. They recruit people, they screen them, they put them in, and they case manage them every step of the way to the point, frankly, of mothering when they have to, right? You like show up on day one of work 'cause they offered you the job. You gotta bring all this stuff. You gotta get there. Not everybody's doing that, and it shows. And so from that world is we know the employer is engaged. We know the people are getting jobs, so we can keep investing in that. We can keep rolling. That's not a research question. The, the research comes later, but it often comes without real research questions. And if the data is not open enough and accessible enough, you can't really ask quality questions. And so that gets back to insights, right? And context. And, we need all of these things, and it doesn't need to be locked away where you can't see it. It certainly needs to be captured. And I think this is where the federal government may have to step in and be the leader on this because it is gonna be really challenging for everybody else, to kinda do this. And frankly, it'll, it could get so discombobulated that it doesn't make an impact. But I just, I know we can do it better, and I am so hopeful and have just committed so much of my professional world to figuring this out. And I think I think as we talk about these data-based decisions, we have to also think about what's the judgment call and what are we trying to produce? I can tell you what we're trying to produce based on what you're funding, okay? And we don't think about it like that, right? But in education, what we are trying to produce is very simply more degrees across the board, okay? Even in states like Texas, where they had just this-- love the performance-based funding. I love that. But it wasn't limited. So they produced a whole lot more degrees, and the colleges were like, "Hell yeah, we'll take that apple," and they go. But were they degrees that mattered or were they just degrees? Sure, their certification's of value, but what's the value? It's just, it is so complicated and I am not, I am absolutely not judging Texas 'cause we love Texas and we've benchmarked so much of what they are doing. But I think it shows you how challenging it is to really get leading and lagging measures, but specifically leading measures And do it the right way to be able to really, finalize what you need to do. I think about it this way. When I say, you're getting what your ultimate results end up being what you're funding. If you're funding enrollment, you probably get enrollment. Do you get completions? You don't know, and you probably don't care. You probably are not asking that question. And when you do ask that question, it becomes so alarming when you see how bad the data is that you freak out about it, right? Or you get lied to, and education loves this first time, full-time, 'cause that's what the feds capture and blah, blah, blah. And it's you know what? What I wanna know is if 28,000 people left high school and went to college, 10 years later, how many of those people have the degree, period? We know that answer, and it is not a fun one for anybody. If you fund based on credentials, one of the things we've seen a lot of states, and we've talked to a lot of states who are like, "Don't make your list bigger." And we're like, "Bet, we're happy with that." Because they're now having to come in and pull things off of their list. And if you wanna have a lot of fun, put something on... Put a low wage, low, genuinely low value credential on a list of any kind and try to take it off. You will be the scum of the earth, and everyone will hate you. Anyway, if you fund credentials, you're gonna get credentials. You damn well better make sure they're credentials you care about. And that's where we make the mistake. If we're gonna fund enrollment, we're gonna get enrollment. But is it enrollment in things we need more of? Probably not. Because you haven't incentivized a quality metric, you've incentivized a quantity metric, and that's ultimately what we're talking about. And just so I am very clear, I am not a fan of traditional performance-based funding models because they get so bastardized by the system that they don't actually produce anything. They produce more, but in these cases, more is just simply more. We're not driving the economy. We're not leading with the answer of, "Hey, companies, I can tell you that we're serious about this 'cause here's what we're doing." Now, we're trying to take a different approach, at least where we're concerned and the funding that, that we are working with in Mississippi, and so we'll see. We'll see. I might come back in, in a couple years and be like that sucked. That didn't work." But I think it will. I think if we focus on the things we need more of, you incentivize the individual and you incentivize the education entity at the same time. They gotta have an aligned, an incentive. Otherwise, you're gonna get a lot of people hopped up about doing something that they can't actually do because there's no room for them. And why would a college increase a program if it's going to continue to lose money on it? Because it's the right thing to do. The right thing doesn't actually keep you in business, right? So there's a lot of complicated things here, so I'm not trying to make this easy. I'm not trying to lecture or criticize, but I do think we gotta talk about it. And we need to talk about it, deeply without getting our feelings hurt. That's the one thing that drives me crazy. I hear... you hear the little rumblings, and it's like, "Oh, Courtney, blah, blah, blah, blah." It's like, "Sweetie, I don't think about you. This is not personal. I literally care about this system that I am charged with taking care of. That's what I care about. I care about what it produces, and I'm not gonna, I'm not gonna hold my words to make you feel comfortable, and I don't expect you to do the same for me. I was asked to disrupt. I'm going to disrupt. My team is going to disrupt. We wanna do it with you, but if you don't want to, that's fine with me. We're still going to do it. You can work with us, and we probably will disrupt less but better, or we'll just do it ourselves," and that's how I live all the time because I just don't have the luxury of slowing down and waiting for the conditions to be right, for the people to finally get it. That's not the luxury of what Mississippi created when they created our office which is why I love it so much. But I think to start wrapping this up a little bit 'cause I'm just talking longer, but I think if you think about, data and what we really need, I think it's simple to just stop and say, what if w- what if? Just take a moment. Close your eyes if you're not driving. Imagine a system where every college president, workforce director, superintendent, nonprofit leader, executive director, legislator, whomever, could hop on a website and just simply see What the data says. The data are what the data are. Okay? That's all you gotta do. Just go onto the website, look at the data What if? This is not a gotcha game. This is not trying to prove a negative point. It's not trying to prove a positive point. It's just trying to prove the point. Here is where we are. Then researchers and professionals and people who are really good at what they do can look at that data. A kindergarten teacher has no idea what her impact is on the students. Okay, saying 98% of the students graduated high school is a lie, because we put them in alternative programs to get them out of our numerator and our denominator, so they just cease to exist. So 98% of the students that we haven't found anything else to do with may graduate, but it's not 98% of the students. How do we help a kindergarten teacher learn how to, know the need to produce a, a higher quality student, at some level, blah, blah. I know this is a horrible example, but you get what I'm saying. If they can't ever see any correlation to their work. So I think... and we believe it because we're humans, right? And we wanna believe the good things. And if we-- and if somebody's pointing out the bad things, they're a jerk. They're trying to catch somebody. They're, name whatever, right? Instead of maybe no one's trying to catch you. Maybe no one's trying to point out the bad stuff. Maybe we're just simply trying to make sure we continually improve. And that is what, literally, I will put my state up against anybody on that. Anybody Is it perfect? Nope. Are we there yet? Nope. Do I hope we get there sooner than later? Yep. And we got a lot of people trying to figure that out. And so when I think about the system that I want to represent, it's this kind of system where accountability is ultimately just a continuous improvement instead of an annual judgment done through some report that nobody ever reads and nobody cares about, right? Oh, okay, here's what the data says for this program. Were there... anything else happened? Okay, something did happen. There was a layoff in the community, and so the program we were training for, okay that, that explains that. Okay, what do we need to do? How do we fix that in the future? All these things. That's what I want, and you need to have whole people dedicated to that, 'cause the people running these programs cannot stop and think about the impact because they're too busy chasing grants, chasing paperwork, chasing their tail, chasing their leaders, chasing humans who keep running off doing stupid things. Like this is the reality. You have to have units who are not responsible for the work analyzing the w- work, and doing so fairly, equitably and with lots and lots of diligence and intention, and I believe that thoroughly. Now, what I believe and what I can accomplish sometimes are two different things, so we'll see where we go. We'll see if we can get there. But I think having the conversation and beginning it is step one. But now I think as we talk about these systems, a key takeaway in this is that none of these systems are working together. They are not designed to work together. They are not funded to work together, and they're not expected or measured to work together. That's what we've got to figure out. Because as we're talking about data, we know that the data can tell us what happens. We know that research should be able to tell us potentially why it happened But the thing that we need most is leadership that decides what to do next, and does so quickly. So the thing I have to keep coming back to is even like I, I think I'd put my- there's very few states like mine. There's a, it's what? Maybe a quarter of the states, I think it's less than that, have some version of a longitudinal data system. So all cool. We know things. What are we doing with those things? And this goes well beyond me. I can do what I can do, but I'm not a legislator. I'm not elected official. Somebody else has to take this baton and run with it. But we're gonna have to run with it, and we need to run fast, and we need to be setting up systems and data systems across the board that can run with us. And so that's ultimately my hope is that we build systems that can provide evidence through legitimate research, through legitimate data collection research, whatever. And the research shouldn't have to... We shouldn't have to employ five researchers to go look at a single program and say, "Did it matter?" The data ought to just produce. But what we gotta define is what is it? What is an acceptable metric? What is the dollar amount? What is the percentage? Da. What is the timeline that's acceptable for the program? Because if we produce a person and eight years later they eventually figure it out, did we do our job? And I'm telling you, no, we didn't, 'cause that's not what we're selling them on the front end, okay? So whatever we're selling them on the front end better be true on the back end, and that's one thing I've told my team over and over again, is as I'm walking through this, their job is to make sure that we are walking my talk. 'Cause what I cannot stand is the people who talk talk, and they're not actually doing the things they say they're doing, right? It is human at some level to talk g- greater than you're actually doing maybe. I don't want that. And so that's where we are. But anyway I think the conversations need to be had. I think they need to be had at a big scale, and I think we have got to figure out how to move this infrastructure towards something a hell of a lot faster than we are right now. So I don't know. We'll see what happens. We've got some things cooking that I'm really excited about in our office that could help help our workforce directors and our college presidents and, our superintendents and whatever, just see the data in a way that they can actually see some context and know "Oh, I thought that was different." I've seen... and this is not all bad, right? I've seen programs where you're like, "I'd have sworn it was going bad." And then you see the data and you're like, "Huh. That's a lot better than I thought it was. That's fantastic." And stop relying on our gut so much for did it, is it good or is it bad, and rely on our gut more for what's next. And that's what I want for the next generation of leaders is that they can spend their time not having to question crap and un- uncover, lifting a brick up and a bunch of snakes run out. You don't have to do any of that stuff. It's just there. It's at your fingertips. We're rolling. We're going. And you can use your skills to move us to the next level. That's where I am on database decision, among other things. Again, as always, I'd love to chat with you about this. I'll give you back the rest of your time and I would just say that this has been Scratching the Surface of Workforce Development with Courtney Taylor, and I am Courtney Taylor. See you on the flip side