TRUE ACCOUNTABILITY BLOG

Getting True Accountability going across the country requires lots of voices and lots of dialogue. The blog below will host those voices and the responses to them. Sign up to be notified of new posts and to add to the dialogue. 

 
Search

I finally managed to squeeze a description of True Accountability onto a single page. You can download it here:

True Accountability System in Schools
.pd
Download PD • 73KB

I've included the text below as well.

John


Organizational accountability is always and only a system that enables you to build trust with your primary stakeholders. Your primary stakeholders are members of the group without whom you would have no reason to exist. The biggest obstacle regarding accountability concerns the amount of technical knowledge stakeholders and those within the organization have. Those in the organization have a great deal of technical knowledge regarding the work. Those outside the organization have very little, if any.

The point at which trust can be built and which must therefore be the basis for an organizational accountability is the equally meaningful point along a continuum between those with all the technical knowledge and those with none. We refer to that equally meaningful point with the term benefit. A stakeholder expects a benefit as a result of their decision to form a relationship with an organization.

The benefit is the point along the continuum that describes what the non-technical stakeholder expects of the organization while also allowing the organization to recognize how to use its technical expertise to deliver it. Both technical and non-technical people can use language at that point and recognize it as the same thing, despite their very different levels of expertise.

A True Accountability System enables an understanding regarding the technical work within an organization for a non-technical audience. Organizations of all kinds use a wide variety of formal systems to do this including but not limited to annual reports or summaries, anticipated benefits through marketing or advertising, new product releases or hints, political campaigns, and communication with boards and the general public.

What each of these has in common is that first, they identify the benefit they expect to provide to their stakeholders in a manner the stakeholders can clearly understand, and second, they attempt to build trust regarding their ability to deliver that benefit into the future. Organizations that then deliver on those benefits going forward build and maintain trust with their stakeholders while those who fail to deliver do not. Positive or negative consequences for the organization and those in it then follow.

It is important to note that the reason a stakeholder forms a relationship with an organization concerns the benefit they expect to receive. They may base that decision on the past, but the benefit is always ahead.

Translating this to schools is a surprisingly easy effort. Imagine yourself in the position of a stakeholder, which for a school is always the students, their parents, and the community. Now imagine all the benefits you expect as a result of having a relationship with your local public school. The language in these responses will be powerfully simple: e.g., “I want my child to be safe,” “I want to be ready for the world when I graduate,” “I need employees who can think critically.”

One striking thing about schools that makes such a system useful is that the benefits are almost always the same from one school to the next. But what is not the same is what that means to a specific school. Each school is a unique organization and will need to do different work to deliver a similar benefit.

Effective organizations all over the world begin and end their accountability conversations with this notion of benefit and when schools can do the same on a regular basis the amount of information and understanding our stakeholders will have will far surpass anything that exists today. And if our stakeholders can understand what is happening on our schools, so can our policy makers.

True Accountability in schools is anything but a participation trophy. Because the language of accountability is now in language the stakeholder understands, the validity and what a school or district reports regarding their effectiveness will also be easy to see. Only the truth in that environment will serve to build trust as what is true and what is not true is readily apparent. Leadership in a True Accountability environment is one of its most critical functions.

Suggesting that this work is doable does not mean it is effortless. It requires discipline, a clear understanding of True Accountability frameworks and processes, and a willingness to constantly learn and grow and shape a school for the future.

The following was derived from an interview and converted into this FAQ.


1. What is Accountability:

The systems and processes used for building trust with your primary stakeholders.

2. Who are the primary stakeholders?

The members of the group without whom your organization would not exist. For a school that is our students, their parents, and their communities.

3. What is True Accountability based on?

Accountability in effective organizations works from the benefits its stakeholders expect as a result of having a relationship with it. Benefits are that almost magical point that lets a non-technical stakeholder articulate what they need—improved health, readiness to be a successful adult, my plumbing fixed—that then lets the technical experts in the organization see how best need to apply their expertise to provide the benefit. At the point of benefit both sides can talk about the same thing and understand each other, even though they come to it with very different levels of technical expertise.

4. I hear you talk a lot about trust. Why is that?

Organizational accountability is properly referred to as a trust-building machine. That can work well for an organization or it can work poorly, and that’s always a choice. Because organizational accountability can never be turned off, you’re either building trust or hurting it. Those are the only choices.

5. You say organizational accountability can’t be turned off, but the states did turn school accountability off last year.

Which tells us very clearly that what states have imposed as school accountability isn’t an organizational accountability but something else. There are other accountabilities that can be turned off in a crisis, but a crisis is the very moment an organizational accountability is most needed.

6. Who does True Accountability?

Virtually every effective organization out there, including hospitals, churches, synagogues, businesses, non-profits, and schools. They don’t often use the same words and they use a huge array of methods and tools, but they are constantly working to help their stakeholders understand their capacity to deliver a benefit into the future.

7. How long has the effort to get True Accountability into schools been around?

The initial idea started more than a decade ago and the more practical tools and solutions were in place roughly five years. This is still very much a new movement and we’re all learning from each other at every step. We’ve kept it at the level of grassroots efforts and to this point haven’t really pushed it. That’s changing this year—we’ve learned enough that it’s time to go full steam ahead. And remember—the frameworks have been around in lots of different organizations for many, many years longer than that. We’re not really inventing much, just learning how it applies in schools.

8. Where is this being done?

Two state consortia have emerged as the leaders in this work, first in Texas and second in Georgia. Their efforts really helped fine tune the tools and processes. Rowan-Salisbury in North Carolina has also really stepped up on this as well. When the pandemic hit, we were working on getting started with consortia in Mississippi, Pennsylvania, Ohio, and Kentucky, and we’re now working with several national organizations to extend this to a national movement.

9. Is it expensive?

No. Compliance-based systems, which have long been in place in schools are very expensive and we’ve become accustomed to the cost and burden of them, but true accountability isn’t expensive for a whole host of reasons. The biggest reason is that compliance-based accountabilities offer real advantages if you can get away with non-compliance since you’ll get the benefit without the effort, and so the metrics they use must be constantly verified. Just getting those metrics is expensive and verifying them ever more so. But True Accountability incents the truth, with huge disadvantages anytime you veer from it. It turns out that an evidence-based truth is simply a lot less expensive to get out there than repeatedly trying to confirm that you’ve complied with something.

10. You talk about the students being the primary stakeholders, but aren’t the state and the legislature stakeholders as well? After all, they fund a lot of what happens in schools.

Of course, but they aren’t the primary stakeholder, and that distinction is huge. Students are the reason schools exist, not the state. If students didn’t exist, we would still have the state, and if the state didn’t exist or fund education that wouldn’t do away with the need. It is true that we have the schools we do because states agreed that education is important and it’s their responsibility to support it, but that importance exists independent of them.

11. What about objectivity? Don’t we need objective metrics for accountability? How else will this be fair?

I understand the question and the reason it gets asked as often as it does, but anytime you place organizational accountability in the technical weeds of an organization—and metrics and measures are always technical—the accountability system will almost always fail in helping the organization achieve what was intended. That flies in the face of everything that’s been done in education for more than twenty years and seems counterintuitive, but it’s still true.

Here’s why. Metrics are always technical, and when they are properly used, they can be useful within the organization. They can also be useful alongside other evidence to determine whether students are benefitting from their education. But metrics regularly lend themselves to two interpretations: the correct, technical interpretation, and an incorrect, non-technical interpretation. The problem with the non-technical interpretation is that to the naked eye it often feels like the one based on common sense and so it’s all too easy for policy makers to use those interpretations in their work.

Watch how easy that mistake is to make. Take graduation rates. The non-technical interpretation is that the school with a higher graduation rate does a better job graduating students. That feels so commonsensical: the school with the higher rate wins.

But the technical interpretation of a graduation rate isn’t simple. Just because one school graduates a higher percentage of students than another does not automatically mean it’s better at graduating students. It may be, but you must look before you know. Otherwise you’re making a pretty big statement without any evidence that its true. If all the students in the school with the higher rate would graduate regardless of where they went to school, that school isn’t adding much value in that regard—it would happen even if the school and those educators in it didn’t exist. That isn’t saying it’s a bad school, just that the metric isn’t very meaningful for them.

But the opposite may be true for the school with the lower graduation rate. It could be that many of the students who graduate from that school wouldn’t graduate if that school and those teachers didn’t exist, but because they do and did a great job adding value in that regard the students benefitted.

The non-technical interpretation of graduation rates in this instance is just plain wrong. The school with the lower graduation rate is better at graduating students, and if your child is unlikely to graduate and really needs some help in that regard, the choice is clear: it’s the school with the lower graduation rate, where they would be more likely to graduate than in a school that wasn’t used to doing that sort of work.

If the goal is to identify the more effective school at graduating kids, the non-technical interpretation fails to do that. And if the goal is to benefit students who are likely to struggle, the non-technical interpretation isn’t going to do that either. The further into the weeds you go with an accountability, and the more the metrics lend themselves to technical verses non-technical interpretations, the worse it gets. That’s why we stay at the level of benefit. It keeps us grounded in a way that doesn’t create that sort of problem.

The irony in this notion of objectivity is that the more objective we try to be in the name of fairness, the less likely we are to get students what they need. Simple is always good, but you must be careful with simple, because it can lead you in the opposite direction you need to go in the name of trying to do the right thing.

12. So how do we compare schools in the True Accountability world? If we can’t do that it's a free-for-all.

The same way you compare all complex things and organizations, whether that’s hospitals, doctors, grocery stores, political candidates (especially in a primary): carefully and thoughtfully. In none of those instances would you reduce them to a single number and compare the number, which is meaningless. Instead, you look at all the benefits you expect from a grocery store or a doctor and base the comparisons on the profile that emerges.

Schools should be no different. We must look at the profile of a school to understand it, not a number. My experience in schools suggests that there are about thirty benefits schools provide students, and at any one point in time they are actively working to improve eight to ten of those. Knowing which benefits are the focus of those changes and their effectiveness at getting to where they said they need to be is a profile that will let me see clearly the effectiveness of that school and those in it. Trying to average those or reduce their work to a number prevents that from happening.

There can be no doubt that some schools will be more or less effective than others, and we should learn from the effective ones and work hard to improve those that are less effective. But the era of crudely ranking schools and imagining you can find meaning in that must end. Ending that practice doesn’t create a free-for-all if you have a true accountability system in place. The fact is that we have now is a free-for-all that can’t identify school effectiveness and we need to treat it that way and move to something better.

13. So how will the public know that their tax dollars are being well spent in the True Accountability world?

That question presumes that they know that now when I hope I just showed that isn’t the case. What they have now are lots of non-technical interpretations of technical information that are misleading at best, even though many of those misunderstandings really are well-intentioned (but certainly not all). The truth is that taxpayers need the information in the question but don’t currently have it. That is a huge and even embarrassing problem given how much we spend on the current system.

Accountability systems based on stakeholder benefit provide this sort of information better than any other kind of system I know of. Imagine the organization of a family for a moment. If not going hungry is a benefit of being a child in a family and we go from 30% of children going to bed hungry down to 15% going to bed hungry, we will consider whatever we did to have had a positive effect. But if the numbers don’t move or went in the opposite direction it will require a rethinking.

If we try a more metrics-based approach, we would have to find a metric, say, the number of meals served each week, and draw a line that represented the difference between being hungry and not being hungry. Eating regularly is a good thing so that metric will appear to make sense. But consider how many families will be misclassified as hungry or not hungry if that metric serves as the formal accountability tool. Some families may meet the number of meals and be above the threshold, but they’re scant meals and the children still go to bed hungry several days a week. That would be a false positive. Or some families may fall below the threshold by serving only two substantial and satisfying meals most days. That would be a false negative.

Those who try and use metrics for accountability get this. As a result, they tend to make lots of adjustments in the name of trying to make the system more accurate which in turn gets really convoluted. Consider a calorie-based accountability for hunger, which would appear even more scientific and objective than the number of meals. Take the calories consumed at each meal, add them up and divide by the number of people in the household. Those who fall below a number per person are presumed to be hungry, while those above a number are not. Simple, right? Only it isn’t. You would then need to account for the fact that children require a different caloric intake than adults. You would need to check if the household had any athletes, who would consume a disproportionate number of calories given their needs and skew the view of that family, or see if anyone was overweight and on a diet, in which case a reduced calorie intake would need to be seen as a positive, not a negative.

And on and on it would go, with the hope of creating of some uber metric or index that could summarize it all. Only when that happens, after all the machinations and manipulations of the date, we would wind up in the dark as to what is really going on with childhood hunger, because we will have strayed so far away from being accountable for the benefit and moved accountability into the technical weeds of a metric.

Our current educational accountability is set up to identify the percentage of kids who pass a test and if that goes up people want to declare the money well spent. It’s logical to wonder why the percentage of children going to bed hungry and the percentage of kids passing a test aren’t the same thing, because they sure feel similar.

Here’s the difference, and its huge: child hunger is at the level of benefit and test scores are in the technical weeds of a school. Just think about what we do with test scores in our metrics-based accountability. We try and account for students who have had fewer opportunities and treat them and their schools differently. We try and account for how many days the student was in that school, and whether they qualified for services such as special education or instruction in their native language. And then someone mushes it altogether into an index and draws a line in the sand and suggests one step above you’re an effective school, and one step below you’re ineffective. That is so far removed from the benefit of learning academics that it’s impossible to know what it means, for any stakeholder or taxpayer.

Here’s the other thing: taxpayers always need to know that their dollars are being well-spent, and especially in a crisis, and yet metrics-based systems can get turned off in a crisis and frequently are. They’re expensive and burdensome and its right to give people a break from them when priorities need to be elsewhere. But you can never turn off a benefit. When do you stop being accountable for making sure children have the nutrition they need, or for seeing that learning never stops? You don’t. A True Accountability system is the best way to make sure taxpayers get what they need because it can speak directly to them at the level of benefit, even during a crisis, which will make it very clear, very quickly, if they should be satisfied or demand a change.

14. What about the timing? Schools have a lot on their plates just now.

That is an understatement. But here’s the issue I want schools and their leaders to consider. Throughout this next year educators will have unprecedented opportunities to build trust with their stakeholders. Since the pandemic hit the moral and intellectual leadership shown by our colleagues is remarkable, and yet just the other day I heard someone express concern that without testing to hold educator’s feet to the fire they’re worried schools will backslide.

School leaders need to speak with the person who made that statement, and everyone else who thinks the same, and share with them in a truthful, evidence-based manner the facts behind their efforts. Some of that will include areas of effectiveness, while some of it will include areas of ineffectiveness, but so long as it's the truth that notion of trust will start to emerge.

School leaders will be dealing with these and many other voices with or without True Accountability. What True Accountability will provide are the frameworks for having those dialogues such that the truth will be known, and trust will be built. Those dialogues are going to happen. That we do not control. That energy will be spent regardless. So why not use that energy to build a larger trust with public education? True Accountability helps organizations of all types do just that when they are in this sort of moment, and now it’s time to do it in schools.

So if not now, when?

Updated: Aug 11

I had a question from someone looking for research regarding the bias in state test scores. I liked the response I wrote and included it below.

What I generally say isn’t that state test scores themselves are biased, but rather they are improperly used as a judgment tool and that results in biased judgments. As a predictive test, a state test is designed to order kids according to narrow underlying constructs (e.g., some aspect of reading and math) relative to their peers as of a particular day. That ordering will be from the students furthest below average to the students furthest above average. It's helpful to realize that the test items on this type of test don’t really result in an accumulated score, like on a quiz or a test you create to check on how much of what was taught was learned. Rather, each item helps the test maker figure out how far from average each student is. 


What the results will show are patterns that exist as of that testing event, and if bias is an underlying cause of those patterns that will become readily apparent. And if we stopped there and did further analyses we would be using the tool as designed. (Of course, we could do that with just a sample of a few thousand students at each grade across a state and have more than enough for such an analysis, but that would be bad for the publishers.)


Where the mistake occurs—and this is in complete violation of what this type of testing can do—is to assign blunt judgments to students and schools, as if the quality of the school is reflected in the test score. Doing that renders a judgment prior to an investigation of what might have caused that pattern. It may be that the school with high test scores is full of students who would score high regardless of the school they attended, and so a judgment of school success based solely on those scores is dead wrong. It may be that a school with low test scores is full of students whose lives are being saved by the teachers. If so, a judgment of failure would be dead wrong. Where causes can be discovered that should be judged, fine, judge them, either good or bad. But until they are discovered any judgments are invalid.


Using a state test score as the basis for invalid judgments creates judgments that are biased because one cause for where a student is on a particular day in reading math concerns socioeconomics. While seeing that pattern is a good thing, you can’t logically leap to an inference that says, “low scores signal failure on the part of the school and high scores signal success,” because you never bothered to look at causes, which are the only thing you can reasonably judge. But the fact that the invalid judgments confirm a bias that exists broadly in society (that is, schools in poor communities are bad and schools in wealthy communities are good), those judgments are presumed, quite wrongly, to be valid.


The worst thing about all of this is that it perpetuates the bias. The one good thing I can say about state testing is it gives us a real chance to see the bias and inequities in our society which should lead us to work towards broader societal solutions. That's what should happen once we see the patterns and then perform our investigations. We perhaps will find things in the school that should be judged, but we would undoubtedly find that judgments regarding our history and the cause of the inequities are valid and useful, and most certainly worthy of a solution.


But the way the results are used allows the whole country to basically avoid the difficulty of solving real problems. It is much easier to judge the schools rather than society, and since those who will be subject to the negative judgments are those with the least amount of power those judgments are unlikely to have negative political consequences. Thus we act pleased with a system that solves nothing, helps no one, and hurts those who can least afford to be hurt further.

 

©2020 by bravEd