John Tanner
May 2025

School leaders face a series of recurring accountability moments where understanding is essential. These include:

1. School board meetings
2. Community events
3. Parent-teacher conferences
4. School planning processes
5. Annual accountability reports
6. Bond elections
7. Conversations with policymakers

In each of these, understanding has been frustratingly elusive. The reasons are clear:

• These engagements often begin with technical data—information meaningful to educators but opaque to those outside the system.
• The lens is typically internal, rooted in the school's point of view rather than that of the community it serves.

They focus mostly on the present or past, even though accountability is fundamentally a future-oriented function.

What’s been harder to see is how to fix this. Most school leaders treat each accountability moment as separate, and understandably so, since each is unique and requires specific technical knowledge. But from the public's perspective, this creates a fragmented, incoherent picture of what schools do.

And incoherence is the enemy of trust.

The solution lies in the concept of “stasis”: which is a moment of stability that grounds complexity in clarity. Other professions have found such a moment—education has not.

Consider the medical field. Doctors operate with extraordinary technical knowledge, but when they communicate with us, they begin with our concerns and our language. Only then do they add the technical details that support the diagnosis or treatment. That’s what makes trust possible.

Education can do the same.

We’ve asked hundreds of parent and student groups a simple question:

What benefits do you expect in return for entrusting your child’s education to a school?

The answers are remarkably consistent, no matter the location or demographics. And notably, standardized test scores never make the list. Not because academics are unimportant—they are. But because tests are technical tools, not benefits. Trust is built around experiences and outcomes that feel real and immediate to stakeholders.

Here’s what typically shows up in the top five expectations from parents:

1. Keep my child safe.
2. Make sure every child has friends and feels connected.
3. Focus on my child's creativity.
4. Engage my child in interesting, meaningful work.
5. Recognize and support the ways my child is smart—not just where they struggle.

Basic academics often land around 20th, not because they don’t care, but because parents see them as one part of a much broader vision of what education should deliver in return for their trust.

They want schools to adapt, innovate, and evolve toward greater benefits over time.

This vision stands in stark contrast to the one held by many policymakers, who define a successful school as one with high or rising test scores. That perspective has narrowed the public narrative around education for years—and yet, it's not aligned with the lived expectations of parents and students.

More importantly, test scores are always a look backward. They say nothing about whether a school is positioned to deliver what matters next.

What schools need is a unifying, forward-looking framework that ties all accountability moments—board meetings, reports, bond campaigns, conferences—back to the benefits stakeholders expect in exchange for their trust.

When every moment of public engagement is grounded in this shared vocabulary, coherence replaces chaos, and trust begins to grow.

Other professions have shown us the way. We would be wise to follow their lead.

John Tanner
Apr 2025

I hear all the time from teachers that the standardized tests their states use don’t seem a fair way to examine student learning, and to be clear, they aren’t, but not because of poor test construction.

Standardized test scores allow a researcher to order students along a scale from the student with the least of a trait (as defined in a state’s content requirements) to the student with the most. That allows for an examination of something, but not on its own learning.

Crudely stated, the goal is to stair step the questions. Students start with the easiest question and then work through increasingly difficult questions until they can’t answer any more. That becomes their place on the scale. Of course, that is not how students experience the stair stepped questions on a test. Test makers mix them up and then afterwards view the results through the stair-stepped lens.

As a result, the percentage or number of items answered correctly doesn’t mean the same as it does in a classroom developed assessment of learning. The questions aren’t there to indicate how much someone learned, but how what each student knows at a moment in time differs from all the other students taking the test. 30/60, or 60% correct, or any other traditional way of understanding a test score doesn’t apply here. In standardized test land, those are just different names for various points along the scale.

Building that sort of test into single forms that all students will take requires asking questions of students that many likely have no shot at answering correctly (unless they guess). Because a test maker is trying to determine what step each student is on, and they want to be as precise as is humanly possible, it wouldn’t make sense to only include the bottom half of the steps or the top half, as those results wouldn’t be very useful. If you only included the bottom half of the steps, you would have a ton of students at the top step and would have no ability to see the differences between them. They would all look similar when that isn’t the case.

The same would happen if you only included the top half. In that case you would have a ton of students who landed at the bottom step who all looked the same when that again isn’t the case.

If that sounds like a cruel exercise by researchers, it has terrific intentions. If you can order students along a scale, you never need do the impossible and attempt to measure how much any student knows at that moment. The fact is it is impossible to measure the amount of grit, literacy, problem solving, or numeracy skill that any student possesses. The best you can observe are the relative differences between students, and standardized testing is a way to make those observations.

Once a set of scores are in hand, a researcher can bring to bear a ton of other information and attempt to work out any underlying patterns that may need to be considered going forward. That is the power in the proper use of a standardized test. It enables a researcher to find patters that need to be changed and argue for changing them. But that may only possible by inconveniencing some students, most notably those who won’t be able to access or understand the more difficult questions at the higher stair steps.

Where teachers feel a great deal of frustration is that these tests are now used in every state to judge what students have learned, when that was never in the design. If a student isn’t ready for complex content, teachers can know that and tailor teaching to student needs and see their effect in a test of that content. It would be pedagogically unsound to judge that student’s learning by giving them content they cannot access and encouraging them to do their best.

But standardized testing isn’t there to be pedagogical. It is there to let researchers see what they otherwise could not. When teachers complain that the tests are unfair to students, what they are mostly saying is the results aren’t a reflection of learning. But then, they never were and never will be.

Note: Adaptive testing relies upon the same stair stepped approach described here but quickly figures out what items would be incredibly difficult or easy for a student and avoids them. This can help students feel less frustrated because they aren’t attempting impossible content, but the result is still a stair-stepped scare with all the limitations in interpretation of a single-form test.

John Tanner
March 2025

Compliance and effectiveness are not the same thing, and we err greatly in thinking that they are. Compliance means that we have prevented a problem or jumped through the necessary hoops to have qualified for inclusion, but preventing a problem or qualifying for inclusion is not an indicator of whether an organization is effective at its purpose. Both occur in organizations, and both are frequently necessary to further its cause.

Nevertheless, one of the great mistakes any leader can make is to let compliance be the model for the mission components of what the organization does. Compliance is reasonable, necessary, and desired when it comes to FERPA, meeting the base requirements for a grant, or qualifying for Federal funding, but it is the wrong model for determining how to use student data, evaluate the effectiveness of a grant, or work towards increasing literacy and numeracy using Federal funds.

What creates a massive disconnect and a great deal of awkwardness is that a great many grants and both Federal and state funds are awarded based on the benefits they can make available for students and then evaluated via compliance. It is not unusual to see an award based on the innovative work planned by a school or district in which the primary evaluation criteria is compliance with an arbitrary rise in standardized test scores or other metrics.

The lack of validity in that is startling: metrics could rise and fall for any number of reasons, some of which may be the effect of the funding and some of which will find their effect elsewhere. That is hardly a reasonable way to know if funds are being properly expended, but it is where we are and likely will be for the foreseeable future.

Sorting this out can be challenging for anyone. Compliance may be used in ways that are highly inappropriate and misleading, and yet it is compliance with bureaucratic requirements for funding and any number of other requirements that keep the doors open. Considerable amounts of effort are required to do that, as are the skills to work in a highly detailed manner. Attention to detail is more than just critical, as errors, missed deadlines, or misunderstandings of requirements can be the difference between having the resources to be effective or not.

Here are a few things that have helped me over the years think through these issues—and just to be clear, I’m not done thinking on any of it:

• Everyone’s function in any organization is to be effective and to become more effective over time. It doesn’t matter what the role is.
• The function of a person with the title Director of Compliance is not to be compliant. Compliance may be the job description, meaning ensuring compliance is what that person should be evaluated on. But their function is to do whatever is necessary to help the organization become more effective.
• Compliance with requirements sets the stage to do just that, no matter how reasonable or unreasonable they may be.
• The function never changes, ever. No one is ever done increasing their effectiveness. But compliance will change. Requirements will come and go. New tweaks to existing compliance efforts happen all the time. New technologies bring with them new requirements.
• Critical to being effective as an organization is doing what is necessary to achieve compliance when required which creates the space to be effective.
• Just as critical is to remember that organizations don’t exist to be compliant. They are compliant where they must be to exist.

John Tanner
February 2025

When Apple, the Cleveland Clinic, an architect, or a plumber account for what they do to those outside their professions, the language they use feels as if it has always been there, not because it has, but because a sense of the anticipated benefits exists in both the organization and the potential customer.

The only questions remaining are the degree to which the organization can deliver on the benefit, and whether the customers will be willing to exchange their trust and/or dollars for it, and with whom.

If schools had the same relationship with students as companies to their customers, we wouldn’t have many of the problems we do.

Benefits emerge in the looping relationship between customers expressing needs and organizations seeking to satisfy them. But also, in the case of new discoveries and innovations, with the organizations seeking to get a leg up and either do a better job delivering on a benefit or attempt to insert a new need/benefit into the market.

Schools do not have that sort of relationship with a student. A four- or five-year-old doesn’t have an ability to view education beyond the simplistic (but not unimportant) lenses of friends and snacks.

Even a high school freshmen will find it a challenge to articulate the benefits they need from a school, having no broader experience than their own.

Thus do parents and other proxies for the student need to step in. And therein lies the problem.

The relationship between client and customer is a symbiotic one that creates a common shared vocabulary based on the benefits one expects, and one provides.

But what of the relationship between parents, policy makers or other proxies and schools? Can that be described as symbiotic? As one from which a set of benefits and a common shared vocabulary has a reasonably good chance to emerge naturally?

It cannot.

The reasons for that are beside the point here, but the evidence is overwhelming: political accountabilities based on test scores, vouchers, anti-public education sentiments, politicized school board meetings, etc. For any number of reasons, the opportunity for the benefits to emerge naturally isn’t there.

But that doesn’t mean they aren’t there. It means unlike Apple or the Cleveland Clinic, we’re going to have to ask, and that feels a bit odd. It would be as if the Cleveland Clinic gathered a group of patients and asked, “what benefits to you expect or anticipate from us in exchange for your trust, dollars, or both?” Everyone would probably feel a bit weird.

Later, when the leaders of the Cleveland Clinic thought about accounting for those benefits, having not done so before, odds are they would think the point is to account for boiling oceans and moving mountains.

Having long recognized the benefits, that isn’t what Cleveland Clinic or any of them do. Rather, they make decisions inside the organization about the pieces of work to be done, and when it is time to account for that work they do so through the lens of the most appropriate benefit.

And they do so simply: “when it comes to X we needed to make a change. We chose to do this piece of work and agreed what a future moment of effectiveness would look like. We’re at that moment, and based on a body of evidence, our efforts regarding that piece of work were effective/not effective.”

The idea behind a benefits-based accountability isn’t to account for the benefit, but through it. While not completely equivalent, it is helpful to think of a benefit in the vein of supplies on an income statement. No one would treat supplies a category through which to find work.

Rather, they would do work and use supplies to help others make sense of what they did.

Supplies and Keeping Kids Safe have no moral equivalence, and a school can’t help but let that benefit affect their efforts, but regardless, it is the specific work that needs to be accounted for through the lens of benefit.

That’s how we avoid boiling oceans.

John Tanner
May 2024

It turns out there are around 30 things that matter to parents and students when it comes to a public-school education. None of them are a surprise to anyone: keep students safe, make sure each child has friends and feels connected to others, deal with uncertainty, focus on the creative, etc.

Coming in at around the 20th position are basic academics, not because basic academics are unimportant, but because at least 20 other things are considered just as important.

The business world agrees. Their studies put dealing with uncertainty in the 1st spot, and basic academics at around the 30th (of around 40 things) again, not because basic academics are unimportant, but because so many other things are just as important.

That leaves school accountability in its current form in a precarious spot because left to its own devices it will be incapable of engendering trust in a school. If trust is to be had then the school must account for all the things that matter to the people for whom they matter. Failure to do that is a recipe for distrust.

The current school accountability environment is not by design an accountability environment capable of creating trust in schools. It only accounts for (at best) a few of the things that matter.

The things that matter we call benefits. Benefits represent the reasons stakeholders entrust the education of their children to a school. Benefits are what they expect and anticipate in return for that trust. For trust to exist educational leaders must account for their effectiveness regarding what matters, or in other words, those benefits.

It is critical educators learn to think differently than they have in the past about how to account for their efforts. When a benefit is discussed the concern most educators have is how to measure it. This is a colossal mistake. To paraphrase Einstein, “most of what matters can't be measured, and a great deal that can be measured doesn't matter.” Nowhere is that truer than when it comes to benefitting our young people.

Benefits-Based Accountings are new to most educators. Learning to account for things using evidence and interpretations requires some getting used to. But it is also something that every person that has engaged with another profession has experienced, so it is easily learned.

You can imagine a basic form of Benefits-Based Accountability if you envision a list of ~30 benefits down the left column of a sheet of paper and information to the right about a school's efforts regarding each. You could signal whether the school is actively engaged in a programmatic change relative to that benefit, whether it is stable in terms of its efforts, or whether it is considering a change once the bandwidth is available.

As simple as that sounds it would create understanding of the school's priorities today and of the school's priorities going forward in language that makes sense to all stakeholders. Understanding is the true definition of transparency, not the data dumps that are commonplace and contribute little if anything to accurate understandings.

What you can say to others
1. Benefits are what stakeholders expect.
2. Evidence is critical.
3. Trust is everything.

Summary
Benefits-based approaches allow school leaders to account for what matters, which is the only way to develop trust in what those schools and their leaders do.

John Tanner
June 2024

In education, the terms testing and accountability have become inseparable. This is a mistake. It reduces and corrupts the accountability function in schools in ways that are harmful, and then asks a research metric that was never designed to evaluate school effectiveness to judge school effectiveness.

Education is so trapped in its accountability notions that nearly every conversation starts and ends with the idea that accountability must be driven by a metric of some sort. Even recent discussions about being accountable for the skills students need to succeed are being steered towards the idea that they can only be included in an accountability system if they can be tested.

This reflects a gross misunderstanding of the accountability function.

Accountability demands that organizations account for all the things that matter to through the lens of their stakeholders, a list that does not include high or rising test scores. School leaders and advocates would be wise to first create an understanding of what the accountability function in great organizations is as a precursor to doing it right.

It is important to note that the real accountability function in schools is separate from policy, meaning in must be done regardless of the policies that exist. It will be easier to do if the policy agrees with the need, but a policy change should never be a precursor to accounting for what matters.

What you can say to others
1. A proper accounting is essential: Accounting for what an organization does is as important as what it does because if it doesn’t properly account for its actions, it risks appearing ineffective to outsiders.
2. Accountability should be comprehensive and about the things that matter: Accounting for everything that matters to stakeholders is paramount. This is the only mechanism by which stakeholders can come to trust the organization. What matters are the benefits parents and communities expect.
3. Dangers of Limited Accountability: Accounting for only one or two important benefits, even if done well, won’t build trust. Instead, it risks creating the perception that the organization has something to hide. That sort of accounting almost always backfires in a negative way
4. Stakeholder Expectations: Parents, communities, and business owners expect certain benefits from schools, such as ensuring children's safety, fostering friendships, teaching students to deal with uncertainty, teaching fairness, and focusing on creativity. Basic academics rank lower on the list of important aspects, not because they are unimportant, but because other factors are equally important.
5. Trust through Comprehensive Accountability: Schools stand a chance to be trusted if they account for their effectiveness in providing a broad range of benefits. If they only account for one or two aspects, trust will not happen.
6. Evidence-Based Effectiveness: Most benefits parents expect cannot be measured via a test score, but they can be observed and presented as evidence. Effectiveness regarding a benefit can be supported through observable evidence, no testing required.

Summary
By expanding the scope of accountability beyond mere test scores and incorporating a comprehensive approach that includes various benefits valued by stakeholders, schools can build trust and demonstrate their true effectiveness. This shift requires a fundamental change in how accountability is perceived and implemented.

John Tanner
July 2024

Accountability is one of the disciplines of leadership. It is the discipline through which leaders build trust in their organizations. That is different than building trust inside their organizations.
Building trust in an organization requires efforts focused on the organization's stakeholders to ensure they have sufficient information to be able to offer that trust.

The efforts of the organization must be based on the things that matter to those stakeholders.
Consider what matters to you as a stakeholder of complex organizations with which you engage and interact. You would have no capacity to trust a hospital that only accounted for its billing efficiency. Billing efficiency may well be a benefit that matters to you but there will be other benefits that matter at least as much. The prerequisite for you to trust the hospital is that it accounts for all the things that matter. Absent an accounting of what matters, your trust won't be possible.

What matters to a school's stakeholders? Providing children safe, conducive learning spaces. Teaching them to deal with uncertainty. Ensuring creativity is a part of the curriculum. Ensuring that each child has have friends and can connect with the adults in the building. Ensuring children have a voice. Parents all over the country have answered the question about what matters over and over, and their answers differ very little.

Most parent groups land on a list of about 30 things with basic academics around number 20 or 21. That just means that while basic academics are important, there are at least twenty things that are just as important.

If the public is to trust public schools the logic is simple to see—public schools must account for all the things that matter. Absent doing that, the most likely outcome is distrust.

Accounting for what matters supersedes any educational policy. Leaders can and must account for what matters regardless of policy if trust is to be had. And if public education is to survive, trust will be the key to seeing that happen.

But not only is a policy change unnecessary to account for what matters, neither is permission.

Accounting for what matters is a discipline, something any leader can practice, and as a discipline of leadership, practicing it makes you a better leader and better able to contribute to your school’s mission.

You don’t even need to tell anyone what you are doing. Just do it, and watch the results speak for themselves.

Finally, policy issues do exist, and we must address them, but we need not, and indeed, cannot wait for a better policy. The young people in this country and the public education system which is a foundation of our democracy need us to account for what matters now.

What you can say to others
1. Accounting for what matters is the key to stakeholder trust: That is how we come to trust complex organizations, and it is the way we can get others to trust our complex organizations.
2. What matters to stakeholders matters a lot. When engaging a stakeholder, ask them. What would build trust in the public schools? What benefits matter to you? What are your hopes and dreams for your child? The answers are what matters.
3. We aren’t waiting for a policy change: Accounting for what matters needs to happen irrespective of policy. We’d like the policy to match, but that isn’t a prerequisite to doing it right.

Summary

Accounting for what matters must happen irrespective of policy, and it doesn’t require permission, just as working hard doesn’t require permission.

John Tanner
August 2024

If vouchers, charters, or education savings accounts (VCES, for short) solved a problem, I would have to be for them. I’m a realist in that way.

But if resorting to VCES happened because someone didn’t want to do the hard work of solving a problem, and the problem persists, that would be clear evidence that VCES didn’t solve anything.

Rather, it had to do with a preference for a few rather than answers for the many.

That’s all VCES has ever been, because the hard work to enable and empower every school and its leaders to satisfy their social contract with their communities and those they serve was never attempted. Policy makers didn’t follow the leads of other countries experiencing a renaissance of learning to imagine what might be possible. They didn’t examine the accountability function in other professions that enabled each contributing organization to be unique in its excellence and still contribute to an overall understanding of that profession.

They, as a matter of course, have continuously underfunded public education, which research shows has direct ties to GDP, maintaining American excellence, and preserving our place in an ever-changing world, which is both short-sighted and embarrassing.

Policy makers rarely, in fact, have asked educational professionals what might work, and instead continuously violate what we know about how to promote excellence in organizations by insisting on a one-sized-fits-all approach that fits almost no one.

Once that one-size-fits-all approach that was doomed in its ability to meet their policy objectives failed to meet them, what did they do? Did they take the rational approach and say, whoa, maybe we made a mistake? No.

For three decades now the response has been to double down on what was never going to work from the start.

And somewhere in there those advocating for VCES saw their chance. And took it.

VCES voices are very loud. And now that they have the upper hand in so many places they don’t want to give that up. Vouchers primarily help those already with children in private schools pay for them. Chartering is a hugely profitable enterprise. Educational savings accounts are right in the thick of all that. The best way to keep all that going is with a continued message that public schooling and failure are synonymous, and VCES can save you.

Promoting educational excellence in all our schools doesn’t seem to have been the point for as long as anyone can remember, because it’s so rarely been attempted. Let’s be clear. Educating a populace as diverse and with as many needs as American students is a hard thing to do. It presents challenges that will appear insurmountable. It requires patience, planning, innovation, and resources, the likes of which policy has never encouraged.

And now that VCES is here, solving real educational issues and educating the entire populace in the public schools to high levels is something their self-interest encourages them to lobby against.

Every “failure” in a public school stands to enrich a charter operator. Every new voucher enriches a private school parent and enables the private school to raise tuition.

To argue for a vibrant and healthy public education system that can help with society’s broader goals runs counter to VCES interests.

It is hardly the mark of American democracy to have created an alternative approach to education whose proponents now have reason to cheer for the demise of a democratic institution, but here we are.

Fighting for a vibrant public education system is the most democratic thing we can do. Arguing that VCES is a “solution” for what ails public education is not. As Americans, we can and must do better.

Summary
To say that VCES solves anything is disingenuous. It may make life easier for a few, but it does so at the expense of the many, which in turn puts the future of our democracy at risk.

John Tanner
August 2024

When educators hear the word accountability what immediately comes to mind is a huge, negatively motivated, amorphous system that reaches into and colors a majority of what happens in a public school. This can make suggestions that it can be changed difficult to hear, because imagining something different feels like imagining the impossible.

It becomes much easier with the recognition that standardized testing is not the primary issue. That isn't to say it is a non-issue.

Rather, even if we had the greatest test in the history of the world that met every promise of perfectly reflecting teaching and learning, all the issues with the current system would remain.

Public education remains the only profession in which compliance with a set of requirements is also perceived as effectiveness.

Clean air standards represent requirements for keeping the air safe to breath. Complying with those requirements is a good thing for manufacturers to do. However, no leader would ever stand before an audience and declare that because their factory was compliant with clean air standards, it must follow that its products are excellent. A factory may be compliant with clean air standards and manufacture terrible products. Or it may be out of compliance and produce great products.

Effectiveness and compliance are two different things. Pretending they are one and the same would be nonsensical.

What policy makers have public schools do is create indices from test scores and sometimes other data, order schools along those indices, and draw compliance lines in the sand, which they then pretend can also signal effectiveness to one side and failure on the other (terms vary).

This substitution of compliance for effectiveness has been difficult to perceive given the complexity in creating those indices, but the complexity of the metric matters not one whit. Effectiveness must always be determined elsewhere.

Consider where that leaves our current school accountability system. We spend billions of dollars a year on tests and related materials that are then placed into a compliance format, which then pretends it can identify school effectiveness, when that would be impossible. This would be laughable if it weren't so illogical and if it didn't have such negative consequences.

The complexity that has been so difficult for educators to see is this compliance component. As a result, the intricacies in standardized testing have wrongly received the lion's share of attention, distracting from the real issue.

Simply put, when educational accountability was invented, a compliance-based approach was selected, which doomed at the outset any chance for the resulting system, no matter what metrics might be included, to ever indicate effectiveness.

What you can say to others
1. No other profession substitutes compliance for effectiveness: The reason for this is simple: it would leave them without an understanding as to where they are or are not effective.
2. We need to know where schools and are not effective. That is the only way we can improve them. We don’t get that information from our current accountability system.
3. Standardized testing is an issue, but there is a far bigger one: It is compliance. And the fact that we substitute compliance for effectiveness means we spend a few billion dollars a year on a system that cannot tell us if schools are or are not effective.

Summary
Compliance has never, will never, and can never also signal effectiveness, no matter how hard we may want it to.

John Tanner
August 2024

When I tell audiences that standardized tests do not, cannot, and have never measured any amount of anything, very few people believe me. At first.

Here is how I win them over. Imagine that you wanted to analyze a human trait that cannot be measured, like comedic talent. No breathalyzer exists for comedic talent. However, it can still be analyzed for the simple reason that we can observe differences in comedic talent between people.

Imagine a line of people that stretches from the least funny person to the funniest person. We call that a scale. The magic is in figuring out where to place each person, which seems rather challenging, but it is surprisingly simple to imagine. Pretend you have a blank scale with the label not funny on the left and hilarious at the right. Take the first two people that come along and determine who is funnier. Now, drop them both somewhere along the scale. Just be sure that the funniest person is to the right of the less funny person. You’re on your way.

Grab a third person. Determine where that person is relative to the first two and place them accordingly. Bring in a fourth. And a hundredth. And a thousandth. If you need to adjust, squeezing here and widening there, do it.

Eventually, a new person will be exactly as funny as someone you’ve already placed, so multiple people will begin to appear at a spot. There will be a lot more folks occupying spots towards the middle and far fewer at the funniest and least funny spots (the same would be true of a scale based on height, weight, altruistic tendencies, etc.)

What can you now do having scaled the human trait of funniness, even though funniness can’t be measured? Look for patterns by bringing in other data.

Do you see a gender or a racial difference? Do funny people live longer? Are less funny people at increased risk for heart disease?

If you find patterns that you don’t like, do something about it. And then in a year repeat the exercise to see if the negative patterns are dissipating.

It is remarkable how much analysis can be done regarding a human trait that cannot be measured.

Guess what else cannot be measured? The amount of numeracy or literacy possessed by anyone. But what can be observed is who has more or less of it than another, which means it can be scaled and analyzed. But how?

The answer: standardized testing. Imagine 10 questions, lined up from the easiest to the hardest. These will have undergone extensive field testing so that a student who answers three correctly is likely to answer the 3 easiest correctly and miss the 7 most difficult. So too with a score of 1, 5, 7, etc. That way all the students who score a 3 can be said to be similar. Also, they can all be said to know a little more than the students to the left and a little less than the students to the right.

Do that and the test can make decent (but not perfect) observations of the differences between students. Just remember that the questions are there to show how students differ from each other, not measure how much anyone knows.

Once you have a set of scores bring in researchers and lots of other data and use it as one source to help analyze and conquer negative patterns.

Just don’t use it to make judgments on its own, or automatically presume high scores mean great schools and low scores mean bad schools, and certainly don’t teach to the questions. All those things require an understanding of how much and where learning occurred, and standardized test scores have no clue about any of that.

Which begs a question: why is it that educational policy that purports to be about what and where students learn relies on an instrument that cannot, by design, tell you anything about how much anyone knows or learned, let alone where?

Summary
Standardized testing doesn’t measure anything. Don’t pretend it does. You’ll always be wrong.

John Tanner
March 2024

It turns out, is that education deals with its technical data in a way no other profession does.

No doctor would hand a patient the results of their lab work and expect the patient to make sense of it on their own. Lab work data are incredibly complex and require a trained interpreter, someone who can perform an assessment of all the data and come to a reasonable interpretation.

That process is everywhere. Engineers process information relative to physics and structural dynamics so they can declare a set of blueprints capable of producing a building that will stand. CEOs stand in front of their boards and make pronouncements about the health or challenges faced by their organizations, and then share as needed the data they used to arrive at their conclusion. The Federal Reserve regularly offers a brief narrative anyone can understand on the state of the economy and then takes those interested on a deep dive into the indicators that led to their interpretation.

If education handled its technical data like its professional peers, evidence would be constantly gathered and regularly analyzed by experts trained in such things, which would enable them to offer interpretations about what would need to come next. Included in the evidence would have to be considerations of the local context, specific student needs, and a much broader understanding of the school’s overall efforts. The result would be a school with the capacity to constantly improve its ability to deliver on its commitments.

That is not how educational accountability works. Instead, complex data produced from extremely complex tests are presented to the world with a line drawn at a score point. One side of that line is presumed to signal failure, while the other side is presumed to signal success. No one checks whether what is labelled success represents real success or what is labelled as failure represents actual failure. No body of evidence is created in the process that is a prerequisite to valid interpretations of complex phenomena. No professional interpretation, which is a hallmark of accountability moments in every other profession, is permitted.

If something similar were done in other professions patients would rarely get healthy, buildings would regularly fall, the stock market would crater, and the economy would be in shambles.

Getting beyond this mess requires educators learn to offer their interpretations first, based on a body of evidence.

It also requires a willingness to recognize the severe limitations in the research instrument we know as standardized testing. To say that we use it off-label (to borrow a phrase from our medical peers) in so understated as to be laughable.

In truth, we rarely use it on-label, and our off-label approaches are so extreme they are akin to believing the size of this summer’s leeches is an indicator of GDP, or pretending gravity is a choice.

Professional accountabilities offer regular interpretations based on bodies of evidence used on-label.

Every educator owes it to themselves, their students, their schools, and their communities to be able to use technical data on-label, following in the footsteps of our professional peers. No policy change is needed to do that.

What you can say to others
1. Professional interpretations are the hallmark of great accountability environments: Without them, effectiveness won’t be possible.
2. Testing is used off-label. That is a big issue.
3. If other professions ran accountability the way we do in schools, those professions would be in real trouble.

Summary
If educators could be more like their professional peers when it comes to accountability, that would be powerful, and they don’t need a policy change to do it.

John Tanner
Dec 2024

It is understandable that school leaders weary of state testing programs would want to transition to something better. That something better is almost always a local testing or accountability program.

This transition needs to be done thoughtfully. The flaws in state testing programs can be hard to see and in a great many cases the shift to local assessments and accountability brings with it many of those flaws. When that happens, the effort and energy required to make the shift will be wasted. The biggest flaws are these:

1. Psychometrically based tests (e.g., norm-referenced, commercially available tests such as MAP or the Iowa Test of Basic Skills, all state testing programs, any test that results in a predictable score) do not measure teaching or learning. They instead enable a researcher to analyze the differences between students regarding the trait. This reveals patterns behind the test scores that likely exist in real life, that can then lead to increased understanding as to the actions that can disrupt negative patterns.
2. State accountability programs are based in compliance. Lines are drawn, with effective and ineffective labels placed on either side. This is so fallacious that no other profession would do it nor accept it being done. The causes underlying any metric are complex. An 85% graduation rate may be a signal of effectiveness for a school serving an at-risk population. A 99% graduation rate may be a sign that a principal unlocked the doors in an affluent neighborhood. A line risks hiding the truth and sending a false message: that the 99% school faculty worked very hard to graduate students, and the 85% school faculty failed.
3. Assessments of learning require that learning be assessed. If Moby Dick was the text, the questions should concern the meaning of the white whale or the narrative approach. Multiple choice tests aren’t particularly helpful here. Generic, or commercially built tests even less so. Assessments of learning rarely produce predictable results. If all students learned well, they should all do well. If all students learned poorly, they should do poorly.
4. Psychometrically based tests can aid in analyzing teaching and learning, but only in the hands of a thoughtful researcher, and only alongside other forms of evidence. On their own, no interpretations regarding what was taught and what was learned are possible.

Thinking that local assessment is better when it comes to teaching and learning is only true when the assessment is about teaching and learning. All too often the local assessment argument simply replaces a state-driven psychometrically based test with a locally determined psychometrically based test. That accomplishes not one bit of the goal and will prove as useless as the state system.

Thinking that a local line drawn in the sand to be preferable to one drawn at the state level ignores the fact that both lines are about compliance and neither of them can identify effectiveness. The line may be controlled locally, but it will be no more useful than the one drawn by the state.

Any shift away from state systems in their current form must dismiss the idea that testing and accountability are commensurate efforts, that accountability is testing. It is so much more, as so many of the papers in this series show.

If policies are desired that can encourage this shift, they must treat accountability as the means to account for what matters and encourage the right kinds of assessments. Otherwise, the massive political capital required to accomplish the shift will be wasted, and in the end, the naysayers of public education will have one more argument against it.