Edudata on Eric Ekholm

AI tools are (mostly) great for retrieval practice

Tue, 11 Aug 2026 12:00:02 +0000

This is the third post in a brief series that contextualizes AI in education within prominent theories and phenomena of educational psychology. The previous posts discussed Flow Theory and Expectancy-Value Theory. Today’s focus is on the testing effect.

The testing effect – also known as retrieval practice – is a well-established phenomenon in educational and cognitive psychology demonstrating that people learn content better if they are “tested” on it frequently. “Testing” here doesn’t mean high-stakes, final-exam-type testing that school conditions us to associate with the term. Rather, it refers to any task that requires someone to retrieve information from long-term memory, including answering practice problems, taking brief quizzes, or teaching others. One of the most-common versions of this, particularly for those of us who grew up before computers and the internet were ubiquitous in schools, is studying with flash cards. I can still vividly remember making periodic table flash cards for chemistry in Mr. Stietzel’s class…

Scaffold or Crutch? Framing AI through Expectancy-Value Theory

Wed, 08 Jul 2026 12:00:40 +0000

This is the second post in a brief series that contextualizes AI in education within prominent theories and phenomena of educational psychology. The first post discussed AI and flow theory. Today we’ll focus on Expectancy-Value Theory.

Expectancy-Value Theory (EVT) describes a learner’s motivation on a given academic task as a function of their expectations of success on the task as well as of their perceived value of the task. The basic premise is that these components combine multiplicatively to yield motivation, i.e.

AI probably inhibits flow states

Fri, 12 Jun 2026 12:00:46 +0000

Earlier this year, I wrote a series of brief white papers at work that attempt to contextualize AI in education through several foundational, well-studied phenomena of educational psychology. The goal of these papers was to provide our school division’s leadership with hype-free, unsensationalized lenses through which to consider both the affordances and drawbacks of AI in the classroom.

Well, I’m presenting this work at the upcoming MERC Summit on June 25th (if you live in the Richmond, VA area, you should definitely consider going – it’s one of my favorite conferences), and so while I’ve been prepping for that presentation, I figured why not turn those briefs into blog posts.

Embracing a Unix-like philosophy toward data

Fri, 13 Mar 2026 12:00:02 +0000

After quite a long hiatus, we’re back! My goal is to write more regularly, but, you know, we’ll see.

When I was younger, I used to regularly watch Alton Brown’s Good Eats with my family. And, as any devotee of Good Eats remembers, Alton absolutely despised single-purpose kitchen gadgets – quesadilla makers, avocado slicers, hot dog toasters. All sorts of over-specialized slicers and pans and cookers and other devices. According to Alton, the true heroes of the kitchen were the multi-taskers – cast iron pans, sharp knives, thick bottomed saucepans, wooden spoons, forks. With a very few exceptions, kitchen tools that couldn’t be used for several tasks were wastes of both space and money.

Being Data-Informed vs Data-Driven

Thu, 30 Oct 2025 12:00:59 +0000

Within the past week or so, I’ve encountered two people who have purposefully distinguished between being “data driven” and “data informed.” First, Rick Saporta differentiates these two approaches in his recent presentation on business data strategy. And second, Tiankai Feng situates both “data informed” and “data driven” decision making on a continuum in his book Humanizing Data Strategy (p. 16-17).

Although they make slightly different points, the heart of both Saporta and Feng’s positions is that “data-informed” decision making entails explicitly combining personal expertise with insights gained from data, whereas “data driven” decision making entails making a decision entirely based on data. In Feng’s argument, these decisions are entirely data-driven because they are made autonomously by some model or algorithm. In Saporta’s, these “data-driven” decisions are still made by people, but they’re made by people who either fail to appropriately weigh their expertise alongside the available data or who don’t have sufficient expertise to factor into the decision at hand.

Bar Graphs Should Start at 0

Fri, 19 Sep 2025 12:00:46 +0000

I was reading an evaluation of a digital learning platform (Imagine Math) the other day, and I came across this graph:

The point of this post isn’t to talk about Imagine Math or whether it’s a good product or how teachers should incorporate it into their math instruction. That the evaluation I was reading was about Imagine Math is completely incidental. Rather, the point of this post is to discuss the sacrilege of creating a bar graph, like the one above, with a y-axis that doesn’t start at 0.

Virginia’s VALLSS 2024-25 Fall-Spring Growth Report

Fri, 22 Aug 2025 12:00:16 +0000

Programming note: I’m back after a long summer hiatus. I honestly just kind of lost motivation to do weekly posts. So, moving forward, I’m not going to promise weekly posts. I’ll shoot for a 1 or 2 times a month, or whenever something strikes me that I want to write about.

I just read the Virginia Language and Literacy Screening System (VALLSS) 2024-25 fall-to-spring growth report, which was sent out by the Virginia Department of Education (VDOE) about a week or so ago.

Are there regional issues in education?

Fri, 04 Jul 2025 12:00:19 +0000

Last week, I gave a talk titled “Data Management Tips for Educators” at the 2025 MERC Summit. For those who don’t live in the Richmond metropolitan area – or for those who do but don’t know about MERC – MERC is the Metropolitan Educational Research Consortium. It’s a partnership between VCU’s School of Education and several surrounding school districts, including Chesterfield County Public Schools (CCPS), where I work.

I won’t restate MERC’s mission, principles, and goals, but one of MERC’s primary functions is to conduct research that benefits its members. And recall that MERC’s members are VCU and the surrounding school districts. This got me thinking about to what extent there are distinct regional issues in education, where “regional” refers to a cluster of school districts/counties within a given state. An relevant example here would be the Richmond metropolitan region.

Data management tips for educators

Fri, 27 Jun 2025 12:00:31 +0000

I gave a presentation at yesterday’s 2025 Metropolitan Educational Research Consortium (MERC) Summit titled “Data Management Tips for Educators.” The slides are available here if you’re interested.

The gist of my talk is that we, as educators, often have lots of data available to us, whether that’s through reports from digital learning platforms, student information system extracts, state reporting, or something else. And we’re being increasingly asked to “use data to make decisions.” But the vast majority of people in schools and school division central offices who are tasked with managing data – aside from the small handful of folks who work on the Database Services or Research teams – don’t have any formal training in data management. My experience is that data management tasks are thrust upon whichever poor soul has some sliver of bandwidth, is the subject matter expert, naively volunteers to help, or is “good with spreadsheets,” and this seems to be especially true in schools. Regardless, the people who end up responsible for “managing their schools’ data” are almost invariably learning as they go.

The narrative fallacy and education

Fri, 20 Jun 2025 12:00:35 +0000

I recently read this article on the Ludicity blog about the narrative fallacy, and it got me thinking about how this applies to education.

Just to make sure we’re all on the same page, the narrative fallacy is a term coined by Nassim Nicholas Taleb in his book, The Black Swan, that refers to people’s proclivity to create (and prefer) straightforward causal stories to explain events, even when reality is complex and random. Taleb’s notion of the narrative fallacy draws from Kahneman and Tversky’s stuff on heuristics and biases (also check out Kahneman’s book, Thinking Fast and Slow), which describes the many reasons people make poor decisions.

Reflecting on the season of standardized testing

Fri, 13 Jun 2025 12:00:54 +0000

And we’re back from a 3-week hiatus that was the result of a guys’ trip, being busy, and being somewhat demotivated. But now we’re back!

Most of my job in late spring and early summer of every year involves analyzing data from standardized tests, and the analyses I’m doing range from back-of-the-napkin estimates of pass rates to careful, rigorous statistical modeling looking for effects of some instructional decision or program. Plus also updating dashboards – there’s lots of that.

Stacked bar charts and data viz tradeoffs

Fri, 16 May 2025 12:00:09 +0000

Stacked bar charts aren’t good plots, and this is a hill I will die on.

Even though everyone over the age of, say, 7, has seen a stacked bar chart at some point in their lives and probably intuits what they are and how to describe them, I’ll start here with an example and some explicit definitions just to make sure we’re all on the same page.

A stacked bar chart is a plot/graph/chart/what-have-you that lets us visualize some quantitative variable by 2 mutually-exclusive categorical variables. A case I encounter often enough in my job is counting student enrollment (a quantitative variable) by school and by race/ethnicity (our 2 categorical variables). Usually, we think of one of these categorical variables as our group (or primary grouping) variable and the other as a subgroup (or secondary grouping) variable. In this example, we might think of school as our group variable and race as our subgroup variable if we’re interested in counting students by race within each school. We could do the inverse, too, but that feels less obvious to me, at least in this case.

AI grifters and some self-reflection

Fri, 09 May 2025 12:00:05 +0000

This week’s post is going to be pretty short.

Over the past week, I’ve had interactions with two different groups of “AI-in -education experts,” and they each left me feeling some combination of despair, disgust, incredulity, and anger. And so I wrote what was essentially a rant dunking on AI grifters whose core competencies sum to little more than cramming as much LinkedIn thought-leader technospeak into a vision statement as is linguistically possible.

Some hopefully useful questions to guide data-driven decisions

Fri, 02 May 2025 12:00:50 +0000

I’ve mentioned in multiple posts on this blog that the primary “point” of data is to help inform decisions. There are probably occasions where it’s useful to just report out on broad compliance-y things, or where we want to explore data with no clear decision in mind in the hopes that some revelation will jump out and whack us over the head. But mostly, if we’re devoting any considerable amount of time to analyzing data, we ought to have some potential decision in mind.

Luxury goods and non-negotiable values

Fri, 25 Apr 2025 10:33:53 +0000

I’ve been very much into the Acquired podcast lately. The premise of the podcast is that each episode describes a single company, usually beginning at some pivotal pre-company occurrence (the birth of the founder, the invention of some critical piece of technology that enables the company, etc) and proceeding to chart the trajectory of the company through to today. The hosts, David and Ben, are these kinda dweeby but ultimately endearing dudes with backgrounds in venture capital but who now just do their podcast full-time. Given this, it’s probably not surprising that the episodes tend to have a finance bent to them. Even so, as someone who has never worked in finance or even in a finance-adjacent industry, I still find the episodes appealing companions during dog walks, commutes to work, laundry folding, and other tasks that allow for auditory entertainment.

I Feel Like I’m Taking Crazy Pills

Fri, 18 Apr 2025 10:42:26 +0000

I recently stumbled onto a preprint research article examining the writing quality of college students before and after the introduction of ChatGPT, as well as some reporting on the same article published on the Hechinger Report. And oh boy did reading it make me feel like Will Ferrell in Zoolander.

The authors of the study analyzed 1.1 million discussion forum posts (i.e. mostly brief, informal writing assignments) from ~16,000 college students submitted between Fall 2021 and Winter 2024, and they estimated the writing quality of each post. The crux of the paper was to compare student writing quality before and after the emergence of ChatGPT (and other LLMs). I’m sure you can guess what the findings are, but I’ll describe them anyway.

RAG time

Fri, 11 Apr 2025 12:00:31 +0000

I’m both an AI optimist and an AI skeptic. I use AI tools in my day-to-day work, mostly to help with coding, but with some other tasks as well. Google Gemini is one of the tabs I open, along with my email and calendar, as soon as I boot up my laptop in the morning, and I keep it open throughout the workday.

And yet. I’m not fully on board the AI hype train, especially for education. Like I wrote about a few weeks ago in my reflection on Grammarly and AI tools that support writing, it feels both shortsighted and naive to try to use AI to solve every problem. Sure, AI tools can help you produce more, but they’re not going to solve issues that are, at their root, cultural, organizational, or interpersonal problems.

Reaction to Emily Oster's "why we need test scores"

Fri, 28 Mar 2025 12:00:56 +0000

Programming Note: I’m doing an intensive AI training next week, so there will be no post on 4/4. I’ll be back again the following week.

Emily Oster recently published an article (post? blog?) on her website, ParentData, titled “Why We Still Need Test Score Data,” and I wanted to both react to that here.

Before that, though, as a brief aside, Emily Oster’s writing is great. If you’re a parent and you don’t subscribe to ParentData, stop reading right now and go subscribe. It’s free (well, there’s a paid plan, too, but it’s optional), and Emily does a great job balancing empirically-based research with the practical day-to-day concerns of parenting. Her book, Cribsheet, is also worth a read if you have young kiddos.

Writing for busy readers, and why not to use AI

Fri, 21 Mar 2025 12:00:18 +0000

I recently read The Productivity Shift (TPS) – a report from Grammarly with the basic premise that businesses should use AI to help employees communicate better, which will increase productivity. You can skim the report if you want, but it’s mostly a lengthy marketing pitch for Grammarly – a company that sells AI-powered software that edits your writing. Maybe Grammarly sells other stuff, too, but I’ve never used any Grammarly product so I’m not sure what their, like, breadth is.

When do predictions about students’ future performance help?

Fri, 14 Mar 2025 12:00:24 +0000

One common use of statistical models is to make predictions about the future. These predictions are ubiquitous in our tech-infused lives today. Meteorologists deliver predictions about the weather to you via the apps on your phone. Your credit card company might flag a transaction as suspicious using a prediction from a model. Your bank might approve or deny a loan based on the predicted probability you’ll default. And if you’ve ever made a household budget in Excel, you’re just making predictions about future expenses.

Does your data’s structure match the question you want to answer?

Fri, 07 Mar 2025 13:00:24 +0000

I have this thesis that people in education often get frustrated with data – or they feel like data is useless – because the structure of the data they’re collecting doesn’t allow them to answer the question they want to answer. This usually happens because someone – a principal, a social worker, a central office administrator, a teacher, whoever – is trying to take an existing report or data product and apply it to their question. The problem is that these reports and data products weren’t designed to answer that specific question. Often, they weren’t designed to answer any question. Many data exports, especially those from learning apps, are just broad, shallow, generic slop that tell us little more than how much time a student spent on an app. Or the reports are compliance-y things that someone is required to submit to fulfill some program guidelines or state mandates.

Reviewing VDOE’s school performance framework (pt 3)

Fri, 28 Feb 2025 13:00:13 +0000

In the last two posts in this mini-series reviewing Virginia’s new school performance framework, I shared my thoughts on its mastery and growth components. I’d encourage you to check out those posts if you haven’t, but my quick summary of them is that I think the mastery component is pretty good (not perfect!) and the growth component is trash.

The final component (well, sort of) in the framework is Readiness. I say “sort of” because Graduation is also a component at the high school level, but that one is kinda self-explanatory. We could get into the minutiae of the business rules in terms of who counts in the calculations and who doesn’t, but that isn’t terribly exciting to me.

Reviewing VDOE’s school performance framework (pt 2)

Fri, 21 Feb 2025 13:00:09 +0000

In my last post, I wrote about the mastery component of Virginia’s new school performance framework. I’m more or less optimistic about that component of the framework in that it’s straightforward to calculate and the weighting of different proficiency levels feels generous (particularly when compared to a binary pass/fail approach).

This post is about the growth component of the framework. I’m…less optimistic about this component. To be honest, it kinda sucks.

Reviewing VDOE’s school performance framework (pt 1)

Fri, 14 Feb 2025 13:00:39 +0000

The Virginia Department of Education (VDOE) recently – within the past year or so – completely revamped their school performance and accreditation system. I’m not going to describe all of the changes in detail, but the gist is that we now have 2 different systems – an accreditation system and a “school performance and support framework” (school performance system). The accreditation system is basically just a measure of inputs – are schools and school divisions providing the inputs they’re required to provide, as described by the VDOE Standards of Quality?

Graphs should be opinionated

Fri, 07 Feb 2025 13:00:16 +0000

I mentioned in last week’s post that people often think of quantitative data as objectively true. Not always, but often. And I’ll grant that there is something compelling about seeing some phenomenon bundled up into a tidy, concise metric, some little nugget that we can carry around with us and recite with authority.

A corollary of this that I run into often is that the data should “speak for itself.” That the job of any data analyst (or person who’s doing some data analysis task) is to simply get the “objectively true” data in front of decision-makers’ faces so it can whisper its truths to them. There’s a logic to this, I suppose. And in some cases, it’s probably the right way to approach data analysis.

Do less, document more

Fri, 31 Jan 2025 13:00:42 +0000

Documentation is like sleep. It’s unsexy. It’s boring. It feels like it gets in the way of doing more. It eats up time that you could be using to do something else, something more productive.

And yet, if you skimp on it, you always regret it. Maybe not immediately, but somewhere down the road, you regret it while you sit, confused, overcaffeinated, unsure of what you should be doing and why you (yet again) put yourself in this position.

Two and a half uses for student addresses

Fri, 24 Jan 2025 13:00:38 +0000

Student addresses are an interesting, and largely untapped, data point. They’re mostly seen as something that a school district’s transportation department needs for creating bus routes…and that’s pretty much it. But there are a handful of cases I can think of where addresses provide us with an additional, useful piece of student data that nothing else really approximates.

First, let’s think about some generic models in which where a student lives might be useful for us, as educators, to know. There are, in my mind, two (and a half) such cases:

Is your data worth its cost?

Fri, 17 Jan 2025 13:00:35 +0000

Back in high school, I ran track and cross country. Every Sunday, my teammates and I would meet somewhere to do our long runs, which typically ranged anywhere from 6 to 15 miles, depending on where we were in our training cycle.

I grew up in Northern Virginia, which, despite its (deserved) reputation as a hellscape of chain restaurants and perpetual traffic, is actually home to some beautiful places to run – including the Manassas Battlefield Park and the Prince William Forest Park.

Updating beliefs via Bayes’ Rule

Fri, 10 Jan 2025 13:00:31 +0000

We talk a lot in education about “using data,” and my general take is that the phrase “using data” doesn’t actually mean anything. But, if we want to be gracious and ignore all of the vagaries that come along with that phrase, then I think that “applying Bayes’ Rule” is what most people mean when they talk about “using data.”

Bayes’ Rule mathematically formalizes a process for updating beliefs that incorporates both prior beliefs and new data. It can be used to estimate lots of things, but we’ll limit this to beliefs here for the sake of concision. If you’re interested in learning more, the book Bayes Rules! by Johnson, Ott, and Dogucu is awesome, and the first chapter is very accessible (subsequent chapters assume some knowledge of programming and mathematical notation).

Stuffed Animals and Too Much Data

Fri, 03 Jan 2025 17:40:26 +0000

I have a 4-year-old daughter who owns a truly staggering number of stuffed animals, which she calls “stuffies.” Probably literally 60 or so. She sleeps in a full-sized bed, and if she puts all of her stuffies on her bed – some of which are bigger than her – there’s very little room for her to actually sleep. She likes having them on her bed, though, and after kicking a few off the end and shoving the others to the side, she can find a nice child-sized pocket to sleep in.