How Abstraction Empowers
How Abstraction Empowers
Hi! 👋 I’m Matt Christie, a data engineer at University of Wisconsin-Madison’s Global Land Use and Environment Lab (GLUE). Recently I had an opportunity to present on the importance of abstraction in software projects to the UW RSE community, a topic that sits close to my heart. While it’s certainly a good design principle to adhere to, I think the value of abstraction goes beyond that; I see it as a north star whose guidance can lead a small team such as ours to a place of maximal agency, self-determination, and control over the software we write and use.
The presentation is on YouTube, but I’ve also used AI to produce a transcript of the talk which is included below. Going through and manually following along with the video, I’m pleased with the results and don’t feel like it takes too many liberties. The place it deviated the most was the Q&A where the audio was quiet in places and there were several people speaking with different trains of thought. In those cases, I’ve gone through and manually edited the transcript to my satisfaction, though it still differs somewhat from the actual conversation which starts at 40:05 in the video.
Enjoy! 🌸
…
All right. Well, yeah — thanks for giving me the opportunity to present. I’m excited to talk to you all today. The topic is going to be about abstraction. We’re often familiar with this as a software design principle, but I’d like to say a little bit more about it beyond just that — specifically how it empowers us to build better, more resilient technology, and how it applies to us and maybe helps us solve some problems we’re experiencing.
I’m Matt Christie. I’m a data engineer at the Gibbs Land Use and Environment Lab at the Nelson Institute here on campus. Let’s take it away.
This is going to be a little unusual — maybe less about introducing a specific thing we’re trying, and more of an editorial. This will be the only time we read off the slides, but I’ll try to use my best disclaimer voice:
“While the content of this presentation is motivated by real-world problems, it largely consists of personal thoughts and feelings. Assertions made herein, aside from theoretical certainties, will not be assumed to be undeniably true, but rather reflect informed opinions derived from firsthand experience. Please interpret at your own discretion — and thank you for attending.”
This is going to be about abstraction. We’re going to start very concrete and go from there.
At the beginning, I’ll go over some of the challenges we’re experiencing at the lab with our own data. After that, I’ll try to diagnose what seems to be responsible for those issues — or at least how we’re thinking about and characterizing them. Then I’ll outline some things I think might solve those issues. Spoiler alert: I think it’s abstraction. And at the end, I’ll broaden way out and show how this is motivated by some deeper structure.
Starting Concrete
At our group, we develop and manage a database that integrates land use data in Brazil to study deforestation. It brings together data on cattle movements, rural properties, and deforestation to build a picture of the cattle sector — which we can then use to study policy. Specifically, right now it’s a Postgres database, built using processes on GitLab, and hosted on AWS.
Development has become hard. We’ve been working on this project for about ten years. It’s survived various grant cycles and phases. While it worked pretty well at its initial conception, we’re starting to run into issues around maintainability. Builds are erroring out in ways we don’t anticipate — even when we’re trying very hard, a detail or two gets past us. These unpredictable code runs mean we have to drill down and address them, which draws out how quickly we can get information from the system about what’s going wrong.
These runs are also unpredictable and go for a very long time, which makes development very difficult. Rather than pushing small changes and automatically creating a new release, we’re batching together issues into single successful runs. Here’s what that looks like in practice: builds of 12 hours, 15 hours. It’s not small issues — it’s meaningful parts of the week devoted to managing this.
We’ve also thought about what might help. In terms of computing, we’re starting to hit a point where we’d like more parallelism. Postgres as our build platform has ACID guarantees, which creates inherent bottlenecks. We’re on a single machine, and scaling it is non-trivial — distributed Postgres exists but is exotic and hard to maintain. And Postgres is a heavy dependency relative to frameworks designed for parallel processes; instead of just importing pandas, you need a Docker container running a full server process for each little task.
There are data challenges too. The final product is a complete database, which is a very large, heavy artifact to manage as a single output. We’ve thought about breaking it into smaller, more manageable artifacts, but the default is still Postgres format — so you’re relying on tooling to dump and restore those artifacts, and there are lots of subtle ways that typing and data formatting can be incompatible.
So that’s the concrete picture. Now I’ll try to articulate what I think underlies some of these challenges.
[Audience question about data sources]
We have thousands of sources — data acquired from websites or endpoints, stored in files. The schemas are quite diverse; if you registered the number of unique schemas we have, it’s in the hundreds. Working with hundreds of Excel files would be… I at least personally would not want to do that.
Our research space is also relatively open-ended. We have some initial projects we’d like to answer, but the data on rural properties in Brazil is general enough to support many kinds of questions. One of the big benefits of Postgres is queryability — the freedom to ask different questions of the data. It’s an open-ended question-answering platform, which you wouldn’t get from something more tailored.
So we like Postgres as a final output. But is it really the right fit for the process of creating the data? Even considering that question, there are a lot of things making it hard to move away from. In some sense, we’re really dependent on it — and even though Postgres is open-source and free to use, there are other factors at play that keep us locked into this specific technology. You usually hear “vendor lock-in” in terms of businesses, but I don’t think it’s restricted to that. And I don’t think Postgres is a unique example of failure — any technology has a specific use case it’s designed for, and that entails pros and cons. The cons are liabilities.
SQLite, for example, has pros — very low dependencies, no server process — but multiple processes can’t access it. If you want parallel computation over SQLite, you either replicate it or devise your own locking system. That’s a liability. AWS has pros too — engineers have developed lots of tools you’d never be able to write on your own. But there’s non-trivial liability in using it: monthly charges that are very uncaring about how you use them. There’s overhead to enacting alerts or engineering cheaper uses, and in worst-case scenarios — like uncertain federal funding — you have to ask whether you even have access to what you’ve stored there. Campus tools like CHTC are also useful, with knowledgeable, helpful facilitators. But if you have one day to do something and you need to write an email for debugging, that’s a liability too.
Usually, when you’re looking at the suite of possibilities, good system design is like shopping — you’re looking at all the options, finding the one that matches your use case, and committing to it. That’s fine. But use cases don’t remain static. If your project has a long time horizon, it’s likely those use cases change, and it becomes unclear how good your initial shopping process will be long-term.
Our use case changed with Postgres. Initially it was great: it was a nice shared execution environment for multiple teams that didn’t know each other well; outputs stored as tables were good for observability and drilling down on issues; SQL as a language gave people from different backgrounds — analysts, software engineers — a common ground. But as some of those reasons have faded, we’re left with the difficulty of migrating away. With smaller teams, you might carve out time to make a change once every few years, but the implications of being stuck with a single system are greater when you have fewer people who can maintain and manage it.
What We’d Want From a Solution
Instead of being tied to a single technology, is there a different way to structure things?
We’re aware there’s risk in adopting any given technology, and it’d be nice to future- proof ourselves. Can we make transitions between technologies a saner, more fluid process?
Some inspiration from CHTC and the Open Science Grid: Miron Livny often gives a talk, and one tagline is that your job should be like water. You don’t want heavy, clunky objects tumbling through the system — you want small, well-scoped tasks with minimal dependencies and modest resource requirements. That makes your work fluid through the system.
Can we make our data flow like water? Can we get away from that cumbersome all-or-nothing build process and have something more fluid? Specifically: seamless transitions between database states rather than heavy version-to-version jumps, smoother and more predictable development, and data that moves more easily between the people who use it and the technologies they use.
What might designing for that fluidity look like? Often fluidity within a given system involves things like replication strategies for reads or asynchronous persistence for scaling writes. But what if the specific component is itself what’s at fault? Can your data flow around it? Can you have fluidity in design decisions themselves?
With fluidity comes liquidity — you want your data to be available no matter what the issue is, whether that’s bugs, infrastructure problems, or institutional uncertainty.
The Case for Abstraction
People are probably generally familiar with abstraction, but let me step through how it as a principle specifically addresses these concerns.
Going back to fluidity: we want processes to be fluid around any given system or technology. What’s getting in the way of that? Why aren’t design decisions fluid?
The question is: what exactly do we need from the things we use? Do we need every feature a system has, implemented in exactly that way — or is there something smaller we actually need? If our applications require tables, do we need a Postgres database specifically, or do we just need something that structures data relationally? There are lots of things that do that. Maybe we’re needlessly depending on everything else Postgres offers when all we need is relational data. Or maybe in some use cases, Excel or SQLite would work fine.
Think of each interaction point with a system as a flag planted in code: “We are committed to using Postgres here.” A single line is fine, but if it’s tens or hundreds of lines, that becomes quite a commitment.
Here’s a concrete example using JSON as a storage technology. Suppose we have code for saving and retrieving variables, and we’ve decided on JSON. There might be six different places where we’ve made a commitment to JSON. If we wanted to change to YAML — which has comments, which is nice for maintenance — we have to go through and change each one. Each one is a dependency that strengthens our reliance on this particular technology.
In a second version of the code, we give ourselves abstract save and get operations, and we scope JSON to an implementation of those operations. Now there are two interaction points instead of six. If we want to change the underlying medium, we go to those two functions, not across the entire codebase. The difference seems small at this scale, but you can imagine how it multiplies as projects grow.
So what do we get with abstraction? We reduce dependencies in code. We become less reliant on specific systems. We use systems only for what we need and nothing more. That gives us more freedom to change the code and more choices for what could plug in where a given functionality is required.
That’s empowering. It gives us the power not to be stuck with one technology, but to have different tools at our fingertips.
The Power Asymmetry
There’s a little asymmetry between technology creators and technology users. As a creator, publishing a tool is relatively easy — you throw it on GitHub and have full control over what you provide to users. By comparison, users have less power to change a given technology. You can submit a pull request, but you don’t have the final say. You can try to make your own technology, but that’s very time-consuming.
With abstraction, that gets flipped. If you can reduce technologies to commodified implementations of core functionality you need, then there’s nothing special about any one technology. You can switch between implementations that serve you, rather than being at the whims of the creator — no longer stuck in legacy maintenance mode.
That’s empowerment through interchangeability. In the best case, the world is your oyster: you’ve identified the core requirements of your application, and you can swap out components with ease.
Where Abstraction Gets Interesting
Up to this point, abstraction is garden-variety best practice — DRY (Don’t Repeat Yourself) and so on. Are there places where it might be new, or where we don’t naturally think of components as candidates for abstraction?
A red flag that abstraction would be useful is when there are many different products that all do roughly the same thing. Consider orchestrators — Airflow, Prefect, Dagster, and others. They all schedule networks of tasks to produce outputs. Each one has its own features, but essentially they’re all orchestrators. Vendor lock-in with orchestrators is very apparent: pick one, and because orchestration is so closely tied to how you execute your code, it gets deeply ingrained.
As a thought experiment: what would abstract orchestration look like? A model of orchestration that doesn’t rely on any specific tool — but if you want to use Airflow, you have an implementation that compiles your abstract tasks to an Airflow DAG, until it stops serving you?
Infrastructure as code is another example. Terraform, AWS CloudFormation, Pulumi — they’re all meant to do the same thing. A few years ago, Terraform changed its license and people scrambled, feeling stuck. Wouldn’t it have been great to just say, “We know what infrastructure as code is in the abstract — you’re one implementation — and if we don’t like it, we’ll switch”?
Even playlists: Spotify has playlists, Apple Music has playlists. If you’re bought into one platform, you’re at their mercy. Recently, two of my favorite artist’s most popular albums just evaporated from Spotify. Is there an abstract way to treat a playlist — a sequence of songs — so you can plug into different systems and route around the liabilities of any given one?
Vertebrate
Easy to abstract: JSON. Harder: orchestrators. Where might we begin?
We started making a package called Vertebrate, a place to store abstractions we think are useful for our use case. We started with a simple abstraction for a computing environment. We hope to expand it with abstractions for data storage, data transfer, and eventually orchestration. If this sounds interesting, I’d recommend checking it out or contributing.
Some Concrete Underpinnings
I’ve framed a lot of this as feelings or opinions, but I want to dip into the concrete underpinnings — showing there are formal ways to make this happen.
Take a simple data structure: a mapping, or key-value pairs. We have an intuitive sense of what this is. Here are some concrete instances: this is JSON, this is YAML, this is a relational table with columns. They’re all expressions of a mapping. But they’re concrete, and in being concrete, they’re somewhat brittle — to work with YAML you need a YAML parser, for relational data you need a relational technology.
What’s essentially “mappy” about these? Maybe it’s less important what the underlying representation is, and more about what information is preserved when you transform it. Instead of focusing on the objects themselves, we can focus on information-preserving transformations — isomorphisms — between them. You can start with a familiar object and define “mappings” as all the objects you can isomorphically transform to. If you can get back to where you started, the path along the way passes through isomorphic objects. This doesn’t depend on any specific concrete requirements — it’s a way to work with information in the abstract, independent of those considerations.
Similarly, abstractions themselves exist in this domain. Two abstractions might look different on their face — one using Python’s __getitem__/__setitem__ dunder methods, another using explicit get and set methods as in Java — but they’re equivalent. You can implement one in terms of the other.
This addresses a key problem with abstraction as currently practiced: people often say “let’s have one standard,” but standards proliferate. That xkcd comic: 14 standards, then someone makes a universal one, now there are 15. These things are hard to maintain top- down.
But maybe you can imagine a world with decentralized abstraction: you have instances shown to be equivalent given one abstraction, and when you see someone in another community doing something similar, you write transformations between your abstractions so each can use the other’s implementations. Developers could plug into a network or web of equivalence — isomorphisms — to achieve that result.
This opens up interesting possibilities. Imagine a data transpiler — someone else has done the work of identifying equivalences between document formats, and you just use the tool rather than writing low-level transformation code yourself. Or cross-environment interfaces: there are already tools that expose Postgres tables and procedures as an API automatically, or that transparently translate Python objects to shell commands, as Python Fire does. This kind of work could enable more of that.
Summary
Abstraction touches all these spheres — it’s intersectional. It’s a way to think about and express attitudes toward technology. There’s theory that grounds it and clarifies what we’re talking about. And it’s actionable: you can write code using abstractions to design systems that are more future-proof, more technology-agnostic, and that serve you better than any one specific tool would. Thank you.
Q&A
[Audience]: (Paraphrasing) You have these ideas about abstraction, but then you just offer up yet another tool, which feels counter to your whole point. How is Vertebrate any different?
[Matt]: That’s a good point. Vertebrate might be more of a lead by example thing. Though Python’s collections.abc is a good model for what Vertebrate is trying to achieve — vanilla, brandless, generic abstractions for people to use in code. One thing I’ve thought about is what we treat as not needing to be reimplemented over and over again versus things that we implicitly understand are useful but haven’t made explicit code objects. We teach design patterns in classes, but you’re expected to rewrite them every time. It’s implicitly understood to be useful, but it doesn’t have a presence as an explicit code object. I feel like there could be more of that. I would be happy if people saw Vertebrate and said, “Okay, I’m going to write my own abstractions, the ones I think are useful” — and get to that decentralized point where we both have our favorite way to express orchestration and there’s a way to translate between them.
[Audience]: So the impression I’m getting: when you’re working with data, even before you know what form it’ll take, think about what you need as the kernel of the information and how you can make that flexible — so that if something changes midway through, you’re not stuck.
[Matt]: Yes — I think it’s about flexibility. One thing that’s interesting about abstractions is that some of those core requirements do converge on abstract objects in almost a Platonic sense. They come up in lots of places because of the nature of the projects we work on and how information is structured. So the goal is: plan for flexibility, but also maybe we can name and identify some of these things more precisely — so that planning for flexibility is less of an open-ended art and more of a process of choosing from sensible, well-known abstractions.
[Audience]: I find myself caught in this frequently. There’s some imagined future where I’ve had the time to think ahead, and I’ve got abstractions for everything, and my code reads like English. But the reality of implementation — especially in science or a university setting — is that I need the thing now, I have to present results today. No one cares what the code looks like; they just want the plot or the dataset. So you don’t get to do the abstraction phase. And then you end up buried under the decisions you made a year ago and have to deal with them because you’re dying. How do you get breathing room to fix it?
[Matt]: A few things on that. The incentives are genuinely hard. My counter is: some projects really are one-off. But in our experience, projects have a habit of scope creep — you do an initial thing, lots of interesting follow-up questions emerge, the project builds in complexity, and at that point you realize you need more structure. There’s often a moment where you didn’t spend time thinking about the future, put a month into making a skeleton, and then found that it produced pretty good results but now you’re stuck with the wrong skeleton.
For that specifically, it is challenging. I think of my own work as carving out channels — one channel for servicing analyses, one for project management and collaborator interactions, one for improvements. And there’s probably no perfect answer. One motivation for making Vertebrate a package is to lower the barrier to better structure. If we can identify broadly applicable abstractions, then restructuring looks more like pulling something out of a toolbox than out of the ether.
As an RSE, you have pressure to get things out quickly, and often the technical internals are hidden from the PI — they’re familiar with the results but not with what makes something easy or hard. Is there a way to empower yourself to build projects and systems you’re happy with, without feeling like you’re permanently underwater? That’s part of what I’m going for.
[Audience]: Is this more of an educational thing, or a technical problem? Research software engineers — especially newer ones — may not know how to implement abstractions, or even if they do, they don’t practice it because you have to get a plot done immediately. I’ve done this myself: make the plot, see it works, expand on it. But from a software engineering background, I know the best thing is: once you have a proof of concept, throw it away and redesign it properly. So is this more educational than technical?
[Matt]: I think it can be both. Education and communication are a big part of it — helping impart the ways in which abstraction is useful, counteracting the incentive to devalue it as over-engineering. But personally, for selfish reasons, I also just intend to use the abstractions. We’re building something that isn’t just principles — it’s “here’s how I want to interact with computing environments, and here’s something that might help us move between them more fluidly.”
[Lab colleague]: Just a biased but important comment: it’s not like the “I need this right now” pressure ever goes away. It’s sort of like this system that maybe six years ago was more easily maintainable — the schema diversity, the volume of data. The process of maintaining it reaches a point where, if researchers and PIs want something right now from a system that’s becoming difficult to maintain, hopefully you have a receptive PI you can sit down with and say: at a certain point, there’s going to be a trade-off — either on what data is available to researchers, or on when they get it. There has to be an understood trade-off. And as software engineers or RSEs broadly, our job is to communicate trade-offs effectively and reason over them, even to non-engineering folks. Communicating those trade-offs is continually important for improving both the software and the results.
[Audience]: Coming from the individual researcher writing code for their projects: even if you’re not doing the full abstraction, you can start laying groundwork — notes to yourself like, “if this becomes a larger project, this is something I should address.” How many times have I had comments like that… six years later they’re still there. (Turn those into GitHub issues.) And: getting results right away is good, to confirm you’re on a viable route. But the more dependent that code is on a very specific implementation of something, the more difficult it is to deploy. When thinking about that for yourself, you might be okay with the headache six months from now. But I encourage people to think about open science — contributing, sharing, distributing. If you as the developer know you’re going to struggle with it in six months, the grad student in a random lab at another university will have no hope in five years if this code ends up in a paper. So it’s important to make sure there’s at least less fragility in your code. Maybe abstraction is one way of doing that, but laying groundwork early — making your work shareable — is important regardless.
[Audience]: So the short answer to “what if you win the lottery and leave the project tomorrow?” is…
[Matt]: No. [Laughter] Yeah. All right. Thanks again.