Drew Dimmery

AI Evaluations for conviviality

Wed, 05 Aug 2026 00:00:00 GMT

This fall, I’m teaching a brand-new course at Hertie on AI evaluation. I think it’s going to be a lot of fun.

We aren’t going to spend the semester running standard model benchmarks or debating sci-fi capabilities. Instead, the angle is to carefully develop experiments to tie these complex systems into the real human lives they affect. This post will dive into how I think about the class, how I’ve designed it and what I’m hoping to get out of it.

Why AI evaluation needs this class

One of my big frustrations with AI evaluation is the way that so much of it is completely stuck in a category error. Put simply, I don’t particularly care about abstract “model capabilities”. I’m a humanist in the sense that I believe the only evaluations of tools that really matter are the ones that measure how they improve the ability of humans to achieve their aims. Far too few AI evaluations focus on this component, as they’re too busy working in a science-fiction world of creating new sentient life and trying to measure… whatever they mean by that. Consider a benchmark like “Humanity’s Last Exam”: the name alone tells you it’s measuring performance on frontier-of-human-knowledge trivia, with no theory at all of which humans are helped by a model answering such questions¹. To me, models are useful when they help me accomplish my goals. This class is about helping me work out more clearly what that should look like in terms of evaluation methodologies.

To take an example of how even AI skeptics miss the mark on this: I respect a lot of Arvind Narayanan’s work on AI and its role in society. The work he’s been doing with Sayash Kapoor on “open world” AI evals is very interesting, but I find that it retains the original sin of AI evaluation, which treats the AI system as largely distinct from the social system in which it operates. The same spirit animates another recent AI evaluation: it’s interesting whether a given model + harness can rewrite a working SQLite clone from the manual, but this doesn’t ultimately tell us much about whether a non-techie home gardener can use an agentic AI system to put together a coherent way of managing their irrigation. “The most important elements of people’s quality of life are plural and qualitatively distinct”² and, in the modern world, increasingly play out in digital spaces. The promise of agentic tools is that they make people better able to personally manipulate these spaces and bend them to their whims. The relevant manipulations we should be testing are not whether Joe Schmoe D.D.S. can rewrite SQLite from scratch (how would he even ask for this? why does he need it?), but whether his office can set up an appointment system exactly tailored to their working style without any technical knowledge. Realism of environment (a la open world evals) is good, but realism of purpose is what I’m looking to engender through this class.

The kind of evaluation I have in mind does exist. METR’s randomized trial of experienced open-source developers is one example³. Whatever you make of the result, it is fundamentally a study of the joint human-AI system doing real work. The gap between perceived and actual benefit is exactly the kind of thing capability benchmarks struggle to surface.

Where the course comes from

In general, the course comes from a place of thinking about AI via experimentation. It started with me deciding I wanted to teach a class on experimentation. Cyrus Samii designed an amazing class called “Quant Field Methods” (note: this is the old version of the course I pivoted off of, but Cyrus has since updated it). The idea was that students would leave the course with a complete pre-analysis plan ready to take into the field: actually multiple plans, as groups of students would repeatedly go through the process of identifying a societal problem, designing an intervention around it, and then working through the design issues to prepare the experiment for the field. I wrote a syllabus roughly based on this idea, which I’ve had in my back pocket for a while. One of the main differences is that Cyrus believes in mechanisms in a way that I don’t. His version emphasized those mechanisms, while I wanted to dig into methods more explicitly. I slowed it down relative to his planned three-cycle class to a two-cycle version that left more room for lectures on methodological problems specifically geared towards experimentation.

I’ve had this syllabus since around 2023, but as I prepared for this fall, I decided that I wanted to target it more directly around a particular problem (rather than leaving it fully generic). Thus, I’ve decided to focus on AI evaluation. It’s very hot right now! Andy Hall recently wrote an argument for why AI evaluation is so important. I think what he’s talking about is important, but it also fails to see exactly what “AI” is: it isn’t just a model, it’s a harness and a large socio-technical system. Things like his dictatorship eval are great, but when I worry about the conviviality of AI systems, I really do think one has to consider the system as a whole, not just “what is emitted by a model”. I’m hoping to push on this through the class, and I think the framing of experimentation is critical to this turn. As Kevin and I write, experiments are ontological tools which create new states of the world. This class is about tracing out and evaluating the ontological interventions in human action that AI systems are making and measuring their performance relative to human goals.

Design decisions

I’ve made a variety of changes to try and bring this together. The main one is that students will work on just a single project, but they will actually field it, too. This was one of the major decisions I agonized over, because while it’s extremely difficult to successfully get an experiment into the field during a one-semester course, I think there are a few reasons to do this anyway:

AI evaluation moves extremely quickly. If you don’t actually do your evaluation quickly, your design will get stale. There isn’t a lot of value in just leaving a plan around for a long time. Students should also learn how to deploy data science skills in the nimble way this requires: it’s how things work in “the real world” of applied data science.
Running the actual experiment is a massive learning opportunity. Almost everyone’s first experiment goes off the rails in some way, and the best way to get past this is through it.
I want to see actual results! Getting a PAP is great, but the feeling of satisfaction from seeing how the experiment actually turns out is beyond compare.
Coding assistants make it easier than ever to do it. I think ramping up expectations is a good way to preserve the value of higher education in the face of AI.

Roughly, my plan is to devote the first two-thirds of the class to developing a really complete pre-analysis plan, and then the last third will be devoted to fielding the experiment, analyzing the results and writing them up.

What’s in the class

Each meeting (we have 12 at Hertie, of ~1h50m each) is split roughly in half: a workshop where students share their work-in-progress (and get feedback), and then a lecture at the end preparing for the next week’s milestone. Topics are reinforced a few times: first in lecture, then in the reading, then in execution and the workshop. The arc runs from a survey of the evaluation landscape (benchmarks versus human experiments) and student pitches, through measurement theory and construct validity, defining the interventional system (a model plus a prompt plus a scaffold plus …), outcome measurement and scorer validation, randomization and threats to validity, and the analysis plan. At this point the pre-analysis plan is registered and the class flips to execution: pilot results, troubleshooting and documented deviations, draft findings, and final presentations. Teams of four carry one project across the whole term and produce four deliverables: an ethics and responsible-evaluation self-assessment, a registered pre-analysis plan, two deliberately informal work-in-progress code demos, and a final report with reproducible code and data. Each of these is coordinated by a different member of the team to ensure shared accountability. There are some other, smaller assignments to make sure students read and reflect on the material as well. All of it happens in the open on GitHub: pull requests, issues, and a project board, so that I can actually see how the work got done and not just what landed at the end⁴.

What counts as a project

The original template assumed human-subjects field experiments, but for a class on AI evaluation, that seemed too prescriptive. I will allow students to focus on developing a well-defined benchmark, a clear annotation task, or a human-subjects experiment. I’m the most skeptical of the first of these, for all the reasons above, but I’d rather leave the option open than rule it out by fiat. The critical thing for each of these is that they must be focused on understanding something about a joint human-AI task, and a benchmark doesn’t get a pass on this. Whatever form a project takes, students will have to justify it on the same human-focused grounds, repeatedly, at every stage of the design. The turn I have in mind here is to focus not just on whether AI can write good code, but on whether it helps a particular population of people better perform whatever they see as their tasks. I suspect a large part of the class will be spent discussing exactly who the audience is, what their goals are and exactly what must be measured about how well they can accomplish them.

Summing up

To bring this back to my big picture, I think the metaphor of tending a garden from a previous post is useful. Suppose we’re enrolled in a quantitative horticulture program. There’s a lot of stuff we can measure about plants: how tall they grow, how much input (water, fertilizer, and so on) they need, how much they yield. These may be cool to measure, and you might be able to optimize any of them, but at some point you have to be really specific about what you’re looking for from the garden. If you grow the largest eggplants in the world, but what you really cared about was how the flowers made you feel, then your highly optimized garden is still useless. Cyrus’s initial framing of the class is ultimately right that theory is critically important, but I think he was wrong about what kind of theory was needed. In contrast to his original pitch, which focused on theoretical mechanisms, I think it might be more important to focus on normative theory⁵. This is roughly the bet behind “Building the Society We Want”, a syllabus I helped Kevin put together a few years back, which organizes everything around Values first and analytical Tools second. That’s the direction I’m pushing this class in, at least; I’m excited to see how it goes!

Indeed, it feels much more like the opposite: measuring whether models are ready to kill us all in our sleep, making it truly our last exam. ↩
Martha Nussbaum, Creating Capabilities: The Human Development Approach ↩
Alas, there were some issues with this trial. Recht has written about them, and I won’t recapitulate that here. The overall idea of measuring this is, in my view, much better than their more famous chart of time horizons insomuch as it focuses on a human reference. ↩
I have a set of scripts that provide a full dashboard of how students worked on projects together, which makes it easy to see how work was distributed. I think this is really important for group projects, especially in an age of AI, so that I can actually grade on progress and process rather than just the final product. ↩
You might even call this the “Problem Solving” approach to AI evaluation. ↩

Better testing through concentration

Tue, 30 Jun 2026 00:00:00 GMT

Basically every piece of software I write eventually uses some form of stochasticity. Writing tests of random functions is frustrating: you write a test that draws 1,000 samples¹ and asserts it’s the right value (or within some tolerance I pick by feel). It passes and I move on with my life. Then, three months later, when I need to push a small fix somewhere, it starts failing and I have no idea why.

The usual response is either (a) crank up the sample count until the failures stop, (b) widen the tolerance until the test is effectively vacuous, or (c) mark the test as flaky and move on with my life². I’m sure this sounds familiar, but these solutions suck. They’re fundamentally symptoms of the same underlying problem: there’s no principled connection between the test’s statistical properties and the sample count and tolerance chosen.

This has always been annoying, but it’s become a more urgent problem recently. When writing code with AI, whether code is actually good tracks the quality of tests and evaluations. Strong tests mean strong code: the model writes something, the tests catch what’s wrong, the model fixes it, and you converge to a working piece of software at the end of the session. Bad, vacuous tests break this loop. The model “passes” the tests by writing a test that always passes (“Oh, this test is flaky. I’ll just crank up the tolerance.” Yes, unfortunately, Claude has the same bad habits I do, sometimes), and you don’t find out until production. Stochastic code is especially vulnerable here, because the failure mode is silent: a test with a too-wide tolerance or a too-small sample count may happily pass code that’s subtly wrong, but you’ll feel confident because you have a test written supposedly verifying its behavior.

This bothered me enough to (try to) do something about it.³

The idea

pytest-stochastic is a pytest plugin that makes the connection between test and statistical guarantee explicit. You declare what you know about your test statistic---its bounds, variance, sub-Gaussian parameter, whatever you’ve got---and the framework does the rest. It selects the tightest applicable concentration inequality from a registry of ten I chose “stochastically”, computes the minimum sample size to hit the acceptable flakiness budget, and runs the test.

from pytest_stochastic import stochastic_test

@stochastic_test(
    expected=0.5,
    atol=0.01,
    bounds=(0, 1),
    failure_prob=1e-8,
)
def test_uniform_mean(rng):
    return rng.uniform(0, 1)

That failure_prob=1e-8 is guaranteed based on the invariants you provide: the test will produce a false failure less than once in 100 million runs (so long as your assumptions are correct). The framework figures out that it needs exactly 7,394 samples to deliver that guarantee via Hoeffding’s inequality, and runs them.

The key insight is that you don’t need to carefully figure out the right concentration inequality, you just express the assumptions that you know. If you only know the range of your statistic, you get Hoeffding and pay for the worst case. Declare the variance and Bernstein kicks in---often much tighter. Know the sub-Gaussian parameter and you can do even better still. In the worked example in the docs, testing an OLS slope estimator goes from 75,000 samples (bounds only) to 100 (sub-Gaussian): a reduction of multiple orders of magnitude for the same probabilistic guarantee.

But I don’t know my variance

Yeah, no shit, why would you? Deriving variance analytically for a unit test is absurdly overkill. This is what tune mode is for:

pytest --stochastic-tune

This profiles each test empirically (50,000 samples by default), computes an upper confidence bound on the variance, and writes the result to .stochastic.toml. Commit that file, and subsequent runs automatically use the tighter Bernstein bound.

What else

There’s also a @distributional_test decorator for goodness-of-fit testing (KS, chi-squared, Anderson-Darling against a reference distribution), automatic RNG injection with seed reporting on failure for reproducibility, and sensible error messages when you misconfigure things. The documentation covers all of this.

Why bother

Ad-hoc thresholds and magic sample counts are technical debt that compounds quietly, and this is an attempt to do better. For many use cases, alas, the sample counts that concentration inequalities require are still too large to be practical. I tried this on a recent project and ended up just fixing a seed anyway, because I couldn’t afford 10—100 evaluations of the test statistic (which each required letting a neural network converge). When individual evaluations are cheap, though, this should be helpful to get faster, more reliable tests (or at least figure out what the tradeoff is between the two). I think the worked example in the docs is a pretty good example of how this could be useful.

This isn’t new math or anything, but maybe someone finds it useful.

The code is on GitHub. Try it out and let me know about how it goes. I’m making no guarantees, here!

Why 1,000? Because it’s a nice round number, and nice round numbers are inherently trustworthy. ↩
Or more likely, just choose a random seed that works for a single run and hardcode it. It’s future-Drew’s problem when a change makes it randomly start failing. ↩
Read as: bothered me enough to make Claude do something about it. Pretty nearly one-shot the thing, too, but as always the long-tail of iteration still took a while. GitHub shows 15 PRs and about 50 commits, so that’s probably about how much prompting I had to do (each PR represented one initial prompt that I iterated on a couple times fixing errors and providing feedback). ↩

Causal ML (the class)

Mon, 16 Feb 2026 00:00:00 GMT

Teaching CausalML

The most important paper I’ve ever read was David Freedman’s Statistical Models and Shoe Leather. As always, when I mention this paper, I must provide the following quote:

Given the limits to present knowledge, I doubt that models can be rescued by technical fixes. Arguments about the theoretical merit of regression or the asymptotic behavior of specification tests for picking one version of a model over another seem like arguments about how to build desalination plants with cold fusion as the energy source. The concept may be admirable, the technical details may be fascinating, but thirsty people should look elsewhere.

I find this to be a critically important lens through which to view the field of Causal Machine Learning, a class I just taught in the Fall at Hertie about a field that forms the core of a lot of my research. In many ways, this class is about me trying to square my love for design-based causal inference with the realities of modern Causal Machine Learning which is maddeningly focused on cold-fusion-powered desalination. A large amount of the literature in this field is beset by strong super-population assumptions, asymptotic theory and, thanks to the fundamental problem of causal inference, little way to determine what will actually be effective on the data that one has in front of them.

As such, my overriding goal for the course was to make students skeptical and informed consumers and producers of CML. I tried to get them to think about actual experiments whenever possible, and to think about the conditions under which methods work and do not work (and how you might be able to know)¹. Then I expected everyone to get their hands dirty by actually implementing these methods and testing them through simulation studies. They did this both as part of in-class demos as well as a final project meant to extend and flesh out the demo (and fix any issues surfaced during the demo).

I expected a lot from students. Of the nearly two-hour class, I lectured for only around 30 minutes, while students led discussion for the remainder. They gave presentations about papers (with my intervention when I wanted to add additional color / complaints) and performed high quality simulation studies set up so their colleagues could tweak things about the setting or the method to see the results.

The course, therefore, operated on a few operating principles:

Read papers: If you can’t read new work, the field will leave you behind.
Present main ideas: Nothing forces you to understand a paper like presenting its main ideas to your classmates.
Write code: Implementing and stress testing methods is a powerful way to get a deep understanding of how they work.
Don’t fight AI: Focus on in-class performance. If students just let AI create a slide deck/demo and don’t understand it, this will be very obvious when they get up to talk about their work. In practice, there were very few moments when students had offloaded too much of their preparation to AI. I think the incentives here are good. Students can use AI however helps them, but they cannot avoid learning enough of what they need to know to feel comfortable standing in front of the room to talk about it.

The nagging problem is this: we spent the semester stress-testing methods through simulation to see how they work and where they break. This is genuinely useful. But it only tells us how methods perform in worlds we can imagine (and the world is much more complicated than that). The whole point of causal inference is that we never observe the counterfactual, so we can never confirm on our actual dataset that a method did what it promised. I don’t think the course resolves this---I don’t think anything really can. What I think it does is produce people who understand the machinery well enough to know exactly where the leap of faith is, and who can be honest about when they’re making it. I think that matters, even if it’s not enough. There’s still a lot of work to be done on understanding what we can actually learn from these methods and what we can’t---and frankly, a lot of the field doesn’t always seem particularly interested in that question.

With that, check out the syllabus I landed on:

Syllabus

Course overview

The syllabus begins with the building blocks---design-based inference, covariate adjustment, and propensity scores---before moving to the doubly robust and semiparametric methods that form the core of modern CML. From there it turns to heterogeneous treatment effects (first with forests and meta-learners, then neural networks), policy learning, and experimental design. The final weeks cover topics where the standard assumptions start to break down: panel data, partial identification, adaptive experimentation, and interference. Students wrote a referee report on one paper, presented another, gave a live code demonstration of one week’s methods, and produced a final expository project extending the demonstration. All readings are papers; there is no textbook.

Session-by-session

Session 1: Design-based Causal Inference and Monte Carlo Simulation

The potential outcomes framework, what randomization buys you, and the ADEMP framework for simulation studies that students use to stress-test every method in the course.

Will Lowe teaches a fantastic Causal course at Hertie that focuses more on DAG-world, so laying out some strong arguments around manipulability and the implied metaphysics of potential outcomes. This comes early to set the stage for the rest of the course. My computational demo aimed at understanding the difference between inference on the SATE and the PATE as a way to get them thinking about sources of randomization (and better prepare them for the many superpopulations to come).

Session 2: Covariate Adjustment

If randomization already gives you unbiased estimates, should you adjust for covariates at all? If not, is there any role for ML? I think there are actually pretty good answers in this setting! So we talked about the Lin-style regression and AIPW as ways to be both safe and efficient.

Session 3: Balancing Weights

Only in this session are we really getting to the observational world at all. What is “balance”? Why might we want it? How should we define it? We went through a variety of modern approaches to this including permutation weighting, which holds a special place in my heart as it reframes the problem as classification.

Session 4: Doubly Robust Methods, Double ML, and TMLE

Sessions 2 and 3 each model one side of the problem; doubly robust methods combine both, giving you two chances to get it right. Is it actually useful to get two chances at this? We work through the error decompositions that show why it might be helpful even when you get neither right. We hammer back on this many times throughout the course, too.

A recent paper we read was Kennedy (2022).

Session 5: Heterogeneous Treatment Effects I

How do effects vary across individuals? This is categorically harder than estimating averages---there is no observed target, no natural loss function, and no straightforward way to validate predictions, which is why my lecture focuses on the Fundamental Problem of Causal Inference. I’m just categorically unable to let people have fun.

Session 6: Heterogeneous Treatment Effects II --- Neural Networks

This session focuses on using neural networks. My lecture focused on representation learning and why this is actually really tricky in the causal setting: what do you mean we require an invertible representation? In this economy?

Some recent papers we read were Nie & Wager (2021) and Ma et al. (2025).

Session 7: Off-Policy Evaluation and Optimization

A pivot from “how large is the effect?” to “who should we treat?” I pull in the discussion over causal decision making (i.e. ignore causal effects, Learning the Sign is All You Need), as I think it’s worth problematizing what the actual task is. Is there actually an important role for understanding HTEs for decision-making? For many reasons, I still think the answer is yes.

Some recent papers we read were Athey & Wager (2021) and Kern et al. (2025).

Session 8: Experimental Design

Probably the session most near and dear to my heart. Always take the opportunity to give ‘em the Fisher quote:

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

Some recent papers we read were Harshaw et al. (2019) and Arbour et al. (2022).

Session 9: Panel Data and Modern Difference-in-Differences

The course turns to settings without randomization (but, of course, problematizes this framing). We go over the “New Diff-in-diff” and pivot quickly to synthetic control, because I want to keep as much attention on the actual identification problems of the setting (which I think are often elided because they do not admit simple technical fixes: they require thinking about the actual problem setting).

Some recent papers we read were Ben-Michael et al. (2021) and Ben-Michael et al. (2023).

Session 10: Partial Identification

We cover Manski’s no-assumptions bounds and look at a variety of ways that ML can be used to support tighter intervals. I find this really important as a way to tie together the class, but I find that students don’t get that excited by the idea of estimating an interval rather than getting a single holy point estimate.

Some recent papers we read were Khan, Saveski & Ugander (2024) and Samii, Wang & Zhou (2023).

Session 11: Adaptive Experimentation and Reinforcement Learning

I always tell students that explore-exploit is a natural framework that can be helpful to them generally as they live their lives (do we go to the same sushi spot as always or try somewhere new?). I spend a lot of time talking about how simple effect estimators fail. We don’t get much into the solutions for this, unfortunately (I am not fully happy with how this has been solved thus far in the literature).

Some recent papers we read were Ouyang et al. (2022) and Hadad et al. (2021).

Session 12: Interference

We end by blowing up the possibility of causal inference via exponential explosion of potential outcomes under interference. Is it possible to do anything about it? I throw in a lot of attention to design-based solutions which (alas, don’t they always?) require understanding things about the world.

A recent paper we read was Shirani & Bayati (2024).

Closing thoughts

That was my attempt to square the circle. I think it went pretty well all things considered. There’s such a huge amount to cover (jamming DR + DML + TMLE into one session is crazy, as is combining CB + RL into one session). Alas, there was a lot I wanted to cover. I think more than half the classes had some kind of important design-based connection, which I think is good. The machinery is all genuinely useful and extremely interesting, but it doesn’t replace knowing where your identification comes from. I would much rather send students out the door a little paranoid than too comfortable that technical fixes will solve their problems.

One of many frustrations I have is with the term “testing assumptions”. If you can test it, then it isn’t an assumption! ↩

Website Refresh

Wed, 19 Nov 2025 00:00:00 GMT

I’ve made a few changes and updates to this website. None of them are particularly earth-shattering, but I think building one’s own tools—rather than accepting the constraints of platforms—is worth the effort.

Here’s what I’ve been working on:

Semantic Scholar integration for updating publication records
Listmonk tools for sharing blog posts as a newsletter
Improvements to the design and performance, particularly around fonts

I think these changes are consistent with the philosophy I introduced in the last post. I’ll introduce each briefly and discuss how they work—perhaps they’ll be useful to others building their own corners of the web.

Semantic Scholar integration

I’ve had a consistent desire to avoid tedious data entry in updating my website. This was part of my intention when I moved it to Quarto in the first place: Quarto gives great opportunity to programmatically generate a static website. I can easily mix in rich documents that combine code (R and Python in particular) and easily formatted Markdown text. This is all great. It let me hack together a nice way to display my publications in a consistent format based on data stored in a hand-curated YAML file. The next step was to automate the process of updating the publication records. I used the Semantic Scholar API to fetch the latest information about my publications and then used Quarto’s templating capabilities to generate the updated publication list.

Most of the work of this system is in two functions which, because I vibe-coded it, I’ve never really even looked at.

See the code

def fetch_author_papers(author_id):
    """Fetch papers from Semantic Scholar API"""
    base_url = "https://api.semanticscholar.org/graph/v1"
    fields = "paperId,title,authors,year,venue,publicationDate,externalIds,openAccessPdf,url"

    url = f"{base_url}/author/{author_id}/papers"
    params = {'fields': fields, 'limit': 1000}

    try:
        response = requests.get(url, params=params)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as e:
        print(f"Error fetching data from Semantic Scholar: {e}")
        return None

def create_yaml_entry_from_ss_paper(paper):
    """Convert Semantic Scholar paper to YAML entry format"""
    paper_id = paper.get('paperId')
    title = paper.get('title', 'Untitled')
    year = paper.get('year')
    venue = paper.get('venue', '')

    # Process authors
    authors = []
    for author in paper.get('authors', []):
        author_name = author.get('name', '')
        if 'Dimmery' in author_name:
            authors.append("me")
        else:
            authors.append(author_name)

    entry = {
        'title': title,
        'authors': authors,
        'year': year,
        'venue': venue if venue else None,
        'visible': False,  # Default to not visible
        'ssid': paper_id,
    }

    # Add external links if available
    external_ids = paper.get('externalIds', {})
    if external_ids.get('DOI'):
        entry['published_url'] = f"https://doi.org/{external_ids['DOI']}"

    if external_ids.get('ArXiv'):
        entry['preprint'] = f"https://arxiv.org/abs/{external_ids['ArXiv']}"

    # Add open access PDF if available
    open_access = paper.get('openAccessPdf')
    if open_access and open_access.get('url'):
        entry['pdf_url'] = open_access['url']

    return entry

Critically, this is all pretty trivial, but it’s also boring, so I’m glad I didn’t need to think particularly hard about it. It also means I didn’t need to personally dig through the Semantic Scholar API documentation. The code quality is not great, but neither was my own code before!

So now when I build the site, it automatically updates the YAML with my latest papers. I can just browse through them and flip the visible flag to show them once I’m satisfied they’re right.

Listmonk tools

The second main improvement I made was to add an email distribution list to my website’s blog. This was pretty easy because of the great open source email distribution software, Listmonk, and the super easy deployment option at Pikapods. The system I’ve come up with is basically the following:

Write a blog post in a Quarto document.
Push it to a branch if I want to have sensitivity readers.
Once I’m satisfied, push it to main, which automatically triggers a Quarto build of the website and makes the blog post “live” on the website (and RSS feed)
Manually trigger a Github Actions workflow to schedule a Listmonk email campaign.

It’s this last part that I think is particularly cool. Basically the way it works is I vibecoded an over-engineered system that pulls the necessary metadata from a given blog post (in the .qmd file), renders the post to an HTML version, extracts the HTML from that, and sends all of this through Listmonk’s API to schedule an email campaign with nicely formatted HTML content. I’m not going to embed all of the scripts for this here, but you can take a look at them in my website’s Github: scripts/.

To actually trigger this monstrosity, I used a nice feature that Github Actions provides: workflow_dispatch event triggers. Basically, what this means is that you can configure a Github Action to trigger manually through the Github UI. It looks something like this:

So basically all I need to do is tell the workflow what post to schedule (the post_slug part), when to schedule it (the send_at). There are also a few optional things I can plug in, such as which list to send to (Listmonk lets me manage multiple lists easily), or I can send test emails.

This is a solution I’m mostly happy with. I think there’s still a bit of tweaking about some of the practicalities. I think there are two things I’m trying to figure out:

Whether I should maintain separate lists for different categories of post. It wouldn’t be hard to maintain category-specific lists so that people could easily opt-out of posts about (e.g.) how I built things on my website.
How exactly to manage the sending of emails. When I sent out the last email, it only sent to like two-thirds of my subscribers because my email provider is Protonmail setup with a custom blog@ email address for sending/receiving. Unfortunately, there were some rate limits that I didn’t adequately respect the first time.

I’m overall pretty pleased with this system. You’re welcome to subscribe here!

Improvements in design and performance

Within me are two wolves: Sometimes I miss the days of HTML-only websites. Other days, I long for the ability to have nicely reactive and responsive design elements that require richer web frameworks. On some personal projects over the last few months, I’ve learned a bit more about modern webdev tools. One thing I’m particularly taken by is the way that modern webdev tools like Vite manage javascript dependencies.

In short, the problem with a lot of these big web frameworks is that they’re really big (duh)! They have a lot of stuff that you never use in a small personal website like mine, but they still get bundled into the library that is sent down to the website viewer. We don’t need to do that: we can be smarter. That’s basically what tools like Vite do. They only bundle together the parts of code that are necessary for the particular site being built. They also can help with lazy loading (so they’re only loaded when needed).

That’s all a bit of a tangent, but it’s gotten me thinking about why I actually need to load so many Javascript libraries. Quarto ultimately doesn’t seem to be particularly optimized on this front. This is kind of the tradeoff you get: sure, it’s easy to throw something together with Quarto that has a lot of the bells and whistles a data scientist craves, but that doesn’t mean it’s going to be super efficient. There’s a really nice post by Emil Hvitfeldt discussing performance considerations in Quarto.

Purge unused resources

I in-lined many icons to embedded svgs, but, alas, Quarto still loads the bootstrap-icons.css that it does not, as far as I can tell, use anywhere. The solution to this was described in a discussion about Quarto performance by Charles Nepote. Quarto makes it possible to add a post-rendering step to builds which purges unused CSS and minifies JS and CSS (see here). I found that this did a really nice job of reducing the size of resources that needed to be fetched from my site. This helped, and brought the critical path latency down a good bit. After reducing as much of these resources as I could, the critical path is currently loading quarto.js. I think this is probably about as good as it gets¹. Or, at least, as good as it gets without me just basically giving up on letting Quarto manage this at all.

Fonts

Here’s some fun Drew-lore: I’m weirdly into fonts. In the summer before my PhD I read like 3 books about fonts that I marked up more than almost anything else I’ve read.

My font choices

Long story short, I’ve currently landed on the following font styling that I intend to use pretty generally across the web, in documents and on presentations²:

Headers in URW Classico. This is a font based largely on Optima.
Body text in Domitian. This is a font based largely on the venerable Palatino.
Code in Monaspace Argon. This is part of the suite of fonts developed by Github Next. It’s also the primary font I use locally in my IDE. It’s a Humanist Sans font (I have tried to use Serifed fonts for coding, but it just looks utterly wrong to me).
Math in $Neo-Euler$ . When I use math, I want it to stand out as distinct from the surrounding text. I think Neo-Euler does a good job of walking the line between being a little weird while remaining easy to read.

Why humanist fonts matter

These fonts are all strongly humanist in orientation and free³. Despite a brief flirtation with Futura, I have come around to the importance of fonts which embrace more complexity than such geometric renderings. Ultimately, I think the omni-presence of sans-serifed fonts on the web is a reflection of its inhumanity. Good text should have character in style as well as in substance. I think the ubiquity of sans-serifs on the web is indicative of the fact that it isn’t really a place for reading. The medium corresponding to sans-serifs is the microblog, not the book. Bringing back subtle variations in stroke width, text-oriented serifs and other organic qualities of humanist fonts is a (very) small step towards reclaiming the web.

Loading fonts performantly

To use these custom fonts on the web without screwing with performance too much, I used the solution detailed here to make sure fonts can be preloaded and also that they aren’t part of the critical path for rendering. When you load a page here, you might see a replacement background font for a split-second before the font faces are loaded into the CSS.

Remaining concerns

I’m happy with a lot of what has come together. There are a few issues that I think can be improved. I’m considering whether it might make sense to switch to a hybrid system which would use Quarto to build minimal HTML or Markdown documents which are then knit together using a modern web development framework like Astro. This would allow much more granular control over the website and how it loads resources, but could still get the benefits of Quarto where it is most useful (for pre-rendering rich documents full of Python and R code). This is probably a bigger project than I imagine it to be, so I don’t intend to commit to it soon⁴.

I’m also considering whether it would make more sense to switch to a different hosting solution. I’m not particularly pleased with the quality of Netlify’s CDN, which seems to be a bit slow in Europe (I suspect they just have fewer edge locations)⁵. I’m considering whether it might make sense to switch to a system which is self-hosted in a VPS like DigitalOcean/Hetzner (possibly with a tool like Coolify to manage deployment) and then add a CDN like Cloudflare over the top. The benefit to this is that it would combine control (from the self-hosted VPS) with the performance benefits of a reliable CDN through Cloudflare, but it would remain quite cheap (albeit not free).

Tending the garden

There’s something satisfying about all of this tinkering that I find hard to articulate. It’s not that any individual piece is particularly impressive or novel. Rather, it’s the act of construction itself—of shaping raw materials into something that reflects intention and care. McLuhan observed that we shape our tools, and thereafter our tools shape us. The platforms we’ve ceded our digital lives to have shaped us into consumers of content, optimized for something outside of our control. Building my own corner of the web is a small act of resistance against this.

I don’t think everyone needs to hand-roll their entire tech-stack or obsess over font choices (I will tell you fonts are a huge net-negative on my measurable productivity). But I do think there’s value in the process of understanding and improving the systems we inhabit, even if only to understand what we’ve given up by outsourcing them. The web was supposed to be a place where anyone could plant a flag and build something. Somewhere along the way, we traded that for the convenience of renting space in someone else’s domain.

These updates—the automation, the newsletters, the performance tweaks, the typography—are all just ways of making this place more my own. The work is never finished. There’s always another edge to smooth, another inefficiency to address. But that’s the point, isn’t it?

It seems like there are also some other problems, as Firefox and Chrome both intermittently show several seconds to establish a SSL connection to the website (i.e. before any resources are actually loaded). Unclear what’s going on with this… ↩
The ease of writing Markdown documents which are rendered into reveal.js HTML presentations is really nice, and yet another way that I’m aiming at convergence between these media. I ask myself whether I should also move away from directly writing LaTeX, too. Ensuring visual consistency by using literally identical styling directives is very appealing. ↩
We’re Hermann Zapf fans over here if you couldn’t tell. ↩
Truly famous last words. ↩
Making a connection via HTTPS occasionally takes 3s or more, which is just absurd and is by far the slowest part of page-load, although it is also intermittent which makes me suspect something with CDNs. ↩

Back to Basics

Mon, 27 Oct 2025 00:00:00 GMT

I started a Substack not that long ago, because I wanted to get back into blogging. I think blogging is great and microblogs are bad. I’ve had a bit of time to think over the past year about my usage of technology. I’ve come to believe that most platforms are (at best) not really for me, or (at worst) broadly considered harmful. If you previously subscribed to my Substack, I’ve migrated you to a subscription service of my own. If this isn’t what you signed up for, please unsubscribe¹.

I think there are a few dimensions to my thinking that I’m aiming to work out in this post:

Platforms constrain freedom
Platforms enshittify
AI coding tools make smaller projects feasible

I’ll then talk about why I think we should fight against the ease of platforms. Let’s get into it.

Platforms constrain freedom

Platforms exist to make it easier to do things by creating one streamlined avenue to accomplish a goal. A canonical example is Facebook winning over other social media sites like MySpace. It might be hard to recall now, but one of the main reasons for this was that, in contrast to MySpace, Facebook provided a simple, clean, consistent user experience. On MySpace you would arrive on a page, have music randomly play at you and be barraged by weird fonts, colors and images. Facebook constrained how users could express themselves quite substantially, and this was very popular. It let people ignore all this wacky design stuff created by thirteen-year-olds and focus on what they wanted to do with social media: poke each other, write weird status updates and check out relationship statuses².

I think this is a pretty common pattern for new platforms: They do, in fact, generally make something that people want to do easier! Of course, given the nature of this post, I need to address the Substack in the room. Substack is generally a good platform. It makes it easy to write (certain kinds of) blogs and it makes (certain kinds of) monetization easier. The parentheticals, however, are what I find frustrating about platforms. By their nature, they constrain the user to a particular form of the thing they provide.

Scale means that Substack can’t and won’t cater to all of the small, weird features that people want. If I were designing my own blog system, I would want (a) control over exactly the fonts and styles my content is displayed with, (b) easy integration of math via the great tools like MathJax, (c) the ability to have code-rich documents like Quarto in which code runs in-line and displays output in-line, (d) the ability to embed any kinds of content I want wherever I want it. Ultimately, I think this is the dream of the web: I can have a little homestead on the internet that is exactly the way I want it: a convivial web³. Platforms cannot provide this: their raison d’être is reducing complexity as they make human behavior digitally legible down a particular prescribed path.

I dislike this constraint: I want to use tools that work the way I choose for them to work.

Platforms enshittify

Even if you feel well-served by a platform now, this is no guarantee that the platform will continue to serve you well in the future once you’re stuck there. I won’t belabor this point, because it’s been talked about ad nauseum by others⁴. Platforms, as they mature, need to monetize more aggressively as well as build moats to competitors. This often leads to more constraining of user freedom. I think the real “break-glass” moment for Substack is the ability to collect and export all subscribers’ email addresses. This is, right now, a defining feature of Substack that makes it safe for people to use. This drastically reduces their moat, however. At the moment, switching to Ghost is (technically, at least) pretty trivial. Switching to a Quarto blog deployed through Github Actions and which sends a newsletter through Listmonk (deployed via Pikapods) when triggered through Github Actions is not trivial, but it also wasn’t exactly hard to setup⁵.

The main way that Substack appears to be aiming at constructing a moat is in their push for discovery. That is, they want to make it easy to find Substack writers on Substack. They have no interest in doing this with writers hosted elsewhere. This provides an incentive for people to stick around, even if the actual writing and publishing experience is worse. I don’t think Substack is made up of evil people, quite the opposite, I think they truly wish to help writers make a living (while taking their cut). The problem is that they have few incentives to make it possible for writers to truly experiment. Why would they? Writers will want to stick around because they can get more subscribers there, not because the CMS is uniquely good or they can uniquely realize their vision there. The one exception is that many writers’ visions are simply “to be able to feed themselves through their writing”. I think this goal would be better served by increased experimentation⁶, but I recognize beggars feel that they can’t be choosers. I think this is a false dilemma, because:

AI coding tools make smaller projects feasible

Platforms work by making a particular workflow easy enough that we accept lock-in. An alternative path exists, however, which is to build our own tools to support that workflow. A critical fact of contemporary society is that this is becoming increasingly easy. I think a lot of people have, thus far, only interacted with LLM systems through the lens of the chat interface. While this is impressive technology, I think it blinds a lot of people to what is now possible. Let me briefly detail an example.

For my website, I decided I didn’t want to manually write out the metadata for my publications anymore. So in one afternoon, I vibecoded a system which connects to the Semantic Scholar API⁷, checks for new papers on my profile and if it finds them, sets up the metadata in papers.yaml. It doesn’t display until I flip a switch from visible: false (the default) to visible: true. Thus, I’m not too worried about the system going haywire: nothing is posted without me getting the chance to review it, but it reduces tedious data entry. The technology is there to vastly reduce the time between an idea and a prototype of that idea. For this Semantic Scholar system, it took me just a few minutes to set Claude up with instructions, and I had a complete and functional prototype no more than one hour later, most of which did not require me to be thinking about the implementation (verified via commit log).

It’s substantially easier to build things now. As the effort to build novel computing systems shrinks, it’s easier to build exactly the systems you want for yourself (or for yourself and a few others). We should do more of this, on the margin. Individuals’ capabilities to personalize their computing environment has literally never been higher than it now is. Choosing not to take advantage of this is a choice to let other people (who do not have your best interests at heart) choose your tools for you. The point of platforms is to capture you by making it easy for you to achieve particular goals (but restrict you to those goals they deem worthy), but the thing that’s different now is that it’s also easier than ever for you to build what you need for yourself.

Technological vanguardism

Open source software is a critical part of this story, as it serves as the backbone for building any kind of new software in the current era. This is recognized in a variety of initiatives as part of EuroStack⁸. That is, the idea is that Europe can’t or shouldn’t try to compete with American and Chinese hyperscalers by trying to incubate European hyperscalers as competitors. Rather, Europe should commit to smaller alternatives based on a robust shared digital infrastructure of open-source software. AI-supported individuals able to mix-and-match together the computing systems they want to support their lives is a rare hopeful vision of the future.

A lot of the maintainers of critical open source projects are getting older⁹. I wonder whether the next generation will have the same commitment to the principles that underlie the open source movement. In particular, we’ve all discussed how “the youths” increasingly don’t understand file-systems or whatever your preferred concern is. My worry is that they have become so used to an Internet made up entirely of platforms that they can’t even imagine anything different: an Internet where when you want to create a space to chat with friends or colleagues the only option is Discord or Slack controlled by someone else, never hosting a Ventrilo server, a phpBB forum, an IRC server or, in modern terms, software like Zulip or Matrix.

Extensible tools like this often have a higher entry cost than platforms do: they may require standing up servers, writing some code or doing some non-obvious configuration. This has been a major problem with Open Source Software for ages: they’re built by developers for developers. Open source developers and maintainers of open source projects are a sort of technological vanguard who accept some of these costs on themselves. They build tools, often just for fun, because they can and it scratches a weird itch they have in their own lives. Sometimes, these projects fit the needs of the moment and take off, but many times they remain personal or niche. It’s important that the people who can build such tools do so to pave a (code)path for others to follow, but they are not usually about making that path highly accessible.

The costs of building one’s own systems are lower than they’ve ever been because of AI code assistance. This provides a critical link between the Open Source Software ecosystem (which is the fundamental infrastructure underlying modern software of all kinds) and personal use. That big open source project which was almost right for you? It’s not hard to make minor modifications to it now to make it just right.

The problem, however, is that every person who accepts Platforms into their hearts does create a social cost as well. If everyone only goes to Substack to get blogs, then blogs hosted elsewhere (no matter their quality) will be read less. This is what Substack is hoping to achieve, and I think it’s important for those of us comfortable with forging a different path to try.

I’m a technological determinist: I believe that the technologies we create fundamentally transform our lives and societies. I also believe strongly in the power of human agency: we can reshape the technostructure to harness these technologies the way that we want. We can build the society we want, but that does mean that we need to actually do the building. I, for one, am planning to become much more serious about putting in the work on this. Moving this blog back to my own website is a first step in this direction, but it is not the last.

Image source: [https://brooklynrail.org/2017/10/criticspage/Hidden-Figures/](Papilla Estelar: Remedios Varo)

The link is at the bottom of the email, or let me know at blog@ddimmery.com ↩
Or “connect the world” depending on how you feel about Facebook ↩
Note also the importance of open-source software in what I’m looking for. ↩
Apoorva Lal has a nice formal model of this ↩
I’ll write a rundown of how I did this in a later post. ↩
crypto! micropayments! easy ways to bundle or unbundle author’s writings! different models of payment beyond mere “subscriptions”! ↩
I am increasingly a fan of the work Semantic Scholar is doing as a more palatable alternative to Google Scholar. Unfortunately, they are probably doomed to failure because Google Scholar uses an expansive definition of citation that means their numbers are bigger. Even as a fan of Semantic Scholar, I definitely provide the Google Scholar citation count and h-index for annual reviews. ↩
Of course, EuroStack is a bit of a classic European idea in which everyone who talks about it means something slightly different than one another. Nevertheless! ↩
In some sense, we all are. ↩

What's the point of RCTs?

Thu, 13 Feb 2025 00:00:00 GMT

Kevin Munger and I recently had the joy to write a response to “Nonparametric Identification is not enough, but randomized controlled trials are” by the all-star team of Aronow, Robins, Saarinen, Sävje and Sekhon (ARSSS). You can tell Kevin and I like this paper because as of this moment, we make up about one quarter of its total citations on Google Scholar. I believe a number of other responses will be coming out soon; I’m very excited to read them.

I’m going to riff a bit more here about our response, “Enough?”. In particular, I want to dig into what I see as the most important perspective we articulate in the paper and some of its further implications. Namely, we argue that the biggest value of experiments is ontological¹.

How to respond to a great paper

This was my first reaction to thinking about how to do this. I love the paper, and I refer to it a lot. But a response isn’t that interesting if you’re just saying “ditto”. We started out with Kevin wanting to talk more about how uniform consistency of the SATE was still insufficient for science (plus various other big picture philosophy of science points) and me wanting to discuss different reasons why RCTs are special (e.g. correct specification through shoe-leather)². Regardless, we quickly latched onto the most provocative word in the title: “enough”. What does it mean for a method to be “enough”? Musing on this formed the core of our response.

Enough for what?

We started off by trying to think through what it would, conceivably, take for an estimate to be generalizable knowledge. We identified 4 things that we certainly want to be able to generalize over: (1) sample, (2) site, (3) realization of a theory, and (4) time. The first two are fairly well-trodden. Convenience samples may make it hard to generalize to populations of interest, and site-selection bias may make it difficult to generalize to the locations we’d like to know for future policy choices. Tal Yarkoni has already talked through the multiple challenges of generalizations across various dimensions in the language of random effects in Psychology.

The third element is kind of weird. What exactly is the set of treatments we could have defined based on a particular theory we’re trying to test? I have a bit of a pre-occupation with this question because it was my experience running message-tests at Facebook that while psychological mechanisms may be important, so are the specifics of wording that are largely unrelated to the mechanisms. If you write text that sounds like a robot, it’s gonna suck even if you’re pulling on the right mechanism. If we want to generalize the results to the theory being good or bad, then you have to be able to abstract away from this.

The final point, on time, is a long-time hobby horse of Kevin’s. One of the main criticisms of this focus is that it isn’t actually any different than other forms of generalizability. We identify one extremely important way that time is different: there’s no ‘shoe-leather’ based solution for it. You can random sample from a population, a set of sites or from potential treatment definitions. You cannot randomly sample time. This is an extraordinarily important difference, because it means, ultimately, that you must rely on modeling assumptions that may not be necessary in these other cases (for exactly the reasons ARSSS spell out: randomization is an extremely powerful tool to eliminate such assumptions).

I strongly believe in the power of shoe-leather³. The fact that it is insufficient for temporal generalization should be concerning⁴. Regardless, our point is just that randomization is good, sure, but it doesn’t get us to where we want to go scientifically. What does makes it good?

Control of what?

I think the best part of the paper focuses on the ontological value of experimentation. That is, experiments create novel states of the world, and this is a powerful way to imagine alternatives to the social reality we currently inhabit. It creates incentives for people to create cooler and more ambitious changes to the world while measuring what those changes do. But I think this gets at another reason that RCTs are particularly useful.

Observational methods assume a complicated ontology in the course of creating a DAG: how to chunk up the social world into nodes and how to assign values for those nodes to each unit. Put simply, the social world is extremely complex, and this process is subject to extraordinary error. RCTs circumvent this difficulty entirely: their accuracy does not depend on such ontological assumptions. Instead, they impose their ontology. In the paper, we focused on the ontology of the treatment, but the argument is much broader and deeper.

For example, suppose that we run some field experiment that changes substantial aspects of people’s media diet and we measure effects on various constructs that we think are interesting: ideology, perhaps, beliefs about facts, etc. It isn’t just that the RCT imposes the ontology of treatment and control, the RCT provides information on all of these constructs that we create (i.e. causal effects on these constructs). In survey research a common recommendation is to avoid regressing one survey construct on another⁵. But the truth is, we do this with all observational research, as we make assumptions about constructs everywhere. With the RCTs, we only do this on one side of the equation. The treatment ontology is imposed, but this allows us to weaken the assumptions of ontology in the outcomes. We will still have valid causal effects on whatever ontology we choose.

Or maybe in our field experiment, we measure something collected administratively about behavior (e.g. turnout in an election). In this case, both the treatment and the outcome have some ontological precision in their meaning⁶. What about the heterogeneity we may observe? Well, we don’t rely on the ontology the same way we do in observational settings: to take an ontology that is currently the subject of political contention, consider measuring heterogeneity by gender identity. There are different ways to construct this ontology, but when we choose such a construction, we can simply measure the heterogeneity according to this ontology. If we’re instead in an observational world, we would need to care much more about getting this ontology “correct” in some way as it pertains to selection into treatment⁷. In an RCT, we do not rely on this correctness. We can explore variation with respect to whatever ontology we choose and, perhaps, even make judgments about which ontology better reflects the variation in response.

RCTs give us a solid foundation from which to explore these questions, while observational work cannot do this so easily.

For whom?

Critically, however, the ontological power of experiments is premised on control of the world. It is power not just in the statistical sense of charts and numbers, but in the sense of guns and laws. Wielding this form of power requires responsibility and humility.

Experimentation can be a tool for achieving our country. By trying out new social possibilities, we can provide the foundation on which a better society can be built. Doing this requires venturing past is and into ought. This is fraught territory for scientists, so must be part of a larger democratic process. As the fabric of society is increasingly constructed of bits and code, we can increasingly implement creative new interventions that are meaningful to people’s lives. I think, in fact, we’re ethically obligated to do so.

By making our focus broader in this framing, we do something that ARSSS could not: draw distinctions between types of experiments and demonstrate the value of experimentation within a larger experimenting society. All experiments enjoy the statistical benefits that ARSSS describe. Not all experiments are similarly powerful ontological tools for considering new social worlds. The experiments that measure up best under this perspective are the ones that attempt ambitious changes to social reality. The experiments that do poorly are, mostly, what we have: survey experiments. These are, surely, not enough.

Read the paper and let us know what you think!

You may have read some about this perspective on Kevin’s blog. ↩
Chris Harshaw has a response to ARSSS that goes more in this direction focused on the importance of the design-based frame of thought (and its rhetorical power). ↩
If you’re reading this blog, I hope you’re already familiar with the beautiful David Freedman paper on this subject, Statistical Models and Shoe Leather. If you aren’t, read it immediately. ↩
Shoutout to the GOAT. ↩
I’ve done a little searching, but I’d be interested in some citation archeology / intellectual history. Where does this recommendation originate? ↩
Or at least something like phenomenological precision. ↩
If you’ve ever read into the literature on measurement error of covariates for observational causal inference (its a nearly intractable problem), this should make you sweat. ↩

What was US2020?

Wed, 29 Jan 2025 00:00:00 GMT

The US 2020 Facebook and Instagram Election Project (US2020) was a sprawling project with dozens of people involved at various levels: like little ol’ me (on the Meta side). Meta spent more than $20 million on it and committed the time of numerous experienced data scientists and engineers. It was huge. But it was also very small compared to the size of Meta.¹ In this post, I want to reflect on what, exactly, it was, particularly in light of the eLetters to one of its marquee studies and Kevin Munger’s nice article asking what we learned from the project. The fairest reading of the project is the result of a dialogue between internal and external researchers, so that’s the direction I’m going to work towards.

The idea here is that I want to try to develop a theory of what this project was². That is, why does the collaboration exist? Should it exist? If and when we want to do Big Social Science like this, what will we be doing? What should we be doing?

To make the argument that US2020 is best understood as a dialogue, I need to start by talking through what and how large Facebook (the company) was and how experimentation works in that context which is defined by the scope of the collective endeavor that is Facebook (the platform). After that I can address what I see as fundamental misperceptions about the nature of US2020: the idea that the project was either industrial research or independent research. It wasn’t, and trying to fit it into one of these molds leads to many confused discussions about the research.

Independence in future projects should not be the goal, but this does not mean that we need to sacrifice reliability or trustworthiness. Quite the opposite. The very human trustworthiness of the project is as a result of the combination of the non-independence (of the internal researchers) and independence (of the academic researchers). The* tension* between the two of these was what made this project valuable. We can accept that this tension will exist, but resolve much of it through the use of pre-analysis plans. This will better allow us to express the agency and expertise of everyone involved in the project and, hopefully, make future projects more likely to exist and be better.

So, where to start?

What was Facebook?

Facebook was an internet technology company from 2004 to 2021. It no longer exists, having been succeeded by “Meta”. At its peak in 2021, it had 70,000 employees and 118 billion in revenue, truly staggering numbers that are difficult to comprehend. Around half of those employees were in technical roles³: they directly worked on changing and improving the functioning of the services that Facebook offered. Can you imagine working on a project with 30,000 other people? It’s critical to understand that this is part of what it means to work at Facebook. And it really felt like this, sometimes! Internal tooling changed often (because people were actively working on it!), the use of a Monorepo meant that all engineers pushed code to the same repository and there were meaningful connections in code between projects. To land a commit at Facebook famously only required you to get one other engineer to sign off on it⁴. The number of commits to the repository was so large that you couldn’t expect to have the same ground truth from one second to the next⁵. For a sense of the speed at which Facebook changed, read this post detailing some of the deployment systems Facebook used written by Chuck Rossi, a famous⁶ release engineering guru at the company.

Experimentation at scale

Facebook changed quickly. Temporal validity is a real problem when things change at this pace. But the problem was actually even worse than the sheer scale would suggest! Many of the biggest changes to how Facebook work were gated not by commits of code, but by A/B testing⁷. The number of active experiments at any one time was on the order of thousands. Even assuming each test had only two groups, this number implies that no two people were likely to experience the same Facebook⁸. I wouldn’t be surprised if, given the increase in employees relative to when we roughed out this number in around 2016, that number was an order of magnitude higher by 2021⁹.

It’s worth thinking about how these experiments worked a little bit. One thing the handful of people implementing a single test absolutely could not do was to control everything about how the other 30k engineers run their tests. This is simply the reality of working on a project with this many other people.¹⁰ This led to difficulties occasionally, in which stuff about the platform changed and rendered a prior test uninformative or incorrect about the current state of the world. On my team, this is why we encouraged two practices: (1) long-run holdouts (where for important features, the rollout of the feature would be to ~99% of users rather than 100%) (2) rerunning important tests to verify results continued to hold.¹¹

Everything is evaluated relative to the sum total of other stuff Facebook is doing on the platform.

The reality of massive simultaneous experimentation and change resulted in internal experiments using the term “status quo” condition rather than “control” condition in experiments. Everything was evaluated relative to the sum total of other stuff Facebook was doing on the platform. As an example, check out the software my former team developed for doing complicated experimentation at platform scale. There is no “control” group, there’s just the “status quo”: what the platform is currently doing. The other groups are potential modifications of that behavior. The reason we evaluate relative to this group is because experimentation is a means of optimization¹²*. *Experimentation was a tool to improve how the system worked, so we wanted to evaluate our one change relative to the otherwise current state of the system. We never care about comparing to the state of the system 1 day or even 5 minutes ago.

This is one reason why I welcomed the eLetter by Bagchi et al at Science: it’s important to remember that “Facebook” (the platform) is a moving target¹³. In short, during the course of US2020, the company implemented a number of other measures aimed at combatting bad behavior during the election (“break-glass” measures). A lot of people forget how large and diverse an organization Facebook (the company) is. The “break-glass” measures discussed in the eLetter are a great example of this. One hand certainly can’t control what the other hand is doing, but it often isn’t even aware of what that hand is doing¹⁴. Unfortunately, I think this important message is undercut by another one in the eLetter:

This can lead to situations where social media companies could conceivably change their algorithms to improve their public image if they know they are being studied.

This implies a single-minded agency of the daily practice of Facebook engineers that is unreasonable (and largely impossible). As I tried to emphasize above, the company is huge, diverse and fast-moving¹⁵. The vision of a monolithic company exercising its will is simply inaccurate, especially when considering a project as small as US2020. Lots of things were changing during the course of this study; that’s (a) entirely normal at this scale and (b) why it’s important for us to be humble about its temporal validity (which I think we were)! The thing to be angry about here is not that the company tried to tweak the results of US2020 to make them come out hunky-dory: the thing to be mad about is that we have no truly reliable measures of the effects of these break-glass measures!¹⁶ We need more testing, not less.

The thing to be angry about here is not that the company tried to tweak the results of US2020 to make them come out hunky-dory: the thing to be mad about is that we have no truly reliable measures of the effects of these break-glass measures!

Was US2020 independent or industrial research?

This leads to what I think is a more important question about the nature of this project. Was US2020 industrial research, guided primarily by the needs of Facebook (the company)? The answer is clearly no. The academics proposed the designs, they had final control rights on the resulting papers and their salaries were in no way dependent on the results of these papers¹⁷. This is all critical, as there is an important difference between the Facebook and academic researchers in the project: Academia has academic freedom, while industry researchers do not. Does that mean that these papers are actually independent, then? Of course not. Seeking independence is a category error for this project. The internal researchers have incentives, as do the academics.

Fundamentally, these parties have different things they bring to the project: internal researchers knew how the platform works in detailed ways that no external person could and they had the capacity to actually work with internal platform data. They had been studying the ins-and-outs of this platform full-time for years. External researchers, on the other hand, were not burdened by corporate incentives and they had the academic freedom to speak freely about the project and their beliefs¹⁸. All of these perspectives are necessary for a successful collaboration, but it would be incorrect to suggest that they were equal partners. They cannot be equal, because they contribute in radically different ways.

The best one can hope for is to reconcile competing interests by writing down what everyone wants from the project, having an argument about which things can be agreed upon and then choosing to do those things or not. We already have a great tool for this kind of collaboration¹⁹.

Pre-registration as a contract

The solution, I think, is setting out a clear contract between the parties at the start of the project. We actually already know what these are: pre-registrations. The pre-registration serves as a contract about exactly what you plan to study and how you plan to do that²⁰. Filing it publicly in advance can then tie your hands and commit you to doing what you said. While problems inevitably arise post-data, it’s easy to file amendments as necessary. Cyrus Samii wrote about how he (and EGAP) sees pre-registrations that I think relates very nicely to this view, as he talks about the value of pre-registration as a conversation about the research you plan to do. It very much strengthens the quality of the work you will end up producing!²¹ When parties are collaborating on research and the parties have different incentives, a pre-registration serves as a Memorandum of Understanding. It gives you a specific document on which you can have an explicit negotiation about what you will do in a transparent way. For example, if the parties can’t agree on a definition of the ‘control’ group, then it must be redefined to reflect what can be agreed on. If in the course of executing the project, one side makes a unilateral decision about something substantively important that would not have been agreed to by the other side, then this is an error in the contract for lacking specificity over something important. Take the lesson and apply it to the next project.

This is the most critical part of the design of US2020. Take expert internal researchers who have a clear conflict of interest (but know more about the actual workings of the platform than anyone outside it) and combine that with independent researchers who are not beholden to the company and are only interested in learning something about how it works. Let these two sides negotiate on what would make a mutually beneficial research project, write it down, do that project²². The appropriate reading, then, of these studies is as the result of a dialogue²³. And to the point of the eLetter which sparked this meditation, the pre-registration documents we put together make clear our choice of comparison group which is exactly why these documents are so valuable.²⁴

Early on, it was decided that the project should speak with a single voice. This makes sense given how an academic paper typically works with only a couple authors: consensus works, and every author can speak their mind about the paper. In my opinion, it doesn’t work for a collaboration as large and diverse as US2020. In US2020, only the academics have the freedom to say what they think, and the design of the project means that they are always the ones who have the final determination over what gets written. In practice, this almost always resulted in decision by consensus. There was really very little disagreement because everyone sought the best scientific result. That said, disagreements existed. For example, from Michael Wagner’s report as the rapporteur of US2020:

Specifically, Meta researchers wanted to be able to expressly disagree with lead author interpretations of findings or other matters in articles they coauthored. Stroud and Tucker opposed this effort, noting that collaborators can remove their name from a paper if they have a fundamental disagreement.

From the way the project was designed, this was the appropriate decision (downstream from choosing to speak with only a single voice), but for the design of future projects, I vehemently disagree with this structure. All members of US2020 are expert scientists, many of whom have different perspectives. The remediation for disagreeing with anything that appears in one of these papers is, for Meta researchers, to entirely take their name off of a project which they have poured, likely, hundreds of hours into: the nuclear option. Much better to have some forum (an appendix?) for authors to include a personal statement clarifying their personal perspective on the paper and how that diverges from the main text²⁵. This acknowledges that no written document can perfectly express the views of dozens of scientists, and provides far more information about what the project shows and what may be debateable matters of interpretation. Anyone who has talked with a scientist knows that if you put two together in a room together you’ll have at least two perspectives on any topic under the sun. It also better expresses the reality that everyone involved in the project is a scientific expert: not merely those in academia. This sets up what I think is a common misapprehension about the project.

Facebook researchers were not robots

In the editorial published along with the eLetters, Holden Thorp and Valda Vinson wrote:

In a statement for this editorial, Chad Kiewiet de Jonge, a research director at the company, Meta, insisted (to H.H.T.) that it had been forthcoming about the emergency measures to the researchers.

Reading this, it might surprise you to learn that not only is Chad an author of the published paper in question, he’s in fact one of the lead authors. That is, he has been in all of these meetings where such things would be discussed! To quote from the detailed list of contributions:

N.J.S. and J.A.T. were joint principal investigators for the academic involvement on this project, responsible for management and coordination. C.K.d.J., A.F., and W.M. led Meta’s involvement on this project and were responsible for management and coordination. [emphasis mine]

Frankly, this has been a problem from the beginning. As I said immediately on publication, “This project was a deep collaboration, and generally, I think, not an adversarial one, either. Everyone wanted to get this right!” Nevertheless, it hasn’t always been presented like this. If you read the project as a dialogue between internal and external researchers (the accurate view, I believe), the only way that this could possibly be “independent” as in the standard set by Michael Wagner is if the Meta coauthors were robots who merely carried out the precise will of the academics²⁶. This is clearly bad for the scientific output, and it isn’t what actually happened, as Wagner notes:

On occasion, this, and advice they received from Meta researchers, led the outside academics to reshape what it is they sought to study, or how they sought to study it, as they learned details about what data were collected and structured by Meta for analysis. For example, the outside academics were not precisely aware of how Facebook groups were joined and, in some cases, left, by users. Coming to understand this led to changes in the working paper examining behavioral polarization.

And yet, there is an implication that Meta researchers were trying to hide things, as in the quote he shares from a former employee saying,

Facebook researchers will answer every question they get from the professors honestly; they are ethical professionals. However, they are also corporate employees. If the professors don’t ask the exact right question, Facebook staff won’t volunteer that they know how to get what you (outside academics) are really asking for.

It’s very weird that this is attributed to a former employee rather than either the academics or to anyone involved in the project. From everything I saw, Meta researchers didn’t just narrowly answer direct questions, but bent over backward to try and re-interpret requests to get at what the academics’ questions intended. For my part, on one paper I spent a lot of time on, I wrote dozens of pages trying to redefine an unworkable analysis an external academic suggested to one that would be workable — while preserving the research questions. Other Meta employees did similar work on basically every single paper. That work is, in effect, an act of persuasion aimed getting the academics, who have the control over the content, to adopt the change or redefinition based on their view of the scientific merit. To what extent was it based on corporate incentives and to what extent was it based on scientific judgment? Likely a combination: I believe more of the latter than the former. But the negotiation of the pre-analysis plan is the way that this is adjudicated, and in US2020, the external academics were the only ones with the power to decide.

I think the problem is that people can’t help but see Meta as a monolithic organization which exhibits a single will. This is seriously incorrect in almost obvious ways: employees disagree about, well, basically everything! Dan Davies’ book on the flows of accountability through the cybernetic lens is a useful correction to this vision. The company “Meta” is distinct from the sum of its individual parts: the procedures and processes that it uses to determine how to operate are largely not under the control of any singular person. In short, it is an organization and this imposes constraints on employees. And yet, journalists and editors of Science seem to have trouble understanding this. Internal researchers must seek permission to do this kind of research (a heavy constraint), but conditional on permission to do so, there is no instrument of Facebook’s will hanging over every discussion and decision they make. This is simply not how information is transmitted within the company²⁷.

Specifically on US2020, employees fought for years to enable this kind of research. There was extreme skepticism among executives precisely due to a belief that there’s no way that the project could be “good” for Facebook. The project exists despite the fact that Facebook (the company) was not particularly in favor of it. Rather, the researchers you see listed as coauthors are on there specifically because they thought it was important to do rigorous, transparent social science about the effects of social media. As a personal anecdote, I was advised against participating in the project if I wanted to optimize my career at Facebook: it’s much more professionally rewarding to make the system work better than it is to do this kind of evaluative work²⁸. That said, I don’t mean to paint all internal researchers as exactly like people who choose not to work for Facebook. The internal researchers are unlikely to have seriously negative beliefs about the platform, so they may demand stronger standards of evidence for harm than might be acceptable to people more willing to see the inherent evil of the company.²⁹ That’s why you have the argument before you see the evidence: in a pre-analysis plan.

What does the future hold?

I think there’s a real danger in misrepresenting what this project was, but that danger isn’t about falling prey to Facebook’s manipulation. Instead, the danger is that there may never be something else like it, even though it’s the single best way we’ve managed to study the crucial question of how social media affects societal outcomes. I think that collapsing all of the internal researchers into a single “Facebook” blob risks poisoning the well. This project exists only because of individual people—many working within Facebook—working hard to make it exist. Mark Zuckerberg, or any other embodiment of Facebook (the company), did not ask for this project. He did not push for it to exist, and he certainly did not ask for the rigorous structure imposed on how the collaboration would work. The right way to think about this project is that Mark Zuckerberg was willing to humor the desires of people within his organization for rigorous social science. After all, how much worse could the results be than what people already believed?

For my part, future research collaborations I have with industry researchers will not pretend to independence. Where possible, I will manage conflicts of interest with pre-analysis plans (advocating for what I see as scientific merit), but I think aiming at subordination of these researchers is ultimately self-defeating. When interpretations of results differ, I will resolve disagreements through an appendix (like a “deviations from PAP” section) identifying the different camps of belief: thus bringing it out into the open of scientific discussion. I will bring the dialogue to the forefront, not hide it out of a misguided (and impossible) search for independence.

By far the most important lesson for social science when faced with a complex entity like the Facebook platform is that we should not be content with a single snapshot about its impact. Acknowledging the size and scale of the engineering effort behind this company must mean acknowledging the necessity to run and rerun tests about important potential interventions.

Would the results of the chronological feed experiment hold in a different election? Without the break-glass measures?³⁰ In a different country? The only way we can ever know is if we run more tests. The only way we can run those tests is if we understand what US2020 was and try to improve it so that the next version is better. Either way, when I was at Facebook, we would never leave something un-tested and un-optimized if we thought it was important. Maybe the only way we’ll get such tests is if governments require them.

On October 29, Senator Markey sent a letter to Zuckerberg urging further independent research on the societal effects of Meta in the vein of US2020. A Meta spokesperson just pointed back at the work from 2020 (much of which is still yet to be published), as if this is the final word on the subject. There won’t be a US2024, and it looks like there won’t be a US2028, either.

Kevin and I were noodling on what kind of commitment US2020 was on the scale of Meta, and he came up with the following: “Millions of dollars and thousands of hours by highly-paid employees — let’s go nuts and say it cost them $25 million. This is compared against $134 billion in revenue for 2023. That’s .019% of their annual revenue devoted to this 6-year project.” [source] This is a huge commitment to social science research, but this is simply not a large scale project by Facebook standards. Reality Labs burns like $15 billion per year.In contrast, financial institutions commit around 15% of their employees to compliance. ↩
See Kevin’s post on this, I think it’s a critical standpoint for understanding modern scientific production. ↩
I can’t find a definitive answer online, but a Quora answer quotes 58% and Gergely Orosz cites 43%. 50% seems like it roughly jibes with my intuition. ↩
The number of major SEVs (read as “major bad thing”) caused by a commit reviewed by a colleague with only the four letters “LGTM” as the review is… a lot.Note: if you do this in “someone else’s code” they are liable to get extremely mad at you. ↩
A lot of the blame for this falls on bots that do “automatic” commits. ↩
Infamous? Certainly had some of the best war stories of deployment at early Facebook I received in bootcamp. ↩
This also led to plenty of bugs that only popped up in particular combinations of features gated by A/B tests, code, and a variety of other technical means. ↩
(2 treatments per test) ^ (number of independent tests) is much larger than 8 billion for any reasonable number of independent tests. There are a lot of independent tests at Facebook due to the “universe” concept (the same basic idea described by Google here). ↩
If A/B tests scale linearly with employees. There were 17k employees in 2016 and 72k in 2021, an increase of 4x. ↩
There are, of course, some exceptions: global holdouts that all employees have to respect, or particular features which interfere with each other strongly. These are both handled with technical solutions. The former by allowing global restrictions on who can be put in experiments and the latter through the use of “universes” (for more, see the paper on PlanOut, for instance) that ensure that users cannot be put in multiple conflicting tests. ↩
If this post wasn’t already way too long, I would provide some examples. One notable one was when, between iterations of text optimization, the translation systems changed, vastly affecting our results. ↩
This perspective, more than maybe any other, was key to what our team was good at. Figuring out what metric to optimize (metric definition) and what levers to pull were often (treatment definition) more essential than the ML toolkit we used for actually doing the optimization, which was fancy and useful, but also ultimately subservient to the problem definition. ↩
They frame this as an issue of “validity”. In our response, we note that they are only referring to external validity, but I find the term misleading in general as Kevin and I discuss in our working paper on partial identification and temporal validity: This word has the unfortunate implication of being binary; computer login passwords and driver’s licenses are either valid or invalid. To say that a driver’s license is “mostly valid” is to say that it is “not valid.” Scientific knowledge is not binary, and while most practitioners can successfully keep this reality in mind when discussing “external validity,” the term introduces unnecessary confusion.And as far as temporal validity goes, as I quoted in my first post on this newsletter:Right now, about 15% of content in a person’s Facebook feed and a little more than that of their Instagram feed is recommended by our AI from people, groups, or accounts that you don’t follow. We expect these numbers to more than double by the end of next year.The current News Feed on Facebook is simply not the same thing we evaluated: the company moves too fast for that. If the only research we deem valid is research which is perfectly temporally valid 4 years on from its setting, we will have no evidence-base on which to make policy at all. This is an important problem we must contend with as a question of meta-science. ↩
It’s hard for me to remember, but despite being an employee and working on US2020 I think I only heard about these break-glass measures through public reporting, rather than from internal communications, but I generally worked on methods stuff rather than specifically on applied civic stuff. ↩
It is not without reason that employees have been described as Chaos Monkeys. ↩
As far as I know, these measures weren’t even implemented in a way to test things internally. but I could be wrong about that, I have no special vision into this. ↩
I suppose you could say that the results would change the amount of prestige afforded to them. If this is the case, would the largely null results we found have been the best for their careers? ↩
Unlike other participants in US2020 from the Meta side, I actually now have the benefit to speak freely, so I’ve got that going for me. ↩
I’m specifically not referring to this as adversarial collaboration, because I don’t think that’s an accurate reading of the reality. It really wasn’t that adversarial, because the internal researchers also wanted to know the answers to these questions. We didn’t collect information on things like race or politics as a matter of course. ↩
Note that my argument here is completely divorced from the statistical arguments in favor of pre-analysis plans. ↩
Note that this justification isn’t a narrow one about getting a hypothesis test to cover at an appropriate rate, it’s about actual substantive concerns of what to study and how to study it. ↩
Of course, no plan survives contact with the enemy and every data scientist knows that the only true enemy is data. ↩
A dialectic? ↩
Some selections from the pre-registrations: The News Feed of users who are assigned to the control condition will not change as a result of the experiment.from the Likeminded exposure PAPUsers assigned to the control condition will see their normal News Feed under existing Facebook policies and procedures.from an as-yet unpublished study’s PAPUsers who have consented are randomized into three treatment conditions (or a control condition shared with other studies).from the Chronofeed PAP (emphasis mine)The second statement, I think, makes this the most clear. Facebook has existing policies and procedures, and the scope of this study will not be changing that. Given the size of Facebook (which I have repeatedly emphasized), I don’t think this should be surprising! ↩
Since the papers were not reviewed by Meta for content, this would give internal researchers some very small measure of the academic freedom that they do not, in general, enjoy. ↩
Academics, mind you, who simply don’t know how internal systems, logging and metrics actually work. There’s no reason they could know this, in fact. ↩
To fully think through the question would require a discussion about how performance is judged at Meta. I think the best way to think about this is that Facebook uses a form of stack ranking, where your manager has about 20 seconds to advocate on your behalf before putting you in a ranked list of your colleagues in the same organization and at the same level. The stuff that will be most persuasive are large impacts of your work on important corporate KPIs. ↩
Because you can measure the value of making a system work better: you run an A/B test and see what your change does. ↩
This is another argument for RCTs, as I think this debate pops up much more clearly in observational causal inference where you have to determine your willingness to balance the potential for bias against the desire to get some answer to a thorny question no matter how flawed. To be explicit, I suspect internal researchers are much less willing to accept any bias, partially due to a strong internal culture in favor of RCTs. Thomas Leavitt has a related argument on this point about the epistemic value of randomization. ↩
Let’s leave a holdout of 0.5% of the Facebook population from receiving these measures next election to find out! ↩

Is Bluesky convivial?

Thu, 21 Nov 2024 00:00:00 GMT

Back in November of 2023, I was lucky enough to participate in a “Morality in Tech” workshop hosted by Princeton’s CITP put on by Kevin Munger. It was a fascinating interdisciplinary group of people trying to wrap our heads around how to grapple with how to tie moral and ethical frameworks to technological change. It culminated in the “Building the Society We Want” syllabus.

This post is a modified version of a talk I gave on the historical development of Smalltalk, and how I see it as a fundamentally different way of thinking about technology than we’re used to today. Unfortunately, I ultimately don’t see it as a way out of the mess we’re in on the social web. Bluesky cannot resolve the fundamental problems with microblogging, which are fundamental to the technology.

What is Smalltalk?

Smalltalk was developed by Alan Kay, Dan Ingalls and Adele Goldberg at Xerox PARC in the 1970s. What it was is a little hard to frame in terms of our contemporary categories around computing. It was more than a programming language and not exactly an operating system as we’ve come to understand them. It was the software part of Alan Kay’s overarching personal computing vision (from back before computing was really personal computing). Smalltalk had a lot of things that are now very common, but were very uncommon when it came out. It was an early iteration of Object-oriented programming¹, it had an early graphical interface, all code was just-in-time compiled (and could be recompiled on the fly), and it integrated a development environment into how the user interacted with the system. But Smalltalk was also a philosophy of computing:

look for the distinction between a programming language and a programming system, and consider the difference in providing a system in which the user can feel individual mastery over complexity

(emphasis mine) The intention was to make the vast complexity of a computer system something that end-users could control. The system should allow the complexity of the system to be comprehensible to the user, and allow them to adapt and reorient it at their will.

A lot of what I’m drawing on here is from the 1981 issue of BYTE Magazine in which PARC chose to write a public introduction to the newest iteration of the Smalltalk system. It’s available online and a truly fascinating read that I can’t recommend enough.

Smalltalk was intended to give the User the power of God over the computing system. This isn’t really subtext. In this introduction to Smalltalk, they provide the following illustration:

On the far left, the System Programmer (God) creates a system (the Taj Mahal). The Programmer creates things (a bridge) for the end-user using the things provided by the System Programmer. Smalltalk was intended to democratize the power of creation, moving into the second figure from the left, in which the Programmer creates kits that the User can use to construct what they want. In Figure 3 this goes even further, as Users can create the kits themselves that they can assemble into what they want. On the far right, however, the idea is that Programmers / Users should be able to climb up above God and impose their own will over that of the System Programmer: to become God.

It’s a vision of a radically empowered user who can manipulate every aspect of their computing environment. In my mind, this vision is exactly what it means to have a “convivial” tool in the sense of Ivan Illich: tools which empower human liberty rather than constrain.

There are costs, however, as Smalltalk asks much more of users, as Kay and Goldberg write in 1977:

The burden of system design and specification is transferred to the user. This approach will only work if we […] allow ordinary users to casually and easily describe their desires for a specific tool.

But the benefit is a fully “reactive” system². What did this look like?

Essentially, you had a browser like the above, through which you could navigate through your system, then the panel at the bottom allowed you to edit the code defining how that part of your system worked. You could then choose to re-compile that part of your system and it would automagically update to follow your new instructions. Voila, complete control over the system, manipulable at runtime.

But there’s one last quote, from Daniel Ingvalls from his article about the design principles of Smalltalk:

Natural Selection: Languages and systems that are of sound design will persist, to be supplanted only by better ones.

Our computing is not based around Smalltalk these days, nor does it look very much like this: it died out in favor of opaque systems with a lack of user-control. It seems that users were, in the end, not willing to take the burden of full responsibility for the functioning of their own computing environment.

Empowerment is still about power

I think the question of inevitability is what I really struggle with here. The dream of a system that radically empowers the user is beautiful. But are users willing to actually take this power? Can they? In grappling with this question, I tried to think about affordances of social media and the extent to which they align with Smalltalky principles.

Bluesky lets users choose their own Feed algorithm.
The hashtag was created as a tool for users to take control of discovery and categorization themselves.
Distributed social media services (e.g. Mastodon) seek to allow individuals to choose the community (and, therefore, community norms) they wish to belong to.
individual choice about network links to create (e.g. friending on Facebook): this is the source of a lot of ideological polarization, rather than something done “to” users by algorithms. In each of these, the user gets pretty profound ability to shape what their social media is. But it’s still not enough! My choice of a feed algorithm doesn’t give me the experience of using Bluesky that I would have if everyone were to use my preferred feed. In order for me to “choose” my experience on the platform, I need the power to choose how everyone experiences the platform. Empowering users doesn’t avoid the critical question of who holds the power to make such changes.

In a previous post, I claimed that there were four main definitional characteristics of microblogging: (1) short texts (2) virality (3) simple engagement possibilities and (4) an infinite feed. The users can’t alter these characteristics on BlueSky or on any other platform. Removing reshares from your own screen doesn’t change whether you experience virality and the associated credibility cascades and everything else associated with virality. To kill virality, you’d need to make aggressive changes like removing the visibility of likes from everyone’s screen, removing ranking based on likes, views, clickthroughs, killing reshares globally and the list of changes goes on. After making all of those changes for everyone then maybe you’ve done a good job of breaking virality. Can you give virality back to a single user, at that point?

It is fundamentally impossible for everyone to alter everything about their social media experience to their whims, while continuing to make it meaningfully social. Even in a world where everyone gets to choose everything about how they interact with social media, they don’t get to choose how everyone else interacts with social media³. Our experiences online aren’t just shaped by our own choices, but by the choices of those around us⁴. We can’t “solve” our experience by changing only our own affordances.⁵ Bluesky can only ever give the illusion of autonomy, it can never provide true control over the environment as envisioned by Smalltalk. That autonomy only works for a computing platform that is fundamentally a-social: it interacts only with you and responds only to your preferences.

Convivial, anarchist empowerment to shape one’s own technical reality doesn’t offer a way out of the shared reality we create with technologies. At best, it can offer a patchwork of small fixes. We cannot fundamentally alter the medium, because the medium is defined by what is shared. The critical question is who has the power to choose the shape of the overall technology, and who gets to study how the world is changed by those choices. Right now, only the platforms have these powers.

In fact, Alan Kay doesn’t actually think modern OOP is at all related to what “OOP” was for Smalltalk. If I had to condense down what I understand he sees as the critical distinction, I’d say that it would be something about autonomy: his vision saw individual “objects” being more-or-less autonomous computing units, which would communicate with one another and this would allow the programmer to create complex systems based on these interactions. Crucially, he saw this as a lot more than a paradigm for writing and organizing a particular piece of software. ↩
“The Smalltalk programming environment is reactive: the user tells it what to do and it reacts, instead of the other way around.” Tesler 1981 ↩
And, in particular, a world in which collaborative filtering continues to play a strong role in what content your network (or other people more broadly) sees. ↩
The most banal and obvious platitude of social science: “other people matter for your experience of life” ↩
Well, you can just sign off completely. And maybe you should! But that’s a different question altogether. For what it’s worth, I’ve basically signed off. I don’t think the bargain is a positive one in my life. ↩

Examining the Apparatus

Tue, 30 Jan 2024 00:00:00 GMT

Kevin Munger has a great article out problematizing the idea of “The Algorithm”. You should read the Mother Jones article and then you should read his Substack¹. They’re both great! And they touch on an important theme I’ll return to a lot, which is about the right ways to conceive of and examine complex systems.

The argument, in brief, is that “the Algorithm” is a misleading frame that elides our participation in the system. He proposes (based on the hottest philosopher of 2024, Vilém Flusser) “the Apparatus” as a better term that acknowledges each of our presence within this greater system which defines much of what we actually mean by “the Algorithm”.

In this blog, I’m going talk about the implications of re-formulating the source of our present weirdness as “the Apparatus”. How is the apparatus constructed? What are the nuts and bolts of how exactly we are implicated in its construction? Shockingly, I’ll argue we need more experiments.

The Apparatus, explained

I think it’s worth talking through how change happens in Tech. I’ll focus on Meta, because it’s the place I know best. This is basically the guts of how the Apparatus gets created.

The problem faced by Tech companies is, put simply “how should we decide what to do?”. There are a number of approaches to this, but I’m going to contrast two:

Build stuff and then ask people how/whether they like it.
Build stuff and then see how/whether people use it. We have to keep in mind, however, that we’re fundamentally talking about making decisions for a product used by (e.g.) 2 billion people. We have to think about how these two approaches would work in that setting, i.e. asking the question “Will it scale?”

The first approach would rely on large-scale surveys. There’s no real way to build up qualitative knowledge about 2 billion people, so we need to be quantitative, and surveys are pretty much the only game in town if you want to get folks’ opinions like this. There are hard limits here, though. At the limit, you can’t (for instance) ask every user how their experience of the site is every five minutes. At Facebook, the standard “cool-down” period after taking a survey was six months when I was there. This imposes hard limits on the precision with which you can actually measure the opinions of users².

The second approach would rely on directly observing behavior and encoding this into meaningful measures about what people are doing. Most of this is just counting things that people are doing³. Because you don’t ask for any of your users’ attention to do this, there aren’t really any practical limits to your measurement of on-platform behaviors⁴.

Given these practicalities, it’s unsurprising that the result is an almost ideological belief in behavioralism: that the only way to understand what users want is to observe what they do⁵. My understanding is that this was part of an inspiring introductory speech that Chris Cox would give to new hires⁶.

The Apparatus, sustained

This process results in a cycle of iterative improvement. Matt Hindman’s book does a good job of describing the power of this cycle. It’s also the basis of the Agile software development philosophy, expressed at Facebook as “ship early, ship often” or, more provocatively, “move fast and break things”.

This saying has, I think, always been misinterpreted. The point is not that you’re okay with breaking things. The point is that you’re okay with trying new things, because you know that you have systems in place for identifying when things break. Many of the neurotic high-performers that get hired in tech are perfectionists, so you need to encourage them to not spend years working on something before launching it and seeing how it works. This should really be interpreted through a mantra of the Agile Manifesto:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Over time, trying a bunch of stuff (some of which might seem crazy ex ante) and watching what users do in response to it is a great way to build a successful product. I think this is one of the defining features of the secret sauce of modern Big Tech.

This perspective flows into incentives throughout the company. Software engineers are, in large part, promoted and compensated based on how well their changes result in user behavior that is deemed “good”. The number went up? Congratulations, you get a bonus. You can break apart this monolith, but this isn’t a bad first order approximation⁷.

The Apparatus, examined

This discussion should make it clear that there’s no sense in which any person understands “the Algorithm”, as Kevin argues. Wouldn’t that be comforting? Stafford Beer had a famous saying, “The purpose of a system is what it does”. This isn’t a reassuring sentiment. We have to work extremely hard to see and measure what a system does. There’s no easy shortcut through “intentions” to understand a system. Even with a truth-serum, we can’t just ask Zuck what he intended to build and then decide if that’s good or bad⁸. No engineer fully understands the entirety of the complex system they’re working on, so we can’t judge the worth of what they’ve built solely on what they intended to build.

So what is the purpose of the apparatus? Well, in one sense it’s essentially the aggregate of the incentives for all the people building the apparatus: making the numbers go up. If we want to understand the purpose of the apparatus, we need to encode the things it does into numbers, and measure how changes to the apparatus result in changes to those numbers. There’s only one consistent way to do that: experimentation⁹. And we need to find the right new numbers to measure, because there will undoubtedly be unintended consequences.

But we cannot forget that the essence of the behavioralism in Tech is that we are those numbers. We are as much a part of the apparatus as the engineers who build the system: every post, every click, every message, every linger over a story is part of the apparatus. This isn’t just true in the sense of our being complicit in the system. It’s true mechanically in the way engineers construct the system. The builders of the system want to give us what we want.

This editorial line of this blog is generally pro-Munger. ↩
It would be interesting to write down, under reasonable assumptions, exactly what this would impose as the minimal detectable effect. It would probably be small (for social science), but implausibly large (for the practical decisions about product made 1000 times a day in tech). ↩
There was a joke at Facebook that this was the main skill of data scientists: we invented a whole job just to do advanced counting. Move fast and count things! ↩
Technical limits exist. Tracking and storing all mouse movement for all 2 billion users would be an absurdly large amount of data collection. More commonly, the practice would be to radically downsample any of this kind of data that is collected. ↩
Of course, there’s also a “revealed preferences” element to this, but even if you ignore that, there’s still a practical matter of scaling. ↩
Somehow, I didn’t get this either as an intern or when I started as a full-time. This is sad, as I’ve heard is was a fascinating romp through the history of technology in a very McLuhan way. Oh well. ↩
I think you can break a lot of work in Tech into three groups (excluding support staff that doesn’t work directly on product): engineers (make numbers go up), UX researchers (make sure we choose the right numbers to go up), internal measurement tool builders (make sure when we say number goes up, it actually went up). ↩
I think many critics would be surprised at the prevalence good intentions here. ↩
There’s an argument here that this is granting too much to behavioralism: that framing the problem in these terms is already giving up the game. I don’t think this is true. A strong argument for behavioralism is that it grants a common epistemological playing field that leads to productive arguments about what should be measured and what should be valued: a regulatory framework. Those are much more productive, I think, than arguing about more basic epistemological points on which there can never be agreement. ↩

Calibration as an HTE diagnostic

Tue, 16 Jan 2024 00:00:00 GMT

In an earlier post, I promised that rather than writing threads about papers, I’d talk about them here. That’s what this post is!

I’ve just had a paper (with Yan Leng) accepted to Information Systems Research titled “Calibration of Heterogeneous Treatment Effects in Randomized Experiments”. The entire motivation of the paper can be summed up in the following chart, showing that the HTEs are not alright:

The data undergirding this chart came from a big (and important) experiment at Facebook which was run around 2019 or so. It is sufficiently sensitive that I (still) can’t talk about what the actual intervention was, but one of the big questions was about how much heterogeneity there was in the intervention. It would have been a big deal if a small fraction of people exhibited very large effects, so we needed to look into whether this was the case. We did this by applying some of the rapidly developing literature on ML estimation of heterogeneous effects.

We threw machine learning at the problem using a bunch of covariates we collected about users’ behavior. We started with the simplest possible model, the venerable T-learner¹. When I was at Facebook, I did a bunch of interviews in which I asked candidates to walk me through their model-building and model-testing process (starting with very simple gut-checks). One of the open-ended questions I would always ask was “how would you know if this particular model was good enough (e.g. to launch to production)?” With ML, you can often give pretty sensible answers to this question depending on what you’re trying to do and thinking carefully about what errors mean². We found this a surprisingly hard question to answer in the causal setting.

How can we have a sense as to whether our HTEs are reasonable? In a standard machine learning setting, you have true labels which you can compare to your estimates to see how well you do at prediction. Even in the case of non-standard prediction models where you only learn about the truth long after you predict, you can use the truth to get a sense of how well you do. A great example of this was when FiveThirtyEight evaluated all of their predictive models they’d ever done. Of course, as you learn in the first lecture of any causal inference class, you never actually observe the true treatment effect: you only have exactly one of the treatment and control potential outcome for a unit.

The standard answer in the causal setting is that while you cannot observe individual effects, you can observe average effects. To bring things back to the motivating graph at the top, we started with the very simple exercise of splitting up our data based on quintiles of the estimated HTEs (from the T-learner) and comparing the average model-based estimates against the standard difference-in-means estimates in each quintile.

The results were pretty startling, as the ML-based HTEs were radically smaller in magnitude than the known unbiased estimates from a difference-in-means. In short, this plot instantly alerted us to the fact that something was going very wrong. As you can see in the plot, these differences cannot simply be explained away as sampling variation in the difference-in-means. Rather, we found the issue to be that the plugin estimators make a fundamentally incorrect bias-variance tradeoff because they’re modeling response-surfaces rather than treatment effects. If you were to have infinite data, this problem might go away, but we’re ultimately never in asymptopia, and we need some heuristics and diagnostics to help us navigate this tradeoff.

The core of our paper is thus to motivate people to look at their data. Have your models actually done a good job of replicating the effects you know you can have some faith in? If not, you need to try and do something else. You can fit a new model (e.g. a DR-learner with tidyhte), or you can use our method to do something basically like Platt scaling, which is what we talk through in the paper: take the aggregated data and find a linear transformation which makes the resulting estimates line up as best they can with the subgroup effects³.

For the linear rescaling approach to work well, you have to make a lot of assumptions. The review process was very focused on demonstrating that this calibration procedure improves the underlying HTE estimates⁴, but I think the best way to think about this is as, fundamentally, a diagnostic method. What we’re doing here is showing you how to look at your data and your model and see whether they align in any way whatsoever⁵. The ultimate plot we landed on (and recommend) looks like the following, which is read basically like a QQ plot:

Note that the axes are scaled by the arcsin. In our settings, at least, this was very important, as estimated HTEs tended to have somewhat long tails (you can see that the vast majority of units are estimated to have effects with magnitude less than 1 or so, but there are still some pretty large estimated effects). To be clear, at the tails of the data, the HTE model understates true effects by around 50% (the difference in the blue dashed line defined by the difference-in-means and the red one from the estimated HTEs). When you care about what your intervention is actually doing, that’s an unacceptable error that you must do something to correct.

In general, I think finite-sample performance for HTEs is vastly underrated in import. There’s some very weird asymptotic shenanigans that happen in some of these HTE papers that I think should be pushed on more. For instance, did you know that causal forests acquire a lot of their nice properties because asymptotically, each terminal node has a constant treatment effect? How often does our data have remotely constant treatment effects within estimated leaves when heterogeneity actually matters? A lot of how well CF will do on your particular data is going to come down to what your true CATE function looks like (just like a lot of other approaches). Luckily, random forests are generally pretty good models, so I don’t think this invalidates much, but it does suggest you need to think hard about model validation. You shouldn’t just rely on asymptotic results.

Another result from our paper I like is that you can exactly characterize the (conditional) bias that you’ll get from fitting a T-learner with ridge regression: it’s purely a function of the regularization parameter: more regularization means more bias⁶. With a T-learner, you typically manage the bias-variance tradeoff on the response surfaces individually, so if you have a very noisy underlying response function, you’ll get a lot of bias even if your true CATE function is very well behaved! If we want to be serious about estimating HTEs, we should take these kinds of finite sample properties seriously. It’s 100% a finite sample issue, as if your data gets big enough, you won’t need to regularize, so you won’t have any bias.

In one of my papers (with David Arbour and Anup Rao), we come at this from a whole different angle, motivating an experimental design based on doing a good job for a particular (knn-based) HTE estimator. We show that one particular view of good design is explicitly trying to minimize the MSE of HTE estimators⁷. I really like this paper, although it’s admittedly pretty weird⁸. In particular, I think the connection between experimental design and the Maxcut problem is a fun insight that makes Kallus’s great design paper a little easier to understand.

Anyway, try talking through all this in a Twitter thread!

There are some reasons to expect that in an RCT a T-learner shouldn’t be too bad: you don’t need to correct for propensity scores (they’re all the same), so you really just want a model to help you extrapolate the tiny amount from nearby factuals to counterfactuals. Soren’s paper on X-learner has some simulations that bear this out. I want to be very clear that I am not recommending that you just stick with T-learners in RCTs, though! I’m only making the very weak argument that it isn’t crazy to think they might do a decent job as a first-cut. ↩
A lot of very good computer scientists completely balked at the question, though. It requires actually thinking about the specifics of the application rather than just looking at an AUC number and making an evaluation. ↩
An alternative approach that may be even better is to use the Chernozhukov et al. procedure. We don’t directly compare to this, largely because our motivation starts from the position of diagnostics rather than estimation. ↩
This was definitely reasonable from their perspective: there’s no sense in which the procedure we’re proposing is a panacea to the problems of HTE estimation (I’m increasingly negative on whether there is such a “correct” procedure). ↩
The GRF package has something they refer to as a “test” of calibration, but I think it’s an important distinction to provide these kinds of tests in a visual way. You will almost certainly turn up more problems when you are looking at more than just a single number (e.g. criticisms of the NHST paradigm compressing all information into a single p-value). ↩
Basically, what you get is the following, using the fact that ridge regression is basically just a linear rescaling of OLS (assuming the covariates form an orthonormal basis). The coefficient vector, β, is the set of “true” linear coefficients of the best linear predictor to the CATE function, λ is the regularization parameter. Technically we’re minimizing a bound on MSE (albeit a tight bound). We have a whole section (4.3 and 4.4) problematizing the idea of optimization here, because ultimately you just can’t know enough about your data a priori to make “optimization” something that fully makes sense. In particular, I like the counterexample we define in Section 4.4 which I think makes the dependence on unknown properties (the conditional variance) very clear. ↩
Footnote 7 ↩
The assignment process is basically not randomized, except up to a relabelling (i.e. swapping treatment and control). There are randomized extensions to this that would probably work quite well and efficiently, but which we haven’t spent the time to put together — feel free to reach out if you want to talk about them. ↩

Plagiarism is bad

Thu, 04 Jan 2024 00:00:00 GMT

I think it’s important to make the simple point that plagiarism is bad¹. I won’t be drawn into specifics of current events which have precipitated this discussion, but there are a lot of academics saying absurd things that I want to publicly disagree with in order to uphold worthwhile norms. This post is less about questions of academic integrity than straightforward questions of what the academic enterprise is.

What are we doing?

First, a digression. Latour and Woolgar (1979) is often credited with the idea that the main thing academics do is publish papers. Our incentive systems are fundamentally built around this idea: if you publish more papers that appear in good journals, you will be more successful. It will cause you to be paid more, have more prestige and, generally, wield more power. I used to think that this meant that what many academics were aiming to do was to simply write more papers. I am increasingly feeling that this is incorrect. Academics want to produce more papers, but they have no interest whatsoever in writing them².

In the course of events, a major perspective has arisen, probably expressed most forcefully like this:

Andrew is a trained economist who does good empirical work in the way trained economists do it. And, as he is telling you here, he does not care that much about his lit review section. To be a bit more blunt, I think the argument is basically that empirical social scientists aren’t writing papers, we’re producing estimates. We produce a couple of these estimates per paper, and they are the only actual information in the paper. We’re forced to surround these numbers with words because of arcane medieval institutions, but we all recognize that this is absurd and nobody cares about the words, only the numbers. The right way to read a paper is to skip all the textual mumbo-jumbo, look at the charts, see the numbers, then maybe read the methods section if their approach isn’t obvious. This is reflected by the way we do meta-analysis: each paper is one dot on a single forest chart.

To be clear, I think this is a heinous misrepresentation of what we should think of ourselves as doing. If what we were doing actually mattered, I might even say that I think this view is evil.

Ok, but what are we doing?

The work of quantitative social science is to take an irreducibly complex system and reduce some aspect of it down to a 20 page pdf. If we have even the slightest amount of humility, we should instantly understand that this is an impossible task. It is an even more impossible task to reduce that same complex system down to a few quantitative estimates: <20 bits of information³. The reason we have 19.9 pages of text around those numbers is for a few reasons:

because we’re trying to be humble by acknowledging all of the uncertainty and context around this impossible task
because we’re trying to acknowledge all of the other human effort that has been put into understanding this aspect of the world before (“standing on the shoulders of giants”)⁴

The idea that either objective is well served by copying from the abstract of another person’s work is starkly offensive. It brings to mind a conversation I was having about Ian Hacking the other day. One of the truly remarkable things he does in his writing is create fluent and meaningful reviews of complex work by other people. That it is so easy to read is because he takes the time and effort to translate that other work to the specific questions and topics he’s aiming to consider. This is a monumental effort! It requires a detailed understanding of both the task he has set before himself (e.g. “what do we know about what science is?”) and each of the distinct efforts before that touch on this question — including all the myriad ways that those prior efforts failed to answer the fundamental question, answered a slightly different question or otherwise went astray.

Is what we’re doing different than this? Fundamentally, I argue it is not. Unless we are completely faithfully replicating an analysis that has been done before⁵, we ought to have some fresh perspective we are bringing to the table: the relationship between the current work being performed and what has been done in the past will not be the same as pre-existing work. I think the discussion about copying summaries of papers back and forth completely misses the task. A much better term than “literature review” is, I think, “related work”. It highlights that what is being done in the section is highlighting connections between works. Why on earth would you just provide a generic summary of someone else’s work in your paper? A good summary of related work should be opinionated and it should deal in comparisons: it should be providing a contribution in its own right as a way of synthesizing existing work to come to conclusions about what is known, what isn’t known, and what the strengths and weaknesses are of the existing body of work.

This is obviously not easy, and for shape-rotators like myself, this isn’t exactly work that is always pleasant. But, like, it’s part of the job, man.

I suppose the counter-argument is that time spent working on related work is time not spent working on the quantitative estimates. That’s true, but there are two major counterarguments: (i) the thing we’re producing is the paper, and the related work section is a meaningful fraction of that: Spend the time to make it better. (ii) Specialization means not everyone has to spread their time among all sections in the paper. If you’re doing a shitty job on one part of a paper, add a coauthor that can do a better job on that part⁶.

In defense of thinking

My point is simple. For historically contingent reasons (not necessarily good reasons), academia has chosen that the primary way that we should communicate is through the medium of the academic paper. Given this choice, I think it’s pretty important that we actually care about what we can communicate through this medium, which has the capacity to share much more than a few quantitative estimates.

An example given by Matt Blackwell⁷ is illustrative:

This was particularly funny to me because I just wrote a paper where this common boilerplate sentence was probably my most highly edited one through the entire revision process. I will soon publish a broader blogpost around this (now forthcoming) paper, but first I will quote the relevant section:

This relationship closely accords with the definition of calibration in classiﬁcation and regression problems (Kuleshov et al. 2018), with the added challenge resulting from the fundamental problem of causal inference: Labels (i.e., ITEs) are never observed (Holland 1986).

The bolded sentence roughly accords with Matt’s example of how people usually cite Holland (1986). Crucially, the boilerplate version is not useful for the specific point I wanted to make in this selection! I specifically wanted to indicate (by my use of the very machine-learning term “label” when referring to individual treatment effects) the challenge this well-trodden conceptual problem poses for simple supervised learning. This required different words. It’s my contention that this is almost always the case. If you find yourself repeating something that has been said many times before, maybe you don’t actually need to say it!

We should therefore not normalize the reuse of other people’s words (suited to one particular context, audience and purpose) and re-use them for our own (different!!) context, audience and purpose. I think it is both true that we have sleep-walked into reusing text pretty commonly, as in Matt’s example, and also that we should not do that.

Climbing off the high horse

I recognize that it isn’t always possible to achieve the kind of writing that I’m talking about here. We live in a fallen world. I don’t particularly consider myself a good writer or editor, and everyone faces constraints that lead us to behave in ways that we don’t think are fully consistent with our values. But that doesn’t mean that it’s good or neutral when we make those compromises. It is bad. Plagiarism is bad.

The real problem here is, of course, systemic. We are rewarded for producing as many papers as possible, not particularly for having a well crafted discussion on related work. I do, however, want to at least reinforce the norm here: you should write your own papers, even the parts you don’t like (or get someone else to write them). You should care about writing those sections well, even if they aren’t your favorite part of the paper. They are all part of the work you are signing your name to.

We should not make light of the actual work we do as academics by implying that it is fine to reuse text that is not our own. It is not.

Yes, this is also surprising to me. ↩
I’m very consciously not talking about GPT here, but, like, also that. ↩
A single number with 4 decimal precision is equal to log₂(10^4) = 13.3 bits, assume there are a couple of estimates. ↩
Citation also lets us bypass the hard limits of page limits, too — citations let us gesture at much more complicated ideas than we would be able to fully explain in our particular pdf. ↩
Even in this case, we are different people than the authors of the replicated work and probably have different opinions about how it relates to the world. At the very least, time has passed and the literature is in a different place! ↩
Harder to do this on a dissertation which may not be able to be coauthored. But honestly, to a first approximation I’ve only cared about one person’s dissertation (Ferenc Huszár’s). ↩
I’m taking this example as a jumping off point, I don’t think it’s like, “academic misconduct” to use boilerplate language, and I think his example is a very good one, because tons of people have copied that kind of language. ↩

Stop looking for the next Twitter

Tue, 03 Oct 2023 00:00:00 GMT

Everyone seems to agree now that Twitter/X is bad. They’re right. But the desire to recapture some idyllic ahistorical version of Twitter is wrongheaded. The nature of the platform has serious problems that cannot be solved by a different owner, CEO or by changes in content moderation/ranking. The medium is the message, and the medium is, overall, pretty bad for a lot of what we use it for.

I’m going to trace through my argument around why the design of microblogging platforms are bad and then sketch out how I’m personally choosing to engage with them in light of this badness.

All microblogging is fundamentally the same

There are differences between microblogging sites (I’m primarily thinking about Twitter/X, BlueSky, Mastodon and Threads, but I imagine anything else basically fits the same mold). Nevertheless, I think the core properties are basically the following:

Short text snippets. I think this is definitional. If it weren’t based around this, it wouldn’t be microblogging.
Virality. Content gets out of the small community in which it’s posted. This is fun (at first) for the poster and fun for the viewer. The nature of a viral platform means when you see stuff from outside your community, it has already been vetted, so it is almost definitionally ‘highly engaging’ for some highly specific definition of those words.
Simple engagement possibilities. There are very simple, easy ways to engage with the content you see. A “like” or a “reskeet” is just a click of the mouse away. You don’t have to think very hard or construct a complete and cogent thought in order to “engage” with what you see¹.
A “feed” of posts. You are deluged with an infinitely scrolling progression of more content (making you infinitely content, right?). There are a lot of other features that matter on the margin. Exactly how long is a post? Can you Quote-tweet / Retweet? What content is allowed / promoted? This stuff matters, but I don’t think it really gets at the core of what microblogging is. (As an aside, I think we as social scientists are much better at testing the differences between possible affordances within a single platform than we are at testing the big meaningful differences between platforms).

The thing that I find really clear about this is that these affordances are not built to enable deep and thoughtful exchange of ideas. They’re built to enable surface-level engagement that you’ll come back for again and again.

Some microblogging is different (but not better)

I think Jack may have learned exactly the wrong lessons from Twitter. It seems that the AT protocol which undergirds BlueSky is fundamentally built around the idea that Jack shouldn’t have to be the person that everyone gets mad at. You don’t like how your feed is ranked? Not his fault, that’s a different layer than he controls. Don’t like how content moderation works? Well, you should be on a different application with different rules. This goes to extremes with platforms like Mastodon, where some servers won’t do any moderation whatsoever, while others will ban you aggressively for not giving adequate content warnings. This is different than other microblogging services, but I don’t think there’s much of a way to say it’s better. For instance, the massive CSAM problems there.

How this has worked in practice with AT/BlueSky is kind of different. BlueSky is the only real application built on AT, which means that Jack doesn’t just get to sit around dealing with AT issues, he has to deal with BlueSky issues, too. All of a sudden (and despite his original intentions), he’s having to manage an actual platform rather than just a protocol. This means that BlueSky is (slowly) developing content moderation standards, some of which are actually implemented at the protocol layer (e.g. in AT rather than in BlueSky). As I understand, CSAM, for instance, is a protocol-level concern for AT, it is not federated out to individual applications. I can’t see how this won’t continue to be a giant wellspring of conflicts.

What are we doing On Here?

There are two main things I think happen on microblogs:

**Discovery. **There’s nothing I’ve seen that’s better at delivering interesting papers and news items to the front of my face (i.e. “pointing at things”). Microblogging is extremely effective at making serendipitous connections. Getting pointers to things of interest from people within a hop or two of you in a network which you form according to your whims is extremely powerful!
Jokes/entertainment. There’s a reason that dril is the greatest poster on microblogs. One-liners are a classic joke format, and they’re enjoyable to consume on a feed (perhaps especially a feed that also has a lot of self-important academics on it). What I think doesn’t happen, is anything very much like “discussion”. Discussion requires grace between interlocutors, which is hard to extend given the threat of virality inherent in the platform. Discussion requires context, which is destroyed with virality and its associated context collapse. And important discussions require depth, which is not really possible in short snippets of text. I also think that privacy is often crucial to good discussion². Real privacy is anathema to virality: you can’t really have both. When discussions are ephemeral and low-cost its much easier to try out ideas to see if they seem compelling and convincing.

The easiest discussions to have on microblogs are ones in which the interlocutors already agree with one another and share full context (or which are just pointing to something and saying, essentially “good” or “bad”). In this case, a few words are all it takes to explain what one means and make one’s position clear. As discussions require more nuance and rely on context that is not shared (i.e. on more complex topics), then the goal should not be to write in chunks of a few hundred characters.

I think that a lot of the appreciation of tweet-length communication is a reflection of a failure of academic writing. For a lot of research, the first time someone takes the time to make it broadly accessible is when a tweet-thread is written out³. I’ll bite the bullet, and freely admit that was true for me with our ICML paper about online balanced experimental design. The version we submitted (and which was accepted) was essentially just math. The paper’s structure was, essentially, introduction → math → simulations → the end. Reviewers didn’t love it, but they were intimidated by it⁴. We cleaned it up for the camera-ready version (yes, that’s the easy-to-digest version), but the first time we really thought about how to communicate our results to a broader audience was when I wrote about it for Twitter. That’s bad!⁵ While one solution to this problem is to just write papers that are more accessible, the incentives are currently entirely against this. Being rigorous but difficult to understand is, often, a very good move if your primary incentive is publication.

Twitter, however, has incentives which at least preference some amount of accessibility. If you want your work to be shared and consumed, you have to hook people and encourage them to “engage.” In contrast to typical academicese, I think this is a nice change of pace, and is one of the reason I sometimes appreciate them. “Engagement,” however, is quite different than “accessible.” To be accessible, I think you want things like easy hyperlinks/citation, formatting, figures, quotations and, yes, more than a couple hundred characters. Maybe you also want to throw in some code or math as a treat! We can obviously do better than tweets for this.

What am I doing?

I’m treating microblogging for what it is. I’m essentially indifferent between services, as I don’t think the differences are that meaningful relative to what’s the same. I enjoy finding things on these platforms, so I will peruse them, but I will not expect discussion, and I won’t use them as if I do. Since I think threads about new work are not a particularly good way to provide digestible versions of papers, I will avoid making them. I don’t want to push for “engagement” with my work; I want people to understand it and its context. Instead, I will share new work, and (if appropriate) will link to a longer-form accessible introduction for that work in this space (or somewhere like it)⁶. In short, I will try to make it easy to discover my work on microblogs, but I will not attempt to explain my work in that setting. The platform is not conducive to such explanation. I may also make jokes, but, well, most of those are for the groupchats where I can be spicy.

I think this is ultimately the kind of recommendation Neil Postman makes about television in *Amusing Ourselves to Death, *too: TV is fantastic entertainment device, but don’t mistake it for something that it’s not. Don’t treat it as a medium of explanation or of news or of education. What it does is entertain.

Microblogging platforms are not built for good discussion. I will avoid a poor facsimile of conversation there, and have conversations in places where those conversations can be at their best. For me, this means that my conversation happens outside of the “clearnet”. I have a variety of WhatsApp / Slack / Discord groups for talking with collaborators or just randomers with shared interests. This is, I believe, a much more productive approach than trying to make microblogging into something that it can’t ever really do well. In these environments, you can set up affordances to enable the kind of discussion you want: you can make it easy to share LaTeX, or code. You can make spaces for highly moderated discussions or free-flowing sharing of memes. But most fundamentally, you can retain so much more control and privacy⁷.

Build alternative communities!

About a year ago I started up a Discord for discussion about experimentation which ultimately hasn’t seen much discussion in it. I’m continuing to hang out there, so if anyone wants somewhere outside of the clearnet to talk about experiments, feel free to join! Or come there to tell me about why this post is bad.

But also, use microblogs to create your own semi-private communities and make them easy to discover! The magic of the internet is that it’s all code and you can create whatever you want within those (weak) bounds. There are a ton of tools out there to create whatever kind of community you want, and you don’t need to troll around for “the next Twitter.” Twitter wasn’t ever all that good, and you can just go ahead and make the better community that you’d rather see! Many of those communities won’t work⁸, and that’s totally fine; the process of figuring out what works is how we find a space for online discussion that’s better than Twitter, which is what we should aspire to, anyway.

What AI thinks “a twitter-like website” looks like. It isn’t wrong.Thank you for reading Drew’s News. Now please microblog this in a long thread.

Note: this is bad. My hot take is that thinking is good and we should try to do more of it and improve how we do it. The simple binary “engagement” actions constrain human behavior into radically low dimensional signals. This is basically what Kevin means when he talks about online platforms making us behave like machines. This is not good! There are no posts for which my reaction can truly be summed up by a binary “Like” action! A RT sometimes is an endorsement, but sometimes it isn’t! None of this is expressible in the platforms! ↩
By privacy, I mean that the ability to pretty strongly control how wide the distribution of your content is (such as by sending something in a group chat where only your wife and brother-in-law can read it). Encryption, cybersecurity and data protection are important components of privacy, but not what I’m talking about. ↩
An analogy I like is that threads are in the style of a poetry slam (derogatory). It’s not inherently “bad”, but it is a very distinct style which focuses on having meaningful “beats” every couple dozen words or so. Good tweets in a thread often are basically glorified figures (with caption). That’s a very constraining style! ↩
Comments were essentially “the problem seems important and there’s a lot of math which seems correct”. Some reviewers even called it easy to read, which we all had a good laugh over. The math was abstruse and we literally introduced a parameter without ever explaining why it was important! ↩
It also hasn’t always been this way. A good example is Norbert Wiener’s Cybernetics, which is a ludicrous book in a lot of ways, but it was also not targeted solely at an in-group of very technical readers. Only some chapters are. ↩
I think an Oped length of about 1000 words is a good one to shoot for (don’t check how long this post is, sorry), e.g. Alex’s nice paper. ↩
Control and privacy seem to cut against the democratizing impulse, but they don’t have to. You can, for example, be gracious in admitting people and harsh in throwing them out when they don’t abide by community norms. The key democratizing element isn’t in making everything public, but in giving people a chance. Being public can actually be much worse in some cases: Imagine saying something kind of dumb and going viral as a grad student (when we are all very silly). Not good! ↩
I don’t even know what “work” means here. Just think about this community formation as expanding the groupchats you’re already having good discussions in. Is it a “failed” groupchat if you stop talking in it at some point? Obviously not! Is a coffee meeting a failure if it doesn’t result in a permanent collaboration? Spin something up and see how it goes! ↩

A Blueprint for the Regulation of Tech

Mon, 11 Sep 2023 00:00:00 GMT

The US2020 Facebook and Instagram Election Project (US2020 from here out), the first four papers of which were recently published in Science and Nature, can be the blueprint for meaningful regulation of large online platforms. It’s crucial not just to be satisfied in access to data, but in access to rigorous tests of how changes on platforms affect society.

The key connection is to avoid examining the project through a purely academic lens about academic research and the knowledge gained through the studies.

The news post at Science by Kai Kupferschmidt focuses on this dimension, with input from Joe Bak-Coleman’s saying that “This is not how research on the potential dangers of social media should be conducted”. He focuses on the research and the project as a model for gaining academic knowledge.

The commentary by project rapporteur Michael Wagner focused, too, on the project as largely academic in nature. Wagner gets one element of the project exactly correct from this dimension: It requires the goodwill of Meta to work correctly, which makes a difficult model for future scholarly work.

Meta wants the project interpreted solely as the results of academic research based on Nick Clegg’s news post: “Its findings will be hugely valuable to us, and we hope they will also help policymakers as they shape the rules of the road for the internet”. The findings of these studies should shape discourse on regulation, and not the structure of the collaboration itself. These findings are (overall) pretty positive for Meta. They show that algorithms are *powerful *(they change a lot about on-platform behavior), but they aren’t scary (they don’t swing elections). There are subtleties here, and I don’t mean to get into a detailed accounting of the exact substantive results, but I think this is the reading of the Project Meta wants to push to advertisers (the former story) and to regulators (the latter one).

Brandon Silverman gets nearly to the quick: “That’s why ultimately we need regulation if we want more of this sort of thing and for me, that represents one of the biggest promises of the Digital Services Act[…]” We should think about regulation as a way to free this kind of information from tech companies. But he doesn’t go nearly far enough. A model of data-sharing (proposed in the next tweet) does not go far enough, and it sets up perverse incentives for companies to neglect rigorous measurements of changes that might place them in a negative light.

My argument in this piece is that the structure of collaboration of US2020—sophisticated experts telling online platforms what they must measure about societal impact—should be the future of the regulation of the internet. Don’t simply accept the framing that Nick Clegg pushes, that we should take these findings and make policies based solely on them (implicitly assuming they are the only evidence we might get). The way platforms work is constantly changing: we should not seek perfect generalizability of the findings, we should instead seek systems which allow us to continue to probe and measure how these platforms work flexibly, as they change.

What do we want to measure?

Many of the most important questions we have about tech’s relationship to society are counterfactuals: If Facebook were otherwise identical but ranked Feed chronologically rather than by an algorithm, would there be more or less hate? If YouTube made different choices in recommendations, would it make people less extreme? A counterfactual, fundamentally, asks what would happen were the world just a little bit different. When thinking about the regulation of technology, we’re asking questions like these: if Facebook were to operate differently in some way, would society be better or would it be worse? This embeds two questions: One is a scientific question: what would society be like if Facebook were different? The second is a question of values: Would that counterfactual society be better or worse?

The scientific question can be answered through randomized control trials (RCTs). Such RCTs, however, can only truly be implemented by and within these large tech platforms, so under the status quo the question of values can only be answered by people within those walls. Engineers, data scientists, managers and executives of tech companies: mostly good hearted people, but people who work within institutions that require they think about the good of the platform rather than the good of society.

I worked within Meta for years on improving the process through which the company evaluated potential counterfactual versions of itself – in other words, testing changes through A/B tests. The entire business of software development is to try lots of stuff out but only keep the things that work towards the business’s goals. These goals are expressed in numbers (“KPIs” or “metrics”). My team built machine learning tools to realize these business goals. When business values change, so too, can the platform. Look no further than Facebook’s 2018 pivot to “meaningful social interactions”. Once new numbers are chosen as the target, machine learning gears turn and the platform (in many ways a black box to everyone) pivots on a dime to optimize these new metrics.

Many of these business values are widely held societal norms: nobody in tech wants more spam or child sexual abuse material, for instance. As such, companies develop sophisticated (albeit imperfect) systems for detecting and removing such content and ensuring that it is not distributed widely. Other metrics, like the time spent on a platform, are less clear cut: essentially all internet platforms prize this as a KPI, but it does not necessarily align with what is best for society.

Democratic societies have systems for answering questions of values: we elect representatives who enact policies so that society embodies the values we care about. When such values are contentious, there is conflict, of course, but there is a peaceful process for resolving that conflict through our institutions. Tech lacks the mechanism to ensure that it embodies societal values: even market pressure isn’t straightforward, since usage is free thanks to advertising.

Who gets to measure them?

The only real counterfactual evidence about tech’s effects is controlled by tech itself. This is the promise of US2020, a project I worked on while at Meta. The unique power of this project is that it reveals the counterfactuals which—but for the press of a button—Facebook can decide to make reality. Asking ‘Should Facebook remove algorithmic ranking?’ contains both a question of science and a question of values — and this project answers the question of science, showing precisely what would happen if Facebook did no algorithmic ranking on Feed. As a result, the question of values is finally exposed to democratic scrutiny. Do we measure the right things to effectively answer that question? The way forward can only be an iterative refinement of asking the question and trying to improve the answers we get.

In short, the Election Project is a blueprint for the effective regulation of technology companies. The approach is straightforward and consistent with the language of Chapter III, Section 5 of the Digital Services Act. Regulators can require that for important, societally relevant changes, Very Large Online Platforms must run and report the results of RCTs. These are far more informative than audits of the processes used in these platforms or providing data access without counterfactuals, or trying to explain the precise details of how underlying machine learning systems work (which are mostly black boxes to everyone internally, too). To reiterate: in general, many of the most important systems at VLOPs are not truly “understood” by those who create them. Rather, they are refined by measuring what they do, and launching the changes that push those measurements in a direction that those measurements deem positive.

Philipp Lorenz-Spreen has already compellingly argued that Article 40 of the DSA is a way to free the results of internal A/B tests to the public. Tech companies are already running lots of A/B tests, so maybe we can just require them to share those results! This is a very positive step, but it isn’t enough. The problem with limiting ourselves to these tests is that platforms can simply stop doing them on issues they feel present a regulation risk. We’re back in the same place we started where our ability to understand counterfactuals is dependent on the platforms’ goodwill.

Platforms already limit what is measured internally for this reason. A direct example of this is race in the US. Put simply, Meta works hard to limit what it knows about race in order to preclude legal risks like it faced around housing discrimination in 2019. When I left Meta, the primary way concerns about disparate impact were examined were through zip-code level demographics. This process ensures that individual users’ race was not explicitly inferred, so race cannot be explicitly taken into account in decisions. To be clear, this does not mean that systems do not have disparate racial impacts. It does, however, make it very hard to measure when those disparate impacts might exist.

There are also precedents for requirements to run RCTs, such as with the UK’s Competition and Markets Authority requirement that Google test changes to third-party cookies. The logic here was that Google’s change to how cookies work might lead to big changes in how the advertising market works. Without understanding that impact, the regulator can not make an informed decision about whether it was anti-competitive or not. Hence, require Google to measure the impact, since they’re the only people who can.

More traditionally, this is key to the FDA’s regulation of food and medicine. The regulator is not generally performing the clinical trials to demonstrate safety itself: it instead requires that the would-be producer carry out the required tests, and ensures that those tests meet the standards of scientific rigor necessary to ensure safety. We think it’s important to verify that drugs meet at least a minimal standard before pushing them out broadly to all citizens. The same should be true for large changes to internet platforms. Note that the structure of clinical trials under the FDA would be exactly subject to the same critiques that are being made of US2020.

The point is not who is actually running the test, but who is defining what particular tests must be conducted and how resulting outcomes must be measured. As an insider to this process, let me be very clear: US2020 took incredible care that external academics understood in detail the process by which concepts were operationalized and measured. A lot of the back-and-forth in the collaboration was specifically about this: academics didn’t understand how data is collected and stored internally, and internal researchers needed clear operationalizations to measure the concepts requested. I think I saw someone throw out that there were around 1000 variables to figure this out for.

The DSA provides an avenue for this kind of accountability through counterfactual evidence, but it isn’t through Article 40. Instead, it’s through Article 34. Allow me to quote:

Providers of very large online platforms and of very large online search engines shall diligently identify, analyse and assess any systemic risks in the Union stemming from the design or functioning of their service and its related systems, including algorithmic systems, or from the use made of their services.

By suggesting that these risks arise from the design or functioning of the service, this sentence invokes counterfactuals. The question at play is whether platform design causes these risks. The only valid way to assess this is through counterfactuals as measured through randomized control trials. The counterfactual risk is what VLOPs should demonstrate in their risk assessments. I recognize that this plain-english reading is not formal legal analysis (and I am not a lawyer). Given the latitude of regulators to choose how this law should be interpreted, I hope that they do so in the way that will give them actual answers to the societally important questions of counterfactual risk. US2020 shows them how they can do that. Unfortunately, based on the Commission’s current example of how to apply the risk management framework, this is not likely to be the approach taken to interpreting the DSA.

When are they measured?

Time is a crucial element of this story. The number one complaint by everyone involved in US2020 is the time it took to make it happen. As Kevin Munger emphasizes, it was 32 months after the US2020 studies were actually performed that they were published. Part of that delay was because of peer review (note that the chronological feed paper was originally received by Science on March 7, 2022, but was only published on July 27, 2023. The paper was definitely improved in that time, but it goes without saying that nothing about the intervention or outcome measurement could have been improved or changed as a result of this peer review. It had already happened.

Surely nothing changed about Facebook in that time, right? Well, shortly after submission of chrono-feed, on July 27, 2022, Mark Zuckerberg made an announcement on the Q2 earnings call: “Right now, about 15% of content in a person’s Facebook feed and a little more than that of their Instagram feed is recommended by our AI from people, groups, or accounts that you don’t follow. We expect these numbers to more than double by the end of next year.” Shit. It’s at this point that I’ll note we found clear heterogeneity of the effects of chronological feed on variables like how much uncivil content and slurs show up on Feed, as well as on-platform political behavior on the dimension of inventory size. When there’s more content Feed could show you, there’s more potential for Feed algorithms to change what you see.

Larger inventory is associated with larger effects on on-platform political behavior.Kevin and I have a new working paper showing exactly how poorly agnostic approaches to generalizability fare when reality refuses to stay fixed. When reality is changing at internet speed, then it’s necessary that our knowledge generation process matches that speed. US2020 could do better on this. A lot of work had to be done to mitigate the risks that were inherent in the project as the first of its kind. There was a whole software infrastructure that had to be built from the ground up for protecting privacy, which was done at a pretty incredible speed. There was also a lot of work to build up the internal processes to make everything work both within the collaboration and within Meta (e.g. legal and privacy reviews to make sure user data was appropriately protected). And as stated above, there was a whole arduous process of defining meaningful operationalizations of everything: to a real extent, the scientific language of internal and external researchers wasn’t exactly the same, so translation was required.

If we’re thinking about an ongoing regulatory process, however, a lot of these concerns go away: an infrastructure and set of systems will be set up (by necessity) to streamline processes. Variable definitions will be negotiated over time, so that’s largely just a fixed cost per variable. If the output is regulatory rather than academic, the results wouldn’t need to be gated by the fickle peer review process (just government bureaucracy!).

Defining appropriate implementation of the regulations, too, must be a negotiated and long-term process. Meta has spent years defining specific internal metrics to understand the best dimensions of platform behavior to measure for business purposes. We cannot expect US2020 to have gotten the right societally important dimensions correct on the first try. DSA requires platforms to measure risks “on civic discourse and electoral processes, and public security”. How should these be operationalized? This can only be an iterative process as we see what kinds of measurements are both informative about what we care about in the world and sensitive to the kinds of changes VLOPs make. Iteration on the US2020 model is how to begin this process of refinement.

Wrapping it up

If we want to truly understand a world where tech is different in some way, we must change it in that way and measure what happens as a result. This is how Meta ensures Feed aligns with its business values, and this is the only way to ensure Feed – alongside all other aspects of online platforms – aligns with our societal values, as well. This may require platforms to run RCTs they would not have already run: we may imagine changes that they have not previously tested. At the very least, it will almost certainly be necessary to require that they measure outcomes they might not otherwise measure. In the US, for instance, this might mean requiring that they collect data on race that they are otherwise reluctant to hold.

US2020 has many of the features that are necessary for this kind of counterfactual accountability. It provides rigorous evidence of how Meta’s products would work if we were to change them. The RCTs were designed by external academics with full control rights, whose incentive is not to improve Meta’s bottom-line, but to expose answers to societally relevant questions. Meta had no substantial freedom to say no to such requests. It was not fast enough, nor was it large enough scale (e.g. it was only in the US). Regulators don’t currently have the expertise in-house to do the work that these academics did in asking good questions and co-designing studies to answer them. This is an important problem to solve, but other fields (like the FDA) have shown that such expertise can be acquired by regulatory agencies.

The biggest problem with US2020 is that it only exists out of Meta’s goodwill. And given how expensive it was for Meta, it’s not clear how much of that there is left. Look through the author list for the Meta employees – most worked primarily on this project for the last 3 years, and lots of other engineers and data scientists contributed substantial time as well. These people are all superstars and just their direct compensation over that time is a huge commitment from Meta: easily millions of dollars, likely tens of millions. Given recent belt-tightening, I wouldn’t count on Meta independently finding this a good tradeoff solely in order to be transparent. If we want to guarantee this kind of transparency from internet platforms, then we shouldn’t just hope that they will keep choosing to bring us rigorous counterfactual evaluation. Government should require it.

Quarto for an Academic Website

Wed, 11 May 2022 00:00:00 GMT

Intro

I’ve never been good at keeping my website updated. I always go through two different phases of maintenance:

Rushing around creating a new website with bells and whistles using whatever the flavor of the month is
Never updating an existing website

I’m hoping to break out of this cycle, but am currently solidly within Phase 1.

A highlight from my time in Phase 2 was when I forgot to update my DNS and I totally lost control of drewdimmery.com (don’t go there, it has a squatter). I think my website at that time was some Octopress monstrosity. There are a few reasons I think Quarto might help with my vicious circle.

Serving static HTML pages is about as easy as it gets
Very little Quarto-specific syntax to recall (e.g. CLI commands or abstruse markup)
Lots of flexibility (Python / R) in how to generate that static content
Full programmability means that generation can be based on arbitrary data structures of my choosing

I previously used Hugo Academic for building my website, which was much better than just editing the content directly, but I never remembered the right way to generate a new publication definition (there was a CLI, but I never remembered the syntax). Each publication got its own file describing its details, and I found this quite clunky. I wanted something extremely lightweight: there isn’t much reason for my individual publications to get pages of their own, and I really don’t need a lot of information on each of them. I just want some basic information about each and a set of appropriate links to more details.

This post will detail how I’ve set up Quarto to accomplish this task. I’ve nearly completely separated the two main concerns around maintaining an academic website / CV, which to me are data on publications and software from the design elements of how to display them. It’s entirely possible that my particular issues are unique and this post won’t be useful to anyone else. Luckily, the marginal cost of words on the internet is essentially zero (and maybe the marginal value is, too).

Setup

Setting up Quarto was very easy, so I won’t belabor this. The combination of the Get Started guide with the Website Creation guide kept everything very straightforward. I also used Danielle Navarro’s post and her blog’s code to get everything set up.

I decided late in the setup process to add a blog, so I will mention that it’s actually very easy to do: it basically just requires adding a Listing page (i.e. the blog’s index), a folder to contain the various posts and a _metadata.yml file in that folder to describe global settings to apply to all posts. I just created these manually without too much trouble. This is one of the great things about building sites with tools like Quarto: everything is extremely transparent: just put a couple files in the right places and you’re good to go.

Site Design

To demonstrate how I’ve set things up to populate the website from data about my academic life, I’ll focus on my publications page. There are two main files undergirding this page:

papers.yaml: a data file in YAML with standardized information on each publication. I chose YAML because it's fairly easy to write correctly formatted YAML by hand (and I'll be updating)
research.qmd: The page which takes the data in papers.yaml and turns it into nicely formatted Markdown / HTML. This is setup as a Jupyter-backed qmd file (essentially a Jupyter notebook).

This idea of separating the data side (information about publications) from formatting is aimed at making my life easier. One of the reasons I often stop updating my website is because when I come back in 3 months with a new publication, I never remember all the details about how I formatted entries in whatever flavor of Bootstrap I happened to be using when I built the website. Moreover, because I know that there’s a barrier to understanding before I can get started, it’s extremely easy to put off (and therefore it never gets done).

By separating out the data entry from the formatting, this simplifies matters substantially.

Data

I put data about each publication in a basic YAML format:

See example data

softblock:
  title: Efficient Balanced Treatment Assignments for Experimentation
  authors:
    - David Arbour
    - me
    - Anup Rao
  year: 2021
  venue: AISTATS
  preprint: https://arxiv.org/abs/2010.11332
  published_url: https://proceedings.mlr.press/v130/arbour21a.html
  github: https://github.com/ddimmery/softblock

This is basically like a simplified bibtex entry with more URLs so I can annotate where to find replication materials for a given paper, as well as distinguish between preprints (always freely accessible) versus published versions (not always open access). A convenience that I add in the markup here is referring to myself as me in the author list (which is an ordered list). This allows me to add in extra post-processing to highlight where I sit in the author list.

Some additional things I considered adding but chose to ignore for a first version:

An abstract
A suggested bibtex entry

Both of these would be easy to add, but I chose to start simpler. I don’t love YAML for entering long blocks of text, which both of these are.

Formatting

Since I can write the generation logic for page in Python, this puts me on comfortable ground to hack something together. To knit the above publication data into HTML, I just literally bind together the programmatically generated raw HTML and print it onto the page.

I do a couple additional useful things in this process:

Separate out working papers or non-archival papers from published work (I make this distinction based on whether I include a published_url field or not).
Order and categorize papers by year
Provide nice Bootstrappy buttons for external links (e.g. to Preprints / Code / etc)

See research.qmd fragment

import yaml
from IPython.display import display, Markdown, HTML

def readable_list(_s):
  if len(_s) < 3:
    return ' and '.join(map(str, _s))
  *a, b = _s
  return f"{', '.join(map(str, a))}, and {b}"

def button(url, str, icon):
    icon_base = icon[:2]
    return f"""<a class="btn btn-outline-dark btn-sm", href="{url}" target="_blank" rel="noopener noreferrer">
        <i class="{icon_base} {icon}" role='img' aria-label='{str}'></i>
        {str}
    </a>"""

yaml_data = yaml.safe_load(open("papers.yaml"))
pub_strs = {"pubs": {}, "wps": {}}
for _, data in yaml_data.items():
    title_str = data["title"]
    authors = data.get("authors", ["me"])
    authors = [
        aut if aut != "me" else "<strong>Drew Dimmery</strong>" for aut in authors
    ]
    author_str = readable_list(authors)
    year_str = data["year"]

    buttons = []
    preprint = data.get("preprint")
    if preprint is not None:
        buttons.append(button(preprint, "Preprint", "bi-file-earmark-pdf"))

    github = data.get("github")
    if github is not None:
        buttons.append(button(github, "Github", "bi-github"))

    pub_url = data.get("published_url")
    venue = data.get("venue")
    working_paper = pub_url is None

    pub_str = f'{author_str}. ({year_str}) "{title_str}."'

    if venue is not None:
        pub_str += f" <em>{venue}</em>"

    if working_paper:
        if year_str not in pub_strs["wps"]:
            pub_strs["wps"][year_str] = []
        pub_strs["wps"][year_str].append(
            "<li class='list-group-item'>" + pub_str + "<br>" + " ".join(buttons) + "</li>"
        )
    else:
        if year_str not in pub_strs["pubs"]:
            pub_strs["pubs"][year_str] = []
        buttons.append(button(pub_url, "Published", "ai-archive"))
        pub_strs["pubs"][year_str].append(
            "<li class='list-group-item'>" + pub_str + "<br>" + " ".join(buttons) + "</li>"
        )

I then print this out using the display functions from the IPython module and using the asis chunk option:

See research.qmd fragment

for year in sorted(pub_strs["pubs"].keys(), reverse=True):
    display(Markdown(f"### {year}" + "{#" + f"published-{year}" + "}"))
    display(HTML(
        "<ul class='list-group list-group-flush'>" + '\n'.join(pub_strs["pubs"][year]) + "</ul>"
    ))

The full code is on GitHub.

It’s worth noting that to get the years to show up in the Table of Contents its necessary to be careful exactly how the content is stuck onto the page. If you don’t use the asis chunk option, you can still get all the right content to show up, but it won’t necessarily appear in the ToC. I also found it necessary to include section-divs: false in the header, or else the output would get wrapped in additional div tags which made it harder to get the right classes in the right divs. There are probably more elegant ways to do all of this.

I use the same basic setup to populate the Software page, albeit with simpler logic.

Additions

I debated adding an abstract that expands out on click (like the code folding above in this post). This would actually be more or less trivial to add using a <details> HTML tag if I wanted to provide the data in the YAML. I’m ignoring this for now because I want to minimize data entry for my future self (and it’s anyway just a click away at the Preprint link).

Deployment

It’s extremely easy to build a new version of the website locally (quarto render from CLI), but there’s no guarantee I’ll remember that off the top of my head in a month without Googling, so I think it’s worthwhile to setup automatic building after I push a commit to GitHub.

GitHub Actions is incredible. I adapted the example config from Quarto to the following (also on GitHub here):

GitHub Actions for Netlify

on:
  push:
    branches: main
  pull_request:
    branches: main
  # to be able to trigger a manual build
  workflow_dispatch:

name: Render and deploy website to Netlify

jobs:
  build-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v3
        with:
          python-version: '3.9'
          cache: 'pip'
      - run: pip install -r requirements.txt

      - uses: r-lib/actions/setup-r@v2
        with:
          use-public-rspm: true

      - uses: r-lib/actions/setup-renv@v2

      - name: Install Quarto
        uses: quarto-dev/quarto-actions/install-quarto@v1
        with:
          # To install LaTeX to build PDF book
          tinytex: true
          # uncomment below and fill to pin a version
          # version: 0.9.105

      - name: Render website
        # Add any command line argument needed
        run: |
          quarto render
      - name: Deploy to Netlify
        id: netlify-deploy
        uses: nwtgck/actions-netlify@v1
        with:
          # The folder the action should deploy. Adapt if you changed in Quarto config
          publish-dir: './_site'
          production-branch: main
          github-token: ${{ secrets.GITHUB_TOKEN }}
          deploy-message:
            'Deploy from GHA: ${{ github.event.pull_request.title || github.event.head_commit.message }} (${{ github.sha }})'
          enable-pull-request-comment: true #  Comment on pull request
          enable-commit-comment: true # Comment on GitHub commit
          enable-commit-status: true # GitHub commit status
        env:
          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
          NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
        timeout-minutes: 1

This Action requires two pieces of information from Netlify entered as secrets in GitHub. The NETLIFY_SITE_ID may be found in the site configuration settings, while the NETLIFY_AUTH_TOKEN may be found in personal settings (the personal access token).

One thing I have not yet done is set up an renv to ensure dependencies for blog posts are taken care of in GitHub Actions. This means that posts like the experimental design demo can’t be knit via GitHub Actions. I did this for two reasons (other than laziness). First, it’s a pain to get GIS tools working on any environment (ok, so its part laziness). I’ve actually done this before for automated R CMD checking of the regweight package, but didn’t feel like it was worthwhile here.

The reason it’s not worth it is that Quarto has a great feature called “freezing”. Essentially, it knits blog posts or pages, and only re-renders them when something about the source changes. This means that the vast majority of posts don’t need to be rendered on each build. If I’m working on a blog post, I can write it locally, render on my machine, commit that pre-rendered post and then all future builds on Actions won’t get held up by their inability to render that post.

As I type this, it becomes clear that I’ll forget how to do this pretty often (given that there’s been about an 8 year delay since my next most recent blog, I likely won’t stay in practice). But blogs aren’t my main concern on my website: keeping a software and publication list up-to-date is.

Setting up Actions means that simple updates to pages (or YAML files) can actually be done directly in the GitHub editing UI, which further lowers the barrier for my future self. I don’t even need to clone the repository to whatever computer I’m working on to add a publication!

Future dreams

I imagine my CV is similar to most academics’ in that it’s built like a house of cards (and overfull hboxs). Whenever I add something new to it, I have to copy some lines from elsewhere and modify them to fit the new entry. This always takes me way more time than I’d like. If I mashed together my current About page with the Research page, it’s like 90% of the way to a full CV. It should presumably be pretty easy to do explicitly combine them and output a reasonable-looking CV.

This is a project for another day, though. Too much of the Research page directly outputs HTML, which makes it difficult to naïvely import into a $L A T E X$ CV.

An almost completely naïve approach to directly importing the relevant pages creates this ugly document.

Naïve CV

---
title: "Curriculum Vitae"
format: pdf
---

{{< include about.qmd >}}

{{< include research.md >}}

It’s definitely possible to improve on this. The easiest hacky approach is to just write a whole alternative version of the HTML formatting code which resides in research.qmd to output appropriately formatted $L A T E X$ markup.

For now, I’m pretty pleased with the system I have, but ask me again in three months.

Using SoftBlock to Design an Experiment

Tue, 10 May 2022 00:00:00 GMT

Abstract. This is a demo of using SoftBlock for experimental design for a notional randomization of precincts in North Carolina. To optimize power, we focus on ensuring very similar patterns of prior voting patterns in treatment and in control. This demo walks through all the necessary steps for this process and shows how to perform estimation of average and heterogeneous effects.

This post was originally written as an executed Quarto document. The code below is shown exactly as it ran; the figures, printed values and tables are the outputs captured from that original render.

Introduction

In particular, I’m going to imagine that I’m designing an experiment in which I assign different treatments to particular precincts in North Carolina. In order to optimize power, of course, we want to make sure that our two test groups look as similar as possible in terms of prior voting patterns.

Thus, the steps in this design will be:

Collect relevant historical data.
Define variables on which we wish to balance.
Allocate treatment assignment using new methods.
Simulate the power of hypothesis tests under the proposed design.
Fake some outcome data and analyze it for average and heterogeneous treatment effects.

Implementation of methods

Description

The relevant API is a function with tidyverse semantics called assign_softblock (or assign_greedy_neighbors). These functions accept a vector of columns to be used in the design. The SoftBlock version additionally accepts two arguments, .s2 for the bandwidth of the RBF kernel to use in the construction of a similarity matrix as well as .neighbors which indicates the number of nearest neighbors to include in the graph on which to construct the spanning tree. These parameters don’t generally need to be modified.

Source Code

See source code

writeLines(readLines("https://raw.githubusercontent.com/ddimmery/softblock/master/r_implementation.R"))

library(Matrix)
library(igraph)
library(FNN)
library(hash)

assign_greedily <- function(graph) {
    adj_mat = igraph::as_adjacency_matrix(graph, type="both", sparse=TRUE) != 0
    N = nrow(adj_mat)
    root_id = make.keys(sample(N, 1))

    a = rbinom(1, 1, 0.5)
    visited = hash()

    random_order = make.keys(sample(N))
    unvisited = hash(random_order, random_order)

    colors = hash()
    stack = hash()

    stack[[root_id]] <- a
    tentative_color = rbinom(N, 1, 0.5)
    while ((!is.empty(unvisited)) || (!is.empty(stack))) {
        if (is.empty(stack)) {
            cur_node = keys(unvisited)[1]
            del(cur_node, unvisited)
            color = tentative_color[as.integer(cur_node)]
        } else {
            cur_node = keys(stack)[1]
            color = stack[[cur_node]]
            del(cur_node, stack)
            del(cur_node, unvisited)
        }
        visited[[cur_node]] = cur_node
        colors[[cur_node]] = color
        children = make.keys(which(adj_mat[as.integer(cur_node), ]))
        for (child in children) {
            if(has.key(child, unvisited)) {
                stack[[child]] = 1 - color
            }
        }
    }
    values(colors, keys=1:N)
}


assign_softblock <- function(.data, cols, .s2=2, .neighbors=6) {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    N = nrow(cov_mat)
    st = lubridate::now()
    knn = FNN::get.knn(cov_mat, k=.neighbors)
    st = lubridate::now()
    knn.adj = Matrix::sparseMatrix(i=rep(1:N, .neighbors), j=c(knn$nn.index), x=exp(-c(knn$nn.dist) / .s2))
    knn.graph <- graph_from_adjacency_matrix(knn.adj, mode="plus", weighted=TRUE, diag=FALSE)
    E(knn.graph)$weight <- (-1 * E(knn.graph)$weight)
    st = lubridate::now()
    mst.graph = igraph::mst(knn.graph)
    E(mst.graph)$weight <- (-1 * E(mst.graph)$weight)
    st = lubridate::now()
    assignments <- assign_greedily(mst.graph)
    .data$treatment <- assignments
    attr(.data, "laplacian") <- igraph::laplacian_matrix(mst.graph, normalize=TRUE, sparse=TRUE)
    .data
}

assign_greedy_neighbors <- function(.data, cols) {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    N = nrow(cov_mat)
    knn = FNN::get.knn(cov_mat, k=1)
    knn.adj = Matrix::sparseMatrix(i=1:N, j=c(knn$nn.index), x=c(knn$nn.dist))
    knn.graph <- graph_from_adjacency_matrix(knn.adj, mode="plus", weighted=TRUE, diag=FALSE)
    assignments <- assign_greedily(knn.graph)
    .data$treatment <- assignments
    attr(.data, "laplacian") <- igraph::laplacian_matrix(knn.graph, normalize=TRUE, sparse=TRUE)
    .data
}

assign_matched_pairs <- function(.data, cols, .s2=2, .neighbors=6) {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    N = nrow(cov_mat)
    knn = FNN::get.knn(cov_mat, k=.neighbors)
    knn.adj = Matrix::sparseMatrix(i=rep(1:N, .neighbors), j=c(knn$nn.index), x=exp(-c(knn$nn.dist) / .s2))
    knn.graph <- graph_from_adjacency_matrix(knn.adj, mode="plus", weighted=TRUE, diag=FALSE)
    E(knn.graph)$weight <- (-1 * E(knn.graph)$weight)
    mwm.graph = igraph::max_bipartite_match(knn.graph)
    E(mwm.graph)$weight <- (-1 * E(mwm.graph)$weight)
    assignments <- assign_greedily(mwm.graph)
    .data$treatment <- assignments
    attr(.data, "laplacian") <- igraph::laplacian_matrix(mwm.graph, normalize=TRUE, sparse=TRUE)
    .data
}

# library(tibble)
# data = tibble(
#     x1=runif(10),
#     x2=runif(10),
#     x3=rbinom(10, 1, 0.5)
# )
# library(dplyr)
# library(tidyr)
# library(ggplot2)
# data %>% assign_softblock(c(x1, x2)) -> newdata

# ggplot(newdata, aes(x=x1, y=x2, color=factor(treatment), shape=factor(x3))) + geom_point() + theme_minimal()

# newdata %>%
#     attr("laplacian") %>%
#     ifelse(lower.tri(.), ., 0) %>%
#     as_tibble() -> adj_df
# names(adj_df) <- paste0(1:ncol(adj_df))

# adj_df %>%
#     group_by(id_1=as.character(row_number())) %>%
#     gather(id_2, weight, -id_1) %>%
#     filter(weight != 0)  %>%
#     mutate(id_2=as.character(id_2), id=paste(id_1, id_2, sep='-')) -> adj_df

# locs = newdata %>% mutate(id=as.character(row_number())) %>% select(id, x1, x2, x3)

# edges=bind_rows(
# adj_df  %>% inner_join(locs, by=c('id_1'='id')),
# adj_df  %>% inner_join(locs, by=c('id_2'='id'))
# ) %>% arrange(id) %>% ungroup()

# pp = ggplot(newdata, aes(x=x1, y=x2)) +
# geom_line(aes(group=id, size=1), data=edges, color='grey') +
# geom_point(aes(color=factor(treatment), shape=factor(x3), size=2), alpha=.9) +
# scale_size_continuous(range=c(1, 3)) +
# theme_minimal() + theme(legend.position='none')

# print(pp)

Data Preparation

This demo will be based on North Carolina data because their Board of Elections makes it very easy to get precinct level data. I’m also only going to use historical data from the most recent election to avoid needing to match precincts across elections.

With access to a full voter file, this section could be drastically improved by incorporating other important elements into the design like demographics.

Get Precinct data

Results Data

url <- "http://dl.ncsbe.gov/ENRS/2020_11_03/results_pct_20201103.zip"
zip_file <- tempfile(fileext = ".zip")
download.file(url, zip_file, mode = "wb")
spec = cols(
  County = col_character(),
  `Election Date` = col_character(),
  Precinct = col_character(),
  `Contest Group ID` = col_double(),
  `Contest Type` = col_character(),
  `Contest Name` = col_character(),
  Choice = col_character(),
  `Choice Party` = col_character(),
  `Vote For` = col_double(),
  `Election Day` = col_double(),
  `One Stop` = col_double(),
  `Absentee by Mail` = col_double(),
  Provisional = col_double(),
  `Total Votes` = col_double(),
  `Real Precinct` = col_character(),
  X16 = col_skip()
)
results <- readr::read_tsv(zip_file, col_types=spec)

Shapefiles

url = "https://s3.amazonaws.com/dl.ncsbe.gov/ShapeFiles/Precinct/SBE_PRECINCTS_20201018.zip"
temp <- tempfile()
temp2 <- tempfile()
download.file(url, temp)
unzip(zipfile = temp, exdir = temp2)
nc_SHP_file <- list.files(temp2, pattern = ".shp$",full.names=TRUE)
shapes <- sf::read_sf(nc_SHP_file)

Aggregate Data and Join

results %>%
    filter(`Real Precinct` == 'Y') %>%
    group_by(County, Precinct) %>%
    summarize(
        total_vote_pres=sum(`Total Votes`[`Contest Name` == 'US PRESIDENT'], na.rm=TRUE),
        dem_share_pres=sum(`Total Votes`[`Contest Name` == 'US PRESIDENT' & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_pres,
        gop_share_pres=sum(`Total Votes`[`Contest Name` == 'US PRESIDENT' & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_pres,
        total_vote_senate=sum(`Total Votes`[`Contest Name` == 'US SENATE'], na.rm=TRUE),
        dem_share_senate=sum(`Total Votes`[`Contest Name` == 'US SENATE' & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_senate,
        gop_share_senate=sum(`Total Votes`[`Contest Name` == 'US SENATE' & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_senate,
        total_vote_gov=sum(`Total Votes`[`Contest Name` == 'NC GOVERNOR'], na.rm=TRUE),
        dem_share_gov=sum(`Total Votes`[`Contest Name` == 'NC GOVERNOR' & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_gov,
        gop_share_gov=sum(`Total Votes`[`Contest Name` == 'NC GOVERNOR' & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_gov,
        total_vote_house=sum(`Total Votes`[grepl('US HOUSE OF REPRESENTATIVES DISTRICT', `Contest Name`)], na.rm=TRUE),
        dem_share_house=sum(`Total Votes`[grepl('US HOUSE OF REPRESENTATIVES DISTRICT', `Contest Name`) & `Choice Party` == 'DEM'], na.rm=TRUE)/total_vote_house,
        gop_share_house=sum(`Total Votes`[grepl('US HOUSE OF REPRESENTATIVES DISTRICT', `Contest Name`) & `Choice Party` == 'REP'], na.rm=TRUE)/total_vote_house
    ) %>% ungroup() -> results_agg
inner_join(results_agg, shapes, by=c('County'='county_nam', 'Precinct'='prec_id')) -> df_joined
DT::datatable(df_joined %>% sample_n(size=100) %>% dplyr::select(-geometry), rownames = FALSE, options=list(scrollX=TRUE, autoWidth = TRUE))

County	Precinct	total_vote_pres	dem_share_pres	gop_share_pres	total_vote_senate	dem_share_senate	gop_share_senate	total_vote_gov	dem_share_gov	gop_share_gov	total_vote_house	dem_share_house	gop_share_house	id	enr_desc	county_id
WAKE	16-07	462	0.3182	0.632	455	0.3099	0.6176	460	0.3978	0.5804	457	0.3414	0.6018	2319	16-07	92
WAKE	20-11	626	0.3371	0.631	621	0.3076	0.6264	622	0.3923	0.5932	605	0.3355	0.6264	2363	20-11	92
SWAIN	WHCH	2005	0.5711	0.405	1976	0.5273	0.3917	1988	0.583	0.3888	1960	0.5485	0.3908	2460	WHCH	87
CRAVEN	20	2615	0.356	0.6333	2598	0.3499	0.6159	2610	0.3782	0.6142	2559	0.3369	0.6631	1498	FAIRFIELD HARBOUR	25
BUNCOMBE	21.1	919	0.7367	0.2448	911	0.7179	0.2448	911	0.7684	0.2151	904	0.7334	0.2367	1808	HAW CREEK ELEMENTARY SCHOOL	11
ORANGE	CS1	176	0.3409	0.625	175	0.3371	0.6286	176	0.3977	0.5511	173	0.3815	0.6185	178	COLES STORE	68
GUILFORD	OR2	291	0.2062	0.7629	292	0.1849	0.75	294	0.2449	0.7279	289	0.2215	0.7785	966	OR2	41
HENDERSON	SM	2513	0.2475	0.7374	2491	0.2316	0.7206	2504	0.276	0.7065	2483	0.2384	0.7334	793	SOUTH MILLS RIVER	45
HOKE	01	274	0.3723	0.6095	270	0.3889	0.5667	273	0.4505	0.5385	269	0.3903	0.6097	736	ALLENDALE	47
GUILFORD	G32	270	0.3481	0.6222	271	0.3284	0.6458	270	0.4111	0.5704	261	0.3602	0.6398	919	G32	41

The original post rendered this as an interactive DT::datatable widget showing a random sample of 100 precincts. It has been replaced here by the first 10 rows as a static table — truncated for the static rebuild, so no client-side JavaScript is needed.

Add some geographic features

df_joined$geometry %>%
    st_centroid() %>%
    st_transform("+init=epsg:4326") %>%
    st_coordinates() -> latlong
df_joined$longitude = latlong[, 'X']
df_joined$latitude = latlong[, 'Y']
area = df_joined$geometry %>% st_transform("+init=epsg:4326") %>% st_area()
df_joined$area_km2 = units::drop_units(area) / 1e6 # convert m^2 to km^2
df_joined <- df_joined %>% mutate(vote_density_pres = total_vote_pres / area_km2)

Assign treatment four ways

Softblock

start_time = lubridate::now()
df_joined %>% assign_softblock(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
)) %>%
rename(treatment_sb=treatment)-> df_joined
softblock_weights <- attr(df_joined, "laplacian")
end_time = lubridate::now()
print(end_time - start_time)

Time difference of 0.7473469 secs

Greedy nearest neighbors

start_time = lubridate::now()
df_joined %>% assign_greedy_neighbors(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
)) %>%
rename(treatment_nn=treatment)-> df_joined
nn_weights <- attr(df_joined, "laplacian")
end_time = lubridate::now()
print(end_time - start_time)

Time difference of 1.482411 secs

QuickBlock

start_time = lubridate::now()
df_joined %>% dplyr::select(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
) -> df_qb
qb_blocks = quickblock(as.data.frame(df_qb), size_constraint = 6L)
df_joined$treatment_qb = as.integer(as.character(assign_treatment(qb_blocks, treatments=c('0', '1'))))
end_time = lubridate::now()
print(end_time - start_time)

Time difference of 0.03046703 secs

Matched Pairs

# This is extremely slow.
# start_time = lubridate::now()
# df_joined %>% assign_matched_pairs(c(
#     longitude, latitude, area_km2, vote_density_pres, # geographic
#     total_vote_pres, dem_share_pres, gop_share_pres, # 2020 presidential
#     total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
#     total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
#     total_vote_house, dem_share_house, gop_share_house # 2020 house
# )) %>%
# rename(treatment_mp=treatment)-> df_joined
# mp_weights <- attr(df_joined, "laplacian")
# end_time = lubridate::now()
# print(end_time - start_time)

Power simulation

To do power calculations consistently across methods, I will do the following:

Calculate the implied regression adjustment for each design, applied to vote share in 2020 as the outcome.
Find the residual for each point.
Permute residuals over units.
Estimate power for a given effect size by pulling additional permutations.
Sweep over a range of effect sizes.

create_power_simulator = function(W, A, Y) {
    dL = diag(W)
    Dinv = (2 * A - 1) / dL
    estimates = drop((Dinv %*% W) %*% Y)
    residuals = Y - estimates
    x_mat = cbind(A, 1)
    xlx = t(x_mat) %*% W %*% x_mat
    bread = MASS::ginv(as.matrix(xlx))
    detect_effect = function(effect=1) {
        sim_outcome = estimates + sample(residuals) + A * effect
        coefs = bread %*% t(x_mat) %*% W %*% sim_outcome
        r = diag(drop((sim_outcome - (x_mat %*% coefs)) ^ 2))
        meat = t(x_mat) %*% (W %*% r %*% W) %*% x_mat
        vcv = bread %*% meat %*% bread
        upr = coefs[1] + qnorm(0.975) * sqrt(vcv[1])
        lwr = coefs[1] + qnorm(0.025) * sqrt(vcv[1])
        lwr > 0
    }
    detect_effect
}
create_power_simulator_qb = function(blocks, A, Y) {
    estimate_by_block = tapply(Y[A==1], blocks[A==1], mean) - tapply(Y[A==0], blocks[A==0], mean)
    estimates = unlist(purrr::map(blocks, function(b) estimate_by_block[as.character(b)]))
    residuals = Y - estimates
    detect_effect = function(effect=1) {
        sim_outcome = estimates + sample(residuals) + A * effect
        result = quickblock::blocking_estimator(sim_outcome, blocks, A)
        upr = result$effects[2,1] + qnorm(0.975) * sqrt(result$effect_variances[2,1])
        lwr = result$effects[2,1] + qnorm(0.025) * sqrt(result$effect_variances[2,1])
        lwr > 0
    }
    detect_effect
}

Simulate each design

Softblock

simulate_power = create_power_simulator(softblock_weights, df_joined$treatment_sb, df_joined$dem_share_pres)
estimate_power_for_effect = function(effect) mean(replicate(25, simulate_power(effect=effect)))
effects = seq(0, 0.05, length=25)
power_sb = unlist(purrr::map(effects, estimate_power_for_effect))

Greedy Neighbors

simulate_power = create_power_simulator(nn_weights, df_joined$treatment_nn, df_joined$dem_share_pres)
estimate_power_for_effect = function(effect) mean(replicate(25, simulate_power(effect=effect)))
power_nn = unlist(purrr::map(effects, estimate_power_for_effect))

QuickBlock

simulate_power = create_power_simulator_qb(qb_blocks, df_joined$treatment_qb, df_joined$dem_share_pres)
estimate_power_for_effect = function(effect) mean(replicate(25, simulate_power(effect=effect)))
effects = seq(0, 0.05, length=25)
power_qb = unlist(purrr::map(effects, estimate_power_for_effect))

Power Comparison

ggplot(
    tibble(
        effects=c(effects, effects, effects),
        power=c(power_sb, power_nn, power_qb),
        design=c(rep('SoftBlock', length(power_sb)), rep('Greedy Neighbors', length(power_sb)), rep('QuickBlock', length(power_sb)))
    ), aes(effects, power, color=design)) + geom_line() +
    scale_x_continuous('Effect (pp)', labels=scales::percent) +
    scale_y_continuous("Power", labels=scales::percent) +
    scale_color_discrete("Design") +
    theme_minimal()

Analysis

Average Effects

The effect estimates here use the appropriate design-based estimators for each design.

First, I’m going to generate a fake outcome to use. I’ll leave the average effect near zero (0.5pp), but individual effects are random draws from around that vale, but with heterogeneous effects based on democratic vote share in 2020 (i.e. positive effects in democratic precincts and vice versa).

df_joined$outcome = df_joined$dem_share_pres
df_joined$ite = with(df_joined, 0.005 + 0.005 * (plogis((dem_share_pres - median(dem_share_pres)) / sd(dem_share_pres))))
df_joined$outcome_sb = with(df_joined, outcome + treatment_sb * ite)
df_joined$outcome_nn = with(df_joined, outcome + treatment_nn * ite)
df_joined$outcome_qb = with(df_joined, outcome + treatment_qb * ite)

Softblock

estimate_effect = function(W, A, Y) {
    dL = diag(W)
    Dinv = (2 * A - 1) / dL
    x_mat = cbind(A, 1)
    xlx = t(x_mat) %*% W %*% x_mat
    bread = MASS::ginv(as.matrix(xlx))
    coefs = bread %*% t(x_mat) %*% W %*% Y
    r = diag(drop((Y - (x_mat %*% coefs)) ^ 2))
    meat = t(x_mat) %*% (W %*% r %*% W) %*% x_mat
    vcv = bread %*% meat %*% bread
    list(estimate=coefs[1], std.error=sqrt(vcv[1,1]))
}
sb_est = estimate_effect(softblock_weights, df_joined$treatment_sb, df_joined$outcome_sb)

Greedy Neighbors

nn_est = estimate_effect(nn_weights, df_joined$treatment_nn, df_joined$outcome_nn)

QuickBlock

result = quickblock::blocking_estimator(df_joined$outcome_qb, qb_blocks, df_joined$treatment_qb)
qb_est = list(estimate=result$effects[2,1], std.error=sqrt(result$effect_variances[2,1]))

Plot Average Effects

tibble(
    estimate=c(sb_est$estimate, nn_est$estimate, qb_est$estimate),
    std.error=c(sb_est$std.error, nn_est$std.error, qb_est$std.error),
    design=c("SoftBlock", "Greedy Neighbors", "QuickBlock")
) %>%
ggplot(aes(x=design, y=estimate, ymin=estimate-1.96*std.error, ymax=estimate+1.96*std.error)) +
    geom_pointrange() +
    scale_x_discrete("Design") +
    scale_y_continuous("ATE (pp)", labels=scales::percent) +
    coord_flip() +
    theme_minimal()

Heterogeneous Effects

These effects will be estimated using DR-learner of Kennedy (2020). For simplicity, I will estimate nuisance functions using glmnet.

predict.hte.split = function(x, a, y, s, predict.s=4) {
    s.pi = (predict.s) %% 4 + 1
    s.mu = (predict.s + 1) %% 4 + 1
    s.dr = (predict.s + 2) %% 4 + 1
    pihat <- predict(cv.glmnet(x[s==s.pi,],a[s==s.pi], family="binomial", nfolds=10), newx=x, type="response", s="lambda.min")
    mu0hat <- predict(cv.glmnet(x[a==0 & s==s.mu,],y[a==0 & s==s.mu], nfolds=10), newx=x, type="response", s="lambda.min")
    mu1hat <- predict(cv.glmnet(x[a==1 & s==s.mu,],y[a==1 & s==s.mu], nfolds=10),newx=x, type="response", s="lambda.min")
    pseudo <- ((a-pihat)/(pihat*(1-pihat)))*(y-a*mu1hat-(1-a)*mu0hat) + mu1hat - mu0hat
    drl <- predict(cv.glmnet(x[s==s.dr,],pseudo[s==s.dr]),newx=x[s==predict.s, ], s="lambda.min")
    drl
}
predict.hte.crossfit = function(x, a, y) {
    N = length(a)
    s = sample(1:4, N, replace=TRUE)
    hte = rep(NA_real_, N)
    for (split in 1:4) {
        hte[s==split] = predict.hte.split(x, a, y, s, predict.s=split)
    }
    hte
}
calculate_hte <- function(.data, cols, .treatment='treatment', .outcome='outcome') {
    expr <- rlang::enquo(cols)
    pos <- tidyselect::eval_select(expr, data = .data)
    df_cov <- rlang::set_names(.data[pos], names(pos))
    cov_mat = scale(model.matrix(~.+0, df_cov))
    .data$hte = predict.hte.crossfit(cov_mat, .data[[.treatment]], .data[[.outcome]])
    .data
}

Estimate HTEs

Softblock

df_joined %>% calculate_hte(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
), .treatment='treatment_sb', .outcome='outcome_sb') %>% rename(hte_sb=hte) -> df_joined
summary(df_joined$hte_sb)

     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-0.010306  0.004135  0.007536  0.007148  0.009029  0.033580

Greedy Neighbors

df_joined %>% calculate_hte(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
), .treatment='treatment_nn', .outcome='outcome_nn') %>% rename(hte_nn=hte) -> df_joined
summary(df_joined$hte_nn)

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.005742 0.006825 0.007388 0.007670 0.008596 0.010863

QuickBlock

df_joined %>% calculate_hte(c(
    longitude, latitude, area_km2, vote_density_pres, # geographic
    total_vote_pres, # 2020 presidential
    total_vote_senate, dem_share_senate, gop_share_senate, # 2020 senate
    total_vote_gov, dem_share_gov, gop_share_gov, # 2020 governor
    total_vote_house, dem_share_house, gop_share_house # 2020 house
), .treatment='treatment_qb', .outcome='outcome_qb') %>% rename(hte_qb=hte) -> df_joined
summary(df_joined$hte_qb)

      Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
-0.0007862  0.0070832  0.0083612  0.0082249  0.0094074  0.0398552

Plot HTE distributions

ggplot(df_joined, aes()) +
geom_histogram(aes(x=hte_sb, fill='SoftBlock'), bins=50, alpha=0.4) +
geom_histogram(aes(x=hte_nn, fill='Greedy Neighbors'), bins=50, alpha=0.4) +
geom_histogram(aes(x=hte_qb, fill='QuickBlock'), bins=50, alpha=0.4) +
scale_x_continuous("Effect (pp)", labels=scales::percent) +
theme_minimal()

Drew Dimmery

AI Evaluations for conviviality

Why AI evaluation needs this class

Where the course comes from

Design decisions

What’s in the class

What counts as a project

Summing up

Footnotes

Better testing through concentration

The idea

But I don’t know my variance

What else

Why bother

Footnotes

Causal ML (the class)

Teaching CausalML

Syllabus

Course overview

Session-by-session

Session 1: Design-based Causal Inference and Monte Carlo Simulation

Session 2: Covariate Adjustment

Session 3: Balancing Weights

Session 4: Doubly Robust Methods, Double ML, and TMLE

Session 5: Heterogeneous Treatment Effects I

Session 6: Heterogeneous Treatment Effects II --- Neural Networks

Session 7: Off-Policy Evaluation and Optimization

Session 8: Experimental Design

Session 9: Panel Data and Modern Difference-in-Differences

Session 10: Partial Identification

Session 11: Adaptive Experimentation and Reinforcement Learning

Session 12: Interference

Closing thoughts

Footnotes

Website Refresh

Semantic Scholar integration

Listmonk tools

Improvements in design and performance

Purge unused resources

Fonts

My font choices

Why humanist fonts matter

Loading fonts performantly

Remaining concerns

Tending the garden

Footnotes

Back to Basics

Platforms constrain freedom

Platforms enshittify

AI coding tools make smaller projects feasible

Technological vanguardism

Footnotes

What's the point of RCTs?

How to respond to a great paper

Enough for what?

Control of what?

For whom?

Footnotes

What was US2020?

What was Facebook?

Experimentation at scale

Was US2020 independent or industrial research?

Pre-registration as a contract

Facebook researchers were not robots

What does the future hold?

Footnotes

Is Bluesky convivial?

What is Smalltalk?

Empowerment is still about power

Footnotes

Examining the Apparatus

The Apparatus, explained

The Apparatus, sustained

The Apparatus, examined

Footnotes

Calibration as an HTE diagnostic

Footnotes

Plagiarism is bad

What are we doing?

Ok, but what are we doing?