We pin CPU cores for Erlang schedulers on WhatsApp in order to get more performance. An Erlang scheduler is an abstract worker that can process arbitrary units of work. Traditionally in Erlang, after 4000 reductions⁴, the schedulers will swap away from the current task and give another task a chance to run.

Schedulers can be configured to be bound to particular CPUs using the sbt setting, which has many options around NUMA, hyperthread spreading, etc. As a bit of foreshadowing, there is a very interesting note in the documentation:

Suddenly one day we had absolutely garbage P99 response times. This took us a while to debug, but ultimately with the help of various people we were able to pin it down to Lavd⁵. So this gives us a good place to start: why the hell does lavd muck up my cpu affinity?

Luckily for us, lavd has excellent public documentation: https://github.com/sched-ext/scx/tree/ad30a290a1531987a434e034e2ab2307880a4a11/scheds/rust/scx_lavd

The first paragraph gives us a good explainer (emphasis mine):

scx_lavd is a BPF scheduler that implements an LAVD (Latency-criticality Aware Virtual Deadline) scheduling algorithm. While LAVD is new and still evolving, its core ideas are 1) measuring how much a task is latency critical and 2) leveraging the task's latency-criticality information in making various scheduling decisions (e.g., task's deadline, time slice, etc.). As the name implies, LAVD is based on the foundation of deadline scheduling. This scheduler consists of the BPF part and the Rust part. The BPF part makes all the scheduling decisions; the Rust part provides high-level information (e.g., CPU topology) to the BPF code, loads the BPF code and conducts other chores (e.g., printing sampled scheduling decisions).

So we have a BPF thing that polls and gets scheduling information, and then we have a main.rs that binds CPU topology. Neat. Before we go further, we need to understand what lavd sits on top of. We are in EEVDF land for the Linux scheduler now, which has completely replaced the old CFS scheduler. So forget your time quanta, now you need to think about lag⁶.

In the classical EEVDF space, we have a per CPU runqueue⁷. Specifically, the runqueue struct has this information:

struct rq {
  struct cfs_rq		cfs;
  struct rt_rq		rt;
  struct dl_rq		dl;
#ifdef CONFIG_SCHED_CLASS_EXT
  struct scx_rq		scx;
#endif

}

So, cool, we can see that there's a runqueue for scx (sched_ext) and a runqueue for eevdf (still called cfs). Given that it's per CPU, this means that each CPU maintains its own rb tree of runnable tasks⁸:

struct cfs_rq {
  /* ... other not interesting stuff ... */
  struct rb_root_cached	tasks_timeline; // rb tree
}

So that's neat. This tree means that all the tasks on a given core only compete against other tasks on that same core.

Now if we look at lavd, specifically the scheduling mechanism⁹:

static u64 calc_virtual_deadline_delta(struct task_struct *p,
                                       task_ctx *taskc)
{
  u64 deadline, adjusted_runtime;
  u32 greedy_penalty;

  /*
   * Calculate the deadline based on runtime,
   * latency criticality, and greedy ratio.
   */
  calc_lat_cri(p, taskc);
  greedy_penalty = calc_greedy_penalty(p, taskc);
  adjusted_runtime = calc_adjusted_runtime(taskc);

  deadline = (adjusted_runtime * greedy_penalty) / taskc->lat_cri;
  return deadline >> LAVD_SHIFT;
}

We can see here that there's a notion of "latency criticality" and "greediness." A higher latency criticality gives a tighter deadline, and the greedy penalty ensures fairness while scheduling.

Lavd's Rust handoff code is in main.rs, where scheduler::run() is the handoff to BPF. If autopower is on, we poll the system power profile and then push to BPF, and then process stats.

The BPF code then wakes up, and performs this gigantic list of operations:

SCX_OPS_DEFINE(lavd_ops,
             .select_cpu		= (void *)lavd_select_cpu,
             .enqueue			= (void *)lavd_enqueue,
             .dequeue			= (void *)lavd_dequeue,
             .dispatch		= (void *)lavd_dispatch,
             .runnable		= (void *)lavd_runnable,
             .running			= (void *)lavd_running,
             .tick			= (void *)lavd_tick,
             .stopping		= (void *)lavd_stopping,
             .quiescent		= (void *)lavd_quiescent,
...

You can look at the individual functions, but:

• lavd_enqueue() calculates the virtual deadline, and inserts it into a queueing structure called the DSQ.
• lavd_dispatch() pulls work from the DSQ
• lavd_running() calculates the time slices, and advances the logical clock
• lavd_tick() has some very key logic that checks if there is any higher priority task left in the DSQ. If there is, it shrinks the current time slice to give the higher priority task a chance to run, effectively yielding the current task.
• lavd_stopping() is a cleanup op
• lavd_quiescent() handles various sleeping stats

So critical to this entire system is the concept of a DSQ (dispatch queue). Unlike EEVDF's per-CPU runqueues, a DSQ is shared across multiple CPUs, arranged either by the last level cache (LLC) or an AMD core complex (CCX).

For example, you can think of a traditional CPU with 4 cores, each with their own L1/L2 caches but a big shared L3 cache as sharing one DSQ. This is a more traditional CPU:

¹⁰

However, with the rise of AMD's packaging, we've effectively added a newer interconnect called the "Infinity Fabric":

So this means that for older, Intel chips, they'll have a single DSQ for all their cores on the CPU, while newer AMD CPUs will have multiple DSQs, one for each CCX.

We now need to take another digression into how the Erlang (BEAM) scheduler works. We talked about this briefly at the start, but I'll go more into detail here.

The BEAM scheduling mechanism is divided into subsystems called schedulers¹¹. There is typically a single scheduler per core, and each "scheduler" is really an OS thread that acts as a worker, doing some amount of work before switching to another piece of work¹². From the Linux kernel's perspective, each Erlang scheduler is just another thread to be scheduled. Because Erlang is an actor based programming paradigm, this ensures that each actor only gets a particular amount of CPU, before yielding to another actor. Erlang uses a dimensionless metric known as a "reduction." A reduction roughly maps to a single function call, and because Erlang is a functional programming language, you're guaranteed to have to make some function calls at some point¹³. After a specific amount of reductions (4k as of writing), the scheduler switches¹⁴.

Because the BEAM has cascading schedulers, this fixed reduction count means that when a scheduler has a bunch of stuff queued up and it doesn't execute, then it piles up the work. Unscheduled schedulers (heh) also cannot have tasks stolen without cooperation from the victim scheduler.¹⁵

Now, as with any scheduling system, we need to implement some form of task stealing. If scheduler A is running something and has a big queue of work while scheduler B has nothing to do, it surely makes sense to allow scheduler B to pick up some of the slack. Erlang does this with single task stealing, which means that a scheduler steals a single task from another scheduler's runqueue after the victim scheduler has acknowledged it^{16, 17}.

Phew! We've finally gotten to the actual issue!

At the start, I mentioned that we bound Erlang schedulers to specific CPUs, pinning specific schedulers to specific cores.

Remember that the Erlang scheduler thread has CPU affinity set, so it can only run on, say, CPU 3. But lavd didn't have per-CPU queues at the time. It dumps the thread into the shared LLC DSQ alongside everything else. So the Erlang scheduler thread only gets picked up when CPU 3 happens to pull from the front of the DSQ, which might not be for a while.

This also causes head-of-line blocking. When CPUs 0, 1, and 2 scan the DSQ for work, they find the Erlang scheduler thread sitting there, pinned to CPU 3 and useless to them. They have to iterate past it. Not catastrophic on its own, but it compounds.

In Erlang land, because Erlang schedulers cannot steal tasks without the acknowledgement of the victim scheduler, starving a particular Erlang scheduler means that its runqueue is effectively frozen. Even worse, this creates a second order effect, where:

1. Lavd would finally allow the BEAM scheduler to run
2. The BEAM scheduler runs for a very long time, exceeding the amount of time it's allocated, because it has so much work to do and hasn't been able to offload any of its work to other schedulers
3. Exceeding this virtual deadline means that lavd naturally pushes back the BEAM scheduler further back into the DSQ, due to the greediness penalty.
4. Because it takes so long for the BEAM scheduler to get a chance to run again, it builds up so much work that it exceeds the virtual deadline even further, worsening the situation!

Now this situation was less serious on AMD chips, because multiple CCXs meant multiple shallower DSQs, so the BEAM schedulers got to run more often. However, on machines with a single LLC, this meant that a fat DSQ could cause the BEAM schedulers to get shoved way into the back of the queue.

This was fixed with per-CPU DSQs for Lavd, which is conceptually simple. But the outage wasn't caused by any single bad decision. Erlang's CPU pinning is reasonable. Lavd's shared DSQ was a reasonable default. The greediness penalty is a fair scheduling mechanism. Each layer was doing something sensible in isolation. The problem is that runtime schedulers and kernel schedulers are mostly unaware of each other, and when their assumptions collide, you get feedback loops that neither system is equipped to break.

One of the core primitives in Erlang is called gen_server. This is effectively a single "actor" if you're familiar with the actor framework. A single gen_server holds some stateful information in a variable called State². A gen_server communicates with other actors through messages, and there's a concept of a mailbox, and performing RPC calls depending on the specifics of a message.

The Erlang Runtime System (ERTS) uses a code server, basically a running process that keeps track of compiled code that's been loaded into the BEAM VM. The system is set up to keep two versions of code, a "current" version and an "old" version. What this means is that if you're an actor running the "old" version of the code, your local calls will still get routed to the "old" version, while newer fully qualified remote calls will get routed to the "current" version of the code.

This plays out like this:

You'll often want to change the information stored in State, or the structure of it. For example, consider this code:

-module(counter).
-behaviour(gen_server).
-vsn("1.0.0").
-export([start_link/0, get_count/0, increment/0]).
-export([init/1, handle_call/3, handle_cast/2, code_change/3, terminate/2, handle_info/2]).

%% -- API --
start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

get_count() ->
    gen_server:call(?MODULE, get_count).

increment() ->
    gen_server:cast(?MODULE, increment).

%% -- Callbacks --
init([]) ->
    {ok, 0}.

handle_cast(increment, Count) ->
    {noreply, Count + 1}.

handle_call(get_count, _From, Count) ->
    {reply, Count, Count}.

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.

This gen_server stores an integer Count that increments when you call counter:increment(), and returns via counter:get_count().

In the shell:

peixian@Mac ~/c/erl-hot-reloading> erl
Erlang/OTP 27 [erts-15.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Eshell V15.2 (press Ctrl+G to abort, type help(). for help)
1> file:copy("counter_v1.erl", "counter.erl"), c(counter).
{ok,counter}
2> counter:start_link().
{ok,<0.95.0>}
3> counter:get_count().
0
4> counter:increment().
ok
5> counter:get_count().
1
6> counter:increment().
ok
7> counter:increment().
ok
8> counter:get_count().
3

The code_change callback handles migration between versions. Right now State is a single integer. Not great if we want to track when it was last updated.

We can change our counter module to:

-module(counter).
-behaviour(gen_server).
-vsn("2.0.0").
-export([start_link/0, get_count/0, increment/0, get_info/0]).
-export([init/1, handle_call/3, handle_cast/2, code_change/3, terminate/2, handle_info/2]).

%% State map
-record(state, {
                count = 0,
                increment_count = 0,
                last_updated
               }).

%% -- API --
start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

get_count() ->
    gen_server:call(?MODULE, get_count).

increment() ->
    gen_server:cast(?MODULE, increment).

get_info() ->
    sys:get_state(?MODULE).

%% -- Callbacks --

init([]) ->
     {ok, #state{}}.

handle_cast(increment, State) ->
    {noreply, State#state{count = State#state.count + 1,
                          increment_count = 1,
                          last_updated = erlang:timestamp()
                         }}.

handle_call(get_count, _From, State) ->
    {reply, State#state.count, State}.

code_change("1.0.0", OldState, _Extra) when is_integer(OldState) ->
    NewState = #state{
                  count = OldState,
                  increment_count = OldState,
                  last_updated = erlang:timestamp()},
    {ok, NewState};

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

handle_info(_Info, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.

We added counter:get_info() to return the state, and this migration clause:

code_change("1.0.0", OldState, _Extra) when is_integer(OldState) ->
    NewState = #state{
                  count = OldState,
                  increment_count = OldState,
                  last_updated = erlang:timestamp()},
    {ok, NewState};

This chunk allows us to migrate old state, checking when the old state was specifically an integer. The vsn tag allows us to easily distinguish which version it's coming from.

In the shell:

peixian@Mac ~/c/erl-hot-reloading> erl
Erlang/OTP 27 [erts-15.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Eshell V15.2 (press Ctrl+G to abort, type help(). for help)
1> file:copy("counter_v1.erl", "counter.erl"), c(counter).
{ok,counter}
2> counter:start_link().
{ok,<0.95.0>}
3> counter:increment().
ok
4> counter:increment().
ok
5> counter:get_count().
2
6> file:copy("counter_v2.erl", "counter.erl"), c(counter).
{ok,counter}
7> counter:get_info().
2
8> sys:suspend(counter).
ok
9> sys:change_code(counter, counter, "1.0.0", []).
ok
10> sys:resume(counter).
ok
11> counter:get_info().
{state,2,2,{1769,908677,996152}}
12>

Commands 1-5: load v1, increment twice, count is 2. Command 6: load v2, which gives us counter:get_info().

The key sequence is 8, 9, and 10: we suspend the process, reload the code (calling code_change), then resume. On 11, we see the state has changed from a single integer into the tuple record.

The sequence diagram then looks like this:

Hot reloading is useful because it's fast and preserves user state. Let's see how this works with TCP.

Here's a demo: V1 is an echo server, V2 upcases everything. We hot reload without dropping the netcat connection:

Erlang implements TCP with gen_tcp. You can read more about the specifics, but the short version is gen_tcp implements a listener that hands off the socket to another process.

V1:

-module(tcp_demo).
-behaviour(gen_server).
-vsn("1.0.0").

-export([start_link/0]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, code_change/3, terminate/2]).

-record(state, {socket}).

%% -- API --

start_link() ->
    spawn(fun() -> start_listener() end).

start_listener() ->
    {ok, Listen} = gen_tcp:listen(9000, [binary, {packet, 0}, {active, true}, {reuseaddr, true}]),
    io:format("Listening on 9000...~n"),
    {ok, Socket} = gen_tcp:accept(Listen),
    gen_tcp:close(Listen),

    %% Start the gen_server (using start, not start_link, so it's independent)
    {ok, Pid} = gen_server:start({local, ?MODULE}, ?MODULE, Socket, []),

    ok = gen_tcp:controlling_process(Socket, Pid),

    io:format("Socket transferred to PID ~p. Listener dying.~n", [Pid]).

%% -- Callbacks --

init(Socket) ->
    io:format("V1 (Echo) Started.~n"),
    {ok, #state{socket = Socket}}.

handle_info({tcp, Socket, Data}, State) ->
    gen_tcp:send(Socket, Data), %% Echo back whatever the user sent
    {noreply, State};

handle_info({tcp_closed, _S}, State) ->
    io:format("Client disconnected~n"),
    {stop, normal, State}.

handle_call(_Req, _From, State) -> {reply, ok, State}.
handle_cast(_Msg, State) -> {noreply, State}.
code_change(_OldVsn, State, _Extra) -> {ok, State}.
terminate(_Reason, _State) -> ok.

Run this and nc localhost 9000. It echoes back whatever you type.

V2:

-module(tcp_demo).
-behaviour(gen_server).
-vsn("2.0.0").

-export([start_link/0]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, code_change/3, terminate/2]).

-record(state, {socket, buffer = <<>>}).


%% -- API --
start_link() ->
    spawn(fun() -> start_listener() end).

start_listener() ->
    {ok, Listen} = gen_tcp:listen(9000, [binary, {packet, 0}, {active, true}, {reuseaddr, true}]),
    io:format("V2 Listener on 9000...~n"),
    {ok, Socket} = gen_tcp:accept(Listen),
    gen_tcp:close(Listen),

    {ok, Pid} = gen_server:start({local, ?MODULE}, ?MODULE, Socket, []),
    ok = gen_tcp:controlling_process(Socket, Pid),
    io:format("Socket transferred to PID ~p. Listener dying.~n", [Pid]).

%% -- Callbacks --

init(Socket) ->
    io:format("V2 (Shouting) Started.~n"),
    {ok, #state{socket = Socket, buffer = <<>>}}.

%% V2 Logic: Buffering & Shouting
handle_info({tcp, Socket, Data}, State = #state{buffer = Buff}) ->
    NewBuff = <<Buff/binary, Data/binary>>,
    {Lines, Rest} = split_lines(NewBuff, []),

    %% Process all complete lines found
    [gen_tcp:send(Socket, <<(string:uppercase(Line))/binary, "\n">>) || Line <- Lines],

    {noreply, State#state{buffer = Rest}};

handle_info({tcp_closed, _S}, State) ->
    io:format("Client disconnected~n"),
    {stop, normal, State}.

%% -- Migration Logic --

%% Upgrade from V1 (Socket only) -> V2 (Socket + Buffer)
code_change("1.0.0", {state, Socket}, _Extra) ->
    io:format("Migrating State: V1 -> V2 (Adding buffer)~n"),
    {ok, #state{socket = Socket, buffer = <<>>}};

code_change(_OldVsn, State, _Extra) -> {ok, State}.

%% -- Helpers --

%% Recursively split binary into lines
split_lines(Bin, Acc) ->
    case binary:split(Bin, <<"\n">>) of
        [Line, Rest] -> split_lines(Rest, [Line | Acc]);
        [Last]       -> {lists:reverse(Acc), Last}
    end.

handle_call(_Req, _From, State) -> {reply, ok, State}.
handle_cast(_Msg, State) -> {noreply, State}.
terminate(_Reason, _State) -> ok.

The V2 code adds a buffer field to the state. Why buffer? TCP is a stream protocol, so it doesn't preserve message boundaries. A single recv might give you half a line, two lines, or one and a half lines. Without buffering, you'd uppercase partial words or chop messages in weird places. The buffer accumulates bytes until we hit a newline, then we process the complete line.

The code_change clause migrates the V1 state (socket only) to V2 (socket + empty buffer). Any partial data in flight gets handled correctly after the reload.

For most situations you probably don't want to be doing this. However, there's a few cases where it's been invaluable:

• heisenbugs that are state dependent, that you're not sure if you're going to be able to reproduce if you do a reconnect/service restart. As much as we want our systems to be stateless, many of them aren't.
• very quick remediations. I've seen this twice, but hot code reloading effectively means you can parallel ssh into all machines and hot reload and patch, avoiding the need for an hours long deploy across tens of thousands of machines.

You can find all the code for this on https://git.sr.ht/~peixian/erlang-hot-reloading/ as well.

• Pieces of the Action, by Vannevar Bush (1970)
• Gathering Moss: A Natural and Cultural History of Mosses, by Robin Wall Kimmerer (2003)
• Instantiation, by Greg Egan (2020)
• The Posthumous Memoirs of Brás Cubas, by Macahdo de Assis (1881)

• Planet of Slums, by Mike Davis (2006) LOVED

• Breakneck: China's Quest to Engineer the Future, by Dan Wang (2025)
• Sarajevo Firewood by Said Khatibi (2021)
• Salvador, by Joan Didion (1983)

• People from Oetimu, by Felix K. Nesi (2019) LOVED
• Perfection, by Vincezo Latronico (2022)
• Shadow of the Sun, by Taleb Alrefai (2012)
• The Living and the Rest, by José Eduardo Agualusa (2020)
• Operating Systems: Three Easy Pieces, by Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau (2012)
• Secular Translations: Nation-State, Modern Self, and Calculative Reason, by Talal Asad (2018) LOVED
• Salvage: a Personal Odyssey, by Ian Tew (2007)

• Wall Street Noir, edited by Stephen Rhodes (2007)
• Delhi Noir, edited by Hirsh Sawhney (2009)
• Talking to Ourselves, by Andrés Neuman (2012)
• A General Theory of Oblivion, by José Eduardo Agualusa (2012)
• Hong Kong Noir, edited by Jason Y Ng (2018)
• Song for the Unraveling of the World: Stories, by Brian Evenson (2019)

• São Paulo Noir, edited by Tony Bellotto (2018)

• Glorious Exploits, by Ferdia Lennon (2024) LOVED

• High Static, Dead Lines: Sonic Spectres & the Object Hereafter, by Kristen Gallerneaux (2018)
• Focus: The ASML way - Inside the power struggle over the most complex machine on earth, by Martin Hijink (2024)

• At the Tomb of the Inflatable Pig: Travels Through Paraguay, by John Gimlette (2003)
• Turn the Ship Around!: A True Story of Turning Followers into Leaders, by L. David Marquet (2013)
• Mona, by Pola Oloixarac (2019)
• Death is Hard Work, by Khaled Khalifa (2016) LOVED
• Last Days, by Brian Evenson (2009)
• The Glassy, Burning Floor of Hell, by Brian Evenson (2021)
• Marrakech Noir, edited by Adnan Yassin (2018)
• The City and Its Uncertain Walls, by Haruki Murakami (2023)

• The Data Center as a Computer: An Introduction to the Design of Warehouse-Scale Machines, by Luiz Andre Barroso (2009) LOVED

• Mountains vs Desire: Climbing vs the End of the World, by Margret Grebowicz (2021)
• Montreal Noir, edited by Jacques Filippi (2017)
• Smoke and Ashes: Opium's Hidden Stories, by Amitav Ghosh (2024)
• Unsong, by Scott Alexander (2017)

• Drug Cartels Do Not Exist: Narcotrafficking in US and Mexican Culture, by Oswaldo Zavala LOVED (2022)
• Moonbound, by Robin Sloan (2024)

• Rio Noir, edited by Luiz Alfredo Garcia-Roza (2014)
• The Book of All Skies, by Greg Egan (2021)
• Distress, by Greg Egan (2013)

• The Mountain in the Sea, by Ray Nayler (2022)
• Wounded Tigris: A River Journey Through the Cradle of Civilization, by Leon McCarron (2023)

• Morphotropic, by Greg Egan (2024)
• Beirut Noir, edited by Rawi Hage (2015)

• The Epiphany of Gliese 581, by Fernando Borretti (2022) LOVED

• Can the Monster Speak? A Report to an Academy of Psychoanalysts, by Paul B. Preciado (2020)
• Kill It with Fire: Manage Aging Computer Systems, by Marianne Bellotti (2021)

• Languages: A Very Short Introduction, by Stephen R. Anderson (2012)
• The Adventure, by Georgio Agamben (2018)
• Cruel Optimism, by Lauren Berlant (2011) LOVED
• Preliminary Materials for a Theory of the Young-Girl, by Tiqqun (1999) LOVED
• Busy Doing Nothing, by Rekka Bellum and Devine Lu Linvega (2021)
• Four Thousand Weeks: Time Management for Mortals, by Oliver Burkeman (2021)

• The Purple Land, by William Henry Hudson (1885)
• Ra, by qntm (2014) LOVED
• Havana Noir, edited by Leonardo Padura Fuentes (2007)
• Wiktopher, by Rekka Bellum (2023)
• Philosophy for Passengers, by Michael Marder (2022)

• Singapore Noir, edited by Lu-Lien Cheryl Tan (2014)
• Tehran Noir, edited by Salar Abdoh (2014)
• Buenos Aires Noir, edited by Ernesto Mallo (2014)

• Norweigan Wood, by Haruki Murakami (1987)

• Logic Beach: Part I, by Exurb1a (2017)
• Stuff Goes Bad: Erlang in Anger, by Fred Hebert (2014)
• There Is No Antimemetics Division, by qntm (2020)
• The Bridge to Lucy Dunne, by Exurb1a (2016)
• Permutation City, by Greg Egan (1994)
• Ed, by qntm (2013)
• Sleep and the Soul, by Greg Egan (2023)
• The Utopia of Rules: On Technology, Stupidity, and the Secret Joys of Bureaucracy, by David Graeber (2013)

• Unwanted Witnesses: Journalists and Conflict in Latin America, by Gabriela Polit Duenas (2019)
• Diaspora, by Greg Egan (1997)

• A Thousand Trails Home, by Seth Kantner (2021)

• The Prince of Milk, by Exurb1a (2017)

• An Anthropology of Nothing in Particular, by Martin Demant Frederiksen (2018)

• Discourse on Colonialism, by Aimé Césaire (1955)
• Hollow Kingdom, by Jane Kira Buxton (2019)

• Valuable Humans in Transit and Other Stories, by qntm (2022)
• Venomous Lumpsucker, by Ned Beauman (2022)
• On Suicide Bombing, by Talal Asad (2007)

• Dark Constellations, Pola Oloxiarac (2019)
• Poems for the Lost Because I'm Lost Too, by Exurb1a (2022)

• Invitation to a Beheading, by Vladimir Nabokov (2011)
• Chasing Homer, by László Krasznahorkai (2021)
• Total Shambles, by F. George (2015)

• Intoxicology: A Cultural History of Drink and Drugs, by Walton Stuart (2016)
• Automatic Religion: Nearhuman Agents of Brazil and France, by Paul Christopher Johnson (2020)

• The Fifth Science, by Exurb1a (2018)
• Pilgrims Until We Die: Unending Pilgrimage in Shikoku, by Ian Reader (2021)
• Raccoon, by Daniel Heath Justice (2021)
• Awake, by Harald Voetmann (2021)

• Dirtbag, Massachusetts: A Confessional, by Isaac Fitzgerald (2022)
• My War Gone By, I Miss It So, by Anthony Loyd (1999)
• The Kind Worth Killing, by Peter Swanson (2015)
• Washington Bullets, by Vijay Prashad (2020)
• Another Bloody Love Letter, by Anthony Loyd (2007)

• Meaty, by Samantha Irby (2013)
• Experimenting with Ethnography: A Companion to Analysis, by Andrea Ballestero (2021)

• The Elephant Vanishes, by Haruki Murakami (1993)
• Redemptive Suffering in Islam: A Study of the Devotional Aspects of Ashura in Twelver Shi'ism, by Mahmoud M. Ayoub (1978)
• How Did You Get This Number: Essays, by Sloan Crosley (2010)
• Unbury Our Dead With Song, by Mukoma wa Ngugi (2021)
• Modularity - The Engines of Cognition, by the LessWrong community (2021)

• What Is Religious Authority?: Cultivating Islamic Communities in Indonesia, by Ismail Fajrie Alatas (2021)
• Trust - The Engines of Cognition, by the LessWrong community (2021)
• The Courage to Act: A Memoir of a Crisis and Its Aftermath, by Ben S. Bernanke (2015)
• Fine Structure, by qntm (2010)
• Carte Blanche: The Erosion of Medical Consent, by Harriet A. Washington (2021)
• After Dark, by Haruki Murakami (2007)

• The Graves of Tarim: Genealogy and Mobility across the Indian Ocean, by Enseng Ho (2006)
• Baghdad Noir, by Samuel Shimon (2018)

• The Man Who Solved the Market: How Jim Simons Launched the Quant Revolution, by Gregory Zuckerman (2019)

• An Elegant Puzzle: Systems of Engineering Management, by Will Larson (2019)
• Staff Engineer: Leadership Beyond the Management Track, by Will Larson (2021)
• Secular Translations: Nation-State, Modern Self, and Calculative Reason, by Talal Asad (2018)
• The Impossibility of Religious Freedom, by Winnifred Fallers Sullivan (2007)
• Sex and Secularism, by Joan Wallach Scott (2017)

• Modern Things on Trial: Islam's Global and Material Reformation in the Age of Rida, 1865-1935, by Leor Halevi (2019)

• The Enlightenment Bible: Translation, Scholarship, Culture, by Jonathan Sheehan (2005)
• Locations of Buddhism: Colonialism and Modernity in Sri Lanka, by Anne M. Blackburn (2010)

• Geometry for Ocelots, by Exurb1a (2021)

• The Kowloon English Club, by Stephen Griffiths (2019)
• I, Who Did Not Die, by Zahed Haftlang and Najah Aboud (2017)
• The Spymaster of Baghdad, by Margaret Coker (2021)

• The Structure of Scientific Revolutions, by Thomas S. Kuhn (1962)
• This City is a Minefield, by Aaron Chan (2019)
• Good Times in Dystopia, by George F. (2020)
• First Person Singular: Stories, by Haruki Murakami (2021)

• Understanding 'sectarianism': Sunni-Shi'a Relations in the Modern Arab World, by Fanar Haddad (2020)

• Beyond Debt: Islamic Experiments in Global Finance, by Daromir Rudnyckyj (2018)

• Out of Mesopotamia, by Salar Abdoh (2020)
• The Bus Driver Who Wanted to be God & Other Stories, by Etgar Keret (2001)
• Scientific Freedom: The Exilir of Civilization, by Donald W. Braben (2008)
• Finite and Infinite Games: A Vision of Life as Play and Possibility, by James P. Carse (1987)

• History Has Begun, by Bruno Macaes (2020)
• War from the Ground Up: Twenty-First Century Combat as Politics, by Emile Simpson (2012)
• Arabian Satire: Poetry from the 18th Century Najd, by Ḥmēdān al-Shwēʿir , Jane Tylus (Foreword), Marcel Kurpershoek (Translator) (2020)
• Savage Theories, by Pola Oloixarac (2008)

• Contending Visions of the Middle East: The History and Politics of Orientialism, by Zachary Lockman (2004)
• The Dip: A Little Book That Teaches You When to Quit, by Seth Godin (2007)

• Ghostwritten, by David Mitchell (1999)

• Baghdad: City of Peace, City of Blood – A History in Thirteen Centuries, by Justin Marozzi (2014)
• Dance Dance Dance, by Haruki Murakami (1988)
• A Wild Sheep Chase (The Rat, #3), by Haruki Murakami (1982)

• Religion and Nationalism in Global Perspective, by J. Christopher Soper and Joel S. Fetzer (2018)
• The Last Wolf / Herman, by László Krasznahorkai (2016)
• New Dark Age: Technology and the End of the Future, by James Bridle (2018)
• Peace Operations Seen From Below: UN Missions and Local People, by Beatrice Pouligny (2006)
• A Concise History of Sunnis and Shi'is, by John McHugo (2017)
• State of Repression: Iraq Under Saddam Hussein, by Lisa Blaydes (2018)
• Goat Days, by Benyamin (2008)

• Ultralearning: Master Hard Skills, Outsmart the Competition, and Accelerate Your Career, by Scott Young (2019)
• Streetwise: How Taxi Drivers Establish Customer's Trustworthiness, by Diego Gambetta and Heather Hamill (2005)
• Look Alive Out There, by Sloane Crosley (2018)
• Chaos, a Fable, by Rodrigo Rey Rosa (2019)
• Iraq After America: Strongmen, Sectarians, Resistance, by Joel Rayburn (2014)

• A PhD Is Not Enough: A Guide To Survival In Science, by Peter J. Feibelman (1993)
• How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking – for Students, Academics and Nonfiction Book Writers, by Sönke Ahrens (2017)
• Ottoman Refugees, 1878-1939: Migration in a Post-Imperial World, by Isa Blumi (2013)
• Afghanistan Rising: Islamic Law and Statecraft Between the Ottoman and British Empires, by Faiz Ahmed (2017)
• Spiritual Subjects: Central Asian Pilgrims and the Ottoman Hajj at the End of Empire, by Lâle Can (2020)
• Training for the New Alpinism: A Manual for the Climber as Athlete, by Steve House & Scott Johnston (2014)
• Training for the Uphill Athlete: A Manual for Mountain Runners and Ski Mountaineers, by Steve House (2019)
• The Universal Enemy: Jihad as Solidarity at the End of Empire, by Darryl Li (2019)
• Deep Work: Rules for Focused Success in a Distracted World, by Cal Newport (2016)
• Our Women on the Ground: Essays by Arab Women Reporting from the Arab World, edited by Zahra Hankir (2019)

• Imagined Communities: Reflections on the Origin and Spread of Nationalism, by Benedict Anderson (1983)
• Humanism in Ruins: Entangled Legacies of the Greek-Turkish Population Exchange, by Asli Igsiz (2019)
• How to Write a Thesis, by Umberto Eco (1977)
• The Last Ottoman Generation and the Making of the Modern Middle East, by Michael Provence (2017)

• Familiar Futures: Time, Selfhood, and Sovereignty in Iraq, by Sara Pursely (2019)
• A Nation of Empire: The Ottoman Legacy of Turkish Modernity, by Michael Meeker (2002)

• The Book of Disappearance, by Ibtisam Azem (2019)
• Scenes from Village Life, by Amos Oz (2011)

• The Great War for Civilization: The Conquest of the Middle East, by Robert Fisk (2005)

• Seven Pillars of Wisdom, by T.E. Lawrence (1922)

• The Wandering Falcon, by Jamil Ahmad (2011)

• First as Tragedy, Then as Farce, by Slavoj Žižek (2009)

• House on Fire: The Fight to Eradicate Smallpox, by William H. Foege (2011)
• Asymmetry, by Lisa Halliday (2018)

• Secession and Security: Explaining State Strategy Against Separatists, by Ahsan Butt (2017)
• The Bookseller of Kabul, by Åsne Siererstad (2004)
• The Burning Shores: Inside the Battle for the New Libya, by Frederic Wehrey (2018)

• The Little Book of Fixers, by Jan Chipchase (2019)
• Sacred Cesium Group and Isa's Deluge: Two Novellas of Japan's 3/11 Disaster, by Yusuke Kimura (2019)
• The Infernal Library: On Dictators, the Books They Wrote, and Other Catastrophes of Literacy, by Daniel Kalder (2018)
• Extremism, by J.M. Berger (2018)
• The Locust Effect: Why the End of Poverty Requires the End of Violence, by Gary A. Haugen (2014)

• Burning Country: Syrians in Revolution and War, by Robin Yassin-Kassab (2016)
• War Reporting for Cowards, by Chris Ayres (2006)
• We Want to Negotiate: The Secret World of Kidnapping, Hostages and Ransom, by Joel Simon (2019)
• Babel: Around the World in Twenty Languages, by Gaston Dorren (2018)

• Literature from the "Axis of Evil": Writing from Iran, Iraq, North Korea, and Other Enemy Nations, edited by Alane Mason (2006)
• To the Mountains: My Life in Jihad, from Algeria to Afghanistan, by Abdullah Anas (2019)

• Ibn Khaldun: An Intellectual Biography, by Robert Irwin (2018)
• Why Don't We Learn from History?, by B.H. Liddel Hart (1944)

• The Wind-Up Bird Chronicle, by Haruki Murakami (1994)
• Colorless Tsukuru Tazaki and His Years of Pilgrimage, by Haruki Murakami (2013)
• The Curse of Bigness: Antitrust in the New Gilded Age, by Tim Wu (2018)
• A Month by the Sea: Encounters in Gaza, by Dervla Murphy (2013)
• Kafka on the Shore, by Haruki Murakami (2003)

• Hezbollah: The Global Footprint of Lebanon's Party of God, by Matthew Levitt (2013)
• Babel No More: The Search for the World's Most Extraordinary Language Learners, by Michael Erard (2012)
• The Master and Margarita, by Mikhail Bulgakov (1967)
• Killing Commendatore, by Haruki Murakami (2017)

• The Baghdad Eucharist: A Novel, by Sinan Antoon (2012)
• The 2020 Commission Report on the North Korean Nuclear Attacks Against the United States: A Speculative Novel, by Jeffery Lewis (2018)
• Judas, by Amos Oz (2015)
• Saudi America: The Truth about Fracking and How It's Changing the World, by Bethany McLean (2018)
• Into the Hands of the Soldiers: Freedom and Chaos in Egypt and the Middle East, by David Kirkpatrick (2018)

• Quicksilver War: Syria, Iraq and the Spiral of Conflict, by William Harris (2017)
• An Intimate War: An Oral History of the Helman Conflict, by Mike Martin (2017)

• Spies in the Family: An American Spymaster, His Russian Crown Jewel, and the Friendship That Helped End the Cold War, by Eva Dillion (2017)
• Ballots, Bullets, and Bargains: American Foreign Policy and Presidential Elections, by Michael H Armacost (2015)
• Ignition!: An Informal History of Liquid Rocket Propellants, by John Drury Clark (1972)
• The Impossible Revolution: Making Sense of the Syrian Tragedy, by Yassin Al-Haj Saleh (2017)
• Aftershocks: Great Powers and Domestic Reforms in the Twentieth Century, by Seva Gunitsky (2017)
• The Taliban Reader: War, Islam, Politics, edited by Alex Strick van Linschoten and Felix Kuehn (2018)
• Couchsurfing in Iran: Revealing a Hidden World, by Stephen Orth (2016)
• Bullets and Bulletins: Media and Politics in the Wake of the Arab Uprisings, edited by Mohamed Zayani and Suzi Mirgani (2016)
• The Caliphate at War: The Ideological, Organisational and Military Innovations of Islamic State, by Ahmed S. Hashim (2017)

• Safe Passage: The Transition from British to American Hegemony, by Kori Schake (2017)
• Seizing Power: The Strategic Logic of Military Coups, by Naunihal Singh (2014)
• Afghanistan from the Cold War through the War on Terror, by Barnett Rubin (2013)
• A People's History of the United States, by Howard Zinn (2005)

• Trees, maps, and theorems, by Jean-luc Doumont (2009)
• Brothers of the Gun: A Memoir of the Syrian War, by Marwan Hisham (2018)
• History of the Silk Road, by Jonathan Clements (2017)
• Losing the Nobel Prize, by Brian Keating (2018)

• The Koran in English: A Biography, by Bruce B. Lawrence (2017)
• The Red Star and the Crescent: China and the Middle East, by James Reardon-Anderson (2018)
• High-Speed Empire: Chinese Expansion and the Future of Southeast Asia, by Will Doig (2018)
• On Gravity: A Brief Tour of a Weighty Subject, by A. Zee (2018)
• Foreign Devils on the Silk Road, by Peter Hopkirk (2006)
• Tribes and Politics in Yemen: A History of the Houthi Conflict, by Marieke Brandt (2017)
• Xinjiang and the Modern Chinese State, by Justin M. Jacbos (2016)

• The Ghost Map: The Story of London's Most Terrifying Epidemic, by Steven Johnson (2007)
• Afghanistan: A Cultural and Political History, by Thomas Barfield (2010)
• Why Nations Fail: The Origins of Power, Prosperity, and Poverty, by Daron Acemoğlu (2012)
• Alone in the Ocean, by Slava Kurilov (2016)
• Memories of a Theoretical Physicist, by Joseph Polchinski (2017)
• Dear Committee Members, by Julie Schumacher, (2014)
• A History of the Middle East, by Peter Mansfield (2013)

• Rise to Globalism: American Foreign Policy Since 1938, by Stephen E. Ambrose (2010)
• A Savage War of Peace: Algeria, 1954-1962, by Alistar Horne (2006)

• Pipe Dreams: The Plunder of Iraq's Oil Wealth, by Erin Banco (2018)
• The Black Banners: The Inside Story of 9/11 and the War Against al-Qaeda, by Ali H Soufan (2011)
• Another Fine Mess: America, Uganda, and the War on Terror, by Helen C. Epstein (2017)
• Career Diplomacy: Life and Work in the US Foreign Service, Harry W. Knopp (2017)
• Becoming Fluent: How Cognitive Science Can Help Adults Learn a Foreign Language, by Richard M. Roberts (2015)
• The Tao of Pooh, by Benjamin Hoff (2003)

In January I went to Syria, spending about a month there and bopping around Sayyida Zaineb. The experience was incredible, especially only a month after the fall of the Assad Regime—plus I had the opportunity to visit Aleppo and Lattakia.

Visiting Iraq (again) was also amusing. In the winter, there was 24 hours of government electricity, which itself was a shocker. I saw old friends in Erbil, and even a weird boxing match series in Ainkawa:

BXF (Babalyon X Fighting)

In May I went off to Algeria for a week, where I was charmed by the mosaics and annoyed by the lack of shisha. This was my first trip to Algeria, and really my first exposure seriously to the Maghreb. I'm quite keen on going back, as it's substantially different than Arab culture in the Mashriq.

El Salvador was also an interesting trip, I relaxed, worked through a small earthquake, and also spent quite a bit of time learning about Romero and thinking about saints. It surprised me with how much depth it had. The nightlife was a little dull, but I enjoyed having a tropical place to focus, work, and think.

This year was really filled with work. Many 60 hour weeks, and I spent a significant amount of time getting up to speed on operations for WA and writing Erlang. Towards the end of this year, I'm happy to say I've mostly succeeded, I'm able to pick up most aspects of WhatsApp as an org and navigate and drive impact. I've been on an absurd number of outages this year, and I've seen all sorts of weird things break in WA¹.

One of our funnier outages even made it to the Linux Plumbers Conference this year, where the sched_ext mechanism for providing external schedulers, using the lavd scheduler, caused conflicts with CPU affinity and wreaked havoc on my tail latencies.

See https://www.youtube.com/watch?v=KFItEHbFEwg&source_ve_path=MTc4NDI0 for the full talk.

Honestly this constant firefighting has gotten me feeling a little burned out. In February I'll be transitioning to a role on HHVM Server, where I'll be working on the JIT and platform for HHVM, which has reinvigorated me a bit.

There's a big AI push at Meta currently, where we're heavily being encouraged to use as much AI as possible. I find Claude Code useful, but I've personally gone through the curve of inflated expectations and then the trough. I've seen people submit some real AI slop for me to review, forcing me to be the first line of quality control, but Claude Code also has made me much more productive in areas of programming I wouldn't have devoted time to before, such as Javascript. I now don't really worry about writing Javascript or having to learn React, or learning the latest Python package manager².

I'm still not quite sure how I feel about AI. The environmental impact I actually don't quite believe as much as the media hypes it up, but I do feel like it is pulling the ladder up behind us, removing the need for junior engineers. I feel like a thief who's made it past state lines, where I can efficiently guide the robot through doing tasks. If I was a junior engineer, I'd find it very difficult to resist the urge of using the robot for everything.

AI is an excellent study buddy. It's useful for reframing my thoughts, and it excels as a vague "vibe based" search engine. I feel like the Oxide RFD on LLMs accurately summarizes my experience with it: it's genuinely a transformative tool, but we need to be wary of how it impacts learning. It's sufficiently lowered the bar for learning new technologies, but I am unsure if I trust it with building serious products. For simpler codebases, I think it is excellent. For other codebases, I think I'd like to use it to churn out scaffolding more than anything else.

Despite all the hubbub about vibe coding, I find myself most efficient with the LLM when I understand the system and I just need the encantations for gluing things together. For example, our CLI tool at WhatsApp is written in Rust. Claude is quite good at writing a lot of Rust, but whether it's good Rust is a different question, as it seems to heavily favor clone() and pass by value. However, for a CLI tool this is nearly irrelevant, so I'm fine having Claude churn this stuff out.

On the learning front, one of my biggest highlights this year was how I managed to read 49(!) papers for SWE Tea. SWE Tea really has been a joy to do, and frankly it's quite high return. We read through a series a topics, from huge pages to LLM post training to data center power and so on, and we dived deep into many of the topics. I've learned a ton from SWE Tea, it's definitely been one of the best decisions I've made in recent history.

On a meta note, I'd say I've learned two things about running a paper reading group:

1. Reschedule, never skip. If you get into the habit of skipping weeks, you'll end up normalizing the deviation and then eventually drop off.
2. Batch themes together. When we first started SWE Tea, we did papers in roughly random topics, jumping around. However, batching papers chronologically allows you to trace the evolution of a technology. For example, in our latest theme of sanitizers, we started with Valgrind, walked through ASAN and TSAN, and worked our way up to memory tagging. This was something we happened upon this year, and it's really helped us read these papers. Not only do you maintain context between each week and the next, you can compare and contrast papers and authors in the same field.

Unfortunately, this year has been somewhat bad for my reading and language studying. I've kept up my Arabic and Spanish studying, but those have effectively been high fluency for a while, and only require maintenance mode. I started the year studying Russian but quickly had to give up due to a lack of time. I'd like to pick it back up next year.

I've only read 23 books this year, although some of the books were excellent. Highlights include:

• Glorious Exploits, by Ferdia Lennon
  • I didn't know what to expect going into this book, but it was an excellent emotional tug on the act of putting on a play. The theme sounds contrived, but it's short and I'd greatly recommend it
• People from Oetimu, by Felix K. Nesi
  • I discovered Archipelago Books relatively recently, and I was enchanted with this particular book. Nesi writes beautifully, and weaves a fascinating historical fiction of the Indonesian occupation of Timor.
• Planet of Slums, by Mike Davis
  • This book was on my to read list for a while, and I kept putting it off. When I finally sat down to read it, I was enchanted by the prose and arguments. If you're into books like Drug Cartels Do Not Exist or similar type of academic books, this one will draw you in as well.
• Secular Translations: Nation-State, Modern Self, and Calculative Reason, by Talal Asad

  • This was technically a re-read, but honestly I got so much more out of the re-read than the first read. One of the biggest benefits of re-reading authors like Asad is that you pick up so much more with time and distance. I read this originally for school in my MA program, picking it back up years later showed me so much more. Asad's description of language and how it gets used and reworded in this book is clearer than most of his other books, with banger quotes such as³:

    The translation of values from one natural language to another is always problematic, as every anthropologist who has carried out research in a society that speaks a language foreign to her own knows. Which is why what makes sense to a Western ethnographer in a nonmodern community sometimes doesn’t do so as easily when she returns home to Euro- America. It is almost impossible to abstract an important idea that indicates something distinctive of a particular form of life and find a ready word in a language belonging to a very different form of life…

Looking back, this year has really been leaning into being a big tech worker. I've ramped up to 4 days of crossfit a week, with the plan of going up to 5 days a week in January next year. I've stayed around New York, feeling some aches to leave but not firm plans on permanently leaving yet. I didn't get a chance to do a big bike trip like my Montreal bike trip last year, but I've really thrown myself into crossfit as a result.

Crossfit has been my point of sanity. With a stressful job, having the ability to just hop over to a gym and lift heavy shit during the middle of the day has kept me grounded. Having good trainers that help drive me towards clear progressions in my goals is a good anchor in the waves at work.

One of the things that you learn quickly when working at big tech is the scale of impact and how many problems really are organizational problems. WhatsApp server is very few people, and yet the product is routinely used by billions of people across the world a day. I spend most of my time consensus building at work, drafting alignment docs and firefighting, and I expect that to only stay constant or grow with time. This is sort of the life of big tech, where people end up fighting over stuff like "scope" or "consensus."

I haven't yet decided how I feel about it. I like the opportunities big tech affords me, I enjoy working on large industrial scale software, and I love the fact that I can sightsee and that my coworkers on WhatsApp are incredibly smart. On the other hand, it is exhausting at a particular point, I somewhat miss my old job, where I was putting in significantly less hours.

Now that I've come back to NYC after a few years away, the last year and a half can be described as a lack of FOMO. When I first moved to NYC, I was desperate to go to all the events, see all the people, so on. Now I'm at the stage where I'd prefer quiet nights and make time to see specific friends, which I suppose means that I have come down with a case of the old.

I recently chatted with a friend about this, who pointed out that I had an interesting, well paid job, and have roughly everything else lined up in life. Which is all true, but I constantly feel the pang of wanting to head back into the chaos of the Middle East.

I've forgotten where this came from (perhaps 4X games), but a while back I heard about the expand/exploit tradeoff. With the limited resources of time, I've felt like the past few years I've been expanding, seeing the world, learning new things, and so on. This last year, I've felt like I've been in the exploit phase, focusing on myself and getting really good at software. Part of it has been accepting that I'm in exploit mode for now, banking skills and resources while the expand urge waits.

For now, I'm choosing depth over breadth. We'll see how long that lasts.

النعجة - اجعوط مصطفى

One of my favorite pieces in the Algiers calligraphy museum, titled "The Sheep" by Mustafa Ajaout.

• El Salvador

• Uruguay
• Argentina

• Iraq
• Syria
• Algeria

Going back to the traceback above, I'll walk through my chain of reasoning and how to debug it. There's already many tutorials on how a traceback is structured, but I'm going to walk through this specific one with an example.

The above traceback is a good example of working through the debugging mentality. The error messages will most likely tell you what the errors are, and the role of a programmer during debugging is to find out the where and the how. As a result, it's important to not approach debugging with a specific hypothesis. Each stage in the debugging process should give you more questions, and a programmer must use their brain in debugging. The goal of debugging is not to come in with a specific hypothesis and attempt to prove/disprove it, but rather to find the correct set of questions to answer, and gradually narrow it down. It's important to remain intellectually curious about how your system can behave pathologically.

If a programmer comes into the system with a predetermined set of hypotheses, it's easy to fiddle around with the code in order to (dis)prove those hypotheses. This is a very "scientific method" point of view, but as programmers, we luckily don't have to be constrained by that! The scientific process operates on information, which is why the mode and technique of inquiry is important. But software is both information and a process², which means that programmers do not have to be bound in the same way. We can observe software in action, and take the logs of software to reconstruct the events, rather than testing specific conditions. As a result, it's important to never let your hypothesis about bugs overshadow what the error messages are saying.

Photographs of various shrines I've seen. A shrine here is basically anything I consider to have spiritual significance.

< 2021:07:21 09:55:40 > - < 36.7586833333333, 43.8936416666667 >

< 2021:07:21 11:25:05 > - < 36.7682833333333, 43.8941166666667 >

< 2025:05:04 09:07:04 > - < 36.7849083333333, 3.06117222222222 >

< 2019:07:05 16:42:06 > - < 31.9500888888889, 35.9348805555556 >

< 2021:08:21 11:26:04 > - < 30.5183805555556, 47.8370138888889 >

< 2021:08:21 16:46:12 > - < 30.4000805555556, 47.7329 >

< 2022:04:05 14:46:38 > - < 52.5051333333333, 13.3355305555556 >

< 2017:11:19 11:49:19 > - < 42.8782055555556, 74.6085277777778 >

< 2023:11:19 16:46:06 > - < -34.5871666666667, -58.4275361111111 >

< 2023:11:19 19:27:05 > - < -34.6173666666667, -58.3731277777778 >

< 2023:05:06 11:17:21 > - < 36.9710305555556, 43.3164527777778 >

< 2021:12:30 21:41:04 > - < 40.7478916666667, -73.9400722222222 >

< 2021:07:10 13:30:36 > - < 36.2012555555556, 44.0189222222222 >

< 2022:11:02 13:34:22 > - < 39.4537305555556, -31.2631083333333 >

< 2017:12:13 15:24:18 > - < 35.2342944444444, 139.095322222222 >

< 2022:07:14 22:57:18 > - < 32.6178972222222, 44.0359388888889 >

< 2022:03:27 16:00:34 > - < 32.6511333333333, 43.984925 >

< 2021:08:17 16:38:29 > - < 32.2266416666667, 44.367325 >

< 2023:04:28 12:21:13 > - < 32.6158361111111, 44.032425 >

< 2022:03:27 16:25:08 > - < 32.5944027777778, 44.0308694444444 >

< 2022:09:13 22:22:00 > - < 32.3225666666667, 44.2666444444444 >

< 2022:09:13 12:33:52 > - < 32.17915, 44.3124555555556 >

< 2021:07:21 14:01:17 > - < 36.771225, 43.3034694444444 >

< 2022:11:11 14:20:35 > - < 38.697475, -9.20533888888889 >

< 2023:06:11 15:11:31 > - < -34.9033777777778, -56.2006861111111 >

< 2023:04:28 17:00:31 > - < 31.996075, 44.3149527777778 >

< 2021:08:20 10:10:30 > - < 30.9798, 47.0426555555556 >

< 2022:04:07 15:15:53 > - < 48.8777916666667, 2.30157222222222 >

< 2022:10:26 17:04:06 > - < 37.7402583333333, -25.6622444444444 >

< 2023:02:11 13:44:19 > - < 33.4045944444444, 43.4746388888889 >

< 2021:07:21 15:02:46 > - < 36.7489916666667, 43.1152416666667 >

< 2025:08:16 11:28:31 > - < 13.6972888888889, -89.1890472222222 >

< 2023:03:11 17:00:44 > - < 37.7554944444444, -122.41735 >

< 2023:03:12 15:53:12 > - < 37.4145777777778, -122.077561111111 >

< 2023:05:15 16:51:08 > - < 35.668, 45.0039027777778 >

< 2023:05:14 01:23:35 > - < 35.5663027777778, 45.3740194444444 >

< 2023:11:05 13:46:30 > - < -54.8397555555556, -68.3120194444444 >

< 2024:01:13 17:32:06 > - < -34.8535166666667, -55.9860916666667 >

Sivarland at night. Rain slick, smoke, Ferris wheel lit.

I came into the festival by the southern gate around 8pm. First hit was sugar and oil from churros, then gunpowder. Generators droned. Cables sagged over the walkway. A kid, fingers sticky, bumped into me and the parent shot me an apologetic look.

Soldiers are present without hurry. Long guns slung, hands light on the grip. At the gate, they asked three questions, looked at my bag, and waved me through. The choreography at gatehouses is quick: a nod, a clipboard, a latch.

Sivarland, the carnival zone, runs loud for six days at the start of August². Ferris wheels, drums, fireworks, vendors in the street. The point is to celebrate the Transfiguration of Christ, with the fiesta leading up to the event mass at the metropolitan church.

“Elotes Locos,” “Papitas Fritas,” “Churros Españoles.” Prices in marker, cables overhead.

I walked the loop twice looking for a beer and failed. Drinks were soda and water; I gave up and had a cold one at home. Maybe that’s policy during the fair. There was a concert going on as well, and carnival rides and families gathered everywhere. Circus animals, and plenty of military strolling around.

It’s a carnival threaded with Jesus: banners on ride fronts, a small altar by the speakers, a prayer before the fireworks.

He is everywhere: bus decals, murals, keychains at the cathedral gate. The strongest encounter was at the hospital chapel where he was killed. A small room holds his vestments and diary pages. The guide speaks softly and points with two fingers. Outside, a white rose bush is kept trimmed. The shadow he casts is practical as much as holy: quotes on banners, names on parish flyers, a tone of moral certainty in casual talk.

I found the Museo de la Palabra y la Imagen to be the default memory/genocide museum. While small compared to the Art Museum or the Dr. David J. Guzmán National Museum, it had a dedicated exhibition on Romero and the civil war.

The Romero section was filled with photos of his life, including the elephant he met at the San Salvador Zoo. Photos of him preaching, busts of his face, his family, his notes. The whole space functioned as a shrine, and that's striking because many religious figures people strive to emulate are far away (temporally speaking). Kobo Daishi, Anastasia, and Hussein exist as canonical figures, but Romero exists in color photographs. You can walk the same paths he did, go to the same restaurants he did. Many people who are still alive today remember him.

There's an art to the construction of a saint, especially one that transgresses borders. Romero functions as a politicized sainthood, and I'd almost place him alongside Sheikh Abdesslam Yassine, the Moroccan cleric who wrote Al-Islam aw al-tufan⁶ to admonish King Hassan II. This form of sainthood is active: Yassine set himself against an oppressive regime, and both he and Romero cast long shadows with religious authority contrasted against political power.

Saints are made by the people before they are made by institutions, which you can see in Romero's late canonization in 2018. But Salvadoran society is saturated with Romero, which raises the question of how a living saint becomes a martyr when reproductions and evidence of their life are so obvious and near. Martyrdom is almost easier without photos and documents; you can dramatize, reproduce, replay the saintly life into thematic principles. But Romero and Yassine are difficult to shape or dramatize, as their lives press so close to believers.

Self-censorship is a spectrum, but I had expected there to be significantly less space dedicated to the civil war than there actually was. Numerous photos, faces in black-and-white prints of fighters, and an old classroom kept as it was. A small candle monument to those who died, and an entire gigantic room dedicated to a reproduction of Radio Venceremos.

Reproduction of the broadcasting room.

Radio Venceremos was one of the largest anti-government radio stations of the FMLN. Famously, the government tried so hard to shut them down that radio broadcasters used themselves as bait to lure a government general, and shot down his helicopter. It was on Venceremos that the end of the civil war was broadcast. This room is remarkable as well in its completeness, one can easily see that this room was carved with care.

I was vaguely aware that Bukele was Palestinian, but I didn't realize how much emigration of Palestinians occurred to El Salvador until I walked by Schafik Hándal's massive grave.

Handal was the general secretary of the FMLN, and fought against the government and then eventually transitioned to becoming a politician.

It's in the food too; I kept seeing quibe (kibbeh) on menus and even a restaurant called "Matkub." The graves of Palestinian resistance fighters are all over the necropolis too.

I liked El Salvador. It's a small country, so you can cross it in a few hours, but there's surprising depth in the politics and history. I especially enjoyed the necropolis; it resembles the one in Buenos Aires.

• More Photos: Link