How to Profile Memory Usage

TL;DR

Set the OAA_RSS_PROFILE=1 environment variable and the runner logs process memory (RSS) at each phase boundary of a run — per source fetch, per source load, per lifecycle hook, and per push step. The per-phase trace tells you exactly which step is responsible for a memory peak, so you can target the right fix instead of guessing. It is off by default and adds no overhead to normal runs.

When to use this

Reach for it when a run uses more memory than expected or is being killed (OOM) on large populations, and you need to know where the memory goes:

  • Is it a source being cached in RAM during fetch?

  • Is it the object graph being built during record load?

  • Is it a lifecycle hook building a large transient lookup?

  • Is it payload serialization at push time?

Steps

Step 1: Enable profiling for one run

Prefix (or export) the variable and tee the output to a file:

OAA_RSS_PROFILE=1 oaa-run --config-file config.yaml run 2>&1 | tee run_rss.log

Any truthy value works: 1, true, yes, on.

Step 2: Read the per-phase trace

Pull just the RSS lines:

grep "RSS \[" run_rss.log

You will see lines like:

peak RSS [transform:users]: ~284.7 MB
peak RSS [transform:memberships]: ~3223.8 MB
peak RSS [load:users]: ~409.3 MB
peak RSS [hook:PRE_PUSH_OAA:add_group_memberships]: ~718.0 MB
peak RSS [push:before staging]: ~1093.8 MB

The phase labels are:

Label

Boundary

transform:<source>

After a source’s stream is fetched/cached

load:<source>

After a source’s OAA objects are created

hook:<EVENT>:<name>

After each lifecycle hook runs (e.g. PRE_PUSH_OAA:add_group_memberships)

push:<step>

Stages of a push-phase common hook (e.g. sqlite_streaming_push)

Step 3: Find the culprit

Read the jump between consecutive lines — the step where the number rises is the step that added the memory. For example, a jump from transform:groups (~400 MB) to transform:memberships (~3.2 GB) points squarely at the memberships source being held in RAM.

Step 4: Apply the matching fix

  • A large transform:<source> jump on a source consumed only by hooks → spill it to disk with spill_to_disk: true (see How to Enable Common Hooks and spill_dict_table()).

  • A jump at push:* (or at the default push) → use the sqlite_streaming_push common hook to avoid building the whole payload in memory.

  • A jump at a specific hook:PRE_PUSH_OAA:* → that hook is building a large transient; bound or stream it.

Reading the numbers correctly

The profiler reports the best signal available:

  • With psutil installed — current RSS, so memory that is freed shows up as a drop between phases.

  • Without psutil (the default) — it falls back to resource.getrusage, which reports the monotonic peak RSS (lines read peak RSS [...]). This never decreases, so read the jumps between lines, not the absolute values, to see which step added memory. To see frees as well, install psutil into the run environment.

Either way the first line where the number crosses your memory ceiling tells you which phase is responsible.

Coverage note: RSS lines are emitted at fetch, load, and lifecycle-hook boundaries, plus inside push-phase common hooks. The default (non-hook) push path is not instrumented, so if you want the true peak of a run that ends with the built-in push, measure the whole process instead — e.g. /usr/bin/time -l on macOS and read maximum resident set size.

How it works

OAA_RSS_PROFILE gates a small helper, oaa.modules.mem_profile.log_rss(), which is called at the phase boundaries above. When the variable is unset the calls are no-ops, so there is zero cost and no log noise on normal runs. The push-phase RSS lines from sqlite_streaming_push are always emitted (they are part of that feature); everything else is gated by the variable.