What is lazy_dict_table()?

Overview

lazy_dict_table() converts a dictionary iterator/generator into a PETL table with lazy evaluation. It’s the recommended way to return data from fetch() functions in custom modules.

Instead of loading all records into memory at once, lazy_dict_table() processes records one at a time, reducing memory usage by approximately 85-90%.

Import

from oaa.modules.lazy_dicts import lazy_dict_table

Basic Usage

def fetch(connection=None, source=None, **kwargs):
    """Return a lazy-evaluated PETL table."""
    def data_generator():
        for record in api.fetch_all():
            yield record
    
    return lazy_dict_table(data_generator())

Why Use This?

Approach

Memory Usage

Best For

petl.fromdicts(list(data))

Loads ALL records into memory

Small datasets (< 10K records)

lazy_dict_table(data)

Processes lazily on-demand

Large datasets (100K+ records)

Example:

  • 1 million records dataset:

    • petl.fromdicts(): ~2GB memory usage

    • lazy_dict_table(): ~200MB memory usage (90% reduction)

Migration from petl.fromdicts()

Old Pattern (Still Works)

import petl

def fetch(connection=None, source=None, **kwargs):
    data = api.fetch_all()
    return petl.fromdicts(list(data))  # Loads ALL into memory

New Pattern (Memory-Efficient)

from oaa.modules.lazy_dicts import lazy_dict_table

def fetch(connection=None, source=None, **kwargs):
    data = api.fetch_all()
    return lazy_dict_table(data)  # Lazy evaluation, minimal memory

How It Works

  1. LazyDict Wrapper: Wraps your generator with logic to extract headers

  2. First Iteration: When first record is accessed, headers are extracted from dictionary keys

  3. Lazy Yielding: Subsequent records are converted to tuples and yielded one at a time

  4. PETL Integration: petl.wrap() provides PETL table interface; cache() adds caching layer

Important — lazy_dict_table() caches in RAM. It is lazy only until the first full iteration: under the hood it wraps the table in petl’s cache(), which materialises every row in memory on that first pass and keeps it there so the table can be iterated again. For sources read once and turned into OAA objects that is exactly what you want. But for a very large source that is read during a run without becoming objects — e.g. a destination: none junction table (a multi-million-row membership or role-assignment table consumed only by a hook) — that cache can hold several GB in RAM for the whole run. That is what spill_dict_table() below is for.

Spilling large sources to disk: spill_dict_table()

spill_dict_table() is the disk-backed counterpart to lazy_dict_table(). It drains the generator once into a temporary SQLite file and returns a re-iterable PETL table backed by that file, so the rows never sit fully in RAM:

  • Bounded memory — rows are streamed back one at a time; the full set is never resident.

  • Re-iterable — header probes and multi-pass hooks just re-query SQLite, so it is a drop-in replacement for lazy_dict_table().

  • Fetched once — the source is read from the network a single time.

  • Auto-cleaned — the temporary file is removed at process exit.

from oaa.modules.sqlite_source import spill_dict_table

def fetch(connection=None, source=None, **kwargs):
    # Same shape as lazy_dict_table(), but backed by disk instead of RAM.
    return spill_dict_table(api.fetch_all())

Choosing between them

Use

When

lazy_dict_table()

Default. The source becomes OAA objects, or it is small/medium.

spill_dict_table()

Very large sources (millions of rows) — especially destination: none tables consumed only by hooks — where holding the rows in RAM is the memory bottleneck.

The built-in concurrent_fetch common hook calls spill_dict_table() automatically when a source sets spill_to_disk: true (no custom code needed) — see How to Enable Common Hooks. When you write your own fetch(), call it directly as shown above.

Backpressure matters. Spilling only bounds memory if the producer doesn’t buffer the whole dataset ahead of the consumer. The framework’s concurrent_paginate helper keeps only a small window of pages in flight for exactly this reason, so rows reach disk instead of piling up in RAM. If you feed spill_dict_table() from your own concurrent fetcher, apply the same bound.

Production Usage

As of June 2026:

  • 11 projects have been migrated to lazy_dict_table() for ~85-90% memory reduction

  • 34+ projects still use petl.fromdicts() (works fine for smaller datasets)

  • 1 project is using spill_dict_table() (ServiceNow DCPS)

  • Best practice for new integrations: Use lazy_dict_table()