Integration Rules

This document describes a list of rules to follow as you build out integrations for Veza using the OAA-Runner framework. It assumes you are a technical person, but not necessarily an expert in OAA-Runner.

To make sense of these rules, here is the pipeline they refer to. A config.yaml file declares the application, its sources, and how source fields map to Veza objects. Data from each source arrives as a streaming petl table, which can be reshaped by transforms and by hooks at different stages of the pipeline. From that stream the framework builds OAA objects (users, groups, roles, custom entities) and pushes the final payload to Veza.

Configure First, Code Second

• If something can be done in the config.yaml file, it must be done there for simplicity and readability. See also exhaust config.yaml before writing a hook.

• If you can avoid writing Python code, avoid it.

• Look for opportunities to create new transforms. See how to add a new transform.

Keep the Data Stream Lazy

• PRE_TRANSFORM and POST_TRANSFORM hooks must never materialize the data stream, only act on the stream using petl functions. See how to use PRE_TRANSFORM hooks. The framework streams data for a reason – see why we use a streaming data approach.

• Use lazy_dict_table, never petl.fromdicts. The lazy table processes records one at a time instead of loading them all into memory.

• Always return a petl iterable from the fetch function. Always.

• A field present with value null is not the same as a field that’s absent – don’t let a truthiness check discard a valid null.

Hooks

• Prefer record_transforms over PRE_PUSH_OAA iterations. See how to use RECORD_TRANSFORM hooks and what are OAA Runner hooks.

• PRE_PUSH_OAA and POST_PUSH_OAA ignore target=; they fire globally once. Scope behavior by iterating sources["name"].records inside the hook.

• RECORD_TRANSFORM hooks do not receive connection=; use config.connections["<name>"] instead.

API Resilience

• Always handle rate limiting, throttling, exponential retries, and backoff. See how to handle API pagination and how to add a REST API source.

• Assume that any API will respond unpredictably.

• Always use recursive pagination; never a while True loop. See how to handle API pagination.

• Use BaseURLSession, never a raw requests.Session() with manual URL joining.

Credentials and Secrets

• Use Python logging, not print statements.

• Never log secrets to stdout.

• Use params_or_env() for all credential and config reads; avoid calling os.getenv() directly. See how to use required_env.

• required_env must be a YAML list (e.g. [MY_KEY]), never a bare string – a bare string is accepted silently and fails silently. See required_env reference.

• Never set veza_url in config.yaml; leave it blank or omit it entirely.

• Never persist MOCK_RESULTS in config.yaml or connection params; pass it explicitly on the command line only. See how to mock API responses for local development and mock framework reference.

• When in doubt about configuration syntax, run oaa validate instead of memorizing every constraint.

Permissions

• Always resolve permissions with config.lookup_perms(); never hardcode OAAPermission lists. See how to model resources with permissions.

• permission_to_veza_map entries must use the permissions: key; the bare list form fails schema validation.

• Every permission value used in a role assignment must be registered once via provider.add_custom_permission() before push – Veza rejects any permission it wasn’t told about, even a canonical value.

The modules/<application name>.py File

• This file’s one responsibility is to house the fetch function. See how to write a fetch function.

• Do no field transforms, data manipulation, or other things here; those live in the hooks file.

• Read values from source.<source name>.params, connection.params, or os.environ.

• fetch must backfill every field named in field_mapping with None when it’s missing from the raw response – not "", False, or 0.

• Always return a petl iterable from the fetch function. Always.

Conventions

• Max line length is 100 characters.

• Every function needs a concise docstring – one line for simple functions, a short paragraph for complex ones.

• Always set a module-level logger: logger = logging.getLogger(__name__).

Development Environment

• Never invoke python, python3, or pip directly; use uv run.

• The mocks module must never be imported by the main module; wire it only through logic_module in config.yaml for testing.

For more on how modules fit into an integration, see what is the modules folder used for.