Learning the OAA Runner Framework: Part 5 — Connecting to a REST API

Calvin holds two tin cans connected by a very long string that runs off into the distance toward a large server tower. He shouts into his can. Hobbes listens at the other can pressed to his ear, tail wagging. The string has a little badge on it labeled "API Key".

This is Part 5 of the Learning the OAA Runner Framework series. It builds on the patterns established in the CSV track (Parts 1–4).

What you’ll learn: Configure a connections block, understand how a custom fetch module handles API key authentication and pagination, and map API response fields to OAA user, group, and role objects. By the end you’ll run the integration against the live BongaWonga App API and see users, groups, and roles load without errors.

Prerequisites

  • Completed or read the CSV track (Part 1 through Part 4).

  • An API key for the BongaWonga App (BW_APP_API_KEY). Get it from https://app.bongawonga.com/profile. See Getting API credentials for all systems.

Setup

Create your working directory:

mkdir -p 05-rest-api/modules
cd 05-rest-api

Set your API key as an environment variable:

export BW_APP_API_KEY=your_api_key

Copy the following code into a new file at modules/app_bw.py:

import logging
import time

from oaa.modules.base_url_session import BaseURLSession
from oaa.modules.lazy_dicts import lazy_dict_table
from oaa.modules.params_or_env import params_or_env

logger = logging.getLogger(__name__)


class BongaWongaAPI:
    def __init__(self, connection, source):
        self.connection = connection
        self.source = source
        self.base_url = params_or_env(connection, source, 'base_url')
        self.api_key = params_or_env(connection, source, 'BW_APP_API_KEY')
        self._session = None

    @property
    def session(self):
        if not self._session:
            self._session = BaseURLSession(base_url=self.base_url)
            self._session.headers.update({
                'Authorization': f'Bearer {self.api_key}',
            })
        return self._session

    def fetch_paginated(self, path, resource_key='data', per_page=100):
        page = 1
        while True:
            params = {'page': page, 'per_page': per_page}
            logger.debug('Fetching %s page=%d', path, page)
            response = self.session.get(path, params=params, timeout=30)

            if response.status_code == 429:
                logger.warning('Rate limited, retrying in 5s')
                time.sleep(5)
                continue

            response.raise_for_status()
            data = response.json()
            records = data.get(resource_key, [])
            if not records:
                break
            yield from records
            pagination = data.get('pagination', {})
            if not pagination.get('has_next', False):
                break
            page += 1


def fetch(connection=None, source=None, **kwargs):
    api = BongaWongaAPI(connection, source)
    resource_path = source.params.get('resource', '')
    resource_key = source.params.get('resource_key', 'data')
    logger.debug('Fetching from resource: %s', resource_path)
    records = api.fetch_paginated(resource_path, resource_key=resource_key)
    return lazy_dict_table(records)

Copy the following into a new file at config.yaml:

version: v2
log_level: INFO

You’ll build it step by step below.

Step 1: Define the application

Open config.yaml and add the app block:

app:
  name: BongaWonga App
  application_type: Custom App

Same pattern as the CSV examples.

Step 2: Register the fetch module

This example uses a Python module to fetch data from the API. Tell oaa-runner where to find it:

module_directory: ./modules

Step 3: Define the connection

A connections block is required for any source that uses a custom Python fetch module. It holds authentication credentials, base URL, and other params the module needs at runtime.

Open config.yaml and add the following connections block:

connections:
  bw_app:
    connection_type: custom
    custom_module: app_bw
    required_env:
      - BW_APP_API_KEY
    params:
      base_url: https://app.bongawonga.com/api/v1

  • connection_type: custom — uses a Python module for data fetching.

  • custom_module: app_bw — filename (without .py) inside module_directory.

  • required_env — env var names checked at startup. oaa-runner exits early if any are missing, preventing a confusing error later in the pipeline.

  • params — arbitrary key/value pairs passed to the module at runtime. base_url is the API root. The module reads it via params_or_env().

Step 4: Understand the fetch module

modules/app_bw.py was created in Setup. It has two parts: a class that handles HTTP and pagination, and a fetch() function that oaa-runner calls once per source.

The fetch() function:

def fetch(connection=None, source=None, **kwargs):
    api = BongaWongaAPI(connection, source)
    resource_path = source.params.get('resource', '')
    results = api.fetch_paginated(resource_path)
    return lazy_dict_table(results)

  • connection — the connection config block from config.yaml (has params).

  • source — the source config block (has its own params like resource).

  • lazy_dict_table() — wraps the results in a lazy iterator for memory efficiency.

The BongaWongaAPI class handles:

  • API key auth — sets Authorization: Bearer <key> once on the session

  • Paginated requests with ?page=1&per_page=100, following has_next to the last page

  • 429 backoff: waits 5 seconds and retries

Two oaa-runner utilities do the heavy lifting:

  • params_or_env(connection, source, key) — resolves a config value from env vars, connection.params, or source.params, in that priority order. This is how the module reads BW_APP_API_KEY from the environment without hard-coding env var names. See How to use oaa-runner utilities.

  • BaseURLSession(base_url=...) — a requests.Session subclass that prepends a base URL to every request, so callers can pass /users instead of the full URL. See How to use oaa-runner utilities.

You don’t need to modify app_bw.py for this example.

Step 5: Add the users source

Add a sources block:

sources:
  users:
    destination: LocalUser
    source_type: custom
    connection: bw_app
    params:
      resource: /users
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: unique_id
      email:
        destination_field: email
        transforms:
          - name: lowercase

  • source_type: custom — calls the fetch() function from the connection’s module.

  • connection: bw_app — references the connection defined in Step 3.

  • params.resource — passed to fetch() as source.params.get('resource').

  • params.resource_key: data — the key in the API response that contains the records array. The BongaWonga API wraps every response in a consistent envelope:

    {
  "data": [
    {"unique_id": "u-1", "email": "alice@example.com", "status": "active"},
    {"unique_id": "u-2", "email": "bob@example.com",   "status": "inactive"}
  ],
  "pagination": {
    "page": 1,
    "per_page": 100,
    "total": 2,
    "has_next": false
  }
}

    resource_key: data tells oaa-runner (and the fetch_paginated method) to extract the array at response["data"]. The pagination.has_next field controls whether to fetch the next page.

Try it: Run with users only

Your config is complete enough to run. Test it now:

uv tool run --from oaa-runner oaa custom_app run --config-file=config.yaml --no-push

You should see users loading from the API. If you see an authentication error, verify that BW_APP_API_KEY is set correctly in your shell.

Step 6: Add the remaining user field mappings

The BongaWonga API uses different field names than OAA expects in a few places:

  • API returns display_name — OAA expects name

  • API returns last_login — OAA expects last_login_at

  • API returns status (a string enum) — OAA expects is_active (a boolean)

Add these mappings to the users source field_mapping block:

      name:
        destination_field: name
        transforms:
          - name: jinja_render
            options:
              template: "{{ row.display_name }}"
      given_name:
        destination_field: given_name
        custom_type: STRING
        transforms:
          - name: jinja_render
            options:
              template: "{{ row.first_name }}"
      family_name:
        destination_field: family_name
        custom_type: STRING
        transforms:
          - name: jinja_render
            options:
              template: "{{ row.last_name }}"
      title:
        destination_field: title
        custom_type: STRING
      department:
        destination_field: department
        custom_type: STRING
      status:
        destination_field: is_active
        transforms:
          - name: jinja_render
            options:
              template: "{{ 'true' if row.status == 'active' else 'false' }}"
          - name: validate_boolean
      created_at:
        destination_field: created_at
        transforms:
          - name: date_convert
      last_login:
        destination_field: last_login_at
        transforms:
          - name: date_convert

Why jinja_render for field renames: Use the OAA field name as the mapping key and read the actual API field via row.<field>. For example, name key with {{ row.display_name }} reads the API’s display_name and stores it as OAA name. The last_login entry inverts this: the key matches the API field and destination_field: last_login_at routes it to the OAA field.

Why status needs a transform chain: The API returns status as a 4-value enum (active, inactive, suspended, pending_activation). OAA’s is_active expects a boolean. The jinja_render converts the string to "true" or "false", then validate_boolean converts it to a proper boolean.

Why date_convert for date-only strings: The BongaWonga App API returns dates as YYYY-MM-DD with no time component. date_convert handles both date-only strings and full ISO 8601 datetimes.

Step 7: Add groups and roles sources

  groups:
    destination: LocalGroup
    source_type: custom
    connection: bw_app
    params:
      resource: /groups
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: unique_id
      name:
        destination_field: name
      is_admin:
        destination_field: is_admin
        custom_type: BOOLEAN

  roles:
    destination: LocalRole
    source_type: custom
    connection: bw_app
    params:
      resource: /roles
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: unique_id
      name:
        destination_field: name

Each source reuses the same bw_app connection. fetch() is called once per source, each time fetching the endpoint specified in that source’s params.resource.

Complete config

version: v2
log_level: INFO

app:
  name: BongaWonga App
  application_type: Custom App

module_directory: ./modules

connections:
  bw_app:
    connection_type: custom
    custom_module: app_bw
    required_env:
      - BW_APP_API_KEY
    params:
      base_url: https://app.bongawonga.com/api/v1

sources:
  users:
    destination: LocalUser
    source_type: custom
    connection: bw_app
    params:
      resource: /users
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: unique_id
      email:
        destination_field: email
        transforms:
          - name: lowercase
      name:
        destination_field: name
        transforms:
          - name: jinja_render
            options:
              template: "{{ row.display_name }}"
      given_name:
        destination_field: given_name
        custom_type: STRING
        transforms:
          - name: jinja_render
            options:
              template: "{{ row.first_name }}"
      family_name:
        destination_field: family_name
        custom_type: STRING
        transforms:
          - name: jinja_render
            options:
              template: "{{ row.last_name }}"
      title:
        destination_field: title
        custom_type: STRING
      department:
        destination_field: department
        custom_type: STRING
      status:
        destination_field: is_active
        transforms:
          - name: jinja_render
            options:
              template: "{{ 'true' if row.status == 'active' else 'false' }}"
          - name: validate_boolean
      created_at:
        destination_field: created_at
        transforms:
          - name: date_convert
      last_login:
        destination_field: last_login_at
        transforms:
          - name: date_convert

  groups:
    destination: LocalGroup
    source_type: custom
    connection: bw_app
    params:
      resource: /groups
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: unique_id
      name:
        destination_field: name
      is_admin:
        destination_field: is_admin
        custom_type: BOOLEAN

  roles:
    destination: LocalRole
    source_type: custom
    connection: bw_app
    params:
      resource: /roles
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: unique_id
      name:
        destination_field: name

Run the complete integration

Before running, make sure your BW_APP_API_KEY is set in your shell:

export BW_APP_API_KEY=your_api_key
uv tool run --from oaa-runner oaa custom_app run --config-file=config.yaml --no-push

If it worked, you’ll see:

INFO - Loading users from /users
INFO - Loading groups from /groups
INFO - Loading roles from /roles
...
status: true
report: []

Users, groups, and roles should all load successfully. User-to-group and user-to-role relationships are not yet wired — that happens in Part 6.

To push to Veza, set VEZA_URL and VEZA_API_KEY and remove --no-push.

What’s next

This example loads one API into one destination type. The next example runs the same BongaWonga APIs three ways: as a Custom App, as an HRIS, and as an IdP — each using its own config file, its own destination type semantics, and its own connection module.

Next: Part 6 — Multiple Systems, Multiple Destinations

See also