How to Create an IDP Integration

TL;DR

Build an OAA integration that connects an identity provider (custom LDAP, in-house SSO, non-native IdP) to Veza using destination_type: IDP. This guide covers User and Group destinations, required fields, role assignments, and end-to-end configuration.

Prerequisites

• OAA runner installed (see How to Install uv and How to Run Integrations Locally)

• API documentation for your identity provider

• API credentials stored in a .env file

• Basic familiarity with config.yaml structure (see What is an OAA config file?)

When to Use IDP vs Native Veza Connectors

Use destination_type: IDP when:

• Your identity provider is not already covered by a native Veza connector

• You have a custom LDAP, in-house SSO, or proprietary IdP

• You need fine-grained control over user/group data extraction

If Veza already provides a native connector for your IdP (Okta, Azure AD, etc.), prefer the native connector. Use this integration type for non-native or custom IdPs only.

For a full comparison of all destination types, see What Are the Runner Types?.

Steps

Step 1: Set Up the IDP Configuration

Create a config.yaml with destination_type: IDP. The app block must include name, idp_type, and domain:

version: v2
log_level: INFO
destination_type: IDP

app:
  name: My Identity Provider
  idp_type: custom_idp     # your IdP type identifier
  domain: idp.example.com  # your IdP domain

module_directory: ./modules

Required app fields for IDP:

Field	Description	Example
name	Display name for the provider in Veza	Corporate SSO
idp_type	Identifier for your IdP type	custom_idp, ldap, internal_sso
domain	Domain of the identity provider	idp.example.com

Step 2: Configure a Connection

Define a custom connection that fetches data from your IdP API:

connections:
  idp_api:
    connection_type: custom
    custom_module: idp_client
    required_env:
      - IDP_API_KEY
    params:
      base_url: https://idp.example.com/api/v1

Store your API key in .env:

# .env
IDP_API_KEY=your_api_key_here

For more details on API connections, see How to Add a REST API Source and How to Authenticate with API Keys.

Step 3: Define a User Source

The User destination is the primary object for IDP integrations. It requires three mandatory fields:

Required Field	Description	Example
identity	Unique identity identifier (username, UPN, etc.)	jsmith@example.com
name	Display name	Jane Smith
email	Email address	jsmith@example.com

sources:
  idp_users:
    destination: User
    source_type: custom
    connection: idp_api
    params:
      resource: /users
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: identity
      name:
        destination_field: name
      email:
        destination_field: email
        transforms:
          - name: null_to_blank
          - name: lowercase
      external_id:
        destination_field: external_id
        custom_type: STRING
      status:
        destination_field: is_active
        transforms:
          - name: jinja_render
            options:
              template: "{{ 'true' if row.status == 'active' else 'false' }}"
          - name: validate_boolean
      mfa_enabled:
        destination_field: mfa_enabled
        custom_type: BOOLEAN
      created_at:
        destination_field: created_at
        transforms:
          - name: date_convert
      last_login:
        destination_field: last_login_at
        transforms:
          - name: date_convert

Step 4: Define a Group Source (Optional)

IDP integrations can include a Group source for groups, roles, or organizational units from your identity provider:

sources:
  idp_groups:
    destination: Group
    source_type: custom
    connection: idp_api
    params:
      resource: /groups
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: identity
      name:
        destination_field: name
      is_admin:
        destination_field: is_admin
        custom_type: BOOLEAN

Group required fields:

• name (required)

• One of identity or unique_id (required — exactly one must be mapped)

Important: If both identity and unique_id are mapped for a Group source, the integration will error. Map exactly one of these fields.

Step 5: Link Users to Groups

To create user-to-group membership relationships, use record_transforms_non_stream with a custom transform function. This pattern links user records to their group memberships after both sources have been fetched:

sources:
  idp_users:
    destination: User
    source_type: custom
    connection: idp_api
    params:
      resource: /users
      resource_key: data
    field_mapping:
      # ... field mappings as above ...
    record_transforms_non_stream:
      - link_idp_user_groups

Then implement the link function in your hooks module:

import logging

logger = logging.getLogger(__name__)


def link_idp_user_groups(source, all_sources=None, **kwargs):
    """Link IdP users to their groups based on membership data."""
    if not all_sources:
        return

    # Get the groups source
    groups_source = all_sources.get('idp_groups')
    if not groups_source:
        logger.warning("No 'idp_groups' source found for linking")
        return

    # Build a lookup of group identities
    group_lookup = {}
    for record in petl.records(groups_source.stream):
        identity = record.get('identity') or record.get('unique_id')
        if identity:
            group_lookup[identity] = record.get('name', identity)

    # Add group membership to each user record
    for record in source.stream:
        user_groups = record.get('group_memberships', '')
        if user_groups:
            group_names = [
                group_lookup.get(g.strip(), g.strip())
                for g in user_groups.split(',')
                if g.strip()
            ]
            record['groups'] = group_names

For more on linking entities, see How to Create User-to-Role Relationships.

Step 6: Add Custom Properties

Fields that are not standard OAA User or Group properties can be added as custom properties using custom_type:

field_mapping:
  # Custom properties for User:
  department:
    destination_field: department
    custom_type: STRING
  mfa_enabled:
    destination_field: mfa_enabled
    custom_type: BOOLEAN
  last_login:
    destination_field: last_login_at
    custom_type: TIMESTAMP
    transforms:
      - name: date_convert
  certifications:
    destination_field: certifications
    custom_type: STRING_LIST

Available custom_type values:

Type	Description	Example
STRING	Text values	"Engineering"
BOOLEAN	True/False values	true
NUMBER	Numeric values	42
STRING_LIST	Array of strings	["admin", "user"]
TIMESTAMP	ISO 8601 date/time	"2024-01-15T10:30:00Z"
TIMESTAMP_AD	Windows AD timestamp	133456789012345678

For complete details, see What are Veza Custom Properties?.

Step 7: Write the Fetch Module

Create modules/idp_client.py with a fetch() function that returns a lazy_dict_table for memory efficiency:

import requests
import logging
from oaa.modules.lazy_dicts import lazy_dict_table

logger = logging.getLogger(__name__)


class IdPAPIClient:
    def __init__(self, connection=None, source=None):
        params = connection.params if connection else {}
        self.base_url = params.get('base_url')
        self.api_key = params.get('api_key')
        self.session = requests.Session()
        self.session.headers.update({
            'Authorization': f'Bearer {self.api_key}'
        })

    def fetch_paginated(self, resource, resource_key='data', page_size=100):
        """Fetch all records from a paginated IdP API endpoint."""
        offset = 0

        while True:
            url = f"{self.base_url}{resource}"
            params = {
                'offset': offset,
                'limit': page_size
            }

            response = self.session.get(url, params=params, timeout=30)
            response.raise_for_status()
            data = response.json()
            items = data.get(resource_key, [])

            if not items:
                break

            for item in items:
                yield item

            offset += page_size
            logger.info(f"Fetched {offset} records from {resource}")


def fetch(connection=None, source=None, **kwargs):
    """
    Fetch the data. This is the function that is called by oaa.

    :param connection: Connection configuration from config.yaml
    :param source: Source configuration from config.yaml
    :returns: PETL table using lazy_dict_table for memory efficiency
    """
    api = IdPAPIClient(connection=connection, source=source)
    resource_path = source.params.get('resource', '')
    resource_key = source.params.get('resource_key', 'data')
    results = api.fetch_paginated(resource_path, resource_key=resource_key)
    return lazy_dict_table(results)

For more details, see How to Write a Fetch Function and How to Handle API Pagination.

Step 8: Validate and Run

Validate your configuration:

uv tool run --from oaa-runner oaa app validate --config=config.yaml

Run a dry run (no push to Veza):

uv tool run --from oaa-runner oaa app run --config=config.yaml --dry_run=True --save-json=True

Inspect output.json to verify users and groups are populated correctly:

# Check user count
jq '.users | length' output.json

# Check first user
jq '.users[0]' output.json

# Check group count
jq '.groups | length' output.json

Run with push to Veza:

uv tool run --from oaa-runner oaa app run --config=config.yaml

End-to-End config.yaml Example

version: v2
log_level: INFO
destination_type: IDP

app:
  name: BongaWonga Identity
  idp_type: bongawonga
  domain: idp.bongawonga.com

module_directory: ./modules
logic_modules:
  - bw_hooks

connections:
  bw_idp:
    connection_type: custom
    custom_module: idp_bw
    required_env:
      - BW_IDP_API_KEY
    params:
      base_url: https://idp.bongawonga.com/api/v1

sources:
  idp_users:
    destination: User
    source_type: custom
    connection: bw_idp
    params:
      resource: /users
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: identity
      name:
        destination_field: name
      email:
        destination_field: email
        transforms:
          - name: null_to_blank
          - name: lowercase
      external_id:
        destination_field: external_id
        custom_type: STRING
      status:
        destination_field: is_active
        transforms:
          - name: jinja_render
            options:
              template: "{{ 'true' if row.status == 'active' else 'false' }}"
          - name: validate_boolean
      mfa_enabled:
        destination_field: mfa_enabled
        custom_type: BOOLEAN
      created_at:
        destination_field: created_at
        transforms:
          - name: date_convert
      last_login:
        destination_field: last_login_at
        transforms:
          - name: date_convert
    record_transforms_non_stream:
      - link_idp_user_groups

  idp_groups:
    destination: Group
    source_type: custom
    connection: bw_idp
    params:
      resource: /groups
      resource_key: data
    field_mapping:
      unique_id:
        destination_field: identity
      name:
        destination_field: name
      is_admin:
        destination_field: is_admin
        custom_type: BOOLEAN

A complete working example is available at docs/howtos/examples/06-multi-source-api/config-idp.yaml.

Examples

Real Integration: BongaWonga Identity

The BongaWonga training series includes a full IDP integration with:

• User source with identity, name, email required fields

• Group source with identity and name fields

• Custom properties for MFA status, external ID, and login timestamps

• User-to-group linking via record_transforms_non_stream

See the example config at docs/howtos/examples/06-multi-source-api/config-idp.yaml.

Next Steps

• How to Create an HRIS Integration - Connect a workforce/HR system

• How to Create a Custom App Integration - Connect any application

• How to Configure Field Mappings - Map source fields to OAA fields

• How to Create User-to-Role Relationships - Link users to groups and roles

• How to Handle API Pagination - Work with large user directories

Related Documentation

• What Are the Runner Types? - Comparison of all destination types

• What are Veza Custom Properties? - Custom field types and timestamp formats

• What is an OAA config file? - Config file structure reference

• How to Write a Fetch Function - Fetch module patterns