How to Use Multiple Datasources¶

Goal¶

Configure individual sources in config.yaml to push their data to different Veza datasources, so that entities from different sources appear under distinct datasource names in Veza.

When to use this¶

You want different entity types (users, groups, roles) to appear in separate Veza datasources.
You have a single data feed that contains logically distinct categories and want each category surfaced in its own datasource without running the integration multiple times.
You need fine-grained access-review scoping in Veza by datasource.

Prerequisites¶

A working config.yaml with at least one source.
Familiarity with field mappings and generators.

How it works¶

OAA-runner resolves the datasource for each source push in this priority order:

datasource: on the source — overrides everything else.
datasource: on the app — fallback for all sources that don’t set their own.
Generated default "{app.name} Datasource" — used when neither of the above is set.

Adding datasource: to a source is all that is required to route it to a specific datasource.

Step-by-step¶

1. Add `datasource:` to a source¶

Set the datasource: field on any source to name the Veza datasource it should push to:

sources:
  users:
    destination: LocalUser
    source_type: csv
    csv_filepath: path/to/users.csv
    datasource: my-datasource      # <-- this source lands in "my-datasource"
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name

Sources without a datasource: field fall back to the app-level default or the generated default.

2. Route different sources to different datasources¶

Give each source its own datasource: value to place them in separate Veza datasources:

sources:
  users:
    destination: LocalUser
    source_type: csv
    csv_filepath: path/to/users.csv
    datasource: users-datasource

  groups:
    destination: LocalGroup
    source_type: csv
    csv_filepath: path/to/groups.csv
    datasource: groups-datasource

  roles:
    destination: LocalRole
    source_type: csv
    csv_filepath: path/to/roles.csv
    datasource: roles-datasource

3. Split one feed into multiple datasources using derived sources¶

If your data comes from a single file or API but you want different subsets in different datasources, use a base source with destination: ~ and derive filtered sources from it:

sources:
  # Base source — loads data but does not push OAA objects
  all_users:
    destination: ~
    source_type: csv
    csv_filepath: path/to/users.csv
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      REGION:
        destination_field: region
        custom_type: STRING

  # Derived source — only EMEA users, pushed to its own datasource
  emea_users:
    destination: LocalUser
    source_type: source
    source: all_users
    datasource: emea
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      REGION:
        destination_field: region
        custom_type: STRING
        transforms:
          - name: filter_by_field
            options:
              options: 'EMEA'

  # Derived source — only APAC users, pushed to a different datasource
  apac_users:
    destination: LocalUser
    source_type: source
    source: all_users
    datasource: apac
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      REGION:
        destination_field: region
        custom_type: STRING
        transforms:
          - name: filter_by_field
            options:
              options: 'APAC'

Note: The filter_by_field transform uses the key options: (a comma-separated string). To match multiple values use options: 'value1,value2'.

4. Attach generators to derived sources¶

Generators run with the per-datasource provider, so group and role memberships are automatically scoped to the correct datasource. Attach generators to the derived sources:

  emea_users:
    destination: LocalUser
    source_type: source
    source: all_users
    datasource: emea
    field_mapping:
      # ...
    generators:
      user_group:
        generator_type: local_group
        source_field: GROUP
      user_role:
        generator_type: local_role
        source_field: ROLE

5. Set an app-level default (optional)¶

To apply a datasource to all sources unless they override it, set datasource: on the app block:

app:
  name: My App
  datasource: my-default-datasource

Sources that set their own datasource: still override this value.

6. Verify with dry run¶

cd src && uv tool run --from oaa-runner oaa app \
  --config-file path/to/config.yaml \
  --dry-run \
  --save-json

This writes JSON payloads to disk without pushing to Veza. Inspect the output files to confirm each source’s entities land in the expected datasource.

Full working example¶

The following example splits users, groups, and roles into separate admin and non-admin datasources. It uses three CSV files: sample_admin_users.csv, sample_groups.csv, and sample_roles.csv.

The complete config is at src/tests/configs/config-v2-multi-datasource-csv.yaml.

sample_admin_users.csv:

ID,FIRST_NAME,LAST_NAME,NAME,EMAIL,IS_ADMIN,GROUP,ROLE
1,Luke,Skywalker,Luke Skywalker,luke.skywalker@bongawonga.com,true,jedi-council,jedi-master
2,Leia,Organa,Leia Organa,leia.organa@bongawonga.com,true,rebel-command,rebel-general
3,Han,Solo,Han Solo,han.solo@bongawonga.com,false,rebel-alliance,rebel-pilot
4,Chewbacca,,Chewbacca,chewbacca@bongawonga.com,false,rebel-alliance,rebel-pilot
5,Ahsoka,Tano,Ahsoka Tano,ahsoka.tano@bongawonga.com,false,outer-rim-scouts,rebel-trooper

sample_groups.csv:

ID,NAME,IS_ADMIN
jedi-council,Jedi Council,true
rebel-command,Rebel Command,true
rebel-alliance,Rebel Alliance,false
outer-rim-scouts,Outer Rim Scouts,false

sample_roles.csv:

ID,NAME,IS_ADMIN
jedi-master,Jedi Master,true
rebel-general,Rebel General,true
rebel-pilot,Rebel Pilot,false
rebel-trooper,Rebel Trooper,false

config.yaml:

version: v2
app:
  name: My App
  application_type: Custom App
sources:
  # ── Users ──────────────────────────────────────────────────────────
  all_users:
    destination: ~
    source_type: csv
    csv_filepath: path/to/sample_admin_users.csv
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      FIRST_NAME:
        destination_field: first_name
        custom_type: STRING
      LAST_NAME:
        destination_field: last_name
        custom_type: STRING
      EMAIL:
        destination_field: email
        custom_type: STRING
      GROUP:
        destination_field: group
        custom_type: STRING
      ROLE:
        destination_field: role
        custom_type: STRING
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: validate_boolean

  admin_users:
    destination: LocalUser
    source_type: source
    source: all_users
    datasource: admins
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      FIRST_NAME:
        destination_field: first_name
        custom_type: STRING
      LAST_NAME:
        destination_field: last_name
        custom_type: STRING
      EMAIL:
        destination_field: email
        custom_type: STRING
      GROUP:
        destination_field: group
        custom_type: STRING
      ROLE:
        destination_field: role
        custom_type: STRING
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: filter_by_field
            options:
              options: 'true'
          - name: validate_boolean
    generators:
      user_group:
        generator_type: local_group
        source_field: GROUP
      user_role:
        generator_type: local_role
        source_field: ROLE

  non_admin_users:
    destination: LocalUser
    source_type: source
    source: all_users
    datasource: non-admins
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      FIRST_NAME:
        destination_field: first_name
        custom_type: STRING
      LAST_NAME:
        destination_field: last_name
        custom_type: STRING
      EMAIL:
        destination_field: email
        custom_type: STRING
      GROUP:
        destination_field: group
        custom_type: STRING
      ROLE:
        destination_field: role
        custom_type: STRING
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: filter_by_field
            options:
              options: 'false'
          - name: validate_boolean
    generators:
      user_group:
        generator_type: local_group
        source_field: GROUP
      user_role:
        generator_type: local_role
        source_field: ROLE

  # ── Groups ─────────────────────────────────────────────────────────
  all_groups:
    destination: ~
    source_type: csv
    csv_filepath: path/to/sample_groups.csv
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: validate_boolean

  admin_groups:
    destination: LocalGroup
    source_type: source
    source: all_groups
    datasource: admins
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: filter_by_field
            options:
              options: 'true'
          - name: validate_boolean

  non_admin_groups:
    destination: LocalGroup
    source_type: source
    source: all_groups
    datasource: non-admins
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: filter_by_field
            options:
              options: 'false'
          - name: validate_boolean

  # ── Roles ──────────────────────────────────────────────────────────
  all_roles:
    destination: ~
    source_type: csv
    csv_filepath: path/to/sample_roles.csv
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: validate_boolean

  admin_roles:
    destination: LocalRole
    source_type: source
    source: all_roles
    datasource: admins
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: filter_by_field
            options:
              options: 'true'
          - name: validate_boolean

  non_admin_roles:
    destination: LocalRole
    source_type: source
    source: all_roles
    datasource: non-admins
    field_mapping:
      ID:
        destination_field: unique_id
      NAME:
        destination_field: name
      IS_ADMIN:
        destination_field: is_admin
        custom_type: BOOLEAN
        transforms:
          - name: filter_by_field
            options:
              options: 'false'
          - name: validate_boolean

This produces two datasources in Veza:

admins — Luke Skywalker, Leia Organa; Jedi Council, Rebel Command; Jedi Master, Rebel General
non-admins — Han Solo, Chewbacca, Ahsoka Tano; Rebel Alliance, Outer Rim Scouts; Rebel Pilot, Rebel Trooper

Each user is also assigned to their group and role via generators, so Veza shows the full membership graph within the correct datasource.

Troubleshooting¶

Wrong YAML key for filter options

The correct key is options: (a nested mapping with key options), not matches:. Using matches: will silently ignore the filter and pass all rows through.

# ✅ Correct
transforms:
  - name: filter_by_field
    options:
      options: 'true'

# ❌ Wrong
transforms:
  - name: filter_by_field
    matches:
      - 'true'

Filter not matching — rows are missing

filter_by_field compares values as strings. CSV boolean fields come through as "true" and "false" (lowercase). Use options: 'true' / options: 'false', not options: 'True' or options: '1'.

Generators and multi_value

If your CSV field contains a comma-separated list of group IDs, set multi_value: true on the generator — but note that generators expect a Python list when multi_value: true. CSV values are plain strings and will not be split automatically. Use a pre-transform hook to split the field first, or keep group/role columns as single values per row.

How to Use Multiple Datasources¶

Goal¶

When to use this¶

Prerequisites¶

How it works¶

Step-by-step¶

1. Add `datasource:` to a source¶

2. Route different sources to different datasources¶

3. Split one feed into multiple datasources using derived sources¶

4. Attach generators to derived sources¶

5. Set an app-level default (optional)¶

6. Verify with dry run¶

Full working example¶

Troubleshooting¶

OAA Runner

Navigation

Related Topics

How to Use Multiple Datasources¶

Goal¶

When to use this¶

Prerequisites¶

How it works¶

Step-by-step¶

1. Add datasource: to a source¶

2. Route different sources to different datasources¶

3. Split one feed into multiple datasources using derived sources¶

4. Attach generators to derived sources¶

5. Set an app-level default (optional)¶

6. Verify with dry run¶

Full working example¶

Troubleshooting¶

1. Add `datasource:` to a source¶