How to Use required_env for Environment Variable Management

TL;DR

Use the required_env field on connections to declare which environment variables your integration needs. oaa-runner validates them before running and auto-loads them from .env — no more calling load_dotenv() in every module.

Prerequisites

  • An integration configured with config.yaml

  • A .env file containing your secrets (or environment variables set in your shell)

Why required_env?

Without required_env, every integration module must:

  1. Call load_dotenv() at the top of the file and again inside fetch()

  2. Use params_or_env() or os.getenv() to read each secret

  3. Hope the developer remembers to set the right variables

With required_env:

  • Validation fails fast — missing variables are caught before any data is fetched

  • No load_dotenv() needed — oaa-runner loads .env from the config directory automatically

  • Auto-injection — variables are injected into connection.params, so you access them directly

  • Self-documenting — the config file explicitly declares which secrets each connection needs

Steps

Step 1: Declare required variables in config.yaml

Add a required_env list under each connection that needs secrets:

connections:
  my_api:
    connection_type: custom
    custom_module: modules.myapp
    required_env:
      - MY_API_BASE_URL
      - MY_API_KEY
    params:
      timeout: 30

VEZA_URL and VEZA_API_KEY are always implicitly required — you do not need to list them in required_env.

Step 2: Add variables to your .env file

Place the .env file in the same directory as your config.yaml:

# .env (never commit this file!)
VEZA_URL=https://myorg.vezacloud.com
VEZA_API_KEY=veza-api-key-here
MY_API_BASE_URL=https://api.example.com
MY_API_KEY=sk-abc123def456

Add .env to .gitignore:

echo ".env" >> .gitignore

Step 3: Access variables in your module

With required_env, variables are auto-injected into connection.params. No load_dotenv() call is needed:

Before (without required_env):

from dotenv import load_dotenv
from oaa.modules.params_or_env import params_or_env

load_dotenv()

class MyAPI:
    def __init__(self, connection, source):
        self.base_url = params_or_env(connection, source, 'MY_API_BASE_URL')
        self.api_key = params_or_env(connection, source, 'MY_API_KEY')

After (with required_env):

# No load_dotenv() needed — oaa-runner handles it

class MyAPI:
    def __init__(self, connection, source):
        self.base_url = connection.params['MY_API_BASE_URL']
        self.api_key = connection.params['MY_API_KEY']

Step 4: Validate your configuration

Before running, validate that all required variables are present:

oaa app validate --config=config.yaml

You will see output like:

==================================================
STEP 2: Validating required environment variables...
==================================================
✅ All required environment variables are set (4 checked)

If a variable is missing:

==================================================
STEP 2: Validating required environment variables...
==================================================
❌ Required environment variable "MY_API_KEY" is not set.
   Declared in connection "my_api" required_env.
   Set it in your .env file or shell environment.
❌ VALIDATION FAILED - Please review errors above

How It Works

Validation

Before each run, oaa-runner checks that every variable listed in required_env is present in the environment. VEZA_URL and VEZA_API_KEY are always checked regardless. If any variable is missing, the run fails immediately with a clear error message naming the variable and the connection that declared it.

Auto-loading

When required_env is present, oaa-runner loads .env from the same directory as the config.yaml file. This replaces the need for individual modules to call load_dotenv().

Injection

After loading, values from the environment are injected into connection.params. Given this config:

connections:
  my_api:
    connection_type: custom
    custom_module: modules.myapp
    required_env:
      - MY_API_KEY
    params:
      base_url: https://api.example.com

And this .env:

MY_API_KEY=sk-abc123

The connection.params dict inside fetch() contains:

{
    "base_url": "https://api.example.com",   # from config
    "MY_API_KEY": "sk-abc123",                # auto-loaded from .env
}

Existing params values are preserved. Environment values override params values when the same key appears in both.

Connection Types

required_env works on all connection types:

Custom connections

connections:
  my_api:
    connection_type: custom
    custom_module: modules.myapp
    required_env:
      - MY_API_KEY
      - MY_API_SECRET
    params:
      base_url: https://api.example.com

API connections

connections:
  external_api:
    connection_type: api
    uri: https://api.example.com
    fetch_module: api_fetch
    required_env:
      - API_TOKEN

DB connections

connections:
  my_db:
    connection_type: db
    DB_DIALECT: POSTGRES
    DB_SERVER: db.example.com
    DB_NAME: mydb
    DB_USER: sa
    DB_PASSWORD: "!env"
    required_env:
      - DB_PASSWORD
      - DB_SERVER

Note: DB connections already support the !env syntax for individual fields. Use required_env when you want pre-flight validation or need to reference the same variable from your module code.

CSV and Veza API connections

connections:
  my_csv:
    connection_type: csv
    required_env:
      - CSV_ENCRYPTION_KEY

Multiple Connections

Each connection declares its own requirements:

connections:
  hr_api:
    connection_type: custom
    custom_module: modules.hr
    required_env:
      - HR_API_KEY
    params:
      base_url: https://hr.example.com

  iam_api:
    connection_type: custom
    custom_module: modules.iam
    required_env:
      - IAM_CLIENT_ID
      - IAM_CLIENT_SECRET
    params:
      base_url: https://iam.example.com

All variables across all connections are validated before the run starts.

Migration Guide

To migrate an existing integration from load_dotenv() to required_env:

  1. Add required_env to config.yaml for each connection that uses secrets

  2. Remove load_dotenv() calls from your module files

  3. Replace params_or_env() calls with direct connection.params[...] access

  4. Remove from dotenv import load_dotenv imports

  5. Validate with oaa app validate --config=config.yaml

Before

# modules/myapp.py
from dotenv import load_dotenv
from oaa.modules.params_or_env import params_or_env

load_dotenv()

class MyAppAPI:
    def __init__(self, connection, source):
        self.base_url = params_or_env(connection, source, 'MYAPP_API_BASE_URL')
        self.api_key = params_or_env(connection, source, 'MYAPP_API_KEY')

# config.yaml
connections:
  myapp:
    connection_type: custom
    custom_module: modules.myapp
    params:
      base_url: https://api.myapp.com

After

# modules/myapp.py — no load_dotenv needed
class MyAppAPI:
    def __init__(self, connection, source):
        self.base_url = connection.params['MYAPP_API_BASE_URL']
        self.api_key = connection.params['MYAPP_API_KEY']

# config.yaml
connections:
  myapp:
    connection_type: custom
    custom_module: modules.myapp
    required_env:
      - MYAPP_API_BASE_URL
      - MYAPP_API_KEY
    params:
      base_url: https://api.myapp.com

Best Practices

1. Use load_dotenv() without override parameter

In your module’s fetch() function, always use load_dotenv() without the override=True parameter. This allows externally-set environment variables (e.g., from test runners or CI/CD) to take precedence over .env file values:

Incorrect (overrides external env vars):

from dotenv import load_dotenv

def fetch(**kwargs):
    load_dotenv(override=True)  # ❌ Overwrites MOCK_RESULTS=true prefix, VEZA_URL, etc.
    ...

Correct (respects external env vars):

from dotenv import load_dotenv

def fetch(**kwargs):
    load_dotenv()  # ✓ External env vars take precedence
    ...

2. Never include testing flags in .env files

Testing flags like MOCK_RESULTS and VEZA_URL should not be in .env. Instead, test via environment variable prefix:

Incorrect .env:

MOCK_RESULTS=false      # ❌ Breaks test prefix pattern
VEZA_URL=https://dummy  # ❌ Interferes with real credentials

Correct .env:

MYAPP_API_KEY=xxxxx
MYAPP_BASE_URL=https://api.myapp.com

Test command:

MOCK_RESULTS=true oaa-run ... run

3. VEZA_URL and VEZA_API_KEY are implicitly required

Even without explicit required_env configuration, these two variables are always validated at startup. Use --no-push=True in dry-run mode to skip Veza credential validation:

oaa app run --config=config.yaml --no-push=True

Troubleshooting

Error: Required environment variable "MY_API_KEY" is not set

  • Verify .env file exists in the same directory as config.yaml

  • Check that the variable name matches exactly (case-sensitive)

  • Ensure no extra whitespace in .env values

Error: Required environment variable "VEZA_URL" is not set

  • VEZA_URL and VEZA_API_KEY are always required, even without required_env

  • Add them to your .env file or set them in your shell environment

Variable injected but empty string

  • Check that your .env file has the value after the = sign

  • Empty strings pass validation (only missing variables fail)

Module still calling load_dotenv()

  • Safe to leave load_dotenv() calls in existing modules for backward compatibility

  • The call is idempotent — calling it twice has no negative effect

Next Steps