Cloud and warehouses

Connect each data system with a dedicated read-only identity. Select the smallest useful catalog scope, then add runtime or cloud-lineage evidence for the jobs that move data through it.

AWS

Google Cloud

BigQuery

Embrasure creates a managed service account for each BigQuery connector. Grant it:

for role in roles/bigquery.jobUser roles/bigquery.dataViewer roles/datalineage.viewer; do
  gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:EMBRASURE_SERVICE_ACCOUNT" \
    --role="$role"
done

Prefer dataset-level Data Viewer grants. Grant Data Lineage Viewer in every project whose links should be read.

Dataproc and Spark

Enable the Data Lineage API and publish Spark lineage:

gcloud dataproc clusters create YOUR_CLUSTER \
  --region=YOUR_REGION \
  --properties=dataproc:dataproc.lineage.enabled=true

Run a job that reads and writes cataloged assets, sync the BigQuery connector, then inspect the output table. New links appear on the connector sync cadence.

Azure

Use managed identity or a dedicated service principal. Azure SQL and Cosmos DB provide catalog identity. Synapse, Data Factory, ADLS, and Power BI lineage arrive through Microsoft Purview or direct OpenLineage producers. Select subscriptions, resource groups, workspaces, and environments explicitly.

Snowflake, Databricks, and SQL databases

Cloud lineage source contract

Cloud lineage feeds use one configuration shape:

{
  "provider": "amazon_datazone",
  "scope": {
    "account_id": "123456789012",
    "region": "us-east-1",
    "domain_id": "dzd_example"
  },
  "identity": {
    "mode": "assume_role",
    "role_arn": "arn:aws:iam::123456789012:role/embrasure-lineage-reader"
  },
  "environment_map": {"prod": "production"},
  "sync_cadence_minutes": 30
}

provider accepts google_data_lineage, amazon_datazone, or microsoft_purview. Provider-specific scope uses projects and locations for Google Cloud, account, region, and domain for AWS, or tenant, subscription, and collection for Azure.

Spark event-log source contract

Use one source for Dataproc, EMR, Glue, Synapse Spark, or self-managed Spark logs:

{
  "kind": "spark_event_log",
  "storage_provider": "s3",
  "uri": "s3://company-spark-events/production/",
  "connector_id": "connector-id",
  "environment": "production",
  "include_prefixes": ["jobs/"],
  "exclude_prefixes": ["tmp/"],
  "sync_cadence_minutes": 15
}

storage_provider accepts s3, gcs, or adls. The source reuses the linked connector identity and never accepts embedded cloud secrets.