Cloud and warehouses

Connect each data system with a dedicated read-only identity. Select the smallest useful catalog scope, then add runtime or transformation evidence for the jobs that move data through it.

Catalog scope controls what Embrasure can see. To control what is served as lineage by schema, query principal, statement kind, provider, or engine, use each connector's lineage policy. Policies retain the underlying evidence and repair the graph retroactively.

AWS

AWS sources use three paths: catalog and warehouse access, runtime lineage, and native lineage. AWS Glue Data Catalog stores table metadata; AWS Glue jobs are Spark compute and use a different setup.

Catalogs and warehouses

Runtime lineage

Amazon EMR and AWS Glue jobs

Keep the existing Spark execution role and send OpenLineage events directly to Embrasure. The role retains its normal data and catalog permissions; it does not need Embrasure access to the EMR or Glue control plane. Configure the listener using the Spark and Flink setup.

Native lineage

Amazon DataZone

Provide the AWS account, region, domain ID, and a read-only assumed role using the same external-ID trust pattern as Athena. Grant datazone:GetLineageEvent, datazone:ListLineageEvents, datazone:GetLineageNode, and datazone:ListLineageNodeHistory; do not grant datazone:PostLineageEvent. Embrasure imports DataZone's versioned dataset and job lineage on the configured sync cadence. See Data lineage in Amazon DataZone.

Google Cloud

BigQuery

Embrasure creates a managed service account for each BigQuery connector. Grant it:

for role in roles/bigquery.jobUser roles/bigquery.dataViewer roles/datalineage.viewer; do
  gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
    --member="serviceAccount:EMBRASURE_SERVICE_ACCOUNT" \
    --role="$role"
done

Prefer dataset-level Data Viewer grants. Grant Data Lineage Viewer in every project whose links should be read.

Enable the Google Data Lineage API in the project that runs the BigQuery jobs (the compute project), then run new supported BigQuery jobs to generate lineage:

gcloud services enable datalineage.googleapis.com \
  --project=YOUR_COMPUTE_PROJECT_ID

BigQuery does not have a per-job lineage flag. After the API is enabled, Google automatically records supported copy, load, query, DDL, and DML jobs. Earlier jobs are not backfilled, and new lineage can take up to 24 hours to appear. If jobs run in a different project from the tables they read or write, enable the API in the job's compute project and grant Embrasure roles/datalineage.viewer there as well.

Dataproc and Spark

Spark lineage is not emitted unless it is enabled. Enable the Data Lineage API in the project, then choose the setup that matches the runtime.

For a Dataproc cluster, enable lineage when creating the cluster. All supported Spark jobs submitted to that cluster then publish lineage:

gcloud dataproc clusters create YOUR_CLUSTER \
  --project=YOUR_PROJECT_ID \
  --region=YOUR_REGION \
  --properties=dataproc:dataproc.lineage.enabled=true

For a Managed Service for Apache Spark serverless batch, enable lineage on the batch itself:

gcloud dataproc batches submit pyspark YOUR_JOB.py \
  --project=YOUR_PROJECT_ID \
  --region=YOUR_REGION \
  --properties=spark.dataproc.lineage.enabled=true

You can instead enable future Spark workloads at the project level with DATAPROC_LINEAGE_ENABLED=true. Managed Service for Apache Spark runtime 3.0 does not inherit that project setting, so enable spark.dataproc.lineage.enabled=true on each 3.0 batch, session, or session template.

Run a job that reads and writes cataloged assets, sync the BigQuery connector, then inspect the output table. New links appear on the connector sync cadence.

Azure

Use the connector's supported read-only identity for Azure SQL, Azure PostgreSQL, Azure MySQL, or Cosmos DB. Select the smallest useful database and schema scope. For jobs that transform these assets, send OpenLineage events from the runtime or import a supported transformation artifact.

Snowflake

Use a dedicated service user, role, and quickly suspending warehouse. Generated key-pair setup is recommended: Embrasure creates an RSA key pair, keeps the private key encrypted, and gives an administrator one SQL block to install the public key and baseline grants. A role-restricted Snowflake programmatic access token is also supported.

Required read access

The generated setup block applies these grants. Replace the example names when managing the role yourself.

GRANT USAGE ON WAREHOUSE COMPUTE_XS TO ROLE EMBRASURE_RO;
GRANT USAGE ON DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT USAGE ON ALL SCHEMAS IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT USAGE ON FUTURE SCHEMAS IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;

GRANT SELECT ON ALL TABLES IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT SELECT ON FUTURE TABLES IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT SELECT ON ALL VIEWS IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT SELECT ON FUTURE VIEWS IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT SELECT ON ALL MATERIALIZED VIEWS IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;
GRANT SELECT ON FUTURE MATERIALIZED VIEWS IN DATABASE ANALYTICS TO ROLE EMBRASURE_RO;

For a smaller scope, grant USAGE and SELECT per schema instead of across the database. Snowflake documents the same container and object grants in its read-only role guidance.

Query history, lineage, and warehouse evidence

These grants are optional, but they unlock stronger observability. Use Snowflake's database roles instead of the broader IMPORTED PRIVILEGES grant.

GRANT DATABASE ROLE SNOWFLAKE.GOVERNANCE_VIEWER TO ROLE EMBRASURE_RO;
GRANT DATABASE ROLE SNOWFLAKE.USAGE_VIEWER TO ROLE EMBRASURE_RO;

GOVERNANCE_VIEWER covers QUERY_HISTORY, QUERY_ATTRIBUTION_HISTORY, and ACCESS_HISTORY; USAGE_VIEWER covers WAREHOUSE_METERING_HISTORY. See Snowflake's database-role mapping for Account Usage. Without these grants, catalog sync and approved read-only queries still work, but history, native lineage, or cost analysis may be unavailable.

Databricks

Use a Databricks service principal with OAuth machine-to-machine authentication. Assign it to the workspace, grant the Databricks SQL access entitlement, and give it CAN USE on one SQL warehouse. Embrasure needs the workspace URL and that warehouse's HTTP path. Databricks documents these steps in its OAuth service-principal guide.

Unity Catalog read access

Grant access only to the catalogs and schemas Embrasure should understand and query. USE CATALOG and USE SCHEMA allow object access; SELECT on the schema is inherited by its current and future tables and views.

GRANT USE CATALOG ON CATALOG analytics TO `SERVICE_PRINCIPAL_APPLICATION_ID`;
GRANT USE SCHEMA ON SCHEMA analytics.production TO `SERVICE_PRINCIPAL_APPLICATION_ID`;
GRANT SELECT ON SCHEMA analytics.production TO `SERVICE_PRINCIPAL_APPLICATION_ID`;

Repeat the schema grants for each selected scope. If the connection is metadata-only, BROWSE on the catalog exposes object metadata without data access. For query execution, keep the explicit USE CATALOG, USE SCHEMA, and SELECT grants. See the Unity Catalog privileges reference.

System lineage and query history

Databricks system tables require a Unity Catalog-enabled workspace. An account and metastore administrator has access by default and can grant the service principal only the tables Embrasure uses.

GRANT USE CATALOG ON CATALOG system TO `SERVICE_PRINCIPAL_APPLICATION_ID`;

GRANT USE SCHEMA ON SCHEMA system.access TO `SERVICE_PRINCIPAL_APPLICATION_ID`;
GRANT SELECT ON TABLE system.access.column_lineage TO `SERVICE_PRINCIPAL_APPLICATION_ID`;
GRANT SELECT ON TABLE system.access.workspaces_latest TO `SERVICE_PRINCIPAL_APPLICATION_ID`;

GRANT USE SCHEMA ON SCHEMA system.query TO `SERVICE_PRINCIPAL_APPLICATION_ID`;
GRANT SELECT ON TABLE system.query.history TO `SERVICE_PRINCIPAL_APPLICATION_ID`;

system.access.column_lineage supplies Unity Catalog-native column lineage, and system.query.history supplies observed SQL. system.access.workspaces_latest lets Embrasure verify the workspace's account identity during setup. System tables can contain account-wide operational data, so omit any optional grant your governance policy does not allow. See Databricks' system-table access guidance.

SQL databases

Google lineage sync

The BigQuery connector reads Google Cloud Data Lineage directly during catalog sync when its service account has roles/datalineage.viewer. There is no separate Google Data Lineage catalog connection to configure.