Spark and Dataproc lineage

A Spark job usually reads from one system and writes to another — BigQuery in, an Iceberg REST catalog out. On Google Cloud, the primary path is Dataproc's managed lineage integration: Dataproc publishes into Google Data Lineage, and Embrasure reads that graph after BigQuery catalog syncs. The Spark job does not need network access or credentials for Embrasure.

Use Google-managed lineage on Dataproc

Enable the Data Lineage API in the project and turn on lineage for the cluster or serverless batch:

gcloud dataproc clusters create "$CLUSTER" \
  --region="$REGION" \
  --properties=dataproc:dataproc.lineage.enabled=true

The Dataproc service agent needs permission to publish lineage events. The service identity used by the Embrasure BigQuery connector needs roles/datalineage.viewer in each project whose lineage should be imported.

Sync the BigQuery connector and the Iceberg REST connector before validating the first job. Embrasure publishes each Iceberg table's canonical object-storage location, normalizes Google gcs: entities to gs://, and binds a Google lineage provider from the BigQuery source connector to the uniquely matching Iceberg destination connector. A later catalog sync also reconciles providers that were created before the Iceberg catalog first published its name.

Google does not expose a lineage-change listener, so Embrasure imports this graph on the BigQuery catalog-sync cadence. A disabled API or missing viewer grant is reported as unavailable without failing catalog discovery.

Direct OpenLineage fallback

Use the direct HTTP receiver for Spark outside Google Cloud, or when an operator deliberately wants a second delivery path. Do not configure both paths for the same jobs unless duplicate evidence is intentional.

Provision the direct provider

Create the provider before the first job runs, so the binding is reviewable and the receiver URL is known.

curl --request POST \
  --url "https://api.embrasure.ai/v1/lineage/providers" \
  --header "Authorization: Bearer $EMBRASURE_API_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "workspace_id": "'"$WORKSPACE_ID"'",
    "provider_key": "dataproc-spark",
    "display_name": "Dataproc Spark",
    "bindings": [
      { "connector_id": "'"$BIGQUERY_CONNECTOR_ID"'", "role": "source" },
      { "connector_id": "'"$ICEBERG_CONNECTOR_ID"'", "role": "destination" }
    ]
  }'

The response carries the provider identity, its bindings, current health, and the receiver URL:

{
  "provider_instance_id": "0f2d3e2e-6f5f-4d09-9a1a-2b0c2f1f2c7e",
  "provider_key": "dataproc-spark",
  "created": true,
  "bindings": [
    { "connector_id": "…", "connector_kind": "bigquery", "binding_role": "source" },
    { "connector_id": "…", "connector_kind": "iceberg_rest", "binding_role": "destination" }
  ],
  "health": { "status": "active", "last_event_at": null, "backlog_count": 0 },
  "receiver_url": "https://api.embrasure.ai/api/v1/lineage?workspace_id=…&connector_id=…&provider_key=dataproc-spark"
}

Pass exactly one binding with role any for a single-connector job, or one source plus one destination for a cross-connector job.

You can also create the provider from the app: open a source's overflow menu and choose Set up OpenLineage. The panel creates the binding, then shows the receiver URL and the Spark configuration below.

Configure the direct Spark listener

Set EMBRASURE_API_TOKEN to a workspace-scoped personal access token with write scope in the job environment. Never inline the token into a job definition.

spark-submit \
  --conf "spark.jars.packages=io.openlineage:openlineage-spark_2.12:1.24.2" \
  --conf "spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener" \
  --conf "spark.openlineage.transport.type=http" \
  --conf "spark.openlineage.transport.url=https://api.embrasure.ai" \
  --conf "spark.openlineage.transport.endpoint=/api/v1/lineage?workspace_id=$WORKSPACE_ID&connector_id=$ICEBERG_CONNECTOR_ID&provider_key=dataproc-spark" \
  --conf "spark.openlineage.transport.auth.type=api_key" \
  --conf "spark.openlineage.transport.auth.apiKey=$EMBRASURE_API_TOKEN" \
  --conf "spark.openlineage.namespace=$AIRFLOW_NAMESPACE" \
  --conf "spark.openlineage.parentJobNamespace=$AIRFLOW_NAMESPACE" \
  --conf "spark.openlineage.parentJobName=$DAG_ID.$TASK_ID" \
  your_job.py

For a deliberate direct-receiver fallback on Dataproc, submit the same properties with an alternate delimiter because the Maven coordinate contains commas:

gcloud dataproc jobs submit pyspark your_job.py \
  --cluster=$CLUSTER --region=$REGION \
  --properties="^#^spark:spark.jars.packages=io.openlineage:openlineage-spark_2.12:1.24.2#spark:spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener#spark:spark.openlineage.transport.type=http#spark:spark.openlineage.transport.url=https://api.embrasure.ai#spark:spark.openlineage.transport.endpoint=/api/v1/lineage?workspace_id=$WORKSPACE_ID&connector_id=$ICEBERG_CONNECTOR_ID&provider_key=dataproc-spark#spark:spark.openlineage.transport.auth.type=api_key#spark:spark.openlineage.transport.auth.apiKey=$EMBRASURE_API_TOKEN#spark:spark.openlineage.namespace=$AIRFLOW_NAMESPACE#spark:spark.openlineage.parentJobNamespace=$AIRFLOW_NAMESPACE#spark:spark.openlineage.parentJobName=$DAG_ID.$TASK_ID"

The parent job namespace and name link Spark runs to the Airflow task that scheduled them, so a table's lineage points at the DAG task an operator can act on rather than an anonymous application id.

How cross-connector events resolve

Each dataset in an event is resolved independently against the provider's binding set. The destination connector is never stamped onto an input.

A connector hint outside the binding set is quarantined rather than searched. An identity that matches nothing, or matches more than one table, is left unresolved rather than guessed — see Coverage and troubleshooting.

Required read-only access

The Iceberg REST connector is metadata-only: it reads table metadata and never reads a row. Querying Iceberg data needs a separate engine connector, such as Athena, Trino, or Spark SQL, pointed at the same data.

By default, catalog and OAuth endpoints must resolve only to public network addresses. A self-hosted Embrasure deployment that intentionally reaches an in-VPC catalog can set ICEBERG_REST_ALLOW_PRIVATE_NETWORK=true; keep network egress limited to the catalog and token endpoints in that mode.

Certificate verification is never disabled for an https catalog connection, and credentials are never sent over plaintext http: any catalog URI configured with authentication — basic, bearer, or oauth2 — must be https, as must the OAuth2 token endpoint. Plain http is accepted only for an unauthenticated in-cluster catalog whose auth mode is none.