Pipelines and applications

Use runtime events when a system can report what actually ran. Add declared artifacts and static analysis to fill gaps without overriding stronger evidence.

Airflow

Install the native OpenLineage provider and point its HTTP transport at Embrasure:

pip install apache-airflow-providers-openlineage

export AIRFLOW__OPENLINEAGE__NAMESPACE='airflow-production'
export AIRFLOW__OPENLINEAGE__TRANSPORT="$(cat <<JSON
{
  "type": "http",
  "url": "https://api.embrasure.ai",
  "endpoint": "/api/v1/lineage",
  "urlParams": {
    "workspace_id": "$EMBRASURE_WORKSPACE_ID",
    "connector_id": "$EMBRASURE_CONNECTOR_ID"
  },
  "auth": {
    "type": "api_key",
    "apiKey": "$EMBRASURE_API_TOKEN"
  }
}
JSON
)"

Keep the namespace stable for one deployment. Run a DAG with input and output datasets, confirm a successful POST in task logs, then inspect the output asset.

Spark and Flink

Use the OpenLineage listener with the same receiver for Dataproc, EMR, Glue, Synapse, Kubernetes, or self-managed compute.

spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener
spark.openlineage.transport.type=http
spark.openlineage.transport.url=https://api.embrasure.ai
spark.openlineage.transport.endpoint=/api/v1/lineage?workspace_id=WORKSPACE_ID&connector_id=CONNECTOR_ID
spark.openlineage.transport.auth.type=api_key
spark.openlineage.transport.auth.apiKey=EMBRASURE_API_TOKEN
spark.openlineage.namespace=spark-production

For Flink, set the equivalent openlineage.transport.* properties. Preserve Airflow parent-run facets when an orchestrator launches the compute job so the graph keeps job hierarchy.

dbt

Upload target/manifest.json after dbt compile or dbt build:

: "${EMBRASURE_LINEAGE_ACQUISITION_ID:?set a unique acquisition ID}"

jq --arg workspace "$WORKSPACE_ID" \
   --arg connector "$CONNECTOR_ID" \
   --slurpfile manifest target/manifest.json \
   '{workspace_id:$workspace, connector_id:$connector, manifest:$manifest[0], completeness:"complete"}' \
  > /tmp/embrasure-dbt-manifest.json

curl --fail-with-body --request POST \
  --url https://api.embrasure.ai/v1/pipelines/dbt/manifest \
  --header "Authorization: Bearer $EMBRASURE_API_TOKEN" \
  --header "Content-Type: application/json" \
  --header "Idempotency-Key: dbt-$BUILD_ID" \
  --data-binary @/tmp/embrasure-dbt-manifest.json

Use complete only for the exact project and connector scope. Use a new idempotency key for each acquisition and reuse it only to retry identical bytes.

SQLMesh

Export and upload

Export the project after SQLMesh applies the target environment, then submit the rendered project to POST /v1/pipelines/sqlmesh/project. Run the exporter with the same Python interpreter that runs SQLMesh, after sqlmesh plan --auto-apply or your normal deployment step. It needs only the standard library and your existing SQLMesh install.

Download the version pinned to the embrasure.sqlmesh-project@v1 export schema into your SQLMesh project:

curl --fail --location \
  --output embrasure_sqlmesh_export.py \
  https://raw.githubusercontent.com/EmbrasureAI/embrasure/67665401b7bb08d49d4912f9d65d8204fe847c39/scripts/sqlmesh/embrasure_sqlmesh_export.py

mkdir -p target
python embrasure_sqlmesh_export.py \
  --project . \
  --environment prod \
  --output target/embrasure-sqlmesh-project.json

The exporter resolves environment-specific physical snapshot names from SQLMesh state. Embrasure keeps each virtual model name as the stable asset identity and records its physical snapshot as an alternate identifier. If state is unavailable, the exporter warns on stderr. Virtual table lineage still imports, but physical-name merging is unavailable.

Wrap the export with its workspace, connector, and project identities, then upload it:

jq --arg workspace "$WORKSPACE_ID" \
   --arg connector "$CONNECTOR_ID" \
   --slurpfile project target/embrasure-sqlmesh-project.json \
   '{workspace_id:$workspace, connector_id:$connector, project_id:"analytics-sqlmesh", completeness:"complete", project:$project[0]}' \
  > /tmp/embrasure-sqlmesh-request.json

curl --fail-with-body --request POST \
  --url https://api.embrasure.ai/v1/pipelines/sqlmesh/project \
  --header "Authorization: Bearer $EMBRASURE_API_TOKEN" \
  --header "Content-Type: application/json" \
  --header "Idempotency-Key: sqlmesh-$EMBRASURE_LINEAGE_ACQUISITION_ID" \
  --data-binary @/tmp/embrasure-sqlmesh-request.json

Use complete only for an entire project in one environment. The route accepts partial or complete, honors Idempotency-Key, and returns the standard lineage delivery receipt. Set the acquisition ID to a unique per-run value, such as $GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT in GitHub Actions. Every acquisition needs a new key, including a return to previously uploaded content. Reuse a key only to retry the same request.

Rendered SQL supplies column mappings only when every read matches a declared dependency. Python models and models without rendered SQL keep table-level dependencies without guessed column lineage. See the SQLMesh project API reference for the request contract.

Runtime events (optional)

Runtime events can add executed evidence alongside the project export. The community-maintained sqlmesh-openlineage package is not an Embrasure emitter; project export remains the supported SQLMesh path. It requires Python-based SQLMesh configuration.

Install the package, then enable it in config.py before creating the SQLMesh Config:

pip install sqlmesh-openlineage

import sqlmesh_openlineage

sqlmesh_openlineage.install()

Use the bare API origin because OpenLineage clients append /api/v1/lineage. Authenticate with a workspace-scoped token and keep the namespace stable for one deployment:

export OPENLINEAGE_URL="https://api.embrasure.ai"
export OPENLINEAGE_NAMESPACE="analytics-sqlmesh-prod"
export OPENLINEAGE_API_KEY="$EMBRASURE_WORKSPACE_SCOPED_TOKEN"

Run sqlmesh run normally. See the lineage receiver for authentication and dataset resolution.

Fivetran

Bind each Fivetran source and destination to its Embrasure connectors. Embrasure records connector assets, sync runs, table mappings, and enabled column mappings. Keep source and destination bindings immutable so a later sync cannot cross workspace or connector scope.

Kafka, Kafka Connect, and Debezium

Configure the Kafka cluster, optional schema registry, and Kafka Connect endpoint. Select connectors and topic prefixes, then map each deployment to an environment. Embrasure joins CDC source tables to topics and topics to sinks using connector metadata, schemas, and OpenLineage run events.

Required fields are bootstrap_servers, environment, and topic_scope. schema_registry_url, connect_url, include_connectors, and exclude_topics are optional. Store credentials in the connector secret store, never in the lineage payload.

Looker, Tableau, Power BI, and QuickSight

Connect with provider OAuth or a service identity, then select the sites, workspaces, projects, or folders to index. The BI connector catalogs models, datasets, reports, dashboards, and refresh runs, then resolves their upstream warehouse objects.

Required fields are provider, identity, workspace_scope, environment, and sync_cadence_minutes. Column mappings come from semantic-model expressions when the provider exposes them.

DataHub and custom applications

Import DataHub exports at POST /v1/pipelines/lineage/datahub. For applications and APIs, send standard OpenLineage datasets, jobs, runs, and optional columnLineage facets to the lineage receiver.

Use stable, case-sensitive namespaces and names. Both endpoints must resolve to cataloged objects before an edge appears in the served graph.