OpenLineage
Your scheduler already knows. Let it say so.
Ingest run events, map namespaces, and issue API keys. Browse jobs, datasets, and events, with links back to Airflow.
The registry knows what your schemas say; your scheduler knows what actually ran. Ingestion joins the two, so a run and the objects it touched sit next to each other.
-
1. Point your producer at the endpoint
Settings → OpenLineage shows the ingestion endpoint and a ready-made curl example, plus the API keys that authenticate it.
-
2. Map namespaces
A namespace the producer reports is matched to one of your instances, by host and port or by an explicit mapping you record.
-
3. Read the run
Job, dataset and event views keep the raw JSON and its facets, so a claim can always be traced back to the event that made it.
Why the facet is not trusted
The producer's column facet is not trusted. An Airflow SQL operator attaches one column facet per task — a run-level guess, not a statement's answer: it merges every statement into one input set and attributes an output column by finding a same-named column anywhere in it. That reads as an answer, and it can be wrong. So when an event carries SQL of its own, that SQL is run through the same statement-scoped analyzers that back view definitions and manual SQL. A producer that states column lineage without SQL keeps its own facet; a run that cannot be analyzed keeps its table-level edges and reports the gap.
Known limits
- The producer's column facet is not trusted when the event carries SQL of its own — the statement is re-analyzed instead.
- A run that cannot be analyzed keeps its table-level edges and reports the gap rather than guessing.