Base on @StuffbyYuki Slack message:
Just a random thought - I wonder how hard it'd be to make sqlmesh dbt-integrated tool compatible where sqlmesh can utilize the kind of integrations dbt has with other tools (BI, data observability, etc) via dbt artifacts. I can see some people are hesitant to try sqlmesh because they already have tools integrated around dbt, so if sqlmesh can use the existing dbt ecosystem, I can see more users would be willing to try sqlmesh.
Problem Statement
A major friction point for users considering adopting—or staying with—SQLMesh is the loss of the broader dbt ecosystem. Many organizations have built extensive data stacks around dbt using data observability platforms, BI tools, and data cataloging solutions that rely heavily on dbt artifacts (such as manifest.json).
If a team starts a project natively in SQLMesh, they currently cannot utilize this vast, pre-existing ecosystem of integrations. Waiting for every third-party data tool to build and maintain native SQLMesh integrations is a slow process and creates a significant bottleneck for adoption. In some cases, this lack of ecosystem support is causing users who have already adopted SQLMesh to consider migrating back to dbt just to regain access to their preferred observability and BI integrations.
Impact / Use Cases
Users are currently blocked from easily integrating SQLMesh with:
- Data Observability: Tools like Monte Carlo, Elementary, or Metaplane that ingest metadata and lineage to monitor data health.
- Data Catalogs: Platforms like Atlan, DataHub, or Select that rely on manifest files to map organizational data assets.
- BI Tools: Dashboards that dynamically update or expose lineage based on transformation tool metadata.
Additional Context
We need a way to bridge this gap so SQLMesh users can utilize the existing dbt ecosystem without being blocked by third-party vendor roadmaps. What exact capabilities and integration points would we need to support to alleviate this ecosystem lock-in?
Base on @StuffbyYuki Slack message:
Problem Statement
A major friction point for users considering adopting—or staying with—SQLMesh is the loss of the broader dbt ecosystem. Many organizations have built extensive data stacks around dbt using data observability platforms, BI tools, and data cataloging solutions that rely heavily on dbt artifacts (such as manifest.json).
If a team starts a project natively in SQLMesh, they currently cannot utilize this vast, pre-existing ecosystem of integrations. Waiting for every third-party data tool to build and maintain native SQLMesh integrations is a slow process and creates a significant bottleneck for adoption. In some cases, this lack of ecosystem support is causing users who have already adopted SQLMesh to consider migrating back to dbt just to regain access to their preferred observability and BI integrations.
Impact / Use Cases
Users are currently blocked from easily integrating SQLMesh with:
Additional Context
We need a way to bridge this gap so SQLMesh users can utilize the existing dbt ecosystem without being blocked by third-party vendor roadmaps. What exact capabilities and integration points would we need to support to alleviate this ecosystem lock-in?