pipelines with Helios
In distributed application environments, to understand how to solve problems in code you need to be able to connect the dots between all the different places where your code runs, including frameworks like Databricks and Apache Airflow. The Databricks pipeline may be one of the most crucial places where your code runs, but the visibility you’re getting is limited. Usually, the pipeline is detached from the rest of your architecture – which makes it nearly impossible to test, monitor, and understand how your code is executed. Databricks notebooks are often triggered by microservices, which also consume their output, but all those components are siloed.
How Helios can help your team
Helios provides actionable insight into your end-to-end flows by adapting OpenTelemetry’s context propagation method to fit the specific mechanism of triggering and running Databricks notebooks.
Helios enables you to: