Visualizing your data
pipelines with Helios
Challenge
In distributed application environments, to understand how to solve problems in code you need to be able to connect the dots between all the different places where your code runs, including frameworks like Databricks and Apache Airflow. The Databricks pipeline may be one of the most crucial places where your code runs, but the visibility you’re getting is limited. Usually, the pipeline is detached from the rest of your architecture – which makes it nearly impossible to test, monitor, and understand how your code is executed. Databricks notebooks are often triggered by microservices, which also consume their output, but all those components are siloed.
How Helios can help your team
Helios provides actionable insight into your end-to-end flows by adapting OpenTelemetry’s context propagation method to fit the specific mechanism of triggering and running Databricks notebooks.
Helios enables you to:
- See downstream how services interact with each other
and the connections between them - See how data flows through your entire application
with interactive trace visualization - Understand issues and where they occur,
and easily resolve them before deploying to any environment - Generate tests across the end-to-end flow,
including your Databricks notebooks
- See downstream how services interact with each other and the connections between them
- See how data flows through your entire application with interactive trace visualization
- Understand issues and where they occur, and easily resolve them before deploying to any environment
- Generate tests across the end-to-end flow, including your Databricks notebooks
Start taking control
of your application now
