Purpose
This article explains the main debugging path for failed runs and where to look first depending on the kind of problem.
Overview
The quickest way to debug a failed run is to combine:
- the workflow context
- the orchestration state
- the relevant logs
In practice, the main debugging path depends on whether the failure happened in an offline or online flow, but the most useful tools are usually:
- Ray, for execution-local logs and runtime state
- VictoriaLogs, for centralized logs across services, including Ray logs
- VictoriaMetrics, for resource and runtime metrics
- Superset health checks, to confirm service availability
- Jaeger, to evaluate the execution path within the orchestrator
Step 1: Identify what failed
Start by answering:
- which job, or service failed?
- did the request fail immediately, later during execution, or appear to hang?
This helps narrow the search before opening multiple tools. If the answer to this question is not clear, then it can be narrowed down by
Investigating the logs via VictoriaLogs
Investigating the Ray server for issues related to the deployment
Step 2: Check centralized logs
Ray logs are also centralized in VictoriaLogs, along with logs from other parts of the platform.
VictoriaLogs is usually the best place to start when:
- you do not yet know which subsystem failed
- the failure may span multiple services
- you want one place to correlate logs across the platform
Step 3: Check Ray if the failure is compute-related
If the problem is related to compute resources then the Ray interface is often the best local debugging view.
It is especially useful when you already know the issue is inside Ray workers or Ray Serve.
Step 4: Check metrics and health if needed
If the run looks stalled, unstable, or unexpectedly slow, check:
- VictoriaMetrics for resource activity such as GPU usage or workload behavior
- Superset health checks to see whether one of the platform services was down
- Jaeger, to check for internal issues in the orchestrator
This helps distinguish between:
- an application-level error
- a resource/runtime issue
- a service outage
Recommended debugging path
A practical default workflow is:
- identify the failing run or service
- inspect the failing stage or deployment in Airflow if applicable
- inspect Ray if the problem is in training, serving, or distributed execution
- inspect VictoriaLogs for centralized context
- inspect metrics and health dashboards if the failure may involve load or service availability
Common patterns
The run fails immediately
This often points to a configuration issue, startup issue, or missing dependency.
The run starts and then crashes
This often points to a runtime problem, a Ray execution failure, or a downstream service problem.
The run appears stuck
This often calls for metrics and health checks in addition to logs.
The failure is hard to localize
Start in VictoriaLogs, since it already aggregates Ray logs and broader platform logs.
Best practices
- keep the YAML configuration nearby while debugging
- start with the failing stage or service, not the whole platform
- correlate time across Airflow, logs, and metrics
- use Ray for local execution detail and VictoriaLogs for broader context
Comments
0 comments
Article is closed for comments.