Skip to main content

How to Debug Common Task Failures

When a remote task execution fails, flyte-sdk converts backend execution errors into concrete Python exception classes in the errors module (such as OOMError, TaskTimeoutError, ImagePullBackOffError, and RetriesExhaustedError) so you can pinpoint whether the issue occurred during module discovery, image building, scheduling, or container execution.

The following example demonstrates catching and handling specific failure types returned when monitoring or executing tasks:

from datetime import timedelta
import flyte
from flyte import Resources, RetryStrategy, Timeout
from flyte.errors import (
    BaseRuntimeError,
    DeploymentError,
    ImageBuildError,
    ImagePullBackOffError,
    ModuleLoadError,
    OOMError,
    RetriesExhaustedError,
    TaskTimeoutError,
)

@flyte.task(
    resources=Resources(memory="4GiB", cpu="2"),
    timeout=Timeout(max_runtime=timedelta(minutes=30), max_queued_time=timedelta(minutes=10)),
    retries=RetryStrategy(count=3, backoff=timedelta(seconds=5), backoff_factor=2),
)
def compute_workload(data_size: int) -> int:
    return data_size * 2

try:
    # Trigger or wait on task execution
    result = compute_workload(100)
except OOMError as e:
    print(f"Task exceeded container memory limits. Code: {e.code}, Worker: {e.worker}, Message: {e}")
except TaskTimeoutError as e:
    print(f"Task exceeded maximum runtime or queued time: {e}")
except ImagePullBackOffError as e:
    print(f"Container runtime failed to pull task image: {e}")
except RetriesExhaustedError as e:
    print(f"Task failed after all retries were exhausted: {e}")
except BaseRuntimeError as e:
    print(f"Runtime error ({e.kind}): {e.code} - {e}")

Exception Hierarchy Overview​

All execution exceptions raised by flyte-sdk inherit from BaseRuntimeError, which derives from Python's standard RuntimeError.

BaseRuntimeError
├── RuntimeSystemError (kind="system")
│   ├── UnionRpcError
│   └── LogsNotYetAvailableError
├── RuntimeUnknownError (kind="unknown")
└── RuntimeUserError (kind="user")
    ├── DeploymentError
    ├── ImageBuildError
    ├── ImagePullBackOffError
    ├── InvalidImageNameError
    ├── ModuleLoadError
    ├── NotInTaskContextError
    ├── OOMError
    ├── PrimaryContainerNotFoundError
    ├── ReferenceTaskError
    ├── RetriesExhaustedError
    ├── RunAbortedError
    ├── RuntimeDataValidationError
    ├── TaskInterruptedError
    └── TaskTimeoutError

Every BaseRuntimeError provides the following attributes:

  • code: The error code string (such as "OOMError", "TaskTimeoutError", "DeploymentError", or "ModuleLoadError").
  • kind: The categorization of the error, set to "user", "system", or "unknown".
  • worker: The worker identifier where the failure was reported (or None).

Deployment and Build Failures​

ModuleLoadError​

ModuleLoadError is raised during module discovery when flyte-sdk scans and dynamically imports Python files specified for deployment (via _utils/module_loader.py).

flyte.errors.ModuleLoadError: Failed to load module from /path/to/workflow.py: No module named 'missing_dependency'

Causes​

  • Syntax errors in the scanned Python file.
  • Uninstalled dependencies imported at the top level of the file before remote environment execution.
  • Code executed at module import time (outside task functions) raising unhandled exceptions.

Resolution​

Ensure all top-level imports exist in the local virtual environment used during registration, or move third-party imports inside the task function body:

import flyte

@flyte.task
def process_data():
    # Defer importing heavy packages if not present during deployment scanning
    import heavy_dependency

    return heavy_dependency.run()

DeploymentError​

DeploymentError is raised when registering a task or deployment bundle with Flyte Admin / Task Service fails (in _deploy.py), or when required deployment arguments are missing.

# Deployment requires version when copy_style is "none"
# If copy_style="none" and version is omitted, DeploymentError is raised

Causes​

  • Setting copy_style="none" during deployment without supplying an explicit version string.
  • Network or gRPC authentication failures while contacting the task service during apply() or flyte deploy.

Resolution​

Provide a concrete version parameter when using copy_style="none", and ensure your active environment configuration credentials are valid:

# Deploying with an explicit version when not bundling source code
flyte.deploy(
    [compute_workload],
    version="v1.2.0",
    copy_style="none",
)

ImageBuildError​

ImageBuildError is raised by the remote image builder (_internal/imagebuild/remote_builder.py) when remote image creation fails or the remote builder service is not accessible.

flyte.errors.ImageBuildError: ❌ Build failed in 0:02:15 at https://<host>/executions/<run_id>

Causes​

  • Invalid Dockerfile instructions, broken pip install commands, or inaccessible private package indexes during remote image building.
  • Remote builder task execution reaching PHASE_FAILED.

Resolution​

Open the build execution link printed in the error message to inspect the container build logs, verify pip package version pins, and ensure base images exist.

Scheduling and Image Resolution Failures​

ImagePullBackOffError​

ImagePullBackOffError is raised when Kubernetes cannot pull the container image specified for the task (mapped in _internal/runtime/convert.py when "ImagePullBackOff" appears in the backend execution error code).

Causes​

  • The image name or tag does not exist in the container registry.
  • Missing container registry authentication credentials or Kubernetes image pull secrets.
  • Network restrictions preventing the cluster nodes from contacting the image repository.

Resolution​

Verify the image URI and configure valid container image references on the task or environment:

import flyte
from flyte import Image

custom_image = Image.from_tag(
    name="my-org/my-task-image",
    tag="v1.0.0",
)

@flyte.task(image=custom_image)
def worker_task() -> str:
    return "completed"

Runtime Execution Failures​

OOMError (Out of Memory)​

OOMError is raised when a task container exceeds memory limits and is terminated by the Linux Out-Of-Memory killer (exit code 137). In _internal/runtime/convert.py, flyte-sdk maps backend execution errors to OOMError when "OOM" is present in err.code.upper().

Causes​

  • Large data frames, tensor allocations, or memory-intensive operations exceeding default container memory requests and limits.
  • Tasks running multiple multiprocessing workers consuming shared memory or system RAM.

Resolution​

Increase the requested memory and memory limits on the task using Resources:

import flyte
from flyte import Resources

# Specify exact request, or a (request, limit) tuple
@flyte.task(
    resources=Resources(
        memory=("4GiB", "16GiB"),  # Request 4GiB, limit 16GiB
        cpu=("2", "4"),
        shm="2GiB",                # Shared memory allocation for multiprocessing / PyTorch
    )
)
def train_model():
    ...

TaskTimeoutError​

TaskTimeoutError is raised when task execution reaches PHASE_TIMED_OUT (monitored in _internal/controllers/remote/_controller.py).

Causes​

  • The task execution duration exceeded the configured max_runtime.
  • The task remained queued waiting for cluster resources longer than max_queued_time.

Resolution​

Configure the Timeout object or pass integer seconds to extend execution limits:

from datetime import timedelta
import flyte
from flyte import Timeout

@flyte.task(
    timeout=Timeout(
        max_runtime=timedelta(hours=2),        # Maximum single-attempt runtime
        max_queued_time=timedelta(minutes=30), # Maximum time in scheduling queue
    )
)
def batch_job():
    ...

RetriesExhaustedError​

RetriesExhaustedError is raised when a task fails repeatedly across all retry attempts (mapped in _internal/runtime/convert.py when "RetriesExhausted" appears in err.code).

Causes​

  • Persistent runtime exceptions (e.g., downstream API outages, unhandled exceptions) that failed on each retry attempt.
  • Insufficient retry counts or absence of backoff delays for transient network errors.

Resolution​

Configure a RetryStrategy with backoff to handle intermittent failures, and inspect the nested root cause in logs:

from datetime import timedelta
import flyte
from flyte import RetryStrategy

@flyte.task(
    retries=RetryStrategy(
        count=5,                               # Retry up to 5 times
        backoff=timedelta(seconds=10),          # Base backoff duration
        backoff_factor=2,                      # Exponential multiplier
    )
)
def network_request_task() -> dict:
    ...

Troubleshooting Summary​

Exception ClassTrigger ConditionPrimary Resolution
ModuleLoadErrorPython module failed to import during discovery / deployment scanning.Check for syntax errors; ensure dependencies are installed locally or import inside task functions.
DeploymentErrorTask registration failed or copy_style="none" used without a version.Set explicit version="<tag>" when copy_style="none"; verify cluster connectivity.
ImageBuildErrorRemote image build task failed (PHASE_FAILED).Inspect build execution URL logs for failing package installs or Dockerfile steps.
ImagePullBackOffErrorContainer image cannot be pulled by Kubernetes runtime.Check image name/tag and verify image pull secret permissions on the cluster.
OOMErrorContainer killed by Linux OOM killer (exit code 137).Increase memory in Resources(memory=...) or configure shared memory (shm=...).
TaskTimeoutErrorTask exceeded max_runtime or max_queued_time.Adjust timeout limits using Timeout(max_runtime=..., max_queued_time=...).
RetriesExhaustedErrorTask failed repeatedly across all configured attempts.Increase retry count in RetryStrategy(count=...), add backoff, or fix underlying root exception.