Understanding Runtime Errors

When a Flyte task fails, the exception text alone does not tell you how flyte-sdk classified the failure. Runtime errors carry three structured fields—kind, code, and worker—alongside the human-readable message. The classification is used by task execution, protocol conversion, and the CLI; it is an operational classification, not always a definitive diagnosis of who caused the failure.

The runtime-error hierarchy

BaseRuntimeError in errors.py is the common base and extends Python’s RuntimeError. Its constructor accepts a machine-readable code, an ErrorKind, a root-cause message, and optional worker metadata:

ErrorKind = Literal["system", "unknown", "user"]

class BaseRuntimeError(RuntimeError):
    def __init__(self, code: str, kind: ErrorKind, root_cause_message: str, worker: str | None = None):
        super().__init__(root_cause_message)
        self.code = code
        self.kind = kind
        self.worker = worker

Because the message is passed to RuntimeError, str(error) contains the root-cause message. The classification and identifier are separate attributes: error.kind is the runtime category, error.code identifies the failure, and error.worker contains optional execution-worker metadata.

The base constructor’s annotation restricts kind to the three literals, but it does not validate the value at runtime. BaseRuntimeError is therefore a common data and handling type, rather than a category by itself.

The three category classes hard-code their category while leaving the code, message, and worker to the caller:

class RuntimeSystemError(BaseRuntimeError):
    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "system", message, worker)

class RuntimeUserError(BaseRuntimeError):
    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "user", message, worker)

class RuntimeUnknownError(BaseRuntimeError):
    def __init__(self, code: str, message: str, worker: str | None = None):
        super().__init__(code, "unknown", message, worker)

RuntimeUserError, RuntimeSystemError, and RuntimeUnknownError are sibling categories. Specialized failures such as OOMError, TaskInterruptedError, and RetriesExhaustedError inherit from RuntimeUserError; UnionRpcError inherits from RuntimeSystemError. The RuntimeSystemError docstring explicitly allows either a Union-system bug or a bug in user code, so kind should be read as the runtime’s classification rather than proof of ownership.

How classifications are created

At task invocation, TaskTemplate.__call__ preserves errors that already have a runtime category. If a task-context invocation has no controller, it raises RuntimeSystemError("BadContext", "Controller is not initialized."). Its exception boundary re-raises RuntimeSystemError and RuntimeUserError unchanged. Any other exception is converted with RuntimeUserError(type(e).__name__, str(e)) and raised from the original exception. Thus an otherwise-unclassified Python exception becomes a user-classified runtime error whose code is the exception class name.

Run creation applies a similar classification to gRPC status codes in _run.py. An unavailable service is classified as a system failure, while invalid arguments and an already-existing run are classified as user failures. The source raises RuntimeSystemError("SystemUnavailableError", ...) for grpc.StatusCode.UNAVAILABLE, RuntimeUserError("InvalidArgumentError", e.details()) for INVALID_ARGUMENT, and RuntimeUserError("RunAlreadyExistsError", ...) for ALREADY_EXISTS. Unexpected run-creation RPC statuses are raised as RuntimeSystemError with code RunCreationError. The classification is consequently separate from the lower-level gRPC exception that triggered it.

Conversion across remote execution

Remote execution reports failures through flyteidl.core.execution_pb2.ExecutionError. _internal/runtime/convert.py maps its protocol kind to the native hierarchy. UNKNOWN becomes RuntimeUnknownError, SYSTEM becomes RuntimeSystemError, and USER normally becomes RuntimeUserError. USER errors can also be mapped to specialized user subclasses by inspecting the error code:

def convert_error_to_native(err: execution_pb2.ExecutionError | Exception | Error) -> Exception | None:
    if not err:
        return None

    if isinstance(err, Exception):
        return err

    if isinstance(err, Error):
        err = err.err

    user_code, server_code = _clean_error_code(err.code)
    match err.kind:
        case execution_pb2.ExecutionError.UNKNOWN:
            return flyte.errors.RuntimeUnknownError(code=user_code, message=err.message, worker=err.worker)
        case execution_pb2.ExecutionError.USER:
            if "OOM" in err.code.upper():
                return flyte.errors.OOMError(code=user_code, message=err.message, worker=err.worker)
            elif "Interrupted" in err.code:
                return flyte.errors.TaskInterruptedError(code=user_code, message=err.message, worker=err.worker)
            elif "RetriesExhausted" in err.code:
                return flyte.errors.RetriesExhaustedError(code=user_code, message=err.message, worker=err.worker)
            elif "Unknown" in err.code:
                return flyte.errors.RuntimeUnknownError(code=user_code, message=err.message, worker=err.worker)
            return flyte.errors.RuntimeUserError(code=user_code, message=err.message, worker=err.worker)
        case execution_pb2.ExecutionError.SYSTEM:
            return flyte.errors.RuntimeSystemError(code=user_code, message=err.message, worker=err.worker)
    return None

The complete classifier also recognizes PrimaryContainerNotFound, InvalidImageName, and ImagePullBackOff. The OOM check is case-insensitive because the code is uppercased; the checks for Interrupted, RetriesExhausted, and Unknown are case-sensitive substring checks. A USER protocol error whose code contains Unknown is converted to RuntimeUnknownError, despite its protocol kind being USER.

Native errors are serialized back with the category, code, stringified message, and worker retained:

def convert_from_native_to_error(err: BaseException) -> Error:
    if isinstance(err, flyte.errors.RuntimeUnknownError):
        return Error(
            err=execution_pb2.ExecutionError(
                kind=execution_pb2.ExecutionError.UNKNOWN,
                code=err.code,
                message=str(err),
                worker=err.worker,
            )
        )
    elif isinstance(err, flyte.errors.RuntimeUserError):
        return Error(
            err=execution_pb2.ExecutionError(
                kind=execution_pb2.ExecutionError.USER,
                code=err.code,
                message=str(err),
                worker=err.worker,
            )
        )
    elif isinstance(err, flyte.errors.RuntimeSystemError):
        return Error(
            err=execution_pb2.ExecutionError(
                kind=execution_pb2.ExecutionError.SYSTEM,
                code=err.code,
                message=str(err),
                worker=err.worker,
            )
        )
    else:
        return Error(
            err=execution_pb2.ExecutionError(
                kind=execution_pb2.ExecutionError.UNKNOWN,
                code=type(err).__name__,
                message=str(err),
                worker="UNKNOWN",
            )
        )

The converter checks RuntimeUnknownError before the other categories so its classification is preserved. Anything that is not one of the three category classes falls through to an UNKNOWN protocol error, using the Python exception class name as code and the literal "UNKNOWN" as worker. A bare BaseRuntimeError therefore uses this fallback rather than its own .kind field.

How failures are consumed

The V2 runtime’s run_task function in _internal/runtime/taskrunner.py catches RuntimeSystemError, RuntimeUnknownError, and RuntimeUserError independently, logs each failure, and returns {}, together with the exception, as the task result’s error. An ordinary exception is logged and converted with CustomError.from_exception, which uses the exception class name as its error code and its string form as the message. The function finalizes the parent action in its finally block.

At the CLI boundary, cli/_common.py catches BaseRuntimeError and renders the structured fields in a stable format: click.ClickException(f"{e.kind} failure, {e.code}. {e}"). InitializationError is handled before this general base-class handler and receives a specific initialization message. Other runtime failures combine category, code, and human-readable text; the CLI does not need to infer the category by parsing the message.

Remote action failures add another fallback. _internal/controllers/remote/_controller.py first uses the action’s inline error or client error. If the server reports PHASE_FAILED without one, it attempts to load an error artifact. Failure to load that artifact becomes a RuntimeSystemError, and failure to convert the resulting error becomes RuntimeSystemError("UnableToConvertError", ...). The final category can therefore describe an inability to retrieve or interpret the original failure rather than the original task cause.

Debugging by fields, not message text

Inspect the structured attributes independently from the message:

try:
    run_result = task()
except BaseRuntimeError as exc:
    print(f"kind={exc.kind}")
    print(f"code={exc.code}")
    print(f"worker={exc.worker}")
    print(f"message={str(exc)}")

This reflects BaseRuntimeError’s actual storage model: str(exc) is the root-cause message, while kind, code, and worker are separate values. In particular, do not classify an error by searching its message for words such as “system” or “user.”

Classification caveats

• A native exception not deriving from RuntimeUnknownError, RuntimeUserError, or RuntimeSystemError is serialized as UNKNOWN, with its exception type as the code and worker="UNKNOWN".
• worker is metadata passed through protocol conversion; it does not determine kind.
• TaskTimeoutError(message) calls its parent with "TaskTimeoutError", the message, and "user" as the third positional argument. The parent interprets that third argument as worker, so this class stores kind="user" and worker="user".
• RuntimeDataValidationError uses the source spelling "DataValiationError" for its code. Existing consumers may depend on that exact value.
• The code classifier’s case-sensitive substring checks can change the selected native class when server error-code casing changes.
• Several runtime paths classify an inability to load or convert an error as a system failure. For remote failures, inspect the message and code in addition to kind when determining what happened originally.