The Pickling Fallback Mechanism

When a task input or output has a Python type for which Flyte cannot find a registered transformer, flyte-sdk does not immediately fail type lookup. TypeEngine.get_transformer first checks the direct registry entry, then walks the type's MRO, and then gives dataclasses a chance through the dataclass transformer. Only after those checks does it log the pickle warning, import FlytePickleTransformer lazily, and return a transformer for the unsupported type:

@classmethod
def get_transformer(cls, python_type: Type) -> TypeTransformer:
    v = cls._get_transformer(python_type)
    if v is not None:
        return v

    if hasattr(python_type, "__mro__"):
        class_tree = inspect.getmro(python_type)
        for t in class_tree:
            v = cls._get_transformer(t)
            if v is not None:
                return v

    if dataclasses.is_dataclass(python_type):
        return cls._DATACLASS_TRANSFORMER

    display_pickle_warning(str(python_type))
    from flyte.types._pickle import FlytePickleTransformer

    return FlytePickleTransformer()

This ordering matters. A custom transformer or an MRO match takes precedence, and an unregistered dataclass is handled by the dataclass transformer rather than being sent to pickle. The fallback is therefore a last-resort conversion path, not a replacement for registering a structured representation.

FlytePickle is the internal representation used by that path. Its class docstring says that users should not use the type directly and that types Flyte cannot recognize become FlytePickle. The public flyte.types package re-exports FlytePickle, but the transformer itself remains in the private flyte.types._pickle module. Importing that module constructs and registers one transformer instance with TypeEngine:

class FlytePickleTransformer(TypeTransformer[FlytePickle]):
    PYTHON_PICKLE_FORMAT = "PythonPickle"

    def __init__(self):
        super().__init__(name="FlytePickle", t=FlytePickle)


TypeEngine.register(FlytePickleTransformer())

From an unsupported Python value to a Flyte literal

The normal conversion path is still used. TypeEngine.to_literal obtains the transformer, calls assert_type when transformer type assertions are enabled, delegates to transformer.to_literal, and then normalizes literal URIs. For the fallback, FlytePickleTransformer.assert_type deliberately does nothing: its source comment states that every type can serialize to pickle, so it does not check the value's type.

FlytePickleTransformer.to_literal rejects None, serializes every other value through FlytePickle.to_pickle, and wraps the resulting URI in a scalar blob. The blob is explicitly single-dimensional and has the format PythonPickle:

async def to_literal(
    self,
    python_val: T,
    python_type: Type[T],
    expected: types_pb2.LiteralType,
) -> literals_pb2.Literal:
    if python_val is None:
        raise AssertionError("Cannot pickle None Value.")
    meta = literals_pb2.BlobMetadata(
        type=types_pb2.BlobType(
            format=self.PYTHON_PICKLE_FORMAT,
            dimensionality=types_pb2.BlobType.BlobDimensionality.SINGLE,
        )
    )
    remote_path = await FlytePickle.to_pickle(python_val)
    return literals_pb2.Literal(
        scalar=literals_pb2.Scalar(
            blob=literals_pb2.Blob(metadata=meta, uri=remote_path)
        )
    )

The serialization work in FlytePickle.to_pickle is asynchronous. It calls cloudpickle.dumps, hashes the serialized bytes with MD5, uses the digest as the filename in a random local path, creates the parent directory, writes the bytes with aiofiles, and uploads the local file with storage.put:

@classmethod
async def to_pickle(cls, python_val: typing.Any) -> str:
    h = hashlib.md5()
    str_bytes = cloudpickle.dumps(python_val)
    h.update(str_bytes)

    uri = storage.get_random_local_path(file_path_or_file_name=h.hexdigest())
    os.makedirs(os.path.dirname(uri), exist_ok=True)
    async with aiofiles.open(uri, "w+b") as outfile:
        await outfile.write(str_bytes)

    return await storage.put(str(uri))

The digest is derived from the serialized bytes and controls the local filename. The method does not check whether a file already exists or remove the temporary file, and it does not supply a destination to storage.put; the active raw-data/storage context selects the final remote location. Consequently, pickle fallback depends on the storage subsystem for normal remote uploads and for provider-specific access such as S3, GCS, or Azure.

The same literal type can be produced before serialization by FlytePickleTransformer.get_literal_type. It creates a single PythonPickle blob type and records the source type as metadata:

def get_literal_type(self, t: Type[T]) -> types_pb2.LiteralType:
    lt = types_pb2.LiteralType(
        blob=types_pb2.BlobType(
            format=self.PYTHON_PICKLE_FORMAT,
            dimensionality=types_pb2.BlobType.BlobDimensionality.SINGLE,
        )
    )
    lt.metadata = {"python_class_name": str(t)}
    return lt

The metadata records str(t); it does not make the serialized value structured or enforce that type during loading.

From the literal back to Python

On the reverse path, FlytePickleTransformer.to_python_value reads lv.scalar.blob.uri and delegates to FlytePickle.from_pickle. If the URI is remote, from_pickle obtains a random local path and downloads it with storage.get. For a local URI it reads the supplied path directly. In both cases it reads the bytes asynchronously and passes them to cloudpickle.loads:

@classmethod
async def from_pickle(cls, uri: str) -> typing.Any:
    if storage.is_remote(uri):
        local_path = storage.get_random_local_path()
        await storage.get(uri, str(local_path), False)
        uri = str(local_path)
    async with aiofiles.open(uri, "rb") as infile:
        data = cloudpickle.loads(await infile.read())
    return data

The expected_python_type argument accepted by to_python_value is not passed to, or checked by, from_pickle. Deserialization therefore trusts the payload and returns whatever cloudpickle.loads produces. A Python annotation does not provide runtime type validation for this fallback, and type mistakes can remain undetected until deserialization—or manifest as a loading failure.

guess_python_type is stricter about recognizing the literal representation. It returns FlytePickle only for a blob whose dimensionality is SINGLE and whose format is exactly PythonPickle; otherwise it raises ValueError. Ordinary file blobs, directory blobs, MessagePack binaries, and blobs using another format are not reverse-mapped by this transformer.

Dictionary fallback is a separate path

Dictionaries do not automatically use the ordinary FlytePickleTransformer fallback. DictTransformer.dict_to_binary_literal first tries MessagePackEncoder.encode. Only when that raises TypeError, and only when the dictionary's Annotated metadata contains an OrderedDict with allow_pickle=True, does it call FlytePickle.to_pickle:

@staticmethod
async def dict_to_binary_literal(v: dict, python_type: Type[dict], allow_pickle: bool) -> Literal:
    from flyte.types._pickle import FlytePickle

    try:
        encoder = MessagePackEncoder(python_type)
        msgpack_bytes = encoder.encode(v)
        return Literal(scalar=Scalar(binary=Binary(value=msgpack_bytes, tag=MESSAGEPACK)))
    except TypeError as e:
        if allow_pickle:
            remote_path = await FlytePickle.to_pickle(v)
            return Literal(
                scalar=Scalar(
                    generic=_json_format.Parse(
                        json.dumps({"pickle_file": remote_path}),
                        struct_pb2.Struct(),
                    )
                ),
                metadata={"format": "pickle"},
            )
        raise TypeTransformerFailedError(f"Cannot convert `{v}` to Flyte Literal.\nError Message: {e}")

This representation is intentionally different from an ordinary FlytePickleTransformer literal. It stores a generic value containing pickle_file and literal metadata format: pickle, rather than a single-dimensional blob with PythonPickle format. Without the opt-in metadata, a MessagePack TypeError becomes TypeTransformerFailedError instead of silently switching to pickle.

Where the fallback appears at runtime

Task default inputs use TypeEngine.to_literal while convert_upload_default_inputs constructs the parameters uploaded to the Flyte backend. For each default value, it derives the literal type and schedules the conversion:

for input_name, (input_type, default_value) in interface.inputs.items():
    if default_value and default_value is not inspect.Parameter.empty:
        lt = TypeEngine.to_literal_type(input_type)
        literal_coros.append(TypeEngine.to_literal(default_value, input_type, lt))
        vars.append((input_name, lt))

Native task outputs follow the same route. _internal.runtime.convert.convert_from_native_to_outputs obtains a literal type from the output annotation, converts the returned value with TypeEngine.to_literal, and attaches the result to its output name. An unsupported output type therefore reaches the same pickle fallback as an unsupported input type.

The command-line parameter conversion also recognizes the literal marker. In cli._params.literal_type_to_click_type, a single-dimensional blob with format FlytePickleTransformer.PYTHON_PICKLE_FORMAT becomes PickleParamType; other single blobs become file parameters, while multidimensional blobs become directory parameters:

if lt.HasField("blob"):
    if lt.blob.dimensionality == BlobType.BlobDimensionality.SINGLE:
        if lt.blob.format == FlytePickleTransformer.PYTHON_PICKLE_FORMAT:
            return PickleParamType()
        return FileParamType()
    return DirParamType()

Performance and compatibility tradeoffs

Pickling is convenient because FlytePickleTransformer.assert_type accepts arbitrary values and cloudpickle can serialize objects that have no structured Flyte transformer. The cost is that the result is an opaque Python artifact. The flyte.types package documentation states that pickle is not human-readable, cannot be represented in the Flyte UI, and may be inefficient for large datasets. A pickled object is therefore a poor substitute for a structured Flyte-supported type when users need inspectable values, efficient transfer, or a representation usable outside Python tasks.

The format is also not language-neutral. The built-in pickle warning says that pickled objects can only be sent between the exact same Python version. In operational terms, the consumer must also be able to resolve compatible class definitions and imported modules, and must use compatible serialization-library behavior; from_pickle simply invokes cloudpickle.loads and does not migrate or validate the payload. Changes to those runtime dependencies can make an old artifact fail to load or produce a value different from the consumer's annotation.

Finally, serialization requires a functioning storage context. to_pickle uploads through storage.put without an explicit destination, while from_pickle downloads remote URIs through storage.get; storage credentials, endpoints, retries, and raw-data configuration consequently affect this fallback. None is not accepted by FlytePickleTransformer.to_literal, and the transformer does not enforce the expected type while loading. For durable, inspectable, cross-language, or large-data workflows, the implementation's behavior supports choosing a registered Flyte type or writing a custom TypeTransformer instead of relying on the pickle fallback.