Data Path Abstraction

Logical paths and physical storage

A flyte://... URI is a logical data location, not a filesystem path that RemoteFSPathResolver opens itself. In flyte-sdk, the resolver keeps an in-memory association between that logical string and a provider- or local-storage path. The marker is declared directly on RemoteFSPathResolver:

class RemoteFSPathResolver:
    protocol = "flyte://"

The associated physical path might be a path generated from the execution context's raw-data prefix. RawDataPath.get_random_remote_path() uses the configured prefix and either returns an absolute local path or joins the prefix with a random suffix using the selected fsspec filesystem's separator. The resolver does not call fsspec, upload data, download data, or inspect a filesystem; it only stores and retrieves strings.

Registering and resolving a path

The class provides two class methods. Register the exact logical URI before it is normalized into a literal:

from flyte.storage._remote_fs import RemoteFSPathResolver

RemoteFSPathResolver.add_mapping("flyte://data", "s3://bucket/run-output/result.parquet")
remote_path = RemoteFSPathResolver.resolve_remote_path("flyte://data")

After the registration above, remote_path is the registered physical path. A lookup for a key that is not present returns None:

missing_path = RemoteFSPathResolver.resolve_remote_path("flyte://data/missing")
assert missing_path is None

add_mapping() replaces an existing value for the same string key. Neither method validates the URI scheme or the physical path, and lookup is an exact string match: the resolver does not parse URIs, perform prefix matching, or normalize slashes or escaping.

The mapping is shared by every RemoteFSPathResolver caller in the process through _flyte_path_to_remote_map. Both operations acquire the class-level threading.Lock while accessing that dictionary:

_flyte_path_to_remote_map: typing.ClassVar[typing.Dict[str, str]] = {}
_lock = threading.Lock()

The lock makes individual registrations and lookups thread-safe. It does not make the mapping persistent or share it between processes, workers, task containers, or execution sessions. In the current flyte-sdk source, no production call site directly invokes add_mapping; the path-population mechanism is therefore absent from the in-repository runtime or supplied externally. The module also defines REMOTE_PLACEHOLDER = "flyte://data", but the resolver does not use that constant to create mappings.

RemoteFSPathResolver is an internal API. flyte.storage.__init__ exports storage configuration and operations such as put, get, and get_underlying_filesystem, but it does not re-export RemoteFSPathResolver or REMOTE_PLACEHOLDER.

Runtime literal normalization

The resolver becomes part of normal serialization through modify_literal_uris() in types/_type_engine.py. This helper imports RemoteFSPathResolver lazily and uses RemoteFSPathResolver.protocol to identify logical URIs. It mutates the Flyte protobuf literal in place:

def modify_literal_uris(lit: Literal):
    """
    Modifies the literal object recursively to replace the URIs with the native paths in case they are of
    type "flyte://"
    """
    from flyte.storage._remote_fs import RemoteFSPathResolver

    if lit.HasField("collection"):
        for literal in lit.collection.literals:
            modify_literal_uris(literal)
    elif lit.HasField("map"):
        for k, v in lit.map.literals.items():
            modify_literal_uris(v)
    elif lit.HasField("scalar"):
        if (
            lit.scalar.HasField("blob")
            and lit.scalar.blob.uri
            and lit.scalar.blob.uri.startswith(RemoteFSPathResolver.protocol)
        ):
            lit.scalar.blob.uri = RemoteFSPathResolver.resolve_remote_path(lit.scalar.blob.uri)
        elif lit.scalar.HasField("union"):
            modify_literal_uris(lit.scalar.union.value)
        elif (
            lit.scalar.HasField("structured_dataset")
            and lit.scalar.structured_dataset.uri
            and lit.scalar.structured_dataset.uri.startswith(RemoteFSPathResolver.protocol)
        ):
            lit.scalar.structured_dataset.uri = RemoteFSPathResolver.resolve_remote_path(
                lit.scalar.structured_dataset.uri
            )

The traversal covers nested collections and map values, descends into union values, and rewrites matching URIs in both blob scalars and structured-dataset scalars. A URI that does not start with flyte:// is not passed to the resolver. Conversely, a matching URI with no mapping receives the resolver's None result because that result is assigned directly to the protobuf URI field; callers should therefore ensure the mapping exists before this normalization runs.

TypeEngine serialization path

TypeEngine.to_literal() invokes the selected transformer first, then normalizes the literal before returning it:

@classmethod
async def to_literal(
    cls, python_val: typing.Any, python_type: Type[T], expected: types_pb2.LiteralType
) -> literals_pb2.Literal:
    transformer = cls.get_transformer(python_type)

    if transformer.type_assertions_enabled:
        transformer.assert_type(python_type, python_val)

    lv = await transformer.to_literal(python_val, python_type, expected)

    modify_literal_uris(lv)
    return lv

This places URI rewriting after transformer-specific literal construction. File, directory, dataframe, and other transformer-produced literals can consequently contain resolver candidates when their blob or structured-dataset URI uses the logical protocol.

Structured-dataset encoding has an additional explicit normalization point. DataFrameTransformer.encode() obtains an encoder, builds a structured-dataset literal, and then calls modify_literal_uris():

sd_model = await handler.encode(sd, structured_literal_type)

# Always set the format here to the format of the handler.
sd_model.metadata.structured_dataset_type.format = handler.supported_format
lit = literals_pb2.Literal(scalar=literals_pb2.Scalar(structured_dataset=sd_model))

# Because the handler.encode may have uploaded something, and because the sd may end up living inside a
# dataclass, we need to modify any uploaded flyte:// urls here.
modify_literal_uris(lit)  # todo: verify that this can be removed.

The comment identifies the current behavior as potentially transitional, but the rewrite is performed today, including for a structured dataset nested in a dataclass.

How physical paths are generated today

Logical URI resolution and ordinary upload-path generation are separate paths in the current source. For example, File.new_remote() creates a File whose path is obtained directly from the execution context:

return cls(path=ctx.raw_data.get_random_remote_path(), hash=known_cache_key, hash_method=method)

Likewise, storage.put() generates a destination when the caller does not provide one, then uploads through the filesystem selected for that destination:

async def put(from_path: str, to_path: Optional[str] = None, recursive: bool = False, **kwargs) -> str:
    if not to_path:
        from flyte._context import internal_ctx

        ctx = internal_ctx()
        name = pathlib.Path(from_path).name if not recursive else None  # don't pass a name for folders
        to_path = ctx.raw_data.get_random_remote_path(file_name=name)

    file_system = get_underlying_filesystem(path=to_path)
    from_path = strip_file_header(from_path)
    if isinstance(file_system, AsyncFileSystem):
        dst = await file_system._put(from_path, to_path, recursive=recursive, **kwargs)  # pylint: disable=W0212
    else:
        dst = file_system.put(from_path, to_path, recursive=recursive, **kwargs)

storage.put() returns the filesystem result when it is a string or pathlib.Path; otherwise it returns to_path. It does not call RemoteFSPathResolver.add_mapping(). Thus the runtime flow in this source snapshot is best understood as:

transformer or dataframe encoder
    -> Flyte Literal containing a URI
    -> modify_literal_uris()
       -> exact RemoteFSPathResolver lookup for flyte:// URIs
    -> Literal containing the resolved physical URI

storage.put() / File.new_remote()
    -> RawDataPath.get_random_remote_path()
    -> fsspec or local filesystem path

The second flow generates and uses physical paths directly; it does not automatically create the first flow's logical-to-physical map. Storage provider credentials, endpoints, retry settings, and the execution context's raw-data path affect physical storage access and path generation, not resolver registration. RemoteFSPathResolver has no environment-variable configuration of its own.

Operational boundaries

• Treat mappings as process-local state. They are not persisted, and a mapping created in one worker or execution session is not available in another.
• Use the identical URI string for registration and lookup. Trailing slashes, escaping, and other spelling differences produce a miss.
• Register mappings before TypeEngine.to_literal() or dataframe encoding reaches modify_literal_uris() when a flyte:// URI is present.
• Do not assume that flyte://data is automatically used for every logical path. It is only the value of the unused REMOTE_PLACEHOLDER module constant in this source snapshot.
• Use the private module import only with awareness that the public flyte.storage package does not expose the resolver and that the current repository does not populate its map from the normal upload APIs.