Representing Files and Directories

The flyte-sdk provides File and Dir types to represent individual files and collections of files. These types allow you to declare file-based inputs and outputs in your tasks while abstracting away the underlying storage (local or remote).

Unlike standard Python file handles, flyte-sdk File and Dir objects are serializable references. They track the location (path) and format of the data, and provide methods to interact with the data when needed.

Representing a Single File

You use the File class to represent a single file. It is a generic type, File[T], where T typically represents the format or the library used to process the file (e.g., File[pandas.DataFrame]).

Declaring File Inputs and Outputs

In flyte-sdk, you declare files in your task signatures. The framework handles the serialization of the path and metadata.

import pandas as pd
from flyte.io import File
from flyte import env

@env.task
async def process_csv(input_file: File) -> File:
    # The 'file' object contains the path to the data
    async with input_file.open("rb") as f:
        df = pd.read_csv(f)
    
    # Perform some processing
    df["processed"] = True
    
    # Create a new remote reference for the output
    output_file = File.new_remote()
    async with output_file.open("wb") as f:
        df.to_csv(f)
        
    return output_file

Opening and Reading Files

The File class provides both asynchronous and synchronous methods for I/O:

• open(): An asynccontextmanager that returns an async file-like object.
• open_sync(): A standard contextmanager for synchronous access.

# Async usage
async with csv_file.open("rb") as f:
    content = await f.read()

# Sync alternative
with csv_file.open_sync("rb") as f:
    content = f.read()

Creating File References

You can create File objects from various sources:

1. File.from_local(path, remote_destination=None): Uploads a local file to the configured remote storage and returns a File reference.
2. File.from_existing_remote(remote_path): Creates a reference to a file that already exists in remote storage (e.g., s3://my-bucket/data.csv).
3. File.new_remote(): Generates a random path in the configured remote storage, useful for streaming outputs directly to the cloud.

Representing Directories

The Dir class represents a directory containing multiple files. Like File, it is generic (Dir[T]) and supports both local and remote paths.

Walking a Directory

You can iterate through the files in a Dir using walk() (async) or walk_sync() (sync). These methods yield File objects for every file found.

from flyte.io import Dir

@env.task
async def process_directory(data_dir: Dir):
    # Asynchronously walk through all files in the directory
    async for file in data_dir.walk(recursive=True):
        print(f"Processing file: {file.path}")
        async with file.open("rb") as f:
            # Process individual file content
            ...

Listing Files

If you only need the files in the top-level directory, use list_files():

files = await data_dir.list_files()
for file in files:
    if file.name.endswith(".json"):
        # Do something
        ...

Downloading Directories

To work with a directory locally (e.g., if a tool requires a local folder path), use the download() method:

# Downloads the entire remote directory to a local temporary path
local_path = await data_dir.download()
# local_path is a string pointing to the local directory

Local vs. Remote Storage

The flyte-sdk uses fsspec under the hood to handle different storage protocols. File and Dir objects automatically detect whether a path is local or remote based on the URI scheme (e.g., s3://, gs://, or a plain local path).

• Local Paths: Handled using standard filesystem operations or aiofiles for async.
• Remote Paths: Handled via the flyte.storage layer, which manages credentials and efficient streaming.

When a task receives a File or Dir as an input, the path attribute will point to the location where the data resides. If the data is remote, you must use open() or download() to access the contents; the framework does not automatically download files to the local disk unless you explicitly call these methods.