Skip to main content

Managing Projects and Secrets

When managing multi-tenant deployments and securing task execution environments in flyte-sdk, administrators need to query project metadata and provision sensitive credentials. The flyte.remote.Project and flyte.remote.Secret classes provide programmatic and CLI interfaces to inspect projects and perform CRUD operations on secrets within the control plane.

Managing Projects Programmatically​

The flyte.remote.Project class provides methods to retrieve project details or iterate through all projects registered on the Flyte control plane.

import flyte
from flyte.remote import Project

# Initialize the Flyte SDK client with target endpoint and credentials
flyte.init(
    endpoint="localhost:8080",
    insecure=True,
    org="my-org",
    project="flytesnacks",
    domain="development",
)

# 1. Fetch details of a specific project by name/id
project = Project.get("flytesnacks")
print(f"Project Name: {project.pb2.name}")
print(f"Project ID: {project.pb2.id}")
print(f"Description: {project.pb2.description}")
print(f"State: {project.pb2.state}")

# Convert the project protobuf payload to dictionary or JSON
project_dict = project.to_dict()
project_json = project.to_json()

# 2. Iterate through all projects with optional sorting and filtering
for p in Project.listall(sort_by=("created_at", "desc")):
    print(p.pb2.id, p.pb2.name)

Key Components of Project Management​

  • Project.get(name: str, org: str | None = None): Calls GetProject on the admin project_domain_service to fetch the project definition protobuf (project_pb2.Project).
  • Project.listall(filters: str | None = None, sort_by: Tuple[str, Literal["asc", "desc"]] | None = None): Streams projects using automatic token-based pagination with a batch size of 100. The default sort order is ("created_at", "asc").
  • ToJSONMixin: Inherited by Project to provide .to_dict() and .to_json() methods that serialize the underlying protobuf message (self.pb2).

Asynchronous Project Retrieval​

All remote methods are decorated with @syncify. In asynchronous contexts, use the .aio attribute:

import asyncio
import flyte
from flyte.remote import Project

async def fetch_projects():
    flyte.init(endpoint="localhost:8080", insecure=True)
    
    # Asynchronous single fetch
    proj = await Project.get.aio("flytesnacks")
    
    # Asynchronous iteration
    async for p in Project.listall.aio(sort_by=("id", "asc")):
        print(p.pb2.id)

asyncio.run(fetch_projects())

Managing Secrets Programmatically​

The flyte.remote.Secret class manages secrets stored in the control plane secrets service (payload_pb2 and definition_pb2). Operations are automatically scoped to the active org, project, and domain set during client initialization.

import flyte
from flyte.remote import Secret

flyte.init(
    endpoint="localhost:8080",
    insecure=True,
    org="my-org",
    project="flytesnacks",
    domain="development",
)

# 1. Create a regular string secret
Secret.create(name="database_password", value="super-secret-password", type="regular")

# 2. Create a binary secret or image pull secret
docker_config_bytes = b'{"auths": {"https://index.docker.io/v1/": {"auth": "dXNlcjpwYXNz"}}}'
Secret.create(name="docker-pull-secret", value=docker_config_bytes, type="image_pull")

# 3. Retrieve a secret by name
sec = Secret.get("database_password")
print(f"Secret Name: {sec.name}")
print(f"Secret Type: {sec.type}")
print(f"Serialized Secret: {sec.to_json()}")

# 4. List all secrets in the current project and domain
for s in Secret.listall(limit=50):
    print(s.name, s.type)

# 5. Delete a secret by name
Secret.delete("database_password")

Secret Types and Data Formats​

The type parameter accepts values of type SecretTypes (Literal["regular", "image_pull"]):

  • "regular": Maps to definition_pb2.SecretType.SECRET_TYPE_GENERIC. Used for application credentials, API tokens, and database passwords.
  • "image_pull": Maps to definition_pb2.SecretType.SECRET_TYPE_IMAGE_PULL_SECRET. Used by cluster container runtimes to authenticate against private image registries.

The value parameter accepts:

  • str: Populates SecretSpec.string_value.
  • bytes: Populates SecretSpec.binary_value.

Control Plane Secrets vs. Task Secret Declarations​

In flyte-sdk, distinguish between administrative secret management and task-level injection:

  • flyte.remote.Secret (src/flyte/remote/_secret.py): Performs CRUD operations against the control plane backend.
  • flyte.Secret (src/flyte/_secret.py): Used inside workflow code within @task(secrets=Secret(key="database_password", as_env="DB_PASSWORD")) or container images to bind remote secrets into task runtime containers.

Command Line Administration​

The Flyte CLI provides administrative commands corresponding to Project and Secret SDK operations.

Listing and Inspecting Projects​

# List all projects in the deployment
flyte get project

# Get details of a specific project
flyte get project flytesnacks

Creating, Viewing, and Deleting Secrets via CLI​

# Create a regular secret by passing the value flag
flyte create secret db-token --value "s3cr3t" --project flytesnacks --domain development

# Create a secret by uploading a file (reads as binary)
flyte create secret regcred --from-file ~/.docker/config.json --type image_pull

# List all secrets in the configured project and domain
flyte get secret --project flytesnacks --domain development

# Inspect details of a single secret in JSON format
flyte get secret db-token --project flytesnacks --domain development

# Delete a secret
flyte delete secret db-token --project flytesnacks --domain development

Troubleshooting​

ClientNotInitializedError​

Calling any method on Project or Secret before calling flyte.init(...) or prior to CLI configuration loading raises ClientNotInitializedError.

flyte.errors.InitializationError: ClientNotInitializedError: Client is not initialized.

Resolution: Ensure flyte.init(endpoint=..., ...) is executed before invoking remote operations.

Secret Scoping Mismatches​

Secret.get(), Secret.listall(), and Secret.delete() resolve the target secret's scope (organization, project, domain) from flyte._initialize.get_common_config(). If a secret was created under domain="production" but the client was initialized with domain="development", Secret.get() will fail to find the secret.

Resolution: Re-initialize the client with the required scope or pass --project and --domain flags in the CLI:

flyte.init(project="my-project", domain="production")

Calling @syncify Methods in Async Event Loops​

Invoking synchronous methods like Project.get(...) inside an active asyncio event loop directly can block the loop or cause synchronization deadlocks.

Resolution: Call .aio(...) on the method when running inside async functions:

await Secret.create.aio(name="api-key", value="token")