Runtime Services
The server lifespan owns a single RuntimeServices object
(src/ark_mcp/runtime.py) that is built once at startup and closed at
shutdown. Every tool retrieves it via get_runtime(ctx). This document
describes its operational services, including concurrency limiting, the daily
budget ledger, provider-task and object-key ownership, persistence caching,
per-task persistence single-flight, and the provider retry policy.
Persistent ownership, budget and cache state lives in a single SQLite database at
<artifact_dir>/runtime.sqlite3 (default artifact_dir is .artifacts),
shared by the ownership stores, budget ledger, and task-artifact cache. The
ownership table also stores MCP task owners and atomic execution claims. The
database is opened in WAL mode (PRAGMA journal_mode=WAL) with a
busy_timeout; each store’s
synchronous sqlite3.Connection is guarded by a per-instance asyncio.Lock
and every query is dispatched to a worker thread via asyncio.to_thread, so
SQLite access never blocks the event loop. These stores are still
single-process only — horizontal scaling requires a distributed
replacement.
RuntimeServices fields
Section titled “RuntimeServices fields”| Field | Type | Built from |
|---|---|---|
settings |
Settings |
input settings |
artifact_store |
FilesystemArtifactStore |
FilesystemArtifactStore(artifact_dir, ttl_seconds, downloader) |
safe_downloader |
SafeDownloader |
SafeDownloader(timeout, connect_timeout) |
ownership_store |
SQLiteTaskOwnershipStore |
SQLiteTaskOwnershipStore(database_path) |
object_key_ownership_store |
SQLiteObjectKeyOwnershipStore |
SQLiteObjectKeyOwnershipStore(database_path) |
budget_ledger |
BudgetLedger |
BudgetLedger(database_path, daily_limit_usd) |
provider_limiters |
ProviderLimiters |
ProviderLimiters(provider_limit, principal_limit) |
moderation_query_limiter |
ModerationQueryLimiter |
process-local rolling-minute guard, ten admitted lookups |
task_artifact_cache |
TaskArtifactCache |
SQLiteTaskArtifactCache(database_path, ttl_seconds, max_size) |
task_artifact_locks |
TaskArtifactPersistenceLocks |
process-local lock registry |
close_runtime_services closes five subsystems in order: artifact_store,
ownership_store, object_key_ownership_store, budget_ledger, and
task_artifact_cache. The SafeDownloader is closed indirectly —
FilesystemArtifactStore.close() calls self._downloader.close(). The
provider/principal limiters, moderation quota guard and per-task lock registry
have no close method.
Concurrency limiters (ProviderLimiters)
Section titled “Concurrency limiters (ProviderLimiters)”Two distinct layers, acquired together for every billable call:
- Provider-level — one global
asyncio.Semaphoreper provider bucket:"modelark","modelark-openapi","seed-speech","vod-mediakit","vision-cv","tos", and"s3". Each is independently sized toprovider_limit, so the default cap is five concurrent calls per provider rather than five shared across all providers. - Principal-level — one
asyncio.Semaphore(principal_limit)per principal, lazily created and stored in aTTLCachekeyed by"<tenant_id>\0<principal_id>"(NUL-separated). Cache get/create is guarded by anasyncio.Lock.
acquire(provider, owner) is an async context manager that holds both
the provider bucket semaphore and the principal’s semaphore for the duration
of the call.
For a local principal (stdio transport, or HTTP local mode), the
per-principal semaphore is skipped: a single trusted local caller is
bounded by the provider limit for concurrency; moderation also has a dedicated
ten-lookups-per-rolling-minute guard. Without this semaphore exemption, the default
PRINCIPAL_MAX_CONCURRENCY=3 would silently cap all concurrent calls across
every provider at 3 — below the per-provider limit of 5 — making parallel
local generation appear to “serialize under load.” The per-principal bound
applies to authenticated shared-key HTTP. Every member is principal and tenant
shared-team, so the default is three concurrent billable calls across the
whole team and all providers, not three per member.
| Env var | Default | Constraint | Used as |
|---|---|---|---|
PROVIDER_MAX_CONCURRENCY |
5 |
ge=1 |
per-bucket provider limit |
PRINCIPAL_MAX_CONCURRENCY |
3 |
ge=1 |
whole-team principal limit on shared-key HTTP; skipped in local mode |
The per-principal semaphore cache defaults to maxsize=10_000 and
ttl=86_400 (24h); idle principal semaphores expire after 24h. A TTLCache
eviction of a held semaphore dereferences it from the cache but does not
cancel its waiters.
For parallel variation tools, a third local
asyncio.Semaphore(max_concurrent=5)is created per batch byrun_variation_batch(see Parallel variations). It composes with — and is independent of — these provider/principal limiters.
Concurrency and timeouts
Section titled “Concurrency and timeouts”Parallel calls use the concurrency limiters above; moderation additionally uses its dedicated rolling-minute guard below. There is no server-wide serialization lock. The other bounds:
| Bound | Default | Code | Effect |
|---|---|---|---|
| Provider request | BYTEPLUS_REQUEST_TIMEOUT_MS=600000 (10 min), connect BYTEPLUS_CONNECT_TIMEOUT_MS=10000 |
config/env.py |
Per HTTP call to a provider; timeouts are classified in Error taxonomy |
| HTTP graceful shutdown | equals BYTEPLUS_REQUEST_TIMEOUT_MS |
__main__.py (timeout_graceful_shutdown) |
In-flight HTTP requests get up to one provider timeout to finish on shutdown |
| SQLite lock wait | busy_timeout=5000 ms |
runtime.py::_open_sqlite_connection |
A blocked writer waits up to 5 s in its worker thread, never on the event loop |
| HTTP request body | MCP_HTTP_MAX_BODY_BYTES=314572800 (300 MiB) |
config/env.py |
A value below the 200 MiB video limit × 4/3 (Base64) logs http_body_limit_below_media_upload_max at startup |
| Variation batch | variation_batch_deadline(...) |
tools/_parallel.py |
See Parallel variations |
- A client’s foreground deadline is not a server timeout. Generation,
understanding, transcription and upload tools use task mode
required, so they run in the background worker and a foreground call is rejected before the provider is called. See SPEC_BACKGROUND_JOBS.md and troubleshooting. - Large media should not travel inside JSON-RPC. Use
media_uploadwith a file path (stdio) or the presign and direct-upload path; the body limit only bounds inlined Base64.
Moderation query quota
Section titled “Moderation query quota”RuntimeServices.moderation_query_limiter admits ten lookups per rolling minute,
shared across callers and ID types in one runtime. ark_moderation_get acquires
the modelark-openapi concurrency slot before reserving quota. Exhaustion
returns LOCAL_RATE_LIMITED promptly with a retry delay and no provider call.
Dispatched failures and cancelled reservations consume quota; cancellation
while waiting for concurrency does not. All paths release concurrency and close
any per-call client. The in-memory guard resets on restart and does not coordinate
replicas or other account clients; provider throttling remains authoritative.
Lookup is foreground-only, account-wide trusted local access, without task ownership filtering, automatic retries or result persistence. It does not use the billable-generation budget flow; pricing is unspecified, not zero. See moderation.md for eligibility and live-unverified status.
Artifact persistence single-flight
Section titled “Artifact persistence single-flight”RuntimeServices.task_artifact_locks is a process-local
TaskArtifactPersistenceLocks registry keyed by (provider, task_id). A lock
entry exists only while a holder or waiter is active and is removed when its
user count reaches zero. Enhancement polling uses it to recheck the durable
task-artifact cache inside the critical section and share a just-created
artifact with concurrent waiters, while different task IDs remain concurrent.
Cache failures are best-effort: they emit safe warnings but do not erase a
successful provider result or an artifact that was already created.
This coordination is intentionally single-process, matching the current SQLite state backend. A future distributed state backend requires a distributed single-flight mechanism.
Daily budget + cost estimation
Section titled “Daily budget + cost estimation”Budget ledger (BudgetLedger)
Section titled “Budget ledger (BudgetLedger)”SQLite-backed, same file as the ownership store. Schema:
CREATE TABLE budget_reservations ( id INTEGER PRIMARY KEY AUTOINCREMENT, usage_date TEXT NOT NULL, -- UTC date (ISO) principal_id TEXT NOT NULL, tenant_id TEXT NOT NULL, product TEXT NOT NULL, amount_usd REAL NOT NULL, status TEXT NOT NULL CHECK(status IN ('reserved', 'committed', 'released')))Three-state lifecycle: reserved → committed (charged) or
reserved → released (rolled back). Only rows with status IN ('reserved', 'committed') count against the limit.
reserve(owner, estimate)— sums the principal’s spend for the current UTC date; if a configured limit exists andcurrent + amountwould exceed it, incrementsark_mcp_budget_rejections_total{product}and raisesBudgetExceededErrorbefore the provider is called. Otherwise inserts areservedrow and returns aBudgetReservation.commit(reservation)— marks the rowcommitted.release(reservation)— marks the rowreleased.
| Env var | Default | Behavior |
|---|---|---|
DAILY_BUDGET_USD |
0.0 |
Per-principal UTC daily limit. 0.0 (and any falsy value) normalizes to None → record-only mode (no rejection; the ledger still records everything). A positive value enables reject mode. |
The daily window is the UTC calendar date, scoped per (principal_id, tenant_id).
On shared-key HTTP, every member uses shared-team for both values, so this is
one team-wide cap. Rotating the MCP key does not reset the budget history.
Cost estimation (tools/_cost.py)
Section titled “Cost estimation (tools/_cost.py)”The estimator behind the daily budget and the cost log. Image and understanding estimates use fresh, recognized prices from the ModelArk pricing catalog when available; unmatched models, unsupported units, and other products use built-in rates.
Live prices. With AK/SK configured, the server loads the account’s prices
(discounts applied) from ListModelChargeItems in the background at startup and refreshes them
every BYTEPLUS_MODELARK_PRICING_REFRESH_SECONDS (default 86400, 24 hours). The snapshot
lives in an in-process price book (providers/modelark/pricing.py) keyed by
catalog name, which is the configured model ID without its trailing version
date. A snapshot older than twice the refresh interval is ignored. If a refresh
fails, the previous snapshot is kept until it expires and the failure is logged
as pricing_refresh_failed. The cost_estimate log line carries
price_source (live or builtin).
The refresher uses the server’s resolved settings for credentials, endpoint,
region, model bindings, timeouts, and snapshot freshness, including when the
server was constructed with explicit settings instead of environment defaults.
Built-in rates (used when there is no fresh price for the model):
| Constant | Value |
|---|---|
COST_PER_IMAGE |
0.03 USD |
COST_PER_AUDIO_SECOND |
0.0031 USD |
COST_PER_VIDEO_TASK |
0.07 USD (0.35 for Seedance 2.5) |
COST_PER_STT_SECOND |
0.0006 USD |
COST_UNDERSTANDING_INPUT_PER_MTOK / _OUTPUT_PER_MTOK |
0.50 / 3.00 USD |
COST_PER_3D_TASK_HYPER3D / _HITEM3D |
0.25 / 1.0 USD |
DEFAULT_MAX_CONCURRENT |
5 |
estimate_cost(product, variations, ..., model_id=None):
product |
Formula |
|---|---|
"image" |
Live: variations * ToICompletion price per piece. Built-in: variations * 0.03. Rounded to 4 decimals |
"understanding" |
Live: InferencePrompt and InferenceCompletion per-K-token prices scaled to millions of tokens. Built-in: the per-Mtok constants. Uses prompt_tokens and max_tokens |
"audio" |
round(variations * max(duration_seconds, 10) * 0.0031, 2) — floor of 10 seconds per variation |
"video" |
round(variations * 0.07, 2) (Seedance 2.5 uses 0.35); live video prices are token-based and not yet used |
"3d" |
Per-task built-in rate by family; live prices are not yet used |
"stt" |
Built-in per-second rate with a 10 second floor |
| (any other) | 0.0 |
Live prices are looked up by model ID. Built-in video estimates retain the selected model family’s rate, including Seedance 2.5.
Provider task ownership (SQLiteTaskOwnershipStore)
Section titled “Provider task ownership (SQLiteTaskOwnershipStore)”Ownership of asynchronous provider task IDs, so one principal cannot read, cancel, retrieve, or delete another’s tasks. Schema:
CREATE TABLE IF NOT EXISTS task_ownership ( provider TEXT NOT NULL, task_id TEXT NOT NULL, principal_id TEXT NOT NULL, tenant_id TEXT NOT NULL, created_at TEXT NOT NULL, PRIMARY KEY(provider, task_id))| Method | Behavior |
|---|---|
record(provider, task_id, owner) |
INSERT ... ON CONFLICT(provider, task_id) DO UPDATE — re-recording upserts to the new owner. |
require_owner(provider, task_id, owner) |
Row missing: if owner.is_local returns silently (local mode is permissive for unrecorded tasks), else PermissionError. Row present but mismatched: PermissionError("Task is not owned by the current principal."). |
list_task_ids(provider, owner) |
returns task IDs for the provider and (principal_id, tenant_id). |
ping() |
SELECT 1 liveness probe (used by the /ready route). |
close() |
closes the connection. |
Every lookup is scoped by provider and task_id; owner listings also require
both principal_id and tenant_id. The is_local shortcut only affects the
missing-row branch — a row that exists and belongs to someone else still raises,
even in local mode.
This is distinct from artifact ownership, which is stored in
.meta.jsonsidecars next to the artifact bytes (artifacts/filesystem_store.py). See artifacts.md.
Persistence cache
Section titled “Persistence cache”RuntimeServices.task_artifact_cache is a SQLiteTaskArtifactCache backed by
the same runtime.sqlite3 database as the ownership store and budget ledger.
Schema:
CREATE TABLE IF NOT EXISTS task_artifacts ( provider TEXT NOT NULL, task_id TEXT NOT NULL, artifacts_json TEXT NOT NULL, created_at TEXT NOT NULL, PRIMARY KEY(provider, task_id))It caches provider task lookups → artifact references (dict[str, ArtifactRef | None]).
The TTL defaults to 86,400 seconds (24h), matching the longest provider media-URL
expiry window (24h for image/video; audio URLs expire in 2h), so resolved artifact
references for a provider task ID are not re-resolved while still valid. Entries
older than PERSISTENCE_CACHE_TTL_SECONDS are lazily ignored on read. When the
row count exceeds PERSISTENCE_CACHE_MAX_SIZE, the oldest entries are evicted.
Unlike the previous in-memory TTLCache, the SQLite-backed cache survives
server restarts — a restart no longer loses cached task→artifact mappings.
| Env var | Default | Used as |
|---|---|---|
PERSISTENCE_CACHE_MAX_SIZE |
10_000 |
max rows before oldest eviction |
PERSISTENCE_CACHE_TTL_SECONDS |
86_400 |
TTL in seconds; entries older than this are ignored on read |
Retry policy (providers/retry.py)
Section titled “Retry policy (providers/retry.py)”RetryPolicy is a frozen dataclass with hard-coded defaults (no env vars):
| Field | Default |
|---|---|
max_attempts |
3 |
base_delay_seconds |
0.25 |
max_delay_seconds |
4.0 |
jitter_ratio |
0.2 |
retry_timeouts |
True |
call_with_retry(operation, *, policy=None, sleep=asyncio.sleep, random_value=random.random):
- Only
ProviderErroris retried, and only whenexc.retryableandnot exc.ambiguous_completionandattempt < max_attemptsand (policy.retry_timeoutsorexc.code != "TIMEOUT").seed_understandandseed_audio_understanduseretry_timeouts=False: a chat timeout is safe to retry but is not retried automatically, because a second full-length thinking run doubles latency and token cost; its 429/5xx responses are still retried.ambiguous_completion=Trueis deliberately non-retryable: the mutation may have already succeeded at the provider, so a blind retry could double-charge or duplicate. Reconcile via the task ID / request ID instead. - Any non-
ProviderErrorpropagates immediately (no retry, no wrapping). - Backoff:
- If
exc.error.retry_after_seconds is not None(providerRetry-After):delay = min(retry_after_seconds, max_delay_seconds)— no jitter. - Otherwise exponential with symmetric jitter:
base = min(base_delay * 2**(attempt-1), max_delay);jitter = base * jitter_ratio * (random*2 - 1);delay = max(0.0, base + jitter).
- If
- Each retry increments
ark_mcp_retry_attempts_total{provider, operation}.
Billable call composition (billed_provider_slot)
Section titled “Billable call composition (billed_provider_slot)”A single async context manager ties the above together:
async with billed_provider_slot(ctx, provider=..., product=..., estimated_cost_usd=...): # 1. reserve budget (raises BudgetExceededError before dispatch if over) # 2. acquire provider + principal semaphores # 3. (caller runs the provider call, typically via call_with_retry)Outcome handling:
| Outcome | Budget action |
|---|---|
| Success | commit (charged) |
ProviderError with ambiguous_completion |
commit (treats as billable — may have succeeded) |
ProviderError with code="TIMEOUT" (any, including retryable read timeouts) |
commit (a timed-out completion may still have consumed billable tokens) |
Other ProviderError |
release |
| Cancellation after the provider slot was acquired (call dispatched) | commit (outcome unknown) |
| Cancellation while still waiting for a limiter slot | release |
| Any other exception | release |
In all error cases the exception is re-raised. The commit/release on cancellation is shielded, so a cancelled call never leaves a dangling reservation.
Error taxonomy
Section titled “Error taxonomy”| Class | Module | Raised by | Meaning |
|---|---|---|---|
ProviderError |
domain/errors.py |
provider gateways + transport normalizers | wraps a NormalizedProviderError; the single retryable error type |
BudgetExceededError |
runtime.py (extends ValueError) |
BudgetLedger.reserve |
daily budget would be exceeded; raised before dispatch |
ValueError |
builtin | ProviderLimiters, RetryPolicy, resolve_prompts |
config/argument validation |
RuntimeError |
builtin | get_runtime, BudgetLedger.reserve |
missing runtime services / reservation ID |
PermissionError |
builtin | get_principal, require_owner |
missing token, missing principal/tenant claims, ownership mismatch |
NormalizedProviderError carries: provider (modelark or seed-speech),
operation, http_status, code, message, request_id, retryable,
ambiguous_completion, retry_after_seconds. Transport-layer normalizers in
BaseHttpGateway:
| Method | code |
retryable |
ambiguous_completion |
|---|---|---|---|
normalize_timeout(op) (mutation, default side_effect=True) |
TIMEOUT |
False |
True |
normalize_timeout(op, side_effect=False) (read-only) |
TIMEOUT |
True |
False |
normalize_connection_error |
CONNECTION_ERROR |
True |
None |
normalize_transport_error |
TRANSPORT_ERROR |
True |
None |
Timeout classification depends on whether the call can create provider state:
| Call | Examples | Timeout result | Auto-retried? |
|---|---|---|---|
| Mutation | Seedance/Seed 3D create, cancel, delete; Seedream/Seed Audio generate; MediaKit submit | retryable=False, ambiguous_completion=True |
No — reconcile by task/request ID |
| Read-only poll | Seedance and Seed 3D get_task/list_tasks, ASR query_asr, MediaKit task polls |
retryable=True, ambiguous_completion=False |
Yes |
| Chat completion | seed_understand, seed_audio_understand |
retryable=True, ambiguous_completion=False |
No (retry_timeouts=False) |
| Moderation lookup | ark_moderation_get |
retryable=True, ambiguous_completion=False |
No; one request per admitted lookup |
MediaKit polls apply the same rule to their own transport normalizer: poll timeouts and connection failures are retryable and non-ambiguous, while submit failures stay ambiguous.
Parallel variations (tools/_parallel.py)
Section titled “Parallel variations (tools/_parallel.py)”Re-exports DEFAULT_MAX_CONCURRENT = 5.
generate_seeds(...)— derives Seedream variation seeds from the request’sseed(the base seed):None→ provider-randomized (seed not recorded);-1→ per-variationsecrets.randbelow(2**31)(recorded);N→ deterministic(N + i) % 2**31.resolve_prompts(base_prompt, variation_prompts, count)— usesvariation_promptsif given; else[base_prompt] * count; raises if neither is provided.gather_with_timeout(coros, timeout)— per-coroutineasyncio.wait_for+asyncio.gather(..., return_exceptions=True); returns exceptions andasyncio.TimeoutErroras elements, never raises. Still exported but no longer used by the variation tools.variation_batch_deadline(count, settings, *, max_concurrent=5, provider_attempts=3, persists=True)—ceil(count / max_concurrent) × (request_timeout × provider_attempts + download_timeout × download_attempts); the download term is dropped whenpersists=False(Seedance variations).run_variation_batch(count, factory, *, batch_deadline, max_concurrent=5)— localasyncio.Semaphore(max_concurrent)aroundfactory(idx, progress). There is no per-variation timeout; slot waiting never counts. Atbatch_deadline, unfinished variations are cancelled and classified byVariationProgress.phase:queued→QUEUE_TIMEOUT(retryable),generating/persisting→TIMEOUT(ambiguous), or the factory’s recordedpartialresult (an unpersisted provider-URL artifact) when persisting. Unexpected exceptions map toGATHER_ERROR. A variation counts as succeeded iff it produced anartifactor atask_id. ReturnsVariationSummary(total, succeeded, failed, variations). One variation failing never breaks the others; the batch always returns exactlycountentries.
See api-reference.md for VariationResult /
VariationSummary / VariationError field tables.
MCP task execution claims
Section titled “MCP task execution claims”TaskOwnershipStore.claim(provider, task_id, owner) -> bool uses an atomic
INSERT ... ON CONFLICT DO NOTHING, returning true only for the first claim.
The mcp namespace binds protocol tasks to their application owners; the
mcp-execution namespace records the first entry into a required-task handler.
Neither namespace changes provider task IDs or provider ownership checks.
Execution claims are not released on failure or cancellation: provider work may already have been accepted. A restarted worker encountering an existing claim fails before entering the handler and asks for reconciliation. Foreground status checks and optional background persistence remain repeatable. Claims are stored in the same SQLite file, share its locking/transaction behavior, and follow the existing ownership retention policy. The queue, database and configured snapshot encryption key must stay together across restarts.