seed_audio_understand
Understand or reason about audio (transcribe, translate, analyze, answer questions).
Returns only the final answer; the server fixes the model (see ark_list_models). For structured output use response_format ‘json_schema’ (‘json_object’ is not enforced by the default model). For plain transcription with word timings use speech_to_text. Oversized URL audio and rejected video inputs can use Files API (delivery); video frames also consume tokens. Requires background execution (ark_job_submit).
| Requirement | Value |
|---|---|
| Registration | Requires BYTEPLUS_MODELARK_API_KEY. |
| Execution | Background required |
| Required permission | understanding:read |
This site lists the full configured capability surface, not the tools enabled on your particular server. Use model discovery and job discovery to check your connection. Credentials are server configuration, never tool arguments.
Supported Models
Section titled “Supported Models”These are supported families and built-in model or service IDs for this site revision, not provider account access or a guarantee that every row is enabled on your server. Check model discovery for configured bindings and service availability.
| Model | Model or service ID | Availability | Capabilities |
|---|---|---|---|
| Seed 2.0 Lite (audio understanding) | seed-2-0-lite-260428 |
Server-configured default; BYTEPLUS_MODELARK_API_KEY required. |
1-8 audio clips; transcription, translation, analysis and Q&A; JSON Schema for structured output; oversized URL inputs can use the Files API (up to 512 MB). |
This tool has no per-call model argument. SEED_AUDIO_UNDERSTANDING_MODEL selects a ModelArk model supporting Chat Completions input_audio with deep thinking; other compatible, authorized IDs may be configured.
Ask your connected AI client to use seed_audio_understand for the operation you want.
The client constructs the arguments and handles tool calls. Review its proposed
operation before approving anything billable. For a non-billable introduction,
start with discovery without generation.
Submit through ark_job_submit with tool_name and the complete target arguments, including its inner input object. Poll ark_job_get using the returned Ark job_id, respecting poll_after_ms. Inspect both terminal status and result.is_error; the original output is in result.structured_content. Do not call this tool in the foreground.
Exact MCP call (for developers)
The following is an MCP tool-call envelope, not a shell command. Example media URLs and identifiers must be replaced with your own accessible resources. Seedream and Seedance examples are non-billable dry runs; other provider operations may incur charges.
{ "name": "ark_job_submit", "arguments": { "input": { "tool_name": "seed_audio_understand", "arguments": { "input": { "prompt": "Summarize this recording.", "audios": [ { "kind": "url", "url": "https://example.com/recording.wav" } ] } } } }}Inputs
Section titled “Inputs”The MCP arguments object wraps these fields as {"input": {...}}.
| Field | Type | Required | Description and constraints |
|---|---|---|---|
audios |
array of UnderstandingAudioInput | Yes | Audio clips (1-8), sent in order. kind=‘url’ (HTTPS) or kind=‘base64’ (raw data without data: prefix, max 10 MB decoded) with mime_type audio/wav, audio/mpeg, audio/flac, audio/aac, or audio/mp4. Upload local files with media_upload first.; minItems: 1; maxItems: 8 |
delivery |
“auto”, “direct”, “files_api” | No | URL delivery: auto retries rejected audio through Files API (up to 512 MB); direct returns the rejection; files_api uploads first. Videos use their soundtrack and sampled frames. Base64 stays inline. Uploads are deleted after use.; Default: “auto” |
json_retry |
integer | No | Extra billed attempts when a JSON answer fails to parse or validate. Not retried after finish_reason=‘length’; raise max_tokens instead.; minimum: 0; maximum: 2; Default: 0 |
max_tokens |
integer or null | No | Maximum output tokens including thinking; raise it if finish_reason=‘length’.; Default: null |
overwrite |
boolean | No | Allow save_to to replace an existing file with different content.; Default: false |
prompt |
string | Yes | The question or task for the model.; minLength: 1; maxLength: 32000 |
reasoning_effort |
“low”, “medium”, “high” | No | Deep-thinking depth; thinking is always on. Higher costs more latency and tokens.; Default: “medium” |
repetition_penalty |
number or null | No | Repetition penalty.; Default: null |
response_format |
UnderstandingResponseFormat or null | No | Force JSON output; the parsed result is returned in choices[].parsed.; Default: null |
return_content |
“full”, “summary”, “none” | No | Answer text to inline: ‘full’, ‘summary’ (first 2,000 characters), or ‘none’ (use with save_to). parsed is always returned when available.; Default: “full” |
save_to |
string or null | No | Absolute file path (stdio only) to write the final answer to (JSON when parsed, else UTF-8 text). Must be inside an allowed output root; checked before the model runs.; Default: null |
system |
string or null | No | System instruction for the model.; Default: null |
temperature |
number or null | No | Sampling temperature.; Default: null |
top_p |
number or null | No | Nucleus sampling probability.; Default: null |
Outputs
Section titled “Outputs”| Field | Type | Required | Description and constraints |
|---|---|---|---|
attempts |
integer | No | Completions that produced an answer: 1 plus json_retry attempts used.; Default: 1 |
audio_delivery |
array of “url”, “base64”, “files_api_audio”, “files_api_video” | No | Delivery route per input: url, base64, files_api_audio, or files_api_video. |
choices |
array of UnderstandingChoice | Yes | Model completion choices (one for non-streaming). |
completion_id |
string or null | No | Provider completion ID (e.g. ‘chatcmpl-…’) for tracing.; Default: null |
delivery_fallback_reason |
string or null | No | Provider rejection that triggered automatic Files API fallback, if any.; Default: null |
model |
string | Yes | Model ID used for the completion. |
provider |
“byteplus-modelark” | No | Provider that generated the response.; Default: “byteplus-modelark” |
request_id |
string or null | No | Provider request ID for support tracing.; Default: null |
saved_bytes |
integer or null | No | Size in bytes of the file written to saved_path.; Default: null |
saved_path |
string or null | No | Absolute path the answer was written to when save_to was set.; Default: null |
usage |
UnderstandingUsage | Yes | Token usage summed over all attempts (including json_retry attempts). |
Nested Types
Section titled “Nested Types”Field types named above refer to these shared definitions. Optional/null fields, enum choices and bounds are retained in the complete schemas below. Schema validation is followed by runtime capability and security checks.
MediaSourceKind
No fields. Pass {}.
SeedAudioUnderstandInput
| Field | Type | Required | Description and constraints |
|---|---|---|---|
audios |
array of UnderstandingAudioInput | Yes | Audio clips (1-8), sent in order. kind=‘url’ (HTTPS) or kind=‘base64’ (raw data without data: prefix, max 10 MB decoded) with mime_type audio/wav, audio/mpeg, audio/flac, audio/aac, or audio/mp4. Upload local files with media_upload first.; minItems: 1; maxItems: 8 |
delivery |
“auto”, “direct”, “files_api” | No | URL delivery: auto retries rejected audio through Files API (up to 512 MB); direct returns the rejection; files_api uploads first. Videos use their soundtrack and sampled frames. Base64 stays inline. Uploads are deleted after use.; Default: “auto” |
json_retry |
integer | No | Extra billed attempts when a JSON answer fails to parse or validate. Not retried after finish_reason=‘length’; raise max_tokens instead.; minimum: 0; maximum: 2; Default: 0 |
max_tokens |
integer or null | No | Maximum output tokens including thinking; raise it if finish_reason=‘length’.; Default: null |
overwrite |
boolean | No | Allow save_to to replace an existing file with different content.; Default: false |
prompt |
string | Yes | The question or task for the model.; minLength: 1; maxLength: 32000 |
reasoning_effort |
“low”, “medium”, “high” | No | Deep-thinking depth; thinking is always on. Higher costs more latency and tokens.; Default: “medium” |
repetition_penalty |
number or null | No | Repetition penalty.; Default: null |
response_format |
UnderstandingResponseFormat or null | No | Force JSON output; the parsed result is returned in choices[].parsed.; Default: null |
return_content |
“full”, “summary”, “none” | No | Answer text to inline: ‘full’, ‘summary’ (first 2,000 characters), or ‘none’ (use with save_to). parsed is always returned when available.; Default: “full” |
save_to |
string or null | No | Absolute file path (stdio only) to write the final answer to (JSON when parsed, else UTF-8 text). Must be inside an allowed output root; checked before the model runs.; Default: null |
system |
string or null | No | System instruction for the model.; Default: null |
temperature |
number or null | No | Sampling temperature.; Default: null |
top_p |
number or null | No | Nucleus sampling probability.; Default: null |
UnderstandingAudioInput
| Field | Type | Required | Description and constraints |
|---|---|---|---|
data |
string or null | No | Base64-encoded media data. Required when kind is ‘base64’.; Default: null |
kind |
MediaSourceKind | Yes | Whether the media is referenced by URL or Base64. |
mime_type |
string or null | No | MIME type of the media (e.g. image/png, audio/wav).; Default: null |
url |
string or null | No | HTTPS URL of the media. Required when kind is ‘url’.; Default: null |
UnderstandingJsonSchema
| Field | Type | Required | Description and constraints |
|---|---|---|---|
description |
string or null | No | Optional description of the schema’s purpose.; Default: null |
name |
string | Yes | Schema name sent to the provider (1-64 chars: letters, digits, '', ’-’).; pattern: ”^[A-Za-z0-9-]{1,64}$” |
schema |
object | Yes | JSON Schema (draft 2020-12) the answer must satisfy; at most 64 KB serialized. |
strict |
boolean | No | Ask the provider to enforce the schema strictly during decoding.; Default: true |
UnderstandingResponseFormat
| Field | Type | Required | Description and constraints |
|---|---|---|---|
json_schema |
UnderstandingJsonSchema or null | No | Required when type=‘json_schema’; must be omitted otherwise.; Default: null |
type |
“text”, “json_object”, “json_schema” | Yes | ‘text’, ‘json_object’ (any JSON object), or ‘json_schema’ (JSON matching json_schema). |
Complete JSON Schemas
Section titled “Complete JSON Schemas”Input schema, including the MCP wrapper
{ "$defs": { "MediaSourceKind": { "description": "Whether a media source is a URL or inline Base64 data.", "enum": [ "url", "base64" ], "type": "string" }, "SeedAudioUnderstandInput": { "description": "Input model for ``seed_audio_understand``.", "properties": { "audios": { "description": "Audio clips (1-8), sent in order. kind='url' (HTTPS) or kind='base64' (raw data without data: prefix, max 10 MB decoded) with mime_type audio/wav, audio/mpeg, audio/flac, audio/aac, or audio/mp4. Upload local files with media_upload first.", "items": { "$ref": "#/$defs/UnderstandingAudioInput" }, "maxItems": 8, "minItems": 1, "type": "array" }, "delivery": { "default": "auto", "description": "URL delivery: auto retries rejected audio through Files API (up to 512 MB); direct returns the rejection; files_api uploads first. Videos use their soundtrack and sampled frames. Base64 stays inline. Uploads are deleted after use.", "enum": [ "auto", "direct", "files_api" ], "type": "string" }, "json_retry": { "default": 0, "description": "Extra billed attempts when a JSON answer fails to parse or validate. Not retried after finish_reason='length'; raise max_tokens instead.", "maximum": 2, "minimum": 0, "type": "integer" }, "max_tokens": { "anyOf": [ { "maximum": 32768, "minimum": 1, "type": "integer" }, { "type": "null" } ], "default": null, "description": "Maximum output tokens including thinking; raise it if finish_reason='length'." }, "overwrite": { "default": false, "description": "Allow save_to to replace an existing file with different content.", "type": "boolean" }, "prompt": { "description": "The question or task for the model.", "maxLength": 32000, "minLength": 1, "type": "string" }, "reasoning_effort": { "default": "medium", "description": "Deep-thinking depth; thinking is always on. Higher costs more latency and tokens.", "enum": [ "low", "medium", "high" ], "type": "string" }, "repetition_penalty": { "anyOf": [ { "maximum": 2, "minimum": 0, "type": "number" }, { "type": "null" } ], "default": null, "description": "Repetition penalty." }, "response_format": { "anyOf": [ { "$ref": "#/$defs/UnderstandingResponseFormat" }, { "type": "null" } ], "default": null, "description": "Force JSON output; the parsed result is returned in choices[].parsed." }, "return_content": { "default": "full", "description": "Answer text to inline: 'full', 'summary' (first 2,000 characters), or 'none' (use with save_to). parsed is always returned when available.", "enum": [ "full", "summary", "none" ], "type": "string" }, "save_to": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Absolute file path (stdio only) to write the final answer to (JSON when parsed, else UTF-8 text). Must be inside an allowed output root; checked before the model runs." }, "system": { "anyOf": [ { "maxLength": 32000, "type": "string" }, { "type": "null" } ], "default": null, "description": "System instruction for the model." }, "temperature": { "anyOf": [ { "maximum": 2, "minimum": 0, "type": "number" }, { "type": "null" } ], "default": null, "description": "Sampling temperature." }, "top_p": { "anyOf": [ { "maximum": 1, "minimum": 0, "type": "number" }, { "type": "null" } ], "default": null, "description": "Nucleus sampling probability." } }, "required": [ "prompt", "audios" ], "type": "object" }, "UnderstandingAudioInput": { "description": "Audio input for audio understanding, as an HTTPS URL or Base64 data.", "properties": { "data": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Base64-encoded media data. Required when kind is 'base64'." }, "kind": { "$ref": "#/$defs/MediaSourceKind", "description": "Whether the media is referenced by URL or Base64." }, "mime_type": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "MIME type of the media (e.g. image/png, audio/wav)." }, "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "HTTPS URL of the media. Required when kind is 'url'." } }, "required": [ "kind" ], "type": "object" }, "UnderstandingJsonSchema": { "description": "JSON Schema the answer must satisfy, enforced by the provider during decoding.", "properties": { "description": { "anyOf": [ { "maxLength": 2000, "type": "string" }, { "type": "null" } ], "default": null, "description": "Optional description of the schema's purpose." }, "name": { "description": "Schema name sent to the provider (1-64 chars: letters, digits, '_', '-').", "pattern": "^[A-Za-z0-9_-]{1,64}$", "type": "string" }, "schema": { "additionalProperties": true, "description": "JSON Schema (draft 2020-12) the answer must satisfy; at most 64 KB serialized.", "type": "object" }, "strict": { "default": true, "description": "Ask the provider to enforce the schema strictly during decoding.", "type": "boolean" } }, "required": [ "name", "schema" ], "type": "object" }, "UnderstandingResponseFormat": { "description": "Output format constraint for the answer.", "properties": { "json_schema": { "anyOf": [ { "$ref": "#/$defs/UnderstandingJsonSchema" }, { "type": "null" } ], "default": null, "description": "Required when type='json_schema'; must be omitted otherwise." }, "type": { "description": "'text', 'json_object' (any JSON object), or 'json_schema' (JSON matching json_schema).", "enum": [ "text", "json_object", "json_schema" ], "type": "string" } }, "required": [ "type" ], "type": "object" } }, "additionalProperties": false, "properties": { "input": { "$ref": "#/$defs/SeedAudioUnderstandInput" } }, "required": [ "input" ], "type": "object"}Output schema, including nested definitions
{ "$defs": { "SchemaViolation": { "description": "Why a JSON answer did not parse or did not satisfy the requested JSON Schema.", "properties": { "finish_reason": { "description": "Finish reason of the completion. 'length' means the answer was cut off by max_tokens; raise max_tokens rather than retrying.", "title": "Finish Reason", "type": "string" }, "message": { "description": "Human-readable validation or parse error.", "title": "Message", "type": "string" }, "path": { "description": "JSON Pointer-style location of the first violation (e.g. '/beats/2/start'), or '' when the answer is not valid JSON at all.", "title": "Path", "type": "string" } }, "required": [ "path", "message", "finish_reason" ], "title": "SchemaViolation", "type": "object" }, "UnderstandingChoice": { "description": "A single completion choice returned by the Seed 2.1 model.\n\nOnly the final answer is returned. The model always thinks, but its\nreasoning trace is discarded and never included in tool output.", "properties": { "content": { "description": "The model's final answer. Truncated to the first 2,000 characters when return_content='summary', and empty when return_content='none'.", "title": "Content", "type": "string" }, "content_chars": { "description": "Length of the full answer in characters, before any truncation.", "title": "Content Chars", "type": "integer" }, "content_truncated": { "default": false, "description": "True when content is shorter than the full answer.", "title": "Content Truncated", "type": "boolean" }, "finish_reason": { "description": "Why generation stopped: 'stop', 'length', 'tool_calls', or 'content_filter'.", "title": "Finish Reason", "type": "string" }, "parsed": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "items": {}, "type": "array" }, { "type": "null" } ], "default": null, "description": "The answer parsed as JSON, set when a json_object or json_schema response_format was requested and the answer parsed (and, for json_schema, validated). Returned even when return_content is 'summary' or 'none'.", "title": "Parsed" }, "role": { "const": "assistant", "default": "assistant", "description": "Message role (always 'assistant').", "title": "Role", "type": "string" }, "schema_violation": { "anyOf": [ { "$ref": "#/$defs/SchemaViolation" }, { "type": "null" } ], "default": null, "description": "Set when a JSON response_format was requested but the answer did not parse or did not satisfy the schema. The raw answer is still returned in content." } }, "required": [ "content", "content_chars", "finish_reason" ], "title": "UnderstandingChoice", "type": "object" }, "UnderstandingUsage": { "description": "Token usage for a Seed 2.1 multimodal understanding call.", "properties": { "completion_tokens": { "description": "Number of output (completion) tokens consumed, including reasoning.", "title": "Completion Tokens", "type": "integer" }, "prompt_tokens": { "description": "Number of input (prompt) tokens consumed.", "title": "Prompt Tokens", "type": "integer" }, "reasoning_tokens": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Completion tokens spent on internal deep thinking, when the provider reports it. The reasoning text itself is never returned.", "title": "Reasoning Tokens" }, "total_tokens": { "description": "Total tokens consumed (prompt + completion).", "title": "Total Tokens", "type": "integer" } }, "required": [ "prompt_tokens", "completion_tokens", "total_tokens" ], "title": "UnderstandingUsage", "type": "object" } }, "description": "Output model for ``seed_audio_understand``.", "properties": { "attempts": { "default": 1, "description": "Completions that produced an answer: 1 plus json_retry attempts used.", "title": "Attempts", "type": "integer" }, "audio_delivery": { "description": "Delivery route per input: url, base64, files_api_audio, or files_api_video.", "items": { "enum": [ "url", "base64", "files_api_audio", "files_api_video" ], "type": "string" }, "title": "Audio Delivery", "type": "array" }, "choices": { "description": "Model completion choices (one for non-streaming).", "items": { "$ref": "#/$defs/UnderstandingChoice" }, "title": "Choices", "type": "array" }, "completion_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Provider completion ID (e.g. 'chatcmpl-...') for tracing.", "title": "Completion Id" }, "delivery_fallback_reason": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Provider rejection that triggered automatic Files API fallback, if any.", "title": "Delivery Fallback Reason" }, "model": { "description": "Model ID used for the completion.", "title": "Model", "type": "string" }, "provider": { "const": "byteplus-modelark", "default": "byteplus-modelark", "description": "Provider that generated the response.", "title": "Provider", "type": "string" }, "request_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Provider request ID for support tracing.", "title": "Request Id" }, "saved_bytes": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Size in bytes of the file written to saved_path.", "title": "Saved Bytes" }, "saved_path": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "Absolute path the answer was written to when save_to was set.", "title": "Saved Path" }, "usage": { "$ref": "#/$defs/UnderstandingUsage", "description": "Token usage summed over all attempts (including json_retry attempts)." } }, "required": [ "model", "choices", "usage" ], "title": "SeedAudioUnderstandOutput", "type": "object"}Troubleshooting
Section titled “Troubleshooting”- Missing tool: check provider credentials and feature flags, then restart the connection.
- Invalid arguments: keep the correct wrapper and follow current model limits from discovery. A valid schema alone does not prove the chosen model supports the combination.
- A job completes with an error: inspect the original tool result, not only the outer job status. Never automatically resubmit potentially billable work.
- Local output writes from jobs require server-configured
OUTPUT_ROOTS; HTTP cannot write to a client’s local disk.
See configuration, background execution, and artifact delivery.