Skip to content

seed_audio_understand

Understand or reason about audio (transcribe, translate, analyze, answer questions).

Returns only the final answer; the server fixes the model (see ark_list_models). For structured output use response_format ‘json_schema’ (‘json_object’ is not enforced by the default model). For plain transcription with word timings use speech_to_text. Oversized URL audio and rejected video inputs can use Files API (delivery); video frames also consume tokens. Requires background execution (ark_job_submit).

Requirement Value
Registration Requires BYTEPLUS_MODELARK_API_KEY.
Execution Background required
Required permission understanding:read

This site lists the full configured capability surface, not the tools enabled on your particular server. Use model discovery and job discovery to check your connection. Credentials are server configuration, never tool arguments.

These are supported families and built-in model or service IDs for this site revision, not provider account access or a guarantee that every row is enabled on your server. Check model discovery for configured bindings and service availability.

Model Model or service ID Availability Capabilities
Seed 2.0 Lite (audio understanding) seed-2-0-lite-260428 Server-configured default; BYTEPLUS_MODELARK_API_KEY required. 1-8 audio clips; transcription, translation, analysis and Q&A; JSON Schema for structured output; oversized URL inputs can use the Files API (up to 512 MB).

This tool has no per-call model argument. SEED_AUDIO_UNDERSTANDING_MODEL selects a ModelArk model supporting Chat Completions input_audio with deep thinking; other compatible, authorized IDs may be configured.

Ask your connected AI client to use seed_audio_understand for the operation you want. The client constructs the arguments and handles tool calls. Review its proposed operation before approving anything billable. For a non-billable introduction, start with discovery without generation.

Submit through ark_job_submit with tool_name and the complete target arguments, including its inner input object. Poll ark_job_get using the returned Ark job_id, respecting poll_after_ms. Inspect both terminal status and result.is_error; the original output is in result.structured_content. Do not call this tool in the foreground.

Exact MCP call (for developers)

The following is an MCP tool-call envelope, not a shell command. Example media URLs and identifiers must be replaced with your own accessible resources. Seedream and Seedance examples are non-billable dry runs; other provider operations may incur charges.

{
"name": "ark_job_submit",
"arguments": {
"input": {
"tool_name": "seed_audio_understand",
"arguments": {
"input": {
"prompt": "Summarize this recording.",
"audios": [
{
"kind": "url",
"url": "https://example.com/recording.wav"
}
]
}
}
}
}
}

The MCP arguments object wraps these fields as {"input": {...}}.

Field Type Required Description and constraints
audios array of UnderstandingAudioInput Yes Audio clips (1-8), sent in order. kind=‘url’ (HTTPS) or kind=‘base64’ (raw data without data: prefix, max 10 MB decoded) with mime_type audio/wav, audio/mpeg, audio/flac, audio/aac, or audio/mp4. Upload local files with media_upload first.; minItems: 1; maxItems: 8
delivery “auto”, “direct”, “files_api” No URL delivery: auto retries rejected audio through Files API (up to 512 MB); direct returns the rejection; files_api uploads first. Videos use their soundtrack and sampled frames. Base64 stays inline. Uploads are deleted after use.; Default: “auto”
json_retry integer No Extra billed attempts when a JSON answer fails to parse or validate. Not retried after finish_reason=‘length’; raise max_tokens instead.; minimum: 0; maximum: 2; Default: 0
max_tokens integer or null No Maximum output tokens including thinking; raise it if finish_reason=‘length’.; Default: null
overwrite boolean No Allow save_to to replace an existing file with different content.; Default: false
prompt string Yes The question or task for the model.; minLength: 1; maxLength: 32000
reasoning_effort “low”, “medium”, “high” No Deep-thinking depth; thinking is always on. Higher costs more latency and tokens.; Default: “medium”
repetition_penalty number or null No Repetition penalty.; Default: null
response_format UnderstandingResponseFormat or null No Force JSON output; the parsed result is returned in choices[].parsed.; Default: null
return_content “full”, “summary”, “none” No Answer text to inline: ‘full’, ‘summary’ (first 2,000 characters), or ‘none’ (use with save_to). parsed is always returned when available.; Default: “full”
save_to string or null No Absolute file path (stdio only) to write the final answer to (JSON when parsed, else UTF-8 text). Must be inside an allowed output root; checked before the model runs.; Default: null
system string or null No System instruction for the model.; Default: null
temperature number or null No Sampling temperature.; Default: null
top_p number or null No Nucleus sampling probability.; Default: null
Field Type Required Description and constraints
attempts integer No Completions that produced an answer: 1 plus json_retry attempts used.; Default: 1
audio_delivery array of “url”, “base64”, “files_api_audio”, “files_api_video” No Delivery route per input: url, base64, files_api_audio, or files_api_video.
choices array of UnderstandingChoice Yes Model completion choices (one for non-streaming).
completion_id string or null No Provider completion ID (e.g. ‘chatcmpl-…’) for tracing.; Default: null
delivery_fallback_reason string or null No Provider rejection that triggered automatic Files API fallback, if any.; Default: null
model string Yes Model ID used for the completion.
provider “byteplus-modelark” No Provider that generated the response.; Default: “byteplus-modelark”
request_id string or null No Provider request ID for support tracing.; Default: null
saved_bytes integer or null No Size in bytes of the file written to saved_path.; Default: null
saved_path string or null No Absolute path the answer was written to when save_to was set.; Default: null
usage UnderstandingUsage Yes Token usage summed over all attempts (including json_retry attempts).

Field types named above refer to these shared definitions. Optional/null fields, enum choices and bounds are retained in the complete schemas below. Schema validation is followed by runtime capability and security checks.

MediaSourceKind

No fields. Pass {}.

SeedAudioUnderstandInput
Field Type Required Description and constraints
audios array of UnderstandingAudioInput Yes Audio clips (1-8), sent in order. kind=‘url’ (HTTPS) or kind=‘base64’ (raw data without data: prefix, max 10 MB decoded) with mime_type audio/wav, audio/mpeg, audio/flac, audio/aac, or audio/mp4. Upload local files with media_upload first.; minItems: 1; maxItems: 8
delivery “auto”, “direct”, “files_api” No URL delivery: auto retries rejected audio through Files API (up to 512 MB); direct returns the rejection; files_api uploads first. Videos use their soundtrack and sampled frames. Base64 stays inline. Uploads are deleted after use.; Default: “auto”
json_retry integer No Extra billed attempts when a JSON answer fails to parse or validate. Not retried after finish_reason=‘length’; raise max_tokens instead.; minimum: 0; maximum: 2; Default: 0
max_tokens integer or null No Maximum output tokens including thinking; raise it if finish_reason=‘length’.; Default: null
overwrite boolean No Allow save_to to replace an existing file with different content.; Default: false
prompt string Yes The question or task for the model.; minLength: 1; maxLength: 32000
reasoning_effort “low”, “medium”, “high” No Deep-thinking depth; thinking is always on. Higher costs more latency and tokens.; Default: “medium”
repetition_penalty number or null No Repetition penalty.; Default: null
response_format UnderstandingResponseFormat or null No Force JSON output; the parsed result is returned in choices[].parsed.; Default: null
return_content “full”, “summary”, “none” No Answer text to inline: ‘full’, ‘summary’ (first 2,000 characters), or ‘none’ (use with save_to). parsed is always returned when available.; Default: “full”
save_to string or null No Absolute file path (stdio only) to write the final answer to (JSON when parsed, else UTF-8 text). Must be inside an allowed output root; checked before the model runs.; Default: null
system string or null No System instruction for the model.; Default: null
temperature number or null No Sampling temperature.; Default: null
top_p number or null No Nucleus sampling probability.; Default: null
UnderstandingAudioInput
Field Type Required Description and constraints
data string or null No Base64-encoded media data. Required when kind is ‘base64’.; Default: null
kind MediaSourceKind Yes Whether the media is referenced by URL or Base64.
mime_type string or null No MIME type of the media (e.g. image/png, audio/wav).; Default: null
url string or null No HTTPS URL of the media. Required when kind is ‘url’.; Default: null
UnderstandingJsonSchema
Field Type Required Description and constraints
description string or null No Optional description of the schema’s purpose.; Default: null
name string Yes Schema name sent to the provider (1-64 chars: letters, digits, '', ’-’).; pattern: ”^[A-Za-z0-9-]{1,64}$”
schema object Yes JSON Schema (draft 2020-12) the answer must satisfy; at most 64 KB serialized.
strict boolean No Ask the provider to enforce the schema strictly during decoding.; Default: true
UnderstandingResponseFormat
Field Type Required Description and constraints
json_schema UnderstandingJsonSchema or null No Required when type=‘json_schema’; must be omitted otherwise.; Default: null
type “text”, “json_object”, “json_schema” Yes ‘text’, ‘json_object’ (any JSON object), or ‘json_schema’ (JSON matching json_schema).
Input schema, including the MCP wrapper
{
"$defs": {
"MediaSourceKind": {
"description": "Whether a media source is a URL or inline Base64 data.",
"enum": [
"url",
"base64"
],
"type": "string"
},
"SeedAudioUnderstandInput": {
"description": "Input model for ``seed_audio_understand``.",
"properties": {
"audios": {
"description": "Audio clips (1-8), sent in order. kind='url' (HTTPS) or kind='base64' (raw data without data: prefix, max 10 MB decoded) with mime_type audio/wav, audio/mpeg, audio/flac, audio/aac, or audio/mp4. Upload local files with media_upload first.",
"items": {
"$ref": "#/$defs/UnderstandingAudioInput"
},
"maxItems": 8,
"minItems": 1,
"type": "array"
},
"delivery": {
"default": "auto",
"description": "URL delivery: auto retries rejected audio through Files API (up to 512 MB); direct returns the rejection; files_api uploads first. Videos use their soundtrack and sampled frames. Base64 stays inline. Uploads are deleted after use.",
"enum": [
"auto",
"direct",
"files_api"
],
"type": "string"
},
"json_retry": {
"default": 0,
"description": "Extra billed attempts when a JSON answer fails to parse or validate. Not retried after finish_reason='length'; raise max_tokens instead.",
"maximum": 2,
"minimum": 0,
"type": "integer"
},
"max_tokens": {
"anyOf": [
{
"maximum": 32768,
"minimum": 1,
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Maximum output tokens including thinking; raise it if finish_reason='length'."
},
"overwrite": {
"default": false,
"description": "Allow save_to to replace an existing file with different content.",
"type": "boolean"
},
"prompt": {
"description": "The question or task for the model.",
"maxLength": 32000,
"minLength": 1,
"type": "string"
},
"reasoning_effort": {
"default": "medium",
"description": "Deep-thinking depth; thinking is always on. Higher costs more latency and tokens.",
"enum": [
"low",
"medium",
"high"
],
"type": "string"
},
"repetition_penalty": {
"anyOf": [
{
"maximum": 2,
"minimum": 0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"description": "Repetition penalty."
},
"response_format": {
"anyOf": [
{
"$ref": "#/$defs/UnderstandingResponseFormat"
},
{
"type": "null"
}
],
"default": null,
"description": "Force JSON output; the parsed result is returned in choices[].parsed."
},
"return_content": {
"default": "full",
"description": "Answer text to inline: 'full', 'summary' (first 2,000 characters), or 'none' (use with save_to). parsed is always returned when available.",
"enum": [
"full",
"summary",
"none"
],
"type": "string"
},
"save_to": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Absolute file path (stdio only) to write the final answer to (JSON when parsed, else UTF-8 text). Must be inside an allowed output root; checked before the model runs."
},
"system": {
"anyOf": [
{
"maxLength": 32000,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "System instruction for the model."
},
"temperature": {
"anyOf": [
{
"maximum": 2,
"minimum": 0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"description": "Sampling temperature."
},
"top_p": {
"anyOf": [
{
"maximum": 1,
"minimum": 0,
"type": "number"
},
{
"type": "null"
}
],
"default": null,
"description": "Nucleus sampling probability."
}
},
"required": [
"prompt",
"audios"
],
"type": "object"
},
"UnderstandingAudioInput": {
"description": "Audio input for audio understanding, as an HTTPS URL or Base64 data.",
"properties": {
"data": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Base64-encoded media data. Required when kind is 'base64'."
},
"kind": {
"$ref": "#/$defs/MediaSourceKind",
"description": "Whether the media is referenced by URL or Base64."
},
"mime_type": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "MIME type of the media (e.g. image/png, audio/wav)."
},
"url": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "HTTPS URL of the media. Required when kind is 'url'."
}
},
"required": [
"kind"
],
"type": "object"
},
"UnderstandingJsonSchema": {
"description": "JSON Schema the answer must satisfy, enforced by the provider during decoding.",
"properties": {
"description": {
"anyOf": [
{
"maxLength": 2000,
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Optional description of the schema's purpose."
},
"name": {
"description": "Schema name sent to the provider (1-64 chars: letters, digits, '_', '-').",
"pattern": "^[A-Za-z0-9_-]{1,64}$",
"type": "string"
},
"schema": {
"additionalProperties": true,
"description": "JSON Schema (draft 2020-12) the answer must satisfy; at most 64 KB serialized.",
"type": "object"
},
"strict": {
"default": true,
"description": "Ask the provider to enforce the schema strictly during decoding.",
"type": "boolean"
}
},
"required": [
"name",
"schema"
],
"type": "object"
},
"UnderstandingResponseFormat": {
"description": "Output format constraint for the answer.",
"properties": {
"json_schema": {
"anyOf": [
{
"$ref": "#/$defs/UnderstandingJsonSchema"
},
{
"type": "null"
}
],
"default": null,
"description": "Required when type='json_schema'; must be omitted otherwise."
},
"type": {
"description": "'text', 'json_object' (any JSON object), or 'json_schema' (JSON matching json_schema).",
"enum": [
"text",
"json_object",
"json_schema"
],
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"additionalProperties": false,
"properties": {
"input": {
"$ref": "#/$defs/SeedAudioUnderstandInput"
}
},
"required": [
"input"
],
"type": "object"
}
Output schema, including nested definitions
{
"$defs": {
"SchemaViolation": {
"description": "Why a JSON answer did not parse or did not satisfy the requested JSON Schema.",
"properties": {
"finish_reason": {
"description": "Finish reason of the completion. 'length' means the answer was cut off by max_tokens; raise max_tokens rather than retrying.",
"title": "Finish Reason",
"type": "string"
},
"message": {
"description": "Human-readable validation or parse error.",
"title": "Message",
"type": "string"
},
"path": {
"description": "JSON Pointer-style location of the first violation (e.g. '/beats/2/start'), or '' when the answer is not valid JSON at all.",
"title": "Path",
"type": "string"
}
},
"required": [
"path",
"message",
"finish_reason"
],
"title": "SchemaViolation",
"type": "object"
},
"UnderstandingChoice": {
"description": "A single completion choice returned by the Seed 2.1 model.\n\nOnly the final answer is returned. The model always thinks, but its\nreasoning trace is discarded and never included in tool output.",
"properties": {
"content": {
"description": "The model's final answer. Truncated to the first 2,000 characters when return_content='summary', and empty when return_content='none'.",
"title": "Content",
"type": "string"
},
"content_chars": {
"description": "Length of the full answer in characters, before any truncation.",
"title": "Content Chars",
"type": "integer"
},
"content_truncated": {
"default": false,
"description": "True when content is shorter than the full answer.",
"title": "Content Truncated",
"type": "boolean"
},
"finish_reason": {
"description": "Why generation stopped: 'stop', 'length', 'tool_calls', or 'content_filter'.",
"title": "Finish Reason",
"type": "string"
},
"parsed": {
"anyOf": [
{
"additionalProperties": true,
"type": "object"
},
{
"items": {},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"description": "The answer parsed as JSON, set when a json_object or json_schema response_format was requested and the answer parsed (and, for json_schema, validated). Returned even when return_content is 'summary' or 'none'.",
"title": "Parsed"
},
"role": {
"const": "assistant",
"default": "assistant",
"description": "Message role (always 'assistant').",
"title": "Role",
"type": "string"
},
"schema_violation": {
"anyOf": [
{
"$ref": "#/$defs/SchemaViolation"
},
{
"type": "null"
}
],
"default": null,
"description": "Set when a JSON response_format was requested but the answer did not parse or did not satisfy the schema. The raw answer is still returned in content."
}
},
"required": [
"content",
"content_chars",
"finish_reason"
],
"title": "UnderstandingChoice",
"type": "object"
},
"UnderstandingUsage": {
"description": "Token usage for a Seed 2.1 multimodal understanding call.",
"properties": {
"completion_tokens": {
"description": "Number of output (completion) tokens consumed, including reasoning.",
"title": "Completion Tokens",
"type": "integer"
},
"prompt_tokens": {
"description": "Number of input (prompt) tokens consumed.",
"title": "Prompt Tokens",
"type": "integer"
},
"reasoning_tokens": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Completion tokens spent on internal deep thinking, when the provider reports it. The reasoning text itself is never returned.",
"title": "Reasoning Tokens"
},
"total_tokens": {
"description": "Total tokens consumed (prompt + completion).",
"title": "Total Tokens",
"type": "integer"
}
},
"required": [
"prompt_tokens",
"completion_tokens",
"total_tokens"
],
"title": "UnderstandingUsage",
"type": "object"
}
},
"description": "Output model for ``seed_audio_understand``.",
"properties": {
"attempts": {
"default": 1,
"description": "Completions that produced an answer: 1 plus json_retry attempts used.",
"title": "Attempts",
"type": "integer"
},
"audio_delivery": {
"description": "Delivery route per input: url, base64, files_api_audio, or files_api_video.",
"items": {
"enum": [
"url",
"base64",
"files_api_audio",
"files_api_video"
],
"type": "string"
},
"title": "Audio Delivery",
"type": "array"
},
"choices": {
"description": "Model completion choices (one for non-streaming).",
"items": {
"$ref": "#/$defs/UnderstandingChoice"
},
"title": "Choices",
"type": "array"
},
"completion_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Provider completion ID (e.g. 'chatcmpl-...') for tracing.",
"title": "Completion Id"
},
"delivery_fallback_reason": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Provider rejection that triggered automatic Files API fallback, if any.",
"title": "Delivery Fallback Reason"
},
"model": {
"description": "Model ID used for the completion.",
"title": "Model",
"type": "string"
},
"provider": {
"const": "byteplus-modelark",
"default": "byteplus-modelark",
"description": "Provider that generated the response.",
"title": "Provider",
"type": "string"
},
"request_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Provider request ID for support tracing.",
"title": "Request Id"
},
"saved_bytes": {
"anyOf": [
{
"type": "integer"
},
{
"type": "null"
}
],
"default": null,
"description": "Size in bytes of the file written to saved_path.",
"title": "Saved Bytes"
},
"saved_path": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Absolute path the answer was written to when save_to was set.",
"title": "Saved Path"
},
"usage": {
"$ref": "#/$defs/UnderstandingUsage",
"description": "Token usage summed over all attempts (including json_retry attempts)."
}
},
"required": [
"model",
"choices",
"usage"
],
"title": "SeedAudioUnderstandOutput",
"type": "object"
}
  • Missing tool: check provider credentials and feature flags, then restart the connection.
  • Invalid arguments: keep the correct wrapper and follow current model limits from discovery. A valid schema alone does not prove the chosen model supports the combination.
  • A job completes with an error: inspect the original tool result, not only the outer job status. Never automatically resubmit potentially billable work.
  • Local output writes from jobs require server-configured OUTPUT_ROOTS; HTTP cannot write to a client’s local disk.

See configuration, background execution, and artifact delivery.