Open-Weight Frontier Models on AWS

01The short answer

Three premise corrections worth knowing up front

02Decision flow

03Six surfaces, not three

04Custom Model Import: the allowlist & four gates

Read verbatim from the CMI documentation on 2026-08-12. This is the decisive gate for everything else on this page.

Gate 1 — Architecture (a closed allowlist)

CMI auto-detects the architecture from your config.json during the import job and matches it against this list. It is exhaustive, not illustrative:

• Qwen3 is restricted to Qwen3ForCausalLM and Qwen3MoeForCausalLM only. Converse is not supported for Qwen3 models.
• GPT-OSS is us-east-1 only.

Gates 2–4 — Size, context, tooling

Tokenizer allowlist

T5Tokenizer · T5TokenizerFast · LlamaTokenizer · LlamaTokenizerFast · CodeLlamaTokenizer · CodeLlamaTokenizerFast · GPT2Tokenizer · GPT2TokenizerFast · GPTNeoXTokenizer · GPTNeoXTokenizerFast · PreTrainedTokenizer · PreTrainedTokenizerFast · Qwen2Tokenizer · Qwen2TokenizerFast

Every Qwen-lineage model in scope uses Qwen2Tokenizer, so the tokenizer is never the blocker — the architecture is.

Required files in S3

*.safetensors            # weights, Safetensors format (GGUF is NOT accepted)
config.json              # architecture auto-detection happens here
tokenizer_config.json
tokenizer.json
tokenizer.model

05Eligibility checker

Runs entirely in your browser against the allowlist above. Paste a config.json, or fill the fields manually.

REPO=Qwen/Qwen3-32B
curl -s https://huggingface.co/$REPO/raw/main/config.json \
  | jq '{arch:.architectures, ctx:.max_position_embeddings, quant:.quantization_config.quant_method}'
curl -s https://huggingface.co/$REPO/raw/main/model.safetensors.index.json \
  | jq '.metadata.total_size'          # bytes; divide by 1e9 for GB
curl -s https://huggingface.co/$REPO/raw/main/tokenizer_config.json | jq '.tokenizer_class'

06What IS importable

Within the five families

Other families that hit the allowlist

07Quantization, smaller sizes & "abstracting the class"

A · Quantization never changes CMI eligibility

architectures[] is a property of the model class, not the weight dtype. Every quantized derivative inherits its parent's string. Verified 2026-08-12:

No quantization of any flagship rescues CMI eligibility. Quantization only relaxes the size gate — and size is never the sole blocker. GGUF builds are doubly excluded: CMI requires safetensors.

B · Smaller sizes fix the wrong gate

All of these clear the size caps comfortably and are still blocked on architecture alone:

C · Why "abstracting the class" cannot work for Qwen3.5+

The idea is legitimate in principle: CMI auto-detects architecture from config.json, so relabelling architectures to an allowlisted string would unlock the import — but only if the tensor topology genuinely matches. It does not. Structural config diff, Qwen3-30B-A3B (allowlisted) vs Qwen3.8-2.4T-A95B (not):

Present in Qwen3.5/3.8, absent from Qwen3: layer_types · linear_conv_kernel_dim · linear_key_head_dim · linear_num_key_heads · linear_num_value_heads · linear_value_head_dim · mamba_ssm_dtype · full_attention_interval · attn_output_gate · output_gate_type · partial_rotary_factor · mtp_num_hidden_layers · mtp_use_dedicated_embeddings · shared_expert_intermediate_size · rope_parameters

Tensor-level confirmation across Qwen3.8's 1,609 tensors:

D · Tweaks that genuinely do work

08Master model table

Every value fetched first-hand on 2026-08-12. Weights from model.safetensors.index.json → metadata.total_size (decimal GB), cross-checked against parameter count × dtype width. Arch is the exact config.json architectures[0] — the string CMI matches on.

Context = max_position_embeddings; — means the field is absent from the top-level config (typical for multimodal wrappers, where it sits in a nested text config). n/a in the Weights column means no index.json and the value came from a paginated LFS blob sum, noted in the methodology section.

09Traps & near-misses

Brands you'd expect to work, but don't

Near-misses worth escalating to AWS

10Bedrock managed catalog

Amazon Bedrock now serves these families first-party and serverless — no weights, no infrastructure. Read verbatim from Endpoint availability by models, 2026-08-12.

Two endpoints — pick per model

Exact model IDs & regions

From the per-model Bedrock model cards, fetched 2026-08-12. Note the trap: for several models bedrock-runtime and bedrock-mantle use different ID strings.

None of GLM-5.2, GLM-5.1, Kimi-K3, Kimi-Dev-72B, MiniMax-M3, DeepSeek-V4-Pro or DeepSeek-V4-Flash has a Bedrock model card — verified absent, not merely unfound. They are not offered as managed Bedrock models.

Resolve exact model IDs in your own account

aws bedrock list-foundation-models --region us-east-1 \
  --query "modelSummaries[?contains(providerName,'DeepSeek')
            ||contains(providerName,'Moonshot')
            ||contains(providerName,'MiniMax')
            ||contains(providerName,'Z.AI')
            ||contains(providerName,'Qwen')].[modelId,modelName]" \
  --output table

The former "models at a glance" page now redirects to the Bedrock overview, so list-foundation-models in your target region is the authoritative source for IDs and regional availability.

11SageMaker

JumpStart — verified roster for these five families

From the authoritative Available foundation models table (436 open-weight models, stated current as of 2026-07-29), read directly on 2026-08-12.

Bedrock Marketplace — narrower still

Checked against the Bedrock Marketplace model-compatibility page: GLM, Kimi/Moonshot, MiniMax and standalone Qwen are all absent. The only models in scope are DeepSeek-R1 plus its six distills (DeepSeek-R1-Distill-Qwen-32B / -14B / -7B / -1.5B, Llama-70B, Llama-8B) — all with Converse API: No, Streaming: Yes. No DeepSeek-V3.x, no V4. So for these five families, Marketplace is not a meaningful third path; the real choice is managed catalog vs JumpStart vs BYOC vs EC2.

BYOC + LMI — the path for GLM-5.2, Kimi K3, MiniMax M3, Qwen 3.8, DeepSeek V4

SageMaker's Large Model Inference containers bundle vLLM and DJL-Serving. Since all these architectures are in vLLM upstream, this is the managed path that actually works for the newest models.

env = {
    "HF_MODEL_ID": "deepseek-ai/DeepSeek-V4-Flash",
    "OPTION_ASYNC_MODE": "true",
    "OPTION_ROLLING_BATCH": "disable",
    "OPTION_ENTRYPOINT": "djl_python.lmi_vllm.vllm_async_service",
    "OPTION_TENSOR_PARALLEL_DEGREE": "8",
    "OPTION_MAX_MODEL_LEN": "131072",     # cap context to fit KV cache
}
# image: 763104351884.dkr.ecr.{region}.amazonaws.com/djl-inference:0.34.0-lmi

For weights in the hundreds of GB, enable fast model loading — SageMaker pre-shards the model and streams weights directly from S3 into GPU memory, skipping the download-to-disk and host-memory hops. At 1.5 TB this is the difference between minutes and tens of minutes per scale-out event.

12EC2 / EKS self-host

Instance reference

Budget ~1.2–1.5× weights for KV cache, activations and CUDA graphs. A 1 M-context model needs far more KV headroom than that multiplier suggests — at 1 M context plan on paged KV offload or serve a reduced context via --max-model-len.

Minimum vLLM version per architecture

Established by bisecting registry.py at each vLLM release tag — not from the docs page, which lags badly (Kimi-K3 and Qwen3.5-MoE are absent from the published support table but present in the registry). Latest release at time of writing: v0.27.1 (2026-08-11).

SGLang as an alternative

All five families are implemented in SGLang on main, with dedicated attention backends and speculative-decoding variants:

Kimi-K3's vision tower confirms it is multimodal, resolving one of the open questions elsewhere on this page. Minimum SGLang release versions were not bisected — presence confirmed on main only, so don't quote a version number.

Multi-node inference — the pattern AWS actually documents

vLLM on EKS, streaming weights from S3

image: public.ecr.aws/deep-learning-containers/vllm:0.21.0-gpu-py312-cu130-ubuntu22.04-ec2-v1.0-soci
args:
  - --model=s3://<MODEL_PATH>
  - --tensor-parallel-size=8
  - --load-format=runai_streamer
  - --model-loader-extra-config={"concurrency":17,"distributed":true}
env:
  - name: RUNAI_STREAMER_CHUNK_BYTESIZE
    value: "4294967296"          # 4 GB chunks — best measured throughput

Set distributed:true only when tensor-parallel > 1: each TP rank then streams its own weight shard straight from S3 instead of rank 0 loading everything and broadcasting. Requires the vLLM V1 engine and is incompatible with --enforce-eager.

Trainium / Inferentia — now resolved

Checked against AWS Neuron SDK 2.31.0 (released 2026-07-07; NxD Inference 0.10.18399, vLLM Neuron plugin 0.16.0 / vllm-neuron 0.5.0) and the NxD Inference supported-architecture list.

13Node sizing helper

14What it costs

Bedrock Custom Model Import — the CMU model

CMI bills per Custom Model Unit (CMU) per minute, metered in 5-minute windows starting from the first successful invocation of each model copy. Importing itself is free. AWS decides how many CMUs your model needs at import time, based on architecture, parameter count and context length.

Published CMU reference points: Llama 3.1 8B at 128 K = 2 CMUs · Llama 3.1 70B at 128 K = 8 CMUs. Extrapolating, an importable ~65 GB model (Qwen3-32B, QwQ-32B) plausibly lands around 4 CMUs — at $0.05718 that is roughly $0.23/min ≈ $13.7/hr while active, plus ~$7.80/month storage. Because billing is per-5-minute-window from first invocation rather than a persistent endpoint, CMI is markedly cheaper than a dedicated GPU endpoint for bursty traffic and more expensive for saturated traffic.

EC2 — on-demand, us-east-1, Linux

Capacity Blocks — mechanics

• Required for p5e.48xlarge, p6e-gb200 UltraServers and trn2.48xlarge — no on-demand rate is published for any of them. Optional for p5, p5en, p6-b200, p6-b300.
• Reserve up to 8 weeks in advance; duration 1–14 days (1-day increments) or 15–182 days (7-day increments).
• Up to 64 instances per block; up to 256 instances across multiple blocks on a given date (minimum 4 blocks). Shareable across AWS Organizations accounts.
• Cost = up-front reservation fee at the rate prevailing at purchase + OS fee while running. Linux OS fee is $0.00/hr on all these types (RHEL $1.84/hr, Ubuntu Pro $0.336/hr).

# 1 - find offerings
aws ec2 describe-capacity-block-offerings \\
  --instance-type p5en.48xlarge --instance-count 2 \\
  --capacity-duration-hours 48 \\
  --start-date-range 2026-09-01T00:00:00Z --end-date-range 2026-09-15T00:00:00Z

# 2 - purchase (creates a billable commitment - confirm the offering first)
aws ec2 purchase-capacity-block \\
  --capacity-block-offering-id cb-0123456789abcdefg \\
  --instance-platform Linux/UNIX

For UltraServers pass --ultraserver-type u-p6e-gb200x72 and an UltraServer count instead of an instance type.

15Licensing — read this before you self-host

HuggingFace reports license: other for most of these models, which is not actionable. Every licence below was fetched and read verbatim on 2026-08-12. The differences are material and they do not follow family lines.

# Always check the actual file, not the HF metadata label
curl -s https://huggingface.co/MiniMaxAI/MiniMax-M3/raw/main/LICENSE | head -40
# and list what licence files actually exist (M2.5 uses LICENSE-MODEL, not LICENSE)
curl -s "https://huggingface.co/api/models/MiniMaxAI/MiniMax-M2.5/tree/main" \
  | jq -r '.[].path | select(test("LICEN|NOTICE"))'

16Guides & resources

Bedrock Custom Model Import

Bedrock managed catalog & Marketplace

SageMaker — JumpStart

SageMaker — BYOC / LMI / vLLM

EC2 / EKS

Multi-node, serving stacks & containers

Neuron / Trainium

Licensing — fetch the actual files

Governance & model sources

17Method & caveats

How every number here was produced

Explicitly NOT verified — do not rely on these

• Whether CMI accepts FP8 weights. Docs are now silent; the 2024 GA post said BF16-only. Moot for the flagships here, material if you import quantized weights generally.
• The exact CMI context ceiling. Reasoned to be 131,072 inclusive from AWS's own supported-model set, but not stated numerically in any doc I could find.
• RESOLVED Neuron / Trainium — now verified against Neuron SDK 2.31.0; see §12. Only Qwen3 and Qwen3-MoE are supported.
• RESOLVED Per-region availability of each Bedrock managed model — see the region counts in §10. Still confirm in your own account, as regional rollout moves.
• RESOLVED Exact Bedrock modelId strings and regions — now fetched from the per-model Bedrock model cards; see §10.
• Two architecture remaps left as untested rather than guessed: MiniMaxM2ForCausalLM → MixtralForCausalLM and Glm4ForCausalLM → LlamaForCausalLM.
• Kimi K3 / K2.5 modality. Both use the multimodal ForConditionalGeneration signature but expose no top-level vision_config, so which size cap would apply is unconfirmed. Moot — both exceed even the 200 GB text cap by 3–8×.