---
title: Ministral-3
metatags:
    description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
---

import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';

## 1. Model Introduction
The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.

The Ministral 3 14B Instruct model offers the following capabilities:

Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
System Prompt: Maintains strong adherence and support for system prompts.
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
Large Context Window: Supports a 256k context window.

For further details, please refer to the [official documentation](https://github.com/mistralai)

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

<Ministral3Deployment />

### 3.2 Configuration Tips
**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.

**Pre-installation steps**: Adding the following steps after launching the docker
```shell Command
pip install mistral-common --upgrade
pip install transformers==5.0.0.rc0
```
## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Launch the docker
```shell Command
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```

```shell Command
docker run -d -it --ipc=host --network=host --privileged \
  --cap-add=CAP_SYS_ADMIN \
  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
  --group-add video --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /:/work \
  -e SHELL=/bin/bash \
  --name Ministral \
 lmsysorg/sglang:v0.5.9-rocm720-mi30x \
  /bin/bash
```

#### 4.2.2 Launch the server
```shell Command
sglang serve \
  --model-path mistralai/Ministral-3-14B-Instruct-2512 \
  --tp 1 \
  --trust-remote-code
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

- Hardware: MI300X GPU (8x)
- Model: mistralai/Ministral-3-14B-Instruct-2512
- Tensor Parallelism: 1
- SGLang Version: 0.5.7

- Model Deployment Command:

```bash Command
sglang serve \
  --model-path mistralai/Ministral-3-14B-Instruct-2512 \
  --tp 1 \
  --trust-remote-code
```

#####  Low Concurrency
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Ministral-3-14B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  65.08
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4218
Request throughput (req/s):              0.15
Input token throughput (tok/s):          93.75
Output token throughput (tok/s):         64.84
Peak output token throughput (tok/s):    151.00
Peak concurrent requests:                2
Total token throughput (tok/s):          158.59
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6505.51
Median E2E Latency (ms):                 3037.37
---------------Time to First Token----------------
Mean TTFT (ms):                          3709.33
Median TTFT (ms):                        53.72
P99 TTFT (ms):                           33320.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.63
Median TPOT (ms):                        6.64
P99 TPOT (ms):                           6.66
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.64
Median ITL (ms):                         6.65
P95 ITL (ms):                            6.75
P99 ITL (ms):                            6.82
Max ITL (ms):                            8.45
==================================================
```

##### Medium Concurrency
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Ministral-3-14B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  31.20
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40783
Request throughput (req/s):              2.56
Input token throughput (tok/s):          1271.38
Output token throughput (tok/s):         1307.82
Peak output token throughput (tok/s):    1760.00
Peak concurrent requests:                22
Total token throughput (tok/s):          2579.20
Concurrency:                             13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5351.07
Median E2E Latency (ms):                 5626.45
---------------Time to First Token----------------
Mean TTFT (ms):                          280.87
Median TTFT (ms):                        68.16
P99 TTFT (ms):                           1194.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.47
Median TPOT (ms):                        10.10
P99 TPOT (ms):                           20.00
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.96
Median ITL (ms):                         9.10
P95 ITL (ms):                            9.87
P99 ITL (ms):                            51.39
Max ITL (ms):                            888.63
==================================================
```

##### High Concurrency
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Ministral-3-14B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  88.75
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    252547
Request throughput (req/s):              5.63
Input token throughput (tok/s):          2815.01
Output token throughput (tok/s):         2846.91
Peak output token throughput (tok/s):    4271.00
Peak concurrent requests:                110
Total token throughput (tok/s):          5661.93
Concurrency:                             93.04
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16514.45
Median E2E Latency (ms):                 15834.45
---------------Time to First Token----------------
Mean TTFT (ms):                          148.57
Median TTFT (ms):                        99.15
P99 TTFT (ms):                           455.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.93
Median TPOT (ms):                        34.73
P99 TPOT (ms):                           38.05
---------------Inter-Token Latency----------------
Mean ITL (ms):                           32.45
Median ITL (ms):                         27.30
P95 ITL (ms):                            71.73
P99 ITL (ms):                            73.45
Max ITL (ms):                            328.10
==================================================
```

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

- Benchmark Command

```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
  --num-shots 8 \
  --num-questions 1316 \
  --parallel 1316
```

**Test Results:**

```text Output
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```
