---
title: "Kimi-K2.6"
metatags:
  description: "Deploy Kimi-K2.6 model with SGLang on Ascend NPUs, including single-node, multi-node, and PD disaggregation modes."
---

## Introduction

Kimi-K2.6 is an open-source, native multimodal agentic model developed by Moonshot AI, built through continual
pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It is a Mixture-of-Experts
(MoE) model featuring Multi-head Latent Attention (MLA) and MoE architecture, with 1T total parameters and 32B
active parameters. The model seamlessly integrates vision and language understanding with advanced agentic
capabilities, supporting both instant and thinking modes as well as conversational and agentic paradigms.

This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance
optimization.

This document is validated and written based on **SGLang v0.5.13**. The current model (Kimi-K2.6) is fully supported in
this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use
v0.5.13 or a later version.

## Supported features

| Feature                       | Example usage                                                                                 |
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism            | `--tp-size 16`                                                                                |
| Data Parallelism              | `--dp-size 16`                                                                                |
| Expert Parallelism            | `--ep-size 16 \`<br/>`--moe-a2a-backend deepep \`<br/>`--deepep-mode auto`                     |
| PD Disaggregation             | `--disaggregation-mode prefill \`<br/>`--disaggregation-transfer-backend ascend`              |
| Quantization                  | `--quantization modelslim`                                                                    |
| Chunked Prefill               | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph                     | enabled by default; disable with `--disable-cuda-graph`;<br/>control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` |
| Speculative Decoding          | `--speculative-algorithm EAGLE3 \`<br/>`--speculative-draft-model-path /path/to/draft-model-weights \`<br/>`--speculative-num-steps 4 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 5 \`<br/>`--speculative-draft-model-quantization unquant` |
| Overlap Schedule              | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1`                                                  |
| DP LM Head                    | `--enable-dp-lm-head`                                                                         |
| MLAPO                         | `export SGLANG_NPU_USE_MLAPO=1`                                                               |
| Multistream MoE               | `export SGLANG_NPU_USE_MULTI_STREAM=1`                                                        |

<Note>
The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
mode, and workload. For parameter details, see
[Feature descriptions](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-descriptions); for
recommended configurations for each deployment scenario, see [Best practices](#best-practices).
</Note>

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

- [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16, 595.21GB)
- [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version, 535.91GB)
- [kimi-k2.6-eagle3](https://www.modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3) (EAGLE3 draft model for speculative decoding)
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`.

<Info>
We recommend deploying the W4A8 variant for reduced resource usage and higher throughput.
It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes.
</Info>

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).

## Installation

<Warning>
Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.
</Warning>

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/ascend_npu_faq#8-docker-image-versions-stable-release-vs-daily-build).

<Tabs>
  <Tab title="Atlas 800I A3">

```bash Command
docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3
```

  </Tab>
  <Tab title="Atlas 800I A2">

```bash Command
docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b
```

  </Tab>
</Tabs>

<Tip>
- If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
  container, for example: `-v /path/to/models:/models`.
- Replace `${NAME}` with your own container name or remove `--name` to use default name.
</Tip>

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Kimi K2.6 Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6#single-node-pd-mixed).

### Multi-node online deployment

Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping
prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single
node can provide. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Kimi-K2.6 Best Practice — Multi-node On A3](/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6#multi-node-pd-mixed).

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Kimi-K2.6 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

For multimodal requests (text + image):

```shell
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Kimi-K2.6-w4a8",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
                    {"type": "text", "text": "Describe this image."}
                ]
            }
        ]
    }'
```

Expected result: an HTTP 200 response with a description of the image.

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Kimi-K2.6 Best Practice](/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/ascend_npu_faq).
