Instructions to use ai-sage/GFusion-10B-A1.8B-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ai-sage/GFusion-10B-A1.8B-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ai-sage/GFusion-10B-A1.8B-bf16", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ai-sage/GFusion-10B-A1.8B-bf16", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("ai-sage/GFusion-10B-A1.8B-bf16", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ai-sage/GFusion-10B-A1.8B-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ai-sage/GFusion-10B-A1.8B-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai-sage/GFusion-10B-A1.8B-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ai-sage/GFusion-10B-A1.8B-bf16

SGLang

How to use ai-sage/GFusion-10B-A1.8B-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ai-sage/GFusion-10B-A1.8B-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai-sage/GFusion-10B-A1.8B-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ai-sage/GFusion-10B-A1.8B-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai-sage/GFusion-10B-A1.8B-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ai-sage/GFusion-10B-A1.8B-bf16 with Docker Model Runner:
```
docker model run hf.co/ai-sage/GFusion-10B-A1.8B-bf16
```

GFusion-10B-A1.8B

GFusion-10B-A1.8B is an experimental instruction-tuned diffusion language model trained by adapting GigaChat3-10B-A1.8B-base to block diffusion generation, followed by context extension, SFT, and confidence tuning.

GFusion uses a block size of 32 tokens and performs decoding with entropy-bounded sampling. In contrast to standard autoregressive generation, the model iteratively refines partially masked token blocks. This allows it to finalize multiple tokens in a single forward pass and provides a controllable trade-off between generation quality and decoding speed.

For architecture details, please refer to the GigaChat3-10B-A1.8B-base model card.

More details about GFusion are available in the Habr article.

Inference

Single-user (concurrency=1) decode throughput comparison using aiperf + sglang.

Benchmarks

Benchmark	GFusion 10B-A1.8B	GFusion + CT 10B-A1.8B	GigaChat3 10B-A1.8B	LLaDA-MoE 7B-A1.4B	LLaDA2.0-mini preview 16B-A1.4B
MMLU	73.38	73.09	71.20	67.18	72.49
MMLU-Pro	58.48	58.04	59.60	44.64	49.22
IFEval	70.38	71.22	66.55	59.33	62.50
GPQA	33.84	32.12	35.02	--	23.74
TruthfulQA	44.84	44.68	45.90	--	56.54
GSM8K	84.48	83.78	85.44	82.41	89.01
MGSM	78.80	79.20	76.80	--	81.44
MATH	68.08	66.86	70.00	58.68	73.50
MBPP+	67.20	65.81	63.60	--	66.67
HumanEval	75.00	71.34	72.56	61.59	80.49
HumanEval+	65.63	63.63	66.46	--	71.95
LCB-Lite	29.10	29.09	31.94	--	29.07
RUBQ	63.49	62.56	65.16	--	16.84
MMLU-RU	67.92	67.74	66.20	--	50.48
IFEval-RU	61.27	64.51	64.19	--	55.75

Quickstart

HF Transformers 🤗

from transformers import AutoTokenizer, AutoModelForCausalLM

device = "auto"
model_path = "ai-sage/GFusion-10B-A1.8B-bf16"

model = AutoModelForCausalLM.from_pretrained(
    model_path, device_map=device, trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_path, device_map=device, trust_remote_code=True
)

messages = [
    {"role": "user", "content": "What are the KKT optimality conditions?"}
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    block_size=32,
    gamma=0.70
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

SGLang

GFusion support is available in SGLang PR #29776:

git clone https://github.com/sgl-project/sglang.git
cd sglang

git fetch origin refs/pull/29776/head:gfusion
git switch gfusion

Install the dependencies:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && source "$HOME/.cargo/env"
pip install --upgrade pip

######### CUDA 13 #########
pip install -e "python[all]"

######### CUDA 12 #########
# 12.6 -> CU=126, KW=cu124
# 12.8 -> CU=128, KW=cu129
# 12.9 -> CU=129, KW=cu129

CU=128 ; KW=cu129
pip install --no-deps --force-reinstall \
  "https://github.com/sgl-project/whl/releases/download/v0.4.4/sglang_kernel-0.4.4+${KW}-cp310-abi3-manylinux2014_$(uname -m).whl"

# sglang + deps
pip install --extra-index-url "https://download.pytorch.org/whl/cu${CU}" -e "python[all]"

# re-pin the wheels to the cu12 build
pip list --format=freeze | awk -F'==' '/-cu13(==|$)/ {print $1}' | xargs -r pip uninstall -y
pip install --index-url "https://download.pytorch.org/whl/cu${CU}" --force-reinstall \
  torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0
pip install --no-deps --force-reinstall \
  "https://github.com/sgl-project/whl/releases/download/v0.1.3/sgl_deep_gemm-0.1.3+cu129-py3-none-manylinux2014_$(uname -m).whl"

Create an EBSampling config file:

# eb_sampling.yaml
gamma: 0.15

Start the server with entropy-bounded sampling and fa3/triton attention backend:

python -m sglang.launch_server \
  --model-path ai-sage/GFusion-10B-A1.8B-bf16 \
  --dllm-algorithm EBSampling \
  --dllm-algorithm-config eb_sampling.yaml \
  --attention-backend <fa3 or triton> \
  --host 0.0.0.0 \
  --port 30000 \
  --dtype float16 \
  --mem-fraction-static 0.88 \
  --cuda-graph-bs-decode 1

Example request for the instruct model:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai-sage/GFusion-10B-A1.8B-bf16",
    "temperature": 0,
    "max_tokens": 512,
    "messages": [
      {
        "role": "user",
        "content": "What are the KKT optimality conditions?"
      }
    ]
  }'

Downloads last month: 1,344

Safetensors

Model size

11B params

Tensor type

BF16

Collection including ai-sage/GFusion-10B-A1.8B-bf16

GFusion

Collection

3 items • Updated 9 days ago • 4