TensorSharp supports Vulkan backend

Vulkan Backend Release

Due to high Vulkan backend demand, I update TensorSharp and release the initial version of GGML Vulkan backend by leveraging the external GGML project. The native Vulkan backend will be implemented later.

I tested it on Nvidia Geforce RTX 3080 Laptop GPU and Intel(R) UHD Graphics on Windows. They all work. However, I do not have an AMD GPU, so I have no way to get it tested. It's really appreciated if you have an AMD GPU and would like to try it out. Any feedback and comments are welcome.

Benchmark Results

Here is the benchmark I ran to compare with llama.cpp:

Performance Ratio - TensorSharp vs Reference Engines

Geomean of TensorSharp's per-scenario speedup over each reference engine on the same backend, across every scenario both engines ran (single-stream, MTP-off). A value > 1.0× means TensorSharp is faster (for decode / prefill throughput) or lower-latency (for TTFT); - = no overlapping cells. Per-scenario ratios are in each model's section below.

Gemma 4 E4B it (Q8_0, dense multimodal) (gemma4-e4b)

Decode throughput (tok/s)

Prefill throughput (tok/s)

Time to first token (ms, lower is better)

Performance ratio - TensorSharp vs reference (> 1.0× = TensorSharp faster)

Decode throughput

Prefill throughput

Time to first token (latency; > 1.0× = TensorSharp lower)

Gemma 4 12B it (QAT UD-Q4_K_XL, dense) (gemma4-12b)

Decode throughput (tok/s)

Prefill throughput (tok/s)

Time to first token (ms, lower is better)

Performance ratio - TensorSharp vs reference (> 1.0× = TensorSharp faster)

Decode throughput

Prefill throughput

Time to first token (latency; > 1.0× = TensorSharp lower)

What is TensorSharp?

TensorSharp is an open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), image edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability (support Cuda, Metal and Vulkan backends). The API is completely compatible with OpenAI and Ollama interface. It has on par performance with llama.cpp.

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implemented CUDA, MLX and GGML backend. The GGML backend refers to the GGML project as an external project, and I build a few fusion operations at a higher level.

I learned a lot from other projects and applied them for TensorSharp, such as:

• Paged KV cache and continuous batching from vLLM
• SSD based cache for MoE model from oMLX
• GGUF quantization from llama.cpp
• Other optimizations for prefill and decode

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can give this project a star on GitHub. Thanks in advance.