C and C++ bindings to Tokenizers by thammegowda · Pull Request #1888 · huggingface/tokenizers

thammegowda · 2025-11-21T09:45:57Z

Adding in bindings for two more languages!

bindings/cpp
bindings/c

C is an intermediate step to bind C++ and Rust: i.e., C++ <--> C <--> Rust.

--

Added tests to c++
Added benchmarks for my sanity checks and the results are as expected.

HuggingFaceDocBuilderDev · 2025-11-21T15:24:42Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ArthurZucker

very open to this! let's make sure we have a big compat with expectations in terms of the funcs we bind

CPP bindings coverage improvement

fixed benchmarks

- Introduced a new submodule for Jinja2Cpp to handle chat template rendering. - Enhanced the C++ bindings to load and apply chat templates from a configuration file. - Added methods to retrieve special tokens and their IDs from the tokenizer configuration. - Updated the CMake configuration to include Jinja2Cpp and link it with the tokenizers_cpp library. - Refactored tests to validate the new chat template functionality and special token handling.

Support chat template in c++ api

thammegowda · 2025-12-01T19:36:40Z

Hi @ArthurZucker I've made some progress with c++ adding more APIs.
But... templating like jinaj2 is giving me a bit of trouble. Wondering how your team is handling templates, e..g. chat_template -- is there a native support in the rust code for chat_template in jinja2 format?

I tried integrating https://github.com/jinja2cpp/Jinja2Cpp/ at c++ bindings layer, but some features are limited and lead to crash (IIRC, negative index like messages[-1])

Any tips/recommendation for handling templates natively would be great.

add chat template jinja rendering with minijinja

thammegowda · 2025-12-07T22:03:07Z

Upon digging https://github.com/huggingface/text-generation-inference codebase, I found it's using minijina in Rust to render chat templates (with some workarounds for unsupported features). So, I've removed jinja rendering at c++ bindings layer, instead moved it to tokenizer core in Rust, using minijinja2. This way, all bindings can access the functionality.

Disclaimer: I'm not proficient in Rust, and most of the code is done by AI agents (though I've tried to closely supervise it/them). Based on my testing, everything seems to work (at least for my usecase and its tests pass).

IvanIsCoding · 2025-12-14T17:03:01Z

I will drop this more as a FYI: check https://github.com/mlc-ai/tokenizers-cpp

There's an existing C++ bindings from Tokenizers that goes through the same Rust -> C -> C++ path. The C++ code also binds more than tokenizers because it includes sentencepiece, but that can be cut off.

I think if you want a more battle tested code, fork tokenizers-cpp/rust into your current bindings/c folder and ship tokenizers_c.h.

Once that is done, fork tokenizers_cpp.h and huggingface_tokenizer.cc into bindings/cpp.

The LICENSE shouldn't be an issue. And I think https://chat.webllm.ai/ has a live deployment of the tokenizer with WASM.

johnmai-dev · 2026-02-26T07:46:09Z

Hi! Is there any plan to merge this PR? I’m currently looking for tokenizers implemented in C.

ArthurZucker · 2026-02-26T10:36:20Z

Actually yeah but I need to take over a little and check a few things but planned for sure

Skip unknown fields in deserialization for experimental wrappers.

ArthurZucker

Late on the review! Confirm we plan on maintaining more bindings, but with a much stricter explicit list of what we maintain.

For chat template its a bit too early, and if it makes it in tokenizer.json we'll have a bigger update / it should be separated from this PR entirely! Its also less prio, but we want to have an easy way to use it with minijinja which is a great lib

ArthurZucker · 2026-03-26T10:07:54Z

+- **Rust** (baseline): 100%
+- **C Bindings**: ~100% (essentially identical to Rust)
+- **C++ Bindings**: 97.6% (only 2.4% slower)
+- **Python**: 66.5% (33.5% slower)


cc @McPatate there's stuff we can do to improve this i'm sure 👁️ 👁️

ArthurZucker · 2026-03-26T10:09:08Z

+/// Tokenizer configuration loaded from tokenizer_config.json
+/// Contains authoritative special token definitions and chat template
+#[derive(Default, Clone)]
+struct TokenizerConfig {
+    bos_token: Option<String>,
+    eos_token: Option<String>,
+    pad_token: Option<String>,
+    unk_token: Option<String>,
+    chat_template: Option<String>,
+    add_bos_token: bool,
+    add_eos_token: bool,
+}


We partially have a PR for #1942 + this is not in the core tokenizers we should not add it !

chat_template potentially we'll add support for it, never add_bos_token as tokenizer.json has that info

ArthurZucker · 2026-03-26T10:10:29Z

+}
+
+impl CTokenizer {
+    fn new_from_file(path: &str, config_path: Option<&str>) -> Option<Self> {


the C binding will be at worst more minimal than the python one, at best the same. So we should not have anything extra like TokenizerConfig

ArthurZucker · 2026-03-26T10:12:26Z

+///
+/// Returns: rendered template string (caller must free with tokenizers_string_free), or null on error
+#[no_mangle]
+pub extern "C" fn tokenizers_apply_chat_template(


for now this should never go here unless its in rust core!

ArthurZucker · 2026-03-26T10:12:54Z

I like the idea of having examples for each bidnings!

ArthurZucker · 2026-03-26T10:15:01Z

let's split this in another PR! 🤗

not 100% sure this makes sense to have in tokenizers

thammegowda added 4 commits October 21, 2025 18:35

add bindings to C and C++

16d61fb

add workflows for cpp bindings

5d91ced

disable cpp tests on windows

af80f8d

add benchmarking for all bindings

7291b96

ArthurZucker reviewed Nov 21, 2025

View reviewed changes

Comment thread bindings/c/src/lib.rs

Comment thread bindings/c/src/lib.rs

ArthurZucker added the Feature Request label Nov 21, 2025

thammegowda added 2 commits November 21, 2025 21:25

bindings/cpp: add missing functions for decoding

759cfe8

bindings/cpp: add batching functions; googletest for testing

f5c5de9

thammegowda marked this pull request as draft November 22, 2025 18:51

thammegowda and others added 8 commits November 22, 2025 10:56

Merge pull request #2 from thammegowda/tg/cpp-fix

a88fb46

CPP bindings coverage improvement

fixed benchmarks

3d076be

Merge pull request #3 from sgowdaks/tg/cpp

5c11039

fixed benchmarks

simplify test resource location

c1e984e

cpp bindings: tests compilation are optional, make creates config files,

011045a

GH workflows for cpp: install cmake 3.x (not the latest i.e., 4.x)

8f2e75a

Merge pull request #4 from thammegowda/tg/cpp-jinja2

71646b8

Support chat template in c++ api

thammegowda and others added 2 commits December 7, 2025 01:45

add chat template jinja rendering with minijinja

1428437

Merge pull request #5 from thammegowda/tg/chat-template

f28fe03

add chat template jinja rendering with minijinja

thammegowda marked this pull request as ready for review December 7, 2025 22:03

thammegowda added 3 commits March 9, 2026 17:04

Add staticlib crate type to Cargo.toml

ff39271

Update release profile settings in Cargo.toml

2be242d

Handle unknown fields in tokenizer deserialization

e7c8f5d

Skip unknown fields in deserialization for experimental wrappers.

ArthurZucker reviewed Mar 26, 2026

View reviewed changes

ArthurZucker added the bindings label Mar 26, 2026

Conversation

thammegowda commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 21, 2025

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

thammegowda commented Dec 1, 2025

Uh oh!

thammegowda commented Dec 7, 2025

Uh oh!

IvanIsCoding commented Dec 14, 2025

Uh oh!

johnmai-dev commented Feb 26, 2026

Uh oh!

ArthurZucker commented Feb 26, 2026

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

thammegowda commented Nov 21, 2025 •

edited

Loading