Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Chat Template Implementation #901

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from
Draft

Initial Chat Template Implementation #901

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a4388e6
initial chat template impl
Mar 1, 2025
be0f462
add phi 3.5 support
Mar 1, 2025
c8176e2
remove unnecessary qualification
Mar 3, 2025
a0d74e3
add llama3 chat template
Mar 4, 2025
4863a91
Merge branch 'main' of https://github.com/microsoft/onnxruntime-exten…
Mar 4, 2025
db9f715
fix typo
Mar 4, 2025
9955f0a
add json loading and unit test
Mar 5, 2025
8bf98ef
Merge branch 'main' of https://github.com/microsoft/onnxruntime-exten…
Mar 5, 2025
75aa05b
add deepseek chat template support
Mar 5, 2025
3043d3c
fix llama3 issue
Mar 6, 2025
d08a617
fix deepseek issues
Mar 6, 2025
21d0245
fix eos handing in some cases
Mar 7, 2025
52de34f
add llama 3.3 support
Mar 7, 2025
3a93e4a
Merge branch 'main' of https://github.com/microsoft/onnxruntime-exten…
Mar 7, 2025
7e6e4b6
add base llama 3 support
Mar 8, 2025
c93486e
add llama 2 support
Mar 8, 2025
f48cc44
Merge branch 'main' of https://github.com/microsoft/onnxruntime-exten…
Mar 8, 2025
231093c
add phi vision support and code clean up
Mar 8, 2025
34b8ed7
add phi 3 small and medium support
Mar 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions operators/tokenizer/tokenizer_jsconfig.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ class TokenJsonConfig final {
bos_token_ = "<s>";
eos_token_ = "</s>";
unk_token_ = "<unk>";
chat_template_ = ""; // can add default chat template
return {};
}

Expand All @@ -90,6 +91,8 @@ class TokenJsonConfig final {
parse_token(json_config, "eos_token", eos_token_);
parse_token(json_config, "unk_token", unk_token_);

parse_token(json_config, "chat_template", chat_template_);

auto pad_iter = json_config.find("pad_token");
if (pad_iter != json_config.end() && pad_iter->is_string()) {
pad_token_ = json_config.value("pad_token", "");
Expand Down Expand Up @@ -244,6 +247,8 @@ class TokenJsonConfig final {
std::string unk_token_;
std::string pad_token_;

std::string chat_template_;

AddedTokenMap added_tokens_;

static AddedToken ParseAddedToken(const json& token) {
Expand Down
Loading