Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix the json_pointer compiler warnings within its latest release #904

Merged
merged 2 commits into from
Mar 5, 2025
Merged

fix the json_pointer compiler warnings within its latest release #904

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eeb8093
fix the json_pointer warnings within its latest release
wenbingl Mar 4, 2025
0f60929
fix version info in cgmanifest as well
wenbingl Mar 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ThirdPartyNotices.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ blingfire 0831265c1aca95ca02eca5bf1155e4251e545328

_____

dlib v19.24.6
dlib v19.24.7

Boost Software License - Version 1.0 - August 17th, 2003

Expand Down Expand Up @@ -122,7 +122,7 @@ Viatcheslav Ostapenko <[email protected]>

_____

nlohmann/json v3.10.5
nlohmann/json v3.11.3

MIT License

Expand Down
2 changes: 1 addition & 1 deletion cgmanifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
"component": {
"type": "git",
"git": {
"commitHash": "v19.24.6",
"commitHash": "v19.24.7",
"repositoryUrl": "https://github.com/davisking/dlib.git"
}
}
Expand Down
4 changes: 2 additions & 2 deletions cmake/externals/dlib.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FetchContent_Declare(
dlib
URL https://github.com/davisking/dlib/archive/d3520131a6e0e0fb62bc556b0222b45d99caf905.zip
URL_HASH SHA1=26b0eb3063da744a11144ae620b61fd1fb90fb39
URL https://github.com/davisking/dlib/archive/refs/tags/v19.24.7.zip
URL_HASH SHA1=6c63ea576e2b525751b0dead27c6c1139c5100ae
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
SOURCE_SUBDIR not_set
)
Expand Down
7 changes: 5 additions & 2 deletions operators/tokenizer/bpe_kernels.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -656,13 +656,16 @@ void JsonFastTokenizer::LoadSpmModelParams(const json& tok_json) {
for (const auto& step : *decoders_node) {
std::string type = step.value("type", "");
if (type == "Replace") {
std::string target = step.value("/pattern/String"_json_pointer, "");
std::string target = "";
if (step.contains("pattern")) {
target = step["pattern"].value("String", "");
}
if (target == spm_escaped_space) {
json_conf_.spm_model_ = true;
}
}
else if (type == "Strip") {
std::string content = step.value("/content"_json_pointer, "");
std::string content = step.value("content", "");
if (content == " ") {
json_conf_.add_dummy_prefix_ = true;
}
Expand Down
1 change: 0 additions & 1 deletion operators/tokenizer/tokenizer_jsconfig.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ class TokenJsonConfig final {
TokenJsonConfig() {}
~TokenJsonConfig() {}
using json = nlohmann::json;
using json_pointer = nlohmann::json_pointer<std::string>;
std::shared_ptr<json> added_tokens_decoder;

public:
Expand Down