Repository navigation
Rust: convert data extensions to JSON - #23087
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
DCA results on real databases and an assessment of changed results or unexpected regressions have not been provided for the final conversion.
1 open issue
What changed in this PR
This pull request migrates Rust CodeQL library and test data extensions from YAML to JSON for faster loading with CodeQL CLI 2.27.3 while preserving YAML discovery compatibility.
Changes:
- Converted Rust library, generated, and test extensions from YAML to JSON.
- Updated test configuration to load the JSON threat-model extension.
- Updated the Rust library pack to discover both JSON and YAML model files.
| File | Description |
|---|---|
| rust/ql/test/utils-tests/modelgenerator/CaptureSourceModels.ext.yml | Removes the legacy YAML source-model fixture. |
| rust/ql/test/utils-tests/modelgenerator/CaptureSourceModels.ext.json | Adds the JSON source-model fixture replacement. |
| rust/ql/test/utils-tests/modelgenerator/CaptureSinkModels.ext.yml | Removes the legacy YAML sink-model fixture. |
| rust/ql/test/utils-tests/modelgenerator/CaptureSinkModels.ext.json | Adds the JSON sink-model fixture replacement. |
| rust/ql/test/qlpack.yml | Updates the test pack to load the JSON default threat-model extension. |
| rust/ql/test/library-tests/dataflow/models/models.ext.yml | Removes the legacy YAML dataflow-model fixture. |
| rust/ql/test/library-tests/dataflow/models/models.ext.json | Adds the JSON dataflow-model fixture replacement. |
| rust/ql/test/library-tests/dataflow/lambdas/inline-flow.ext.yml | Removes the legacy YAML lambda-flow fixture. |
| rust/ql/test/library-tests/dataflow/lambdas/inline-flow.ext.json | Adds the JSON lambda-flow fixture replacement. |
| rust/ql/test/library-tests/dataflow/barrier/inline-flow.ext.yml | Removes the legacy YAML barrier-flow fixture. |
| rust/ql/test/library-tests/dataflow/barrier/inline-flow.ext.json | Adds the JSON barrier-flow fixture replacement. |
| rust/ql/test/default-threat-models.model.yml | Removes the legacy YAML default threat-model extension. |
| rust/ql/test/default-threat-models.model.json | Adds the JSON default threat-model extension replacement. |
| rust/ql/lib/qlpack.yml | Adds discovery for JSON model extensions while retaining YAML discovery. |
| rust/ql/lib/ext/generated/modelgenerator/smallvec.model.yml | Removes the legacy generated SmallVec YAML models. |
| rust/ql/lib/ext/generated/modelgenerator/smallvec.model.json | Adds the generated SmallVec JSON models. |
| rust/ql/lib/ext/generated/modelgenerator/once_cell.model.yml | Removes the legacy generated once_cell YAML models. |
| rust/ql/lib/ext/generated/modelgenerator/once_cell.model.json | Adds the generated once_cell JSON models. |
| rust/ql/lib/ext/generated/modelgenerator/log.model.yml | Removes the legacy generated log YAML models. |
| rust/ql/lib/ext/generated/modelgenerator/log.model.json | Adds the generated log JSON models. |
| rust/ql/lib/codeql/rust/frameworks/warp.model.yml | Removes the legacy Warp YAML models. |
| rust/ql/lib/codeql/rust/frameworks/warp.model.json | Adds the Warp JSON models. |
| rust/ql/lib/codeql/rust/frameworks/url.model.yml | Removes the legacy URL YAML models. |
| rust/ql/lib/codeql/rust/frameworks/url.model.json | Adds the URL JSON models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/postgres.model.yml | Removes the legacy Tokio PostgreSQL YAML models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/postgres.model.json | Adds the Tokio PostgreSQL JSON models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/net.model.yml | Removes the legacy Tokio networking YAML models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/net.model.json | Adds the Tokio networking JSON models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/io.model.yml | Removes the legacy Tokio I/O YAML models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/io.model.json | Adds the Tokio I/O JSON models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/fs.model.yml | Removes the legacy Tokio filesystem YAML models. |
| rust/ql/lib/codeql/rust/frameworks/tokio/fs.model.json | Adds the Tokio filesystem JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/process.model.yml | Removes the legacy standard-library process YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/process.model.json | Adds the standard-library process JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/net.model.yml | Removes the legacy standard-library networking YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/net.model.json | Adds the standard-library networking JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/io.model.yml | Removes the legacy standard-library I/O YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/io.model.json | Adds the standard-library I/O JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/fs.model.yml | Removes the legacy standard-library filesystem YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/fs.model.json | Adds the standard-library filesystem JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/ffi.model.yml | Removes the legacy standard-library FFI YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/ffi.model.json | Adds the standard-library FFI JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/env.model.yml | Removes the legacy standard-library environment YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/env.model.json | Adds the standard-library environment JSON models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/alloc.model.yml | Removes the legacy standard-library allocation YAML models. |
| rust/ql/lib/codeql/rust/frameworks/stdlib/alloc.model.json | Adds the standard-library allocation JSON models. |
| rust/ql/lib/codeql/rust/frameworks/sqlx.model.yml | Removes the legacy SQLx YAML models. |
| rust/ql/lib/codeql/rust/frameworks/sqlx.model.json | Adds the SQLx JSON models. |
| rust/ql/lib/codeql/rust/frameworks/serde.model.yml | Removes the legacy Serde YAML models. |
| rust/ql/lib/codeql/rust/frameworks/serde.model.json | Adds the Serde JSON models. |
| rust/ql/lib/codeql/rust/frameworks/rustls.model.yml | Removes the legacy Rustls YAML models. |
| rust/ql/lib/codeql/rust/frameworks/rustls.model.json | Adds the Rustls JSON models. |
| rust/ql/lib/codeql/rust/frameworks/rustcrypto/rustcrypto.model.yml | Removes the legacy RustCrypto YAML models. |
| rust/ql/lib/codeql/rust/frameworks/rustcrypto/rustcrypto.model.json | Adds the RustCrypto JSON models. |
| rust/ql/lib/codeql/rust/frameworks/rusqlite.model.yml | Removes the legacy Rusqlite YAML models. |
| rust/ql/lib/codeql/rust/frameworks/rusqlite.model.json | Adds the Rusqlite JSON models. |
| rust/ql/lib/codeql/rust/frameworks/reqwest.model.yml | Removes the legacy Reqwest YAML models. |
| rust/ql/lib/codeql/rust/frameworks/reqwest.model.json | Adds the Reqwest JSON models. |
| rust/ql/lib/codeql/rust/frameworks/regex.model.yml | Removes the legacy Regex YAML models. |
| rust/ql/lib/codeql/rust/frameworks/regex.model.json | Adds the Regex JSON models. |
| rust/ql/lib/codeql/rust/frameworks/postgres.model.yml | Removes the legacy PostgreSQL YAML models. |
| rust/ql/lib/codeql/rust/frameworks/postgres.model.json | Adds the PostgreSQL JSON models. |
| rust/ql/lib/codeql/rust/frameworks/poem.model.yml | Removes the legacy Poem YAML models. |
| rust/ql/lib/codeql/rust/frameworks/poem.model.json | Adds the Poem JSON models. |
| rust/ql/lib/codeql/rust/frameworks/native-tls.model.yml | Removes the legacy native-TLS YAML models. |
| rust/ql/lib/codeql/rust/frameworks/native-tls.model.json | Adds the native-TLS JSON models. |
| rust/ql/lib/codeql/rust/frameworks/mysql.model.yml | Removes the legacy MySQL YAML models. |
| rust/ql/lib/codeql/rust/frameworks/mysql.model.json | Adds the MySQL JSON models. |
| rust/ql/lib/codeql/rust/frameworks/mysql-async.model.yml | Removes the legacy asynchronous MySQL YAML models. |
| rust/ql/lib/codeql/rust/frameworks/mysql-async.model.json | Adds the asynchronous MySQL JSON models. |
| rust/ql/lib/codeql/rust/frameworks/log.model.yml | Removes the legacy logging YAML models. |
| rust/ql/lib/codeql/rust/frameworks/log.model.json | Adds the logging JSON models. |
| rust/ql/lib/codeql/rust/frameworks/libc.model.yml | Removes the legacy libc YAML models. |
| rust/ql/lib/codeql/rust/frameworks/libc.model.json | Adds the libc JSON models. |
| rust/ql/lib/codeql/rust/frameworks/http.model.yml | Removes the legacy HTTP YAML models. |
| rust/ql/lib/codeql/rust/frameworks/http.model.json | Adds the HTTP JSON models. |
| rust/ql/lib/codeql/rust/frameworks/genericarray.model.yml | Removes the legacy GenericArray YAML models. |
| rust/ql/lib/codeql/rust/frameworks/genericarray.model.json | Adds the GenericArray JSON models. |
| rust/ql/lib/codeql/rust/frameworks/futures.model.yml | Removes the legacy Futures YAML models. |
| rust/ql/lib/codeql/rust/frameworks/futures.model.json | Adds the Futures JSON models. |
| rust/ql/lib/codeql/rust/frameworks/empty.model.yml | Removes the legacy empty YAML extension definitions. |
| rust/ql/lib/codeql/rust/frameworks/empty.model.json | Adds the JSON empty extension definitions. |
| rust/ql/lib/codeql/rust/frameworks/cookie.model.yml | Removes the legacy Cookie YAML models. |
| rust/ql/lib/codeql/rust/frameworks/cookie.model.json | Adds the Cookie JSON models. |
| rust/ql/lib/codeql/rust/frameworks/biscotti.model.yml | Removes the legacy Biscotti YAML models. |
| rust/ql/lib/codeql/rust/frameworks/biscotti.model.json | Adds the Biscotti JSON models. |
| rust/ql/lib/codeql/rust/frameworks/axum.model.yml | Removes the legacy Axum YAML models. |
| rust/ql/lib/codeql/rust/frameworks/axum.model.json | Adds the Axum JSON models. |
| rust/ql/lib/codeql/rust/frameworks/asyncstd/net.model.yml | Removes the legacy async-std networking YAML models. |
| rust/ql/lib/codeql/rust/frameworks/asyncstd/net.model.json | Adds the async-std networking JSON models. |
| rust/ql/lib/codeql/rust/frameworks/asyncstd/io.model.yml | Removes the legacy async-std I/O YAML models. |
| rust/ql/lib/codeql/rust/frameworks/asyncstd/io.model.json | Adds the async-std I/O JSON models. |
| rust/ql/lib/codeql/rust/frameworks/asyncstd/fs.model.yml | Removes the legacy async-std filesystem YAML models. |
| rust/ql/lib/codeql/rust/frameworks/asyncstd/fs.model.json | Adds the async-std filesystem JSON models. |
| rust/ql/lib/codeql/rust/frameworks/actix-web.model.yml | Removes the legacy Actix Web YAML models. |
| rust/ql/lib/codeql/rust/frameworks/actix-web.model.json | Adds the Actix Web JSON models. |
| rust/ql/lib/codeql/rust/dataflow/internal/empty.model.yml | Removes the legacy internal empty YAML extension definitions. |
| rust/ql/lib/codeql/rust/dataflow/internal/empty.model.json | Adds the JSON internal empty extension definitions. |
Review effort: Lite (auto)
| "extensible": "sourceModel" | ||
| }, | ||
| "data": [ | ||
| ["tokio::fs::read::read", "ReturnValue.Future.Field[core::result::Result::Ok(0)]", "file", "manual"], |
Contributor
|
For the generated models, shouldn't we also modify the model generator to output JSON? |
Contributor
Author
I assume the generator is the one in |
Contributor
|
I see. Thanks :) |
paldepind
approved these changes
Nov 11, 2026
Contributor
Author
|
That follow-up PR: #23097 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CodeQL CLI 2.27.3 supports JSON data extensions (aka MaD models), and can load them significantly faster than YAML. This PR converts those in Rust's library pack (first commit), along with
$TEST.ext.ymlfiles (second commit). The conversions were automated by an LLM-generated Python script (which I will add in a separate PR).Although JSON is preferred going forward,
rust/ql/lib/qlpack.ymlnow has patterns that will pick up both.model.ymland.model.jsonfiles, so that any YAML files that do get checked in won't get silently dropped.To verify successful conversion in the first commit, I cherry-picked it onto #23086, which has a temporary test that enumerates all tuples from data extensions. The test still passes after conversion.