Skip to content

Fix silent rule loss when compiling from an advanced file object - #282

Open
Abood059 wants to merge 2 commits into
VirusTotal:masterfrom
Abood059:fix/compile-file-stream
Open

Abood059 wants to merge 2 commits into
VirusTotal:masterfrom
Abood059:fix/compile-file-stream

Conversation

@Abood059

Copy link
Copy Markdown

Summary

Fix a silent failure in yara.compile(file=fp) where reading from an
advanced file descriptor offset caused zero or partial rules to be compiled
without any error or warning.

Root cause

The file != NULL branch used PyObject_AsFileDescriptor + dup() +
fdopen() + yr_compiler_add_file(). Because dup() shares the file
offset with the original descriptor (POSIX), and Python's BufferedReader
prefetches up to 8192 bytes, the C-level read started from an advanced
kernel offset and received empty or partial content.

Fix

Replace the raw-fd path with PyObject_CallMethod(file, "read", NULL) and
pass the buffer to yr_compiler_add_string(). This:

  • Respects the Python stream's logical position (consistent with json.load,
    pickle.load, csv.reader).
  • Works with io.BytesIO (previously raised TypeError).
  • Aligns behavior with yara-x, which already reads via the Python stream.
  • Keeps filepath= and source= branches untouched.

Handles str (via PyUnicode_AsUTF8AndSize), bytes (via
PyBytes_AS_STRING + PyBytes_GET_SIZE), and raises TypeError for other
types. Py_DECREF is called on every exit path; only
yr_compiler_add_string runs with the GIL released.

Tests

  • test_compile_from_advanced_file_object_small
  • test_compile_from_advanced_file_object_large
  • test_compile_from_bytesio

pytest tests/ tests.py -v → 50 passed in 1.78s.

Notes for reviewers

This is a bugfix, not a feature. The public API is unchanged. The diff
is confined to the file != NULL branch of yara-python.c plus the new
test file. testCompileFile in tests.py was switched to 'w+t' mode so
the stream is readable after write() + flush() + seek(0).

Read rules via Python stream's read() method instead of extracting and
duplicating the raw OS file descriptor. This respects Python stream semantics
and logical position (tell()), prevents silent rule loss on advanced file
pointers, and enables in-memory streams such as io.BytesIO.
@google-cla

google-cla Bot commented Nov 14, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant