[XeonFX]

set_memory_limit caps the memory of the whole Lua state, but a single block can be as large as that cap. For a host running untrusted scripts, this means the cost of one library call (utf8.len, string.packsize, string.lower and so on over the largest string the script can build) is bounded only by the total limit: with a 6 MiB limit, one utf8.len call takes over 10 ms. A smaller per-block limit bounds how large a string, table or buffer can be, and so how much work a single call can do on it.

API

/// Returns previous limit (zero means no limit).
pub fn set_allocation_limit(&self, limit: usize) -> Result<usize>

It mirrors set_memory_limit: 0 removes the limit, and it returns Error::MemoryControlNotAvailable where mlua's allocator is not used (module mode, LuaJIT's own allocator).

• A new block, or a realloc that grows a block, larger than the limit fails like any allocation over the memory limit: Lua raises a memory error (Error::MemoryError on the Rust side, catchable by pcall in Lua).
• Shrinking always succeeds, including a block that was allocated before the limit was set.
• It respects ignore_limit (used for lua_pushcfunction on Lua 5.1/LuaJIT/Luau) and sets limit_reached on Luau, the same way the memory limit does.
• unlikely_memory_error is now false while either limit is set (MemoryState::has_limit), so operations that skip protection when no limit is set stay protected.
• When unset, the limit is stored as usize::MAX, so the allocator does one extra comparison.

Tests

• memory::tests::test_allocation_limit (unit test calling the allocator directly): no limit by default, below/at/one past the limit, realloc growing past the limit fails and keeps the block, shrinking from above the limit succeeds, used memory is unchanged by refused requests, 0 removes the limit.
• tests/memory.rs::test_allocation_limit: strings at half the limit and over it, the largest string that fits grows byte for byte with the limit (checked at +64 so LuaJIT's 4-byte string rounding doesn't matter), a growing table fails, pcall(string.rep, ...) over the limit returns false and the state keeps working, removing the limit restores everything.
• tests/memory.rs::test_allocation_limit_with_memory_limit: each limit applies on its own, and removing the memory limit keeps the allocation limit (and keeps create_string protected rather than aborting).

Ran tests/memory.rs on lua55, lua54, lua53, lua52, lua51, luajit, luau and luau-jit; the full suite (with async,serde,macros,anyhow,userdata-wrappers,send) on lua54 and luau, also with --cfg=force_memory_limit; clippy and nightly rustfmt.