Skip to content

feat(optim): add AdamWScheduleFree and AdamWScheduleFree8bit - #2142

Open
shotsan wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
shotsan:feat/schedule-free-8bit
Open

shotsan wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
shotsan:feat/schedule-free-8bit

Conversation

@shotsan

@shotsan shotsan commented Nov 13, 2026

Copy link
Copy Markdown

Closes #1950

Summary

Adds Schedule-Free AdamW optimizers (AdamWScheduleFree, AdamWScheduleFree8bit, and AdamWScheduleFree32bit) to bitsandbytes.optim, implementing the algorithm from "The Road Less Scheduled" (Defazio et al., 2024).

Details

  • 8-bit blockwise state quantization: Stores both optimizer states (z with signed dynamic map, exp_avg_sq with unsigned udynamic map) in uint8 via F.quantize_blockwise / F.dequantize_blockwise when optim_bits == 8 and p.numel() >= min_8bit_size, falling back to float32 for smaller tensors or optim_bits == 32.
  • Train/eval mode switching: Provides optimizer.train() and optimizer.eval() to swap parameters between the interpolated training iterate y and the averaged evaluation iterate x, with a runtime guard if step() is called while in eval mode.
  • FSDP compatibility: Inherits from Optimizer8bit so state_dict() / load_state_dict() wrap quantization tensors under __bnb_optimizer_quant_state__.

Testing

Ran PYTHONPATH=. pytest tests/test_schedule_free.py -v on CPU:

  • 7/7 tests passed (reference numerical equivalence across warmup/weight-decay configs, 8-bit MLP convergence and uint8/absmax state verification, eval()/train() guard, state_dict() round-trip, and argument validation).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8bit Schedule-Free Optimizer

1 participant