Skip to content

feat(optim): add Cautious AdamW (CAdamW, CAdamW8bit, CAdamW32bit) - #2143

Open
shotsan wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
shotsan:feat/cautious-adamw-8bit
Open

shotsan wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
shotsan:feat/cautious-adamw-8bit

Conversation

@shotsan

@shotsan shotsan commented Nov 14, 2026

Copy link
Copy Markdown

Closes #1430

Summary

Adds 8-bit and 32-bit Cautious AdamW optimizers (CAdamW, CAdamW8bit, and CAdamW32bit) to bitsandbytes.optim, implementing the C-AdamW algorithm from "Cautious Optimizers: Improving Training with One Line of Code" (Liang et al., 2024; https://github.com/kyleliang919/C-Optim).

Details

  • Cautious alignment mask: Applies mask = (exp_avg * grad > 0).to(float32) rescaled by 1 / clamp(mean(mask), min=1e-3) to the AdamW update step.
  • 8-bit blockwise state quantization: Stores first and second moments (exp_avg with signed dynamic map, exp_avg_sq with unsigned udynamic map) in uint8 via F.quantize_blockwise / F.dequantize_blockwise when optim_bits == 8 and p.numel() >= min_8bit_size, falling back to float32 for smaller tensors or optim_bits == 32.
  • FSDP compatibility: Inherits from Optimizer8bit so state_dict() / load_state_dict() wrap quantization tensors under __bnb_optimizer_quant_state__.

Testing

Ran PYTHONPATH=. pytest tests/test_c_adamw.py -v on CPU:

  • 4/4 tests passed (reference numerical equivalence with/without weight decay, 8-bit MLP convergence and uint8/absmax state verification, state_dict() round-trip, and argument validation).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8-bit C-Optim optimizers

1 participant