-
DeMo: Decoupled Momentum Optimization
Authors:
Bowen Peng,
Jeffrey Quesnelle,
Diederik P. Kingma
Abstract:
Training large neural networks typically requires sharing gradients between accelerators through specialized high-speed interconnects. Drawing from the signal processing principles of frequency decomposition and energy compaction, we demonstrate that synchronizing full optimizer states and model parameters during training is unnecessary. By decoupling momentum updates and allowing controlled diver…
▽ More
Training large neural networks typically requires sharing gradients between accelerators through specialized high-speed interconnects. Drawing from the signal processing principles of frequency decomposition and energy compaction, we demonstrate that synchronizing full optimizer states and model parameters during training is unnecessary. By decoupling momentum updates and allowing controlled divergence in optimizer states across accelerators, we achieve improved convergence compared to state-of-the-art optimizers. We introduce {\textbf{De}}coupled {\textbf{Mo}}mentum (DeMo), a fused optimizer and data parallel algorithm that reduces inter-accelerator communication requirements by several orders of magnitude. This enables training of large neural networks even with limited network bandwidth and heterogeneous hardware. Our method is topology-agnostic and architecture-independent and supports scalable clock-synchronous distributed training with negligible compute and memory overhead. Empirical results show that models trained with DeMo match or exceed the performance of equivalent models trained with AdamW, while eliminating the need for high-speed interconnects when pre-training large scale foundation models. An open source reference PyTorch implementation is published on GitHub at https://github.com/bloc97/DeMo
△ Less
Submitted 29 November, 2024;
originally announced November 2024.
-
Hermes 3 Technical Report
Authors:
Ryan Teknium,
Jeffrey Quesnelle,
Chen Guang
Abstract:
Instruct (or "chat") tuned models have become the primary way in which most people interact with large language models. As opposed to "base" or "foundation" models, instruct-tuned models are optimized to respond to imperative statements. We present Hermes 3, a neutrally-aligned generalist instruct and tool use model with strong reasoning and creative abilities. Its largest version, Hermes 3 405B,…
▽ More
Instruct (or "chat") tuned models have become the primary way in which most people interact with large language models. As opposed to "base" or "foundation" models, instruct-tuned models are optimized to respond to imperative statements. We present Hermes 3, a neutrally-aligned generalist instruct and tool use model with strong reasoning and creative abilities. Its largest version, Hermes 3 405B, achieves state of the art performance among open weight models on several public benchmarks.
△ Less
Submitted 15 August, 2024;
originally announced August 2024.
-
YaRN: Efficient Context Window Extension of Large Language Models
Authors:
Bowen Peng,
Jeffrey Quesnelle,
Honglu Fan,
Enrico Shippole
Abstract:
Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps…
▽ More
Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. The models fine-tuned using YaRN has been made available and reproduced online up to 128k context length at https://github.com/jquesnelle/yarn
△ Less
Submitted 1 November, 2023; v1 submitted 31 August, 2023;
originally announced September 2023.
-
On the linkability of Zcash transactions
Authors:
Jeffrey Quesnelle
Abstract:
Zcash is a fork of Bitcoin with optional anonymity features. While transparent transactions are fully linkable, shielded transactions use zero-knowledge proofs to obscure the parties and amounts of the transactions. First, we observe various metrics regarding the usage of shielded addresses. Moreover, we show that most coins sent to shielded addresses are later sent back to transparent addresses.…
▽ More
Zcash is a fork of Bitcoin with optional anonymity features. While transparent transactions are fully linkable, shielded transactions use zero-knowledge proofs to obscure the parties and amounts of the transactions. First, we observe various metrics regarding the usage of shielded addresses. Moreover, we show that most coins sent to shielded addresses are later sent back to transparent addresses. We then search for round-trip transactions, where the same, or nearly the same number of coins are sent from a transparent address, to a shielded address, and back again to a transparent address. We argue that such behavior exhibits high linkability, especially when they occur nearby temporally. Using this heuristic our analysis matched 31.5% of all coins sent to shielded addresses.
△ Less
Submitted 4 December, 2017;
originally announced December 2017.
