-
MultiBalance: Multi-Objective Gradient Balancing in Industrial-Scale Multi-Task Recommendation System
Authors:
Yun He,
Xuxing Chen,
Jiayi Xu,
Renqin Cai,
Yiling You,
Jennifer Cao,
Minhui Huang,
Liu Yang,
Yiqun Liu,
Xiaoyi Liu,
Rong Jin,
Sem Park,
Bo Long,
Xue Feng
Abstract:
In industrial recommendation systems, multi-task learning (learning multiple tasks simultaneously on a single model) is a predominant approach to save training/serving resources and improve recommendation performance via knowledge transfer between the joint learning tasks. However, multi-task learning often suffers from negative transfer: one or several tasks are less optimized than training them…
▽ More
In industrial recommendation systems, multi-task learning (learning multiple tasks simultaneously on a single model) is a predominant approach to save training/serving resources and improve recommendation performance via knowledge transfer between the joint learning tasks. However, multi-task learning often suffers from negative transfer: one or several tasks are less optimized than training them separately. To carefully balance the optimization, we propose a gradient balancing approach called MultiBalance, which is suitable for industrial-scale multi-task recommendation systems. It balances the per-task gradients to alleviate the negative transfer, while saving the huge cost for grid search or manual explorations for appropriate task weights. Moreover, compared with prior work that normally balance the per-task gradients of shared parameters, MultiBalance is more efficient since only requiring to access per-task gradients with respect to the shared feature representations. We conduct experiments on Meta's large-scale ads and feeds multi-task recommendation system, and observe that MultiBalance achieves significant gains (e.g., 0.738% improvement for normalized entropy (NE)) with neutral training cost in Queries Per Second (QPS), which is significantly more efficient than prior methods that balance per-task gradients of shared parameters with 70~80% QPS degradation.
△ Less
Submitted 3 November, 2024;
originally announced November 2024.
-
Preconditioned Nonlinear Conjugate Gradient Method for Real-time Interior-point Hyperelasticity
Authors:
Xing Shen,
Runyuan Cai,
Mengxiao Bi,
Tangjie Lv
Abstract:
The linear conjugate gradient method is widely used in physical simulation, particularly for solving large-scale linear systems derived from Newton's method. The nonlinear conjugate gradient method generalizes the conjugate gradient method to nonlinear optimization, which is extensively utilized in solving practical large-scale unconstrained optimization problems. However, it is rarely discussed i…
▽ More
The linear conjugate gradient method is widely used in physical simulation, particularly for solving large-scale linear systems derived from Newton's method. The nonlinear conjugate gradient method generalizes the conjugate gradient method to nonlinear optimization, which is extensively utilized in solving practical large-scale unconstrained optimization problems. However, it is rarely discussed in physical simulation due to the requirement of multiple vector-vector dot products. Fortunately, with the advancement of GPU-parallel acceleration techniques, it is no longer a bottleneck. In this paper, we propose a Jacobi preconditioned nonlinear conjugate gradient method for elastic deformation using interior-point methods. Our method is straightforward, GPU-parallelizable, and exhibits fast convergence and robustness against large time steps. The employment of the barrier function in interior-point methods necessitates continuous collision detection per iteration to obtain a penetration-free step size, which is computationally expensive and challenging to parallelize on GPUs. To address this issue, we introduce a line search strategy that deduces an appropriate step size in a single pass, eliminating the need for additional collision detection. Furthermore, we simplify and accelerate the computations of Jacobi preconditioning and Hessian-vector product for hyperelasticity and barrier function. Our method can accurately simulate objects comprising over 100,000 tetrahedra in complex self-collision scenarios at real-time speeds.
△ Less
Submitted 6 May, 2024;
originally announced May 2024.
-
Lobachevsky-type Formulas via Fourier Analysis
Authors:
Runze Cai,
Horst Hohberger,
Mian Li
Abstract:
Recently renewed interest in the Lobachevsky-type integrals and interesting identities involving the cardinal sine motivate an extension of the classical Parseval formula involving both periodic and non-periodic functions. We develop a version of the Parseval formula that is often more practical in applications and illustrate its use by extending recent results on Lobachevsky-type integrals. Some…
▽ More
Recently renewed interest in the Lobachevsky-type integrals and interesting identities involving the cardinal sine motivate an extension of the classical Parseval formula involving both periodic and non-periodic functions. We develop a version of the Parseval formula that is often more practical in applications and illustrate its use by extending recent results on Lobachevsky-type integrals. Some previously known, interesting identities are re-proved in a more transparent manner and new formulas for integrals involving cardinal sine and Bessel functions are given.
△ Less
Submitted 16 June, 2020;
originally announced June 2020.
-
State transitions in the Morris-Lecar model under stable Lévy noise
Authors:
Rui Cai,
Yancai Liu,
Jinqiao Duan,
Almaz Tesfay Abebe
Abstract:
This paper considers the state transition of the stochastic Morris-Lecar neuronal model driven by symmetric $α$-stable Lévy noise. The considered system is bistable: a stable fixed point (resting state) and a stable limit cycle (oscillating state), and there is an unstable limit cycle (borderline state) between them. Small disturbances may cause a transition between the two stable states, thus a d…
▽ More
This paper considers the state transition of the stochastic Morris-Lecar neuronal model driven by symmetric $α$-stable Lévy noise. The considered system is bistable: a stable fixed point (resting state) and a stable limit cycle (oscillating state), and there is an unstable limit cycle (borderline state) between them. Small disturbances may cause a transition between the two stable states, thus a deterministic quantity, namely the maximal likely trajectory, is used to analyze the transition phenomena in non-Gaussian stochastic environment. According to the numerical experiment, we find that smaller jumps of the Lévy motion and smaller noise intensity can promote such transition from the sustained oscillating state to the resting state. It also can be seen that larger jumps of the Lévy motion and higher noise intensity are conducive for the transition from the borderline state to the sustained oscillating state. As a comparison, Brownian motion is also taken into account. The results show that whether it is the oscillating state or the borderline state, the system disturbed by Brownian motion will be transferred to the resting state under the selected noise intensity.
△ Less
Submitted 18 June, 2019;
originally announced June 2019.
-
Regional gradient controllability of ultra-slow diffusions involving the Hadamard-Caputo time fractional derivative
Authors:
Ruiyang Cai,
Fudong Ge,
YangQuan Chen,
Chunhai Kou
Abstract:
This paper investigates the regional gradient controllability for ultra-slow diffusion processes governed by the time fractional diffusion systems with a Hadamard-Caputo time fractional derivative. Some necessary and sufficient conditions on regional gradient exact and approximate controllability are first given and proved in detail. Secondly, we propose an approach on how to calculate the minimum…
▽ More
This paper investigates the regional gradient controllability for ultra-slow diffusion processes governed by the time fractional diffusion systems with a Hadamard-Caputo time fractional derivative. Some necessary and sufficient conditions on regional gradient exact and approximate controllability are first given and proved in detail. Secondly, we propose an approach on how to calculate the minimum number of $ω-$strategic actuators. Moreover, the existence, uniqueness and the concrete form of the optimal controller for the system under consideration are presented by employing the Hilbert Uniqueness Method (HUM) among all the admissible ones. Finally, we illustrate our results by an interesting example.
△ Less
Submitted 15 April, 2019;
originally announced April 2019.
-
Lévy noise induced escape in the Morris-Lecar model
Authors:
Yancai Liu,
Rui Cai,
Jinqiao Duan
Abstract:
The phenomenon of an excitable system producing a pulse under external or internal stimulation may be interpreted as a stochastic escape problem. This work addresses this issue by examining the Morris-Lecar neural model driven by symmetric α-stable Lévy motion. Two deterministic indices: the first escape probability and the mean first exit time, are adopted to analyse the state transition in this…
▽ More
The phenomenon of an excitable system producing a pulse under external or internal stimulation may be interpreted as a stochastic escape problem. This work addresses this issue by examining the Morris-Lecar neural model driven by symmetric α-stable Lévy motion. Two deterministic indices: the first escape probability and the mean first exit time, are adopted to analyse the state transition in this stochastic model. We calculate both indices in order to understand the transition from the escape region to the target region, and the area of higher indices in escape region. Additionally, we consider the special case of (Gaussian) Brownian motion to compare with (non-Gaussian) Lévy motion case. Our main results indicate that higher first escape probability promotes the transition, while the mean first exit time reflects the stability of the rest state with the selected escape region. The higher non-Gaussianity index and relatively small noise intensity are more prone to produce spikes. Moreover, by calculating both deterministic indices as functions of noise intensity ratio and non-Gaussianity index, we find that the effect of ion channel noise is more pronounced on the stochastic Morris-Lecar model than noise in the current. This work provides some mathematical understanding about the impact of non-Gaussian, heavy-tailed, burst-like fluctuations on excitable systems such as the Morris-Lecar system.
△ Less
Submitted 17 June, 2019; v1 submitted 27 November, 2018;
originally announced November 2018.
-
A role of random slow manifolds in detecting stochastic bifurcation
Authors:
Ziying He,
Rui Cai,
Jinqiao Duan,
Xianming Liu
Abstract:
We consider the relation for the stochastic equilibrium states between the reduced system on a random slow manifold and the original system. This provides a theoretical basis for the reduction about sophisti- cated detailed models by the random slow manifold without significant damage to the overall qualitative properties. Based on this result, we reveal a role of random slow manifolds in detectin…
▽ More
We consider the relation for the stochastic equilibrium states between the reduced system on a random slow manifold and the original system. This provides a theoretical basis for the reduction about sophisti- cated detailed models by the random slow manifold without significant damage to the overall qualitative properties. Based on this result, we reveal a role of random slow manifolds in detecting stochastic bifurca- tion by an example. The example exhibits a stochastic bifurcation phenomenon and possesses a random slow manifold that carries the stochastic bifurcation information of the system. Specifically, the lower dimensional reduced system on the random slow manifold retains the stochastic bifurcation phenomenon from the original system.
△ Less
Submitted 12 May, 2018;
originally announced May 2018.
