-
A Bundle-based Augmented Lagrangian Framework: Algorithm, Convergence, and Primal-dual Principles
Authors:
Feng-Yi Liao,
Yang Zheng
Abstract:
We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a $\textit{single-loop}$ process. Motivated by the proximal bundle method (PBM), we use a $\textit{bundle}$ of past iterates to approximate the subproblem in ALM to get…
▽ More
We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a $\textit{single-loop}$ process. Motivated by the proximal bundle method (PBM), we use a $\textit{bundle}$ of past iterates to approximate the subproblem in ALM to get a computationally efficient update at each iteration. We establish sub-linear convergences for primal feasibility, primal cost values, and dual iterates under mild assumptions. With further regularity conditions, such as quadratic growth, our algorithm enjoys $\textit{linear}$ convergences. Importantly, this linear convergence can happen for a class of conic optimization problems, including semidefinite programs. Our proof techniques leverage deep connections with inexact ALM and primal-dual principles with PBM.
△ Less
Submitted 12 February, 2025;
originally announced February 2025.
-
Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence
Authors:
Feng-Yi Liao,
Lijun Ding,
Yang Zheng
Abstract:
Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates…
▽ More
Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new $\textit{quadratic growth}$ and $\textit{error bound}$ properties for primal and dual SDPs under the strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization.
△ Less
Submitted 30 October, 2024;
originally announced October 2024.
-
Large-Language-Model Enabled Semantic Communication Systems
Authors:
Zhenyi Wang,
Li Zou,
Shengyun Wei,
Kai Li,
Feifan Liao,
Haibo Mi,
Rongxuan Lai
Abstract:
Large language models (LLMs) have recently demonstrated state-of-the-art performance across various natural language processing (NLP) tasks, achieving near-human levels in multiple language understanding challenges and aligning closely with the core principles of semantic communication. Inspired by LLMs' advancements in semantic processing, we propose an innovative LLM-enabled semantic communicati…
▽ More
Large language models (LLMs) have recently demonstrated state-of-the-art performance across various natural language processing (NLP) tasks, achieving near-human levels in multiple language understanding challenges and aligning closely with the core principles of semantic communication. Inspired by LLMs' advancements in semantic processing, we propose an innovative LLM-enabled semantic communication system framework, named LLM-SC, that applies LLMs directly to the physical layer coding and decoding for the first time. By analyzing the relationship between the training process of LLMs and the optimization objectives of semantic communication, we propose training a semantic encoder through LLMs' tokenizer training and establishing a semantic knowledge base via the LLMs' unsupervised pre-training process. This knowledge base aids in constructing the optimal decoder by providing the prior probability of the transmitted language sequence. Based on this foundation, we derive the optimal decoding criterion for the receiver and introduce the beam search algorithm to further reduce the complexity. Furthermore, we assert that existing LLMs can be employed directly for LLM-SC without additional re-training or fine-tuning. Simulation results demonstrate that LLM-SC outperforms classical DeepSC at signal-to-noise ratios (SNR) exceeding 3 dB, enabling error-free transmission of semantic information under high SNR, which is unattainable by DeepSC. In addition to semantic-level performance, LLM-SC demonstrates compatibility with technical-level performance, achieving approximately 8 dB coding gain for a bit error ratio (BER) of $10^{-3}$ without any channel coding while maintaining the same joint source-channel coding rate as traditional communication systems.
△ Less
Submitted 5 July, 2025; v1 submitted 19 July, 2024;
originally announced July 2024.
-
Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods
Authors:
Feng-Yi Liao,
Lijun Ding,
Yang Zheng
Abstract:
Many practical optimization problems lack strong convexity. Fortunately, recent studies have revealed that first-order algorithms also enjoy linear convergences under various weaker regularity conditions. While the relationship among different conditions for convex and smooth functions is well-understood, it is not the case for the nonsmooth setting. In this paper, we go beyond convexity and smoot…
▽ More
Many practical optimization problems lack strong convexity. Fortunately, recent studies have revealed that first-order algorithms also enjoy linear convergences under various weaker regularity conditions. While the relationship among different conditions for convex and smooth functions is well-understood, it is not the case for the nonsmooth setting. In this paper, we go beyond convexity and smoothness, and clarify the connections among common regularity conditions in the class of weakly convex functions, including $\textit{strong convexity}$, $\textit{restricted secant inequality}$, $\textit{subdifferential error bound}$, $\textit{Polyak-Łojasiewicz inequality}$, and $\textit{quadratic growth}$. In addition, using these regularity conditions, we present a simple and modular proof for the linear convergence of the proximal point method (PPM) for convex and weakly convex optimization problems. The linear convergence also holds when the subproblems of PPM are solved inexactly with a proper control of inexactness.
△ Less
Submitted 13 August, 2024; v1 submitted 27 December, 2023;
originally announced December 2023.
-
Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning
Authors:
Feng-Ting Liao,
Yung-Chieh Chan,
Yi-Chang Chen,
Chan-Jan Hsu,
Da-shan Shiu
Abstract:
In this work, we propose a method to create domain-sensitive speech recognition models that utilize textual domain information by conditioning its generation on a given text prompt. This is accomplished by fine-tuning a pre-trained, end-to-end model (Whisper) to learn from demonstrations with prompt examples. We show that this ability can be generalized to different domains and even various prompt…
▽ More
In this work, we propose a method to create domain-sensitive speech recognition models that utilize textual domain information by conditioning its generation on a given text prompt. This is accomplished by fine-tuning a pre-trained, end-to-end model (Whisper) to learn from demonstrations with prompt examples. We show that this ability can be generalized to different domains and even various prompt contexts, with our model gaining a Word Error Rate (WER) reduction of up to 33% on unseen datasets from various domains, such as medical conversation, air traffic control communication, and financial meetings. Considering the limited availability of audio-transcript pair data, we further extend our method to text-only fine-tuning to achieve domain sensitivity as well as domain adaptation. We demonstrate that our text-only fine-tuned model can also attend to various prompt contexts, with the model reaching the most WER reduction of 29% on the medical conversation dataset.
△ Less
Submitted 5 October, 2023; v1 submitted 18 July, 2023;
originally announced July 2023.
-
An Overview and Comparison of Spectral Bundle Methods for Primal and Dual Semidefinite Programs
Authors:
Feng-Yi Liao,
Lijun Ding,
Yang Zheng
Abstract:
The spectral bundle method developed by Helmberg and Rendl is well-established for solving large-scale semidefinite programs (SDPs) in the dual form, especially when the SDPs admit $\textit{low-rank primal solutions}$. Under mild regularity conditions, a recent result by Ding and Grimmer has established fast linear convergence rates when the bundle method captures…
▽ More
The spectral bundle method developed by Helmberg and Rendl is well-established for solving large-scale semidefinite programs (SDPs) in the dual form, especially when the SDPs admit $\textit{low-rank primal solutions}$. Under mild regularity conditions, a recent result by Ding and Grimmer has established fast linear convergence rates when the bundle method captures $\textit{the rank of primal solutions}$. In this paper, we present an overview and comparison of spectral bundle methods for solving both $\textit{primal}$ and $\textit{dual}$ SDPs. In particular, we introduce a new family of spectral bundle methods for solving SDPs in the $\textit{primal}$ form. The algorithm developments are parallel to those by Helmberg and Rendl, mirroring the elegant duality between primal and dual SDPs. The new family of spectral bundle methods also achieves linear convergence rates for primal feasibility, dual feasibility, and duality gap when the algorithm captures $\textit{the rank of the dual solutions}$. Therefore, the original spectral bundle method by Helmberg and Rendl is well-suited for SDPs with $\textit{low-rank primal solutions}$, while on the other hand, our new spectral bundle method works well for SDPs with $\textit{low-rank dual solutions}$. These theoretical findings are supported by a range of large-scale numerical experiments. Finally, we demonstrate that our new spectral bundle method achieves state-of-the-art efficiency and scalability for solving polynomial optimization compared to a set of baseline solvers $\textsf{SDPT3}$, $\textsf{MOSEK}$, $\textsf{CDCS}$, and $\textsf{SDPNAL+}$.
△ Less
Submitted 14 July, 2023;
originally announced July 2023.
-
Iterative Inner/outer Approximations for Scalable Semidefinite Programs using Block Factor-width-two Matrices
Authors:
Feng-Yi Liao,
Yang Zheng
Abstract:
In this paper, we propose iterative inner/outer approximations based on a recent notion of block factor-width-two matrices for solving semidefinite programs (SDPs). Our inner/outer approximating algorithms generate a sequence of upper/lower bounds of increasing accuracy for the optimal SDP cost. The block partition in our algorithms offers flexibility in terms of both numerical efficiency and solu…
▽ More
In this paper, we propose iterative inner/outer approximations based on a recent notion of block factor-width-two matrices for solving semidefinite programs (SDPs). Our inner/outer approximating algorithms generate a sequence of upper/lower bounds of increasing accuracy for the optimal SDP cost. The block partition in our algorithms offers flexibility in terms of both numerical efficiency and solution quality, which includes the approach of scaled diagonally dominance (SDD) approximation as a special case. We discuss both the theoretical results and numerical implementation in detail. Our main theorems guarantee that the proposed iterative algorithms generate monotonically decreasing upper (increasing lower) bounds. Extensive numerical results confirm our findings.
△ Less
Submitted 29 September, 2022; v1 submitted 14 April, 2022;
originally announced April 2022.
-
Dynamic Placement of Rapidly Deployable Mobile Sensor Robots Using Machine Learning and Expected Value of Information
Authors:
Alice Agogino,
Hae Young Jang,
Vivek Rao,
Ritik Batra,
Felicity Liao,
Rohan Sood,
Irving Fang,
R. Lily Hu,
Emerson Shoichet-Bartus,
John Matranga
Abstract:
Although the Industrial Internet of Things has increased the number of sensors permanently installed in industrial plants, there will be gaps in coverage due to broken sensors or sparse density in very large plants, such as in the petrochemical industry. Modern emergency response operations are beginning to use Small Unmanned Aerial Systems (sUAS) that have the ability to drop sensor robots to pre…
▽ More
Although the Industrial Internet of Things has increased the number of sensors permanently installed in industrial plants, there will be gaps in coverage due to broken sensors or sparse density in very large plants, such as in the petrochemical industry. Modern emergency response operations are beginning to use Small Unmanned Aerial Systems (sUAS) that have the ability to drop sensor robots to precise locations. sUAS can provide longer-term persistent monitoring that aerial drones are unable to provide. Despite the relatively low cost of these assets, the choice of which robotic sensing systems to deploy to which part of an industrial process in a complex plant environment during emergency response remains challenging.
This paper describes a framework for optimizing the deployment of emergency sensors as a preliminary step towards realizing the responsiveness of robots in disaster circumstances. AI techniques (Long short-term memory, 1-dimensional convolutional neural network, logistic regression, and random forest) identify regions where sensors would be most valued without requiring humans to enter the potentially dangerous area. In the case study described, the cost function for optimization considers costs of false-positive and false-negative errors. Decisions on mitigation include implementing repairs or shutting down the plant. The Expected Value of Information (EVI) is used to identify the most valuable type and location of physical sensors to be deployed to increase the decision-analytic value of a sensor network. This method is applied to a case study using the Tennessee Eastman process data set of a chemical plant, and we discuss implications of our findings for operation, distribution, and decision-making of sensors in plant emergency and resilience scenarios.
△ Less
Submitted 15 November, 2021;
originally announced November 2021.
-
Selective Information Passing for MR/CT Image Segmentation
Authors:
Qikui Zhu,
Liang Li,
Jiangnan Hao,
Yunfei Zha,
Yan Zhang,
Yanxiang Cheng,
Fei Liao,
Pingxiang Li
Abstract:
Automated medical image segmentation plays an important role in many clinical applications, which however is a very challenging task, due to complex background texture, lack of clear boundary and significant shape and texture variation between images. Many researchers proposed an encoder-decoder architecture with skip connections to combine low-level feature maps from the encoder path with high-le…
▽ More
Automated medical image segmentation plays an important role in many clinical applications, which however is a very challenging task, due to complex background texture, lack of clear boundary and significant shape and texture variation between images. Many researchers proposed an encoder-decoder architecture with skip connections to combine low-level feature maps from the encoder path with high-level feature maps from the decoder path for automatically segmenting medical images. The skip connections have been shown to be effective in recovering fine-grained details of the target objects and may facilitate the gradient back-propagation. However, not all the feature maps transmitted by those connections contribute positively to the network performance. In this paper, to adaptively select useful information to pass through those skip connections, we propose a novel 3D network with self-supervised function, named selective information passing network (SIP-Net). We evaluate our proposed model on the MICCAI Prostate MR Image Segmentation 2012 Grant Challenge dataset, TCIA Pancreas CT-82 and MICCAI 2017 Liver Tumor Segmentation (LiTS) Challenge dataset. The experimental results across these data sets show that our model achieved improved segmentation results and outperformed other state-of-the-art methods. The source code of this work is available at https://github.com/ahukui/SIPNet.
△ Less
Submitted 10 October, 2020;
originally announced October 2020.
-
Sample Mixed-Based Data Augmentation for Domestic Audio Tagging
Authors:
Shengyun Wei,
Kele Xu,
Dezhi Wang,
Feifan Liao,
Huaimin Wang,
Qiuqiang Kong
Abstract:
Audio tagging has attracted increasing attention since last decade and has various potential applications in many fields. The objective of audio tagging is to predict the labels of an audio clip. Recently deep learning methods have been applied to audio tagging and have achieved state-of-the-art performance, which provides a poor generalization ability on new data. However due to the limited size…
▽ More
Audio tagging has attracted increasing attention since last decade and has various potential applications in many fields. The objective of audio tagging is to predict the labels of an audio clip. Recently deep learning methods have been applied to audio tagging and have achieved state-of-the-art performance, which provides a poor generalization ability on new data. However due to the limited size of audio tagging data such as DCASE data, the trained models tend to result in overfitting of the network. Previous data augmentation methods such as pitch shifting, time stretching and adding background noise do not show much improvement in audio tagging. In this paper, we explore the sample mixed data augmentation for the domestic audio tagging task, including mixup, SamplePairing and extrapolation. We apply a convolutional recurrent neural network (CRNN) with attention module with log-scaled mel spectrum as a baseline system. In our experiments, we achieve an state-of-the-art of equal error rate (EER) of 0.10 on DCASE 2016 task4 dataset with mixup approach, outperforming the baseline system without data augmentation.
△ Less
Submitted 11 August, 2018;
originally announced August 2018.
-
Compact Formulation of the First Evolution Equation for Optimal Control Computation
Authors:
Sheng Zhang,
Fei Liao,
Wei-Qi Qian
Abstract:
The first evolution equation is derived under the Variation Evolving Method (VEM) that seeks optimal solutions with the variation evolution principle. To improve the performance, its compact form is developed. By replacing the states and costates variation evolution with that of the controls, the dimension-reduced Evolution Partial Differential Equation (EPDE) only solves the control variables alo…
▽ More
The first evolution equation is derived under the Variation Evolving Method (VEM) that seeks optimal solutions with the variation evolution principle. To improve the performance, its compact form is developed. By replacing the states and costates variation evolution with that of the controls, the dimension-reduced Evolution Partial Differential Equation (EPDE) only solves the control variables along the variation time to get the optimal solution, and its definite conditions may be arbitrary. With this equation, the scale of the resulting Initial-value Problem (IVP), transformed via the semi-discrete method, is significantly reduced. Illustrative examples are solved and it is shown that the compact form evolution equation outperforms the primary form in the precision, and the efficiency may be higher for the dense discretization. Moreover, in discussing the connections to the classic iteration methods, it is uncovered that the computation scheme of the gradient method is the discrete implementation of the third evolution equation, and the compact form of the first evolution equation is a continuous realization of the Newton type iteration mechanism.
△ Less
Submitted 20 February, 2025; v1 submitted 9 April, 2018;
originally announced April 2018.
-
The Third Evolution Equation for Optimal Control Computation
Authors:
Sheng Zhang,
Fei Liao,
Kai-Feng He
Abstract:
The Variation Evolving Method (VEM) that originates from the continuous-time dynamics stability theory seeks the optimal solutions with variation evolution principle. After establishing the first and the second evolution equations within its frame, the third evolution equation is developed. This equation only solves the control variables along the variation time to get the optimal solution, and it…
▽ More
The Variation Evolving Method (VEM) that originates from the continuous-time dynamics stability theory seeks the optimal solutions with variation evolution principle. After establishing the first and the second evolution equations within its frame, the third evolution equation is developed. This equation only solves the control variables along the variation time to get the optimal solution, and its definite conditions may be arbitrary since the equation can eliminate possible infeasibilities. With this equation, the dimension of the resulting Initial-value Problem (IVP), transformed via the semi-discrete method, is greatly reduced. Therefore it might relieve the computation burden in seeking solutions. Illustrative examples are solved and it is shown that the proposed equation may produce more precise numerical solutions than the second evolution equation, and its computation time may be shorter for the dense discretization.
△ Less
Submitted 25 January, 2025; v1 submitted 11 February, 2018;
originally announced February 2018.
-
Computation of Optimal Control Problems with Terminal Constraint via Modified Evolution Partial Differential Equation
Authors:
Sheng Zhang,
Kai-Feng He,
Fei Liao
Abstract:
The Variation Evolving Method (VEM), which seeks the optimal solutions with the variation evolution principle, is further developed to be more flexible in solving the Optimal Control Problems (OCPs) with terminal constraint. With the first-order stable dynamics to eliminate the infeasibilities, the Modified Evolution Partial Differential Equation (MEPDE) that is valid in the infeasible solution do…
▽ More
The Variation Evolving Method (VEM), which seeks the optimal solutions with the variation evolution principle, is further developed to be more flexible in solving the Optimal Control Problems (OCPs) with terminal constraint. With the first-order stable dynamics to eliminate the infeasibilities, the Modified Evolution Partial Differential Equation (MEPDE) that is valid in the infeasible solution domain is proposed, and a Lyapunov functional is constructed to theoretically ensure its validity. In particular, it is proved that even with the infinite-time convergence dynamics, the violated terminal inequality constraints, which are inactive for the optimal solution, will enter the feasible domain in finite time. Through transforming the MEPDE to the finite-dimensional Initial-value Problem (IVP) with the semi-discrete method, the OCPs may be solved with common Ordinary Differential Equation (ODE) numerical integration methods. Illustrative examples are presented to show the effectiveness of the proposed method.
△ Less
Submitted 29 January, 2018;
originally announced January 2018.
-
Computation of Optimal Control Problems with Terminal Constraint via Variation Evolution
Authors:
Sheng Zhang,
Bo Liao,
Fei Liao
Abstract:
Enlightened from the inverse consideration of the stable continuous-time dynamics evolution, the Variation Evolving Method (VEM) analogizes the optimal solution to the equilibrium point of an infinite-dimensional dynamic system and solves it in an asymptotically evolving way. In this paper, the compact version of the VEM is further developed for the computation of Optimal Control Problems (OCPs) w…
▽ More
Enlightened from the inverse consideration of the stable continuous-time dynamics evolution, the Variation Evolving Method (VEM) analogizes the optimal solution to the equilibrium point of an infinite-dimensional dynamic system and solves it in an asymptotically evolving way. In this paper, the compact version of the VEM is further developed for the computation of Optimal Control Problems (OCPs) with terminal constraint. The corresponding Evolution Partial Differential Equation (EPDE), which describes the variation motion towards the optimal solution, is derived, and the costate-free optimality conditions are established. The explicit analytic expressions of the costates and the Lagrange multipliers adjoining the terminal constraint, related to the states and the control variables, are presented. With the semi-discrete method in the field of PDE numerical calculation, the EPDE is discretized as finite-dimensional Initial-value Problems (IVPs) to be solved, with common Ordinary Differential Equation (ODE) numerical integration methods.
△ Less
Submitted 2 January, 2018;
originally announced January 2018.
-
Aircraft trajectory control with feedback linearization for general nonlinear system
Authors:
Sheng Zhang,
Fei Liao,
Yanqing Chen,
Kaifeng He
Abstract:
The feedback linearization method is further developed for the controller design on general nonlinear systems. Through the Lyapunov stability theory, the intractable nonlinear implicit algebraic control equations are effectively solved, and the asymptotically tracking performance is guaranteed. Moreover, it is proved that the controller may be used in an inverse-free version to the set-point contr…
▽ More
The feedback linearization method is further developed for the controller design on general nonlinear systems. Through the Lyapunov stability theory, the intractable nonlinear implicit algebraic control equations are effectively solved, and the asymptotically tracking performance is guaranteed. Moreover, it is proved that the controller may be used in an inverse-free version to the set-point control. With this method, a nonlinear aircraft outer-loop trajectory controller is developed. For the concern regarding the controller's robustness, the integral control technique is combined to counteract the adverse effect from modeling errors. Simulation results verify the well performance of the proposed controller.
△ Less
Submitted 28 December, 2017;
originally announced December 2017.
-
Variation Evolving for Optimal Control Computation, a Compact Way
Authors:
Sheng Zhang,
Jiang-Tao Huang,
Kai-Feng He,
Fei Liao
Abstract:
A compact version of the variation evolving method (VEM) is developed in the primal variable space for optimal control computation. Following the idea that originates from the Lyapunov continuous-time dynamics stability theory in the control field, the optimal solution is analogized to the stable equilibrium point of a dynamic system and obtained asymptotically through the variation motion. With t…
▽ More
A compact version of the variation evolving method (VEM) is developed in the primal variable space for optimal control computation. Following the idea that originates from the Lyapunov continuous-time dynamics stability theory in the control field, the optimal solution is analogized to the stable equilibrium point of a dynamic system and obtained asymptotically through the variation motion. With the introduction of a virtual dimension, namely the variation time, the evolution partial differential equation (EPDE), which seeks the optimal solution with a theoretical guarantee, is developed for the optimal control problem (OCP) with free terminal states, and the equivalent optimality conditions with no employment of costates are established in the primal space. These conditions show that the optimal feedback control law is generally not analytically available because the optimal control is related to the future states. Since the derived EPDE is suitable to be computed with the semi-discrete method in the field of PDE numerical calculation, the optimal solution may be obtained by solving the resulting finite-dimensional initial-value problem (IVP).
△ Less
Submitted 21 November, 2020; v1 submitted 5 September, 2017;
originally announced September 2017.
