-
Tempo: Application-aware LLM Serving with Mixed SLO Requirements
Authors:
Wei Zhang,
Zhiyu Wu,
Yi Mu,
Banruo Liu,
Myungjin Lee,
Fan Lai
Abstract:
The integration of Large Language Models (LLMs) into diverse applications, ranging from interactive chatbots and cloud AIOps to intelligent agents, has introduced a wide spectrum of Service Level Objectives (SLOs) for responsiveness. These workloads include latency-sensitive requests focused on per-token latency in streaming chat, throughput-intensive requests that require rapid full responses to…
▽ More
The integration of Large Language Models (LLMs) into diverse applications, ranging from interactive chatbots and cloud AIOps to intelligent agents, has introduced a wide spectrum of Service Level Objectives (SLOs) for responsiveness. These workloads include latency-sensitive requests focused on per-token latency in streaming chat, throughput-intensive requests that require rapid full responses to invoke tools, and collective requests with dynamic dependencies arising from self-reflection or agent-based reasoning. This workload diversity, amplified by unpredictable request information such as response lengths and runtime dependencies, makes existing schedulers inadequate even within their design envelopes.
In this paper, we define service gain as the useful service delivered by completing requests. We observe that as SLO directly reflects the actual performance needs of requests, completing a request much faster than its SLO (e.g., deadline) yields limited additional service gain. Based on this insight, we introduce Tempo, the first systematic SLO-aware scheduler designed to maximize service gain across diverse LLM workloads. Tempo allocates just enough serving bandwidth to meet each SLO, maximizing residual capacity for others best-effort workloads. Instead of assuming request information or none at all, it adopts a hybrid scheduling strategy: using quantile-based response upper bounds and dependency-graph matching for conservative initial estimates, prioritizing requests by service gain density, and refining decisions online as generation progresses. Our evaluation across diverse workloads, including chat, reasoning, and agentic pipelines, shows that Tempo improves end-to-end service gain by up to 8.3$\times$ and achieves up to 10.3$\times$ SLO goodput compared to state-of-the-art designs
△ Less
Submitted 24 April, 2025;
originally announced April 2025.
-
A Dual-Purpose Deep Learning Model for Auscultated Lung and Tracheal Sound Analysis Based on Mixed Set Training
Authors:
Fu-Shun Hsu,
Shang-Ran Huang,
Chang-Fu Su,
Chien-Wen Huang,
Yuan-Ren Cheng,
Chun-Chieh Chen,
Chun-Yu Wu,
Chung-Wei Chen,
Yen-Chun Lai,
Tang-Wei Cheng,
Nian-Jhen Lin,
Wan-Ling Tsai,
Ching-Shiang Lu,
Chuan Chen,
Feipei Lai
Abstract:
Many deep learning-based computerized respiratory sound analysis methods have previously been developed. However, these studies focus on either lung sound only or tracheal sound only. The effectiveness of using a lung sound analysis algorithm on tracheal sound and vice versa has never been investigated. Furthermore, no one knows whether using lung and tracheal sounds together in training a respira…
▽ More
Many deep learning-based computerized respiratory sound analysis methods have previously been developed. However, these studies focus on either lung sound only or tracheal sound only. The effectiveness of using a lung sound analysis algorithm on tracheal sound and vice versa has never been investigated. Furthermore, no one knows whether using lung and tracheal sounds together in training a respiratory sound analysis model is beneficial. In this study, we first constructed a tracheal sound database, HF_Tracheal_V1, containing 10448 15-s tracheal sound recordings, 21741 inhalation labels, 15858 exhalation labels, and 6414 continuous adventitious sound (CAS) labels. HF_Tracheal_V1 and our previously built lung sound database, HF_Lung_V2, were either combined (mixed set), used one after the other (domain adaptation), or used alone to train convolutional neural network bidirectional gate recurrent unit models for inhalation, exhalation, and CAS detection in lung and tracheal sounds. The results revealed that the models trained using lung sound alone performed poorly in tracheal sound analysis and vice versa. However, mixed set training or domain adaptation improved the performance for 1) inhalation and exhalation detection in lung sounds and 2) inhalation, exhalation, and CAS detection in tracheal sounds compared to positive controls (the models trained using lung sound alone and used in lung sound analysis and vice versa). In particular, the model trained on the mixed set had great flexibility to serve two purposes, lung and tracheal sound analyses, at the same time.
△ Less
Submitted 4 January, 2023; v1 submitted 9 July, 2021;
originally announced July 2021.
-
Multi-path Convolutional Neural Networks Efficiently Improve Feature Extraction in Continuous Adventitious Lung Sound Detection
Authors:
Fu-Shun Hsu,
Shang-Ran Huang,
Chien-Wen Huang,
Chun-Chieh Chen,
Yuan-Ren Cheng,
Feipei Lai
Abstract:
We previously established a large lung sound database, HF_Lung_V2 (Lung_V2). We trained convolutional-bidirectional gated recurrent unit (CNN-BiGRU) networks for detecting inhalation, exhalation, continuous adventitious sound (CAS) and discontinuous adventitious sound at the recording level on the basis of Lung_V2. However, the performance of CAS detection was poor due to many reasons, one of whic…
▽ More
We previously established a large lung sound database, HF_Lung_V2 (Lung_V2). We trained convolutional-bidirectional gated recurrent unit (CNN-BiGRU) networks for detecting inhalation, exhalation, continuous adventitious sound (CAS) and discontinuous adventitious sound at the recording level on the basis of Lung_V2. However, the performance of CAS detection was poor due to many reasons, one of which is the highly diversified CAS patterns. To make the original CNN-BiGRU model learn the CAS patterns more effectively and not cause too much computing burden, three strategies involving minimal modifications of the network architecture of the CNN layers were investigated: (1) making the CNN layers a bit deeper by using the residual blocks, (2) making the CNN layers a bit wider by increasing the number of CNN kernels, and (3) separating the feature input into multiple paths (the model was denoted by Multi-path CNN-BiGRU). The performance of CAS segment and event detection were evaluated. Results showed that improvement in CAS detection was observed among all the proposed architecture-modified models. The F1 score for CAS event detection of the proposed models increased from 0.445 to 0.491-0.530, which was deemed significant. However, the Multi-path CNN-BiGRU model outperformed the other models in terms of the number of winning titles (five) in total nine evaluation metrics. In addition, the Multi-path CNN-BiGRU model did not cause extra computing burden (0.97-fold inference time) compared to the original CNN-BiGRU model. Conclusively, the Multi-path CNN layers can efficiently improve the effectiveness of feature extraction and subsequently result in better CAS detection.
△ Less
Submitted 9 July, 2021;
originally announced July 2021.
-
An Update on a Progressively Expanded Database for Automated Lung Sound Analysis
Authors:
Fu-Shun Hsu,
Shang-Ran Huang,
Chien-Wen Huang,
Yuan-Ren Cheng,
Chun-Chieh Chen,
Jack Hsiao,
Chung-Wei Chen,
Feipei Lai
Abstract:
Purpose: We previously established an open-access lung sound database, HF_Lung_V1, and developed deep learning models for inhalation, exhalation, continuous adventitious sound (CAS), and discontinuous adventitious sound (DAS) detection. The amount of data used for training contributes to model accuracy. Herein, we collected larger quantities of data to further improve model performance. Moreover,…
▽ More
Purpose: We previously established an open-access lung sound database, HF_Lung_V1, and developed deep learning models for inhalation, exhalation, continuous adventitious sound (CAS), and discontinuous adventitious sound (DAS) detection. The amount of data used for training contributes to model accuracy. Herein, we collected larger quantities of data to further improve model performance. Moreover, the issues of noisy labels and sound overlapping were explored. Methods: HF_Lung_V1 was expanded to HF_Lung_V2 with a 1.45x increase in the number of audio files. Convolutional neural network-bidirectional gated recurrent unit network models were trained separately using the HF_Lung_V1 (V1_Train) and HF_Lung_V2 (V2_Train) training sets and then tested using the HF_Lung_V1 (V1_Test) and HF_Lung_V2 (V2_Test) test sets, respectively. Segment and event detection performance was evaluated using the F1 scores. Label quality was assessed. Moreover, the overlap ratios between inhalation, exhalation, CAS, and DAS labels were computed. Results: The model trained using V2_Train exhibited improved F1 scores in inhalation, exhalation, and CAS detection on both V1_Test and V2_Test but not in DAS detection. Poor CAS detection was attributed to the quality of CAS labels. DAS detection was strongly influenced by the overlapping of DAS labels with inhalation and exhalation labels. Conclusion: Collecting greater quantities of lung sound data is vital for developing more accurate lung sound analysis models. To build real ground-truth labels, the labels must be reworked; this process is ongoing. Furthermore, a method for addressing the sound overlapping problem in DAS detection must be formulated.
△ Less
Submitted 29 September, 2021; v1 submitted 8 February, 2021;
originally announced February 2021.
-
Benchmarking of eight recurrent neural network variants for breath phase and adventitious sound detection on a self-developed open-access lung sound database-HF_Lung_V1
Authors:
Fu-Shun Hsu,
Shang-Ran Huang,
Chien-Wen Huang,
Chao-Jung Huang,
Yuan-Ren Cheng,
Chun-Chieh Chen,
Jack Hsiao,
Chung-Wei Chen,
Li-Chin Chen,
Yen-Chun Lai,
Bi-Fang Hsu,
Nian-Jhen Lin,
Wan-Lin Tsai,
Yi-Lin Wu,
Tzu-Ling Tseng,
Ching-Ting Tseng,
Yi-Tsun Chen,
Feipei Lai
Abstract:
A reliable, remote, and continuous real-time respiratory sound monitor with automated respiratory sound analysis ability is urgently required in many clinical scenarios-such as in monitoring disease progression of coronavirus disease 2019-to replace conventional auscultation with a handheld stethoscope. However, a robust computerized respiratory sound analysis algorithm has not yet been validated…
▽ More
A reliable, remote, and continuous real-time respiratory sound monitor with automated respiratory sound analysis ability is urgently required in many clinical scenarios-such as in monitoring disease progression of coronavirus disease 2019-to replace conventional auscultation with a handheld stethoscope. However, a robust computerized respiratory sound analysis algorithm has not yet been validated in practical applications. In this study, we developed a lung sound database (HF_Lung_V1) comprising 9,765 audio files of lung sounds (duration of 15 s each), 34,095 inhalation labels, 18,349 exhalation labels, 13,883 continuous adventitious sound (CAS) labels (comprising 8,457 wheeze labels, 686 stridor labels, and 4,740 rhonchi labels), and 15,606 discontinuous adventitious sound labels (all crackles). We conducted benchmark tests for long short-term memory (LSTM), gated recurrent unit (GRU), bidirectional LSTM (BiLSTM), bidirectional GRU (BiGRU), convolutional neural network (CNN)-LSTM, CNN-GRU, CNN-BiLSTM, and CNN-BiGRU models for breath phase detection and adventitious sound detection. We also conducted a performance comparison between the LSTM-based and GRU-based models, between unidirectional and bidirectional models, and between models with and without a CNN. The results revealed that these models exhibited adequate performance in lung sound analysis. The GRU-based models outperformed, in terms of F1 scores and areas under the receiver operating characteristic curves, the LSTM-based models in most of the defined tasks. Furthermore, all bidirectional models outperformed their unidirectional counterparts. Finally, the addition of a CNN improved the accuracy of lung sound analysis, especially in the CAS detection tasks.
△ Less
Submitted 12 July, 2022; v1 submitted 5 February, 2021;
originally announced February 2021.
-
Development of a Respiratory Sound Labeling Software for Training a Deep Learning-Based Respiratory Sound Analysis Model
Authors:
Fu-Shun Hsu,
Chao-Jung Huang,
Chen-Yi Kuo,
Shang-Ran Huang,
Yuan-Ren Cheng,
Jia-Horng Wang,
Yi-Lin Wu,
Tzu-Ling Tzeng,
Feipei Lai
Abstract:
Respiratory auscultation can help healthcare professionals detect abnormal respiratory conditions if adventitious lung sounds are heard. The state-of-the-art artificial intelligence technologies based on deep learning show great potential in the development of automated respiratory sound analysis. To train a deep learning-based model, a huge number of accurate labels of normal breath sounds and ad…
▽ More
Respiratory auscultation can help healthcare professionals detect abnormal respiratory conditions if adventitious lung sounds are heard. The state-of-the-art artificial intelligence technologies based on deep learning show great potential in the development of automated respiratory sound analysis. To train a deep learning-based model, a huge number of accurate labels of normal breath sounds and adventitious sounds are needed. In this paper, we demonstrate the work of developing a respiratory sound labeling software to help annotators identify and label the inhalation, exhalation, and adventitious respiratory sound more accurately and quickly. Our labeling software integrates six features from MATLAB Audio Labeler, and one commercial audio editor, RX7. As of October, 2019, we have labeled 9,765 15-second-long audio files of breathing lung sounds, and accrued 34,095 inhalation labels,18,349 exhalation labels, 13,883 continuous adventitious sounds (CASs) labels and 15,606 discontinuous adventitious sounds (DASs) labels, which are significantly larger than previously published studies. The trained convolutional recurrent neural networks based on these labels showed good performance with F1-scores of 86.0% on inhalation event detection, 51.6% on CASs event detection and 71.4% on DASs event detection. In conclusion, our results show that our proposed respiratory sound labeling software could easily pre-define a label, perform one-click labeling, and overall facilitate the process of accurately labeling. This software helps develop deep learning-based models that require a huge amount of labeled acoustic data.
△ Less
Submitted 4 January, 2021;
originally announced January 2021.
-
Deep Learning Based Segmentation of Various Brain Lesions for Radiosurgery
Authors:
Siang-Ruei Wu,
Hao-Yun Chang,
Florence T Su,
Heng-Chun Liao,
Wanju Tseng,
Chun-Chih Liao,
Feipei Lai,
Feng-Ming Hsu,
Furen Xiao
Abstract:
Semantic segmentation of medical images with deep learning models is rapidly developed. In this study, we benchmarked state-of-the-art deep learning segmentation algorithms on our clinical stereotactic radiosurgery dataset, demonstrating the strengths and weaknesses of these algorithms in a fairly practical scenario. In particular, we compared the model performances with respect to their sampling…
▽ More
Semantic segmentation of medical images with deep learning models is rapidly developed. In this study, we benchmarked state-of-the-art deep learning segmentation algorithms on our clinical stereotactic radiosurgery dataset, demonstrating the strengths and weaknesses of these algorithms in a fairly practical scenario. In particular, we compared the model performances with respect to their sampling method, model architecture, and the choice of loss functions, identifying the suitable settings for their applications and shedding light on the possible improvements.
△ Less
Submitted 22 July, 2020;
originally announced July 2020.
-
A Novel Massive MIMO Beam Domain Channel Model
Authors:
Fan Lai,
Cheng-Xiang Wang,
Jie Huang,
Xiqi Gao,
Fu-Chun Zheng
Abstract:
A novel beam domain channel model (BDCM) for massive multiple-input multiple-output (MIMO) communication systems has been proposed in this paper. The near-field effect and spherical wavefront are firstly assumed in the proposed model, which is different from the conventional BDCM for MIMO based on the far-field effect and plane wavefront assumption. The proposed novel BDCM is the transformation of…
▽ More
A novel beam domain channel model (BDCM) for massive multiple-input multiple-output (MIMO) communication systems has been proposed in this paper. The near-field effect and spherical wavefront are firstly assumed in the proposed model, which is different from the conventional BDCM for MIMO based on the far-field effect and plane wavefront assumption. The proposed novel BDCM is the transformation of an existing geometry-based stochastic model (GBSM) from the antenna domain into beam domain. The space-time non-stationarity is also modeled in the novel BDCM. Moreover, the comparison of computational complexity for both models is studied. Based on the numerical analysis, comparison of cluster-level statistical properties between the proposed BDCM and existing GBSM has shown that there exists little difference in the space, time, and frequency correlation properties for two models. Also, based on the simulation, coherence bandwidths of the two models in different scenarios are almost the same. The computational complexity of the novel BDCM is much lower than the existing GBSM. It can be observed that the proposed novel BDCM has similar statistical properties to the existing GBSM at the clusterlevel. The proposed BDCM has less complexity and is therefore more convenient for information theory and signal processing research than the conventional GBSMs.
△ Less
Submitted 31 January, 2020;
originally announced February 2020.
