-
LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models
Authors:
Mohammadreza Nemati,
Zhipeng Huang,
Kevin S. Xu
Abstract:
Some of the simplest, yet most frequently used predictors in statistics and machine learning use weighted linear combinations of features. Such linear predictors can model non-linear relationships between features by adding interaction terms corresponding to the products of all pairs of features. We consider the problem of accurately estimating coefficients for interaction terms in linear predicto…
▽ More
Some of the simplest, yet most frequently used predictors in statistics and machine learning use weighted linear combinations of features. Such linear predictors can model non-linear relationships between features by adding interaction terms corresponding to the products of all pairs of features. We consider the problem of accurately estimating coefficients for interaction terms in linear predictors. We hypothesize that the coefficients for different interaction terms have an approximate low-dimensional structure and represent each feature by a latent vector in a low-dimensional space. This low-dimensional representation can be viewed as a structured regularization approach that further mitigates overfitting in high-dimensional settings beyond standard regularizers such as the lasso and elastic net. We demonstrate that our approach, called LIT-LVM, achieves superior prediction accuracy compared to elastic net and factorization machines on a wide variety of simulated and real data, particularly when the number of interaction terms is high compared to the number of samples. LIT-LVM also provides low-dimensional latent representations for features that are useful for visualizing and analyzing their relationships.
△ Less
Submitted 18 June, 2025;
originally announced June 2025.
-
Quantize What Counts: Bit Allocation Insights Informed by Spectral Gaps in Keys and Values
Authors:
Mohsen Hariri,
Alan Luo,
Mohammadreza Nemati,
Lam Nguyen,
Shaochen Zhong,
Qifan Wang,
Xia Hu,
Xiaotian Han,
Vipin Chaudhary
Abstract:
Large Language Models (LLMs) have introduced significant advancements to the capabilities of Natural Language Processing (NLP) in recent years. However, as these models continue to scale in size, memory constraints pose substantial challenge. Key and Value cache (KV cache) quantization has been well-documented as a promising solution to this limitation. In this work, we provide two novel theorems…
▽ More
Large Language Models (LLMs) have introduced significant advancements to the capabilities of Natural Language Processing (NLP) in recent years. However, as these models continue to scale in size, memory constraints pose substantial challenge. Key and Value cache (KV cache) quantization has been well-documented as a promising solution to this limitation. In this work, we provide two novel theorems aimed at enhancing KV quantization methods. Our first theorem, termed Key-Value Norm Disparity, states that the key weight matrices by nature carry richer information compared to the value weight matrices, as evidenced by higher spectral and Frobenius norms across most of the layers. Our second theorem, Key-Driven Quantization, posits that prioritizing the quantization precision of keys over values induces significant improvements to the overall quantization performance. In particular, assigning greater precision to the keys compared to the values achieves a higher degree of precision reduction with minimal impact on model accuracy. We validate these theorems through theory and extensive experiments on several state-of-the-art LLM architectures and benchmarks. These findings offer valuable guidelines for improving KV cache quantization strategies, facilitating more efficient memory utilization without compromising model performance across diverse NLP tasks. Source code is available at https://github.com/mohsenhariri/spectral-kv.
△ Less
Submitted 23 May, 2025; v1 submitted 20 February, 2025;
originally announced February 2025.
-
Energy-Efficient UAV-Assisted IoT Data Collection via TSP-Based Solution Space Reduction
Authors:
Sivaram Krishnan,
Mahyar Nemati,
Seng W. Loke,
Jihong Park,
Jinho Choi
Abstract:
This paper presents a wireless data collection framework that employs an unmanned aerial vehicle (UAV) to efficiently gather data from distributed IoT sensors deployed in a large area. Our approach takes into account the non-zero communication ranges of the sensors to optimize the flight path of the UAV, resulting in a variation of the Traveling Salesman Problem (TSP). We prove mathematically that…
▽ More
This paper presents a wireless data collection framework that employs an unmanned aerial vehicle (UAV) to efficiently gather data from distributed IoT sensors deployed in a large area. Our approach takes into account the non-zero communication ranges of the sensors to optimize the flight path of the UAV, resulting in a variation of the Traveling Salesman Problem (TSP). We prove mathematically that the optimal waypoints for this TSP-variant problem are restricted to the boundaries of the sensor communication ranges, greatly reducing the solution space. Building on this finding, we develop a low-complexity UAV-assisted sensor data collection algorithm, and demonstrate its effectiveness in a selected use case where we minimize the total energy consumption of the UAV and sensors by jointly optimizing the UAV's travel distance and the sensors' communication ranges.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
Predicting Kidney Transplant Survival using Multiple Feature Representations for HLAs
Authors:
Mohammadreza Nemati,
Haonan Zhang,
Michael Sloma,
Dulat Bekbolsynov,
Hong Wang,
Stanislaw Stepkowski,
Kevin S. Xu
Abstract:
Kidney transplantation can significantly enhance living standards for people suffering from end-stage renal disease. A significant factor that affects graft survival time (the time until the transplant fails and the patient requires another transplant) for kidney transplantation is the compatibility of the Human Leukocyte Antigens (HLAs) between the donor and recipient. In this paper, we propose 4…
▽ More
Kidney transplantation can significantly enhance living standards for people suffering from end-stage renal disease. A significant factor that affects graft survival time (the time until the transplant fails and the patient requires another transplant) for kidney transplantation is the compatibility of the Human Leukocyte Antigens (HLAs) between the donor and recipient. In this paper, we propose 4 new biologically-relevant feature representations for incorporating HLA information into machine learning-based survival analysis algorithms. We evaluate our proposed HLA feature representations on a database of over 100,000 transplants and find that they improve prediction accuracy by about 1%, modest at the patient level but potentially significant at a societal level. Accurate prediction of survival times can improve transplant survival outcomes, enabling better allocation of donors to recipients and reducing the number of re-transplants due to graft failure with poorly matched donors.
△ Less
Submitted 5 July, 2022; v1 submitted 4 March, 2021;
originally announced March 2021.
-
Varactor-Based Dynamic Load Modulation of High Power Amplifiers
Authors:
Ali Soltani Tehrani,
Hossein Mashad Nemati,
Haiying Cao,
Thomas Eriksson,
Christian Fager
Abstract:
In this work, dynamic load modulation of high power amplifiers using a varactor-based tunable matching network is presented. The feasibility of dynamic tuning and efficiency enhancement of this technique is demonstrated using a modular design approach for two existing high efficiency power amplifiers (PA), a 7-W class-E, and a 10-W class-J power amplifier PA at 1 GHz. For this purpose and for each…
▽ More
In this work, dynamic load modulation of high power amplifiers using a varactor-based tunable matching network is presented. The feasibility of dynamic tuning and efficiency enhancement of this technique is demonstrated using a modular design approach for two existing high efficiency power amplifiers (PA), a 7-W class-E, and a 10-W class-J power amplifier PA at 1 GHz. For this purpose and for each of the PAs, a simple quasi-static inverse model is developed allowing an efficiency-optimized control of the PA and the varactor-based tunable matching network. Modulated measurements using a single carrier WCDMA signal with 11.3 dB peak-to-average ratio (PAR) indicate about 10 to 14 percentage units improvements in the average power-added efficiency (PAE) for the complete architecture.
△ Less
Submitted 24 October, 2012; v1 submitted 12 October, 2012;
originally announced October 2012.
