Showing 1–50 of 51 results for author: Venkataramanan, R

NSF-MAP: Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction in Assembly Pipelines

Authors: Chathurangi Shyalika, Renjith Prasad, Fadi El Kalach, Revathy Venkataramanan, Ramtin Zand, Ramy Harik, Amit Sheth

Abstract: In modern assembly pipelines, identifying anomalies is crucial in ensuring product quality and operational efficiency. Conventional single-modality methods fail to capture the intricate relationships required for precise anomaly prediction in complex predictive environments with abundant data and multiple modalities. This paper proposes a neurosymbolic AI and fusion-based approach for multimodal a… ▽ More In modern assembly pipelines, identifying anomalies is crucial in ensuring product quality and operational efficiency. Conventional single-modality methods fail to capture the intricate relationships required for precise anomaly prediction in complex predictive environments with abundant data and multiple modalities. This paper proposes a neurosymbolic AI and fusion-based approach for multimodal anomaly prediction in assembly pipelines. We introduce a time series and image-based fusion model that leverages decision-level fusion techniques. Our research builds upon three primary novel approaches in multimodal learning: time series and image-based decision-level fusion modeling, transfer learning for fusion, and knowledge-infused learning. We evaluate the novel method using our derived and publicly available multimodal dataset and conduct comprehensive ablation studies to assess the impact of our preprocessing techniques and fusion model compared to traditional baselines. The results demonstrate that a neurosymbolic AI-based fusion approach that uses transfer learning can effectively harness the complementary strengths of time series and image data, offering a robust and interpretable approach for anomaly prediction in assembly pipelines with enhanced performance. \noindent The datasets, codes to reproduce the results, supplementary materials, and demo are available at https://github.com/ChathurangiShyalika/NSF-MAP. △ Less

Submitted 9 May, 2025; originally announced May 2025.

Comments: 9 pages, 7 figures, 2 tables, IJCAI 2025 (International Joint Conferences on Artificial Intelligence) Special Track on AI4Tech: AI Enabling Critical Technologies

A Survey on Food Ingredient Substitutions

Authors: Hyunwook Kim, Revathy Venkataramanan, Amit Sheth

Abstract: Diet plays a crucial role in managing chronic conditions and overall well-being. As people become more selective about their food choices, finding recipes that meet dietary needs is important. Ingredient substitution is key to adapting recipes for dietary restrictions, allergies, and availability constraints. However, identifying suitable substitutions is challenging as it requires analyzing the f… ▽ More Diet plays a crucial role in managing chronic conditions and overall well-being. As people become more selective about their food choices, finding recipes that meet dietary needs is important. Ingredient substitution is key to adapting recipes for dietary restrictions, allergies, and availability constraints. However, identifying suitable substitutions is challenging as it requires analyzing the flavor, functionality, and health suitability of ingredients. With the advancement of AI, researchers have explored computational approaches to address ingredient substitution. This survey paper provides a comprehensive overview of the research in this area, focusing on five key aspects: (i) datasets and data sources used to support ingredient substitution research; (ii) techniques and approaches applied to solve substitution problems (iii) contextual information of ingredients considered, such as nutritional content, flavor, and pairing potential; (iv) applications for which substitution models have been developed, including dietary restrictions, constraints, and missing ingredients; (v) safety and transparency of substitution models, focusing on user trust and health concerns. The survey also highlights promising directions for future research, such as integrating neuro-symbolic techniques for deep learning and utilizing knowledge graphs for improved reasoning, aiming to guide advancements in food computation and ingredient substitution. △ Less

Submitted 25 October, 2024; originally announced January 2025.

Comments: 27 pages

Many-User Multiple Access with Random User Activity: Achievability Bounds and Efficient Schemes

Authors: Xiaoqi Liu, Pablo Pascual Cobo, Ramji Venkataramanan

Abstract: We study the Gaussian multiple access channel with random user activity, in the regime where the number of users is proportional to the code length. The receiver may know some statistics about the number of active users, but does not know the exact number nor the identities of the active users. We derive two achievability bounds on the probabilities of misdetection, false alarm, and active user er… ▽ More We study the Gaussian multiple access channel with random user activity, in the regime where the number of users is proportional to the code length. The receiver may know some statistics about the number of active users, but does not know the exact number nor the identities of the active users. We derive two achievability bounds on the probabilities of misdetection, false alarm, and active user error, and propose an efficient CDMA-type scheme whose performance can be compared against these bounds. The first bound is a finite-length result based on Gaussian random codebooks and maximum-likelihood decoding. The second is an asymptotic bound, established using spatially coupled Gaussian codebooks and approximate message passing (AMP) decoding. These bounds can be used to compute an achievable trade-off between the active user density and energy-per-bit, for a fixed user payload and target error rate. The efficient CDMA scheme uses a spatially coupled signature matrix and AMP decoding, and we give rigorous asymptotic guarantees on its error performance. Our analysis provides the first state evolution result for spatially coupled AMP with matrix-valued iterates, which may be of independent interest. Numerical experiments demonstrate the promising error performance of the CDMA scheme for both small and large user payloads, when compared with the two achievability bounds. △ Less

Submitted 2 December, 2024; originally announced December 2024.

Comments: 59 pages, 14 figures, a shorter version appeared at IEEE ISIT 2024

Neurosymbolic AI approach to Attribution in Large Language Models

Authors: Deepa Tilwani, Revathy Venkataramanan, Amit P. Sheth

Abstract: Attribution in large language models (LLMs) remains a significant challenge, particularly in ensuring the factual accuracy and reliability of the generated outputs. Current methods for citation or attribution, such as those employed by tools like Perplexity.ai and Bing Search-integrated LLMs, attempt to ground responses by providing real-time search results and citations. However, so far, these ap… ▽ More Attribution in large language models (LLMs) remains a significant challenge, particularly in ensuring the factual accuracy and reliability of the generated outputs. Current methods for citation or attribution, such as those employed by tools like Perplexity.ai and Bing Search-integrated LLMs, attempt to ground responses by providing real-time search results and citations. However, so far, these approaches suffer from issues such as hallucinations, biases, surface-level relevance matching, and the complexity of managing vast, unfiltered knowledge sources. While tools like Perplexity.ai dynamically integrate web-based information and citations, they often rely on inconsistent sources such as blog posts or unreliable sources, which limits their overall reliability. We present that these challenges can be mitigated by integrating Neurosymbolic AI (NesyAI), which combines the strengths of neural networks with structured symbolic reasoning. NesyAI offers transparent, interpretable, and dynamic reasoning processes, addressing the limitations of current attribution methods by incorporating structured symbolic knowledge with flexible, neural-based learning. This paper explores how NesyAI frameworks can enhance existing attribution models, offering more reliable, interpretable, and adaptable systems for LLMs. △ Less

Submitted 29 September, 2024; originally announced October 2024.

Comments: Six pages, three figures, Paper under review

AssemAI: Interpretable Image-Based Anomaly Detection for Manufacturing Pipelines

Authors: Renjith Prasad, Chathurangi Shyalika, Ramtin Zand, Fadi El Kalach, Revathy Venkataramanan, Ramy Harik, Amit Sheth

Abstract: Anomaly detection in manufacturing pipelines remains a critical challenge, intensified by the complexity and variability of industrial environments. This paper introduces AssemAI, an interpretable image-based anomaly detection system tailored for smart manufacturing pipelines. Utilizing a curated image dataset from an industry-focused rocket assembly pipeline, we address the challenge of imbalance… ▽ More Anomaly detection in manufacturing pipelines remains a critical challenge, intensified by the complexity and variability of industrial environments. This paper introduces AssemAI, an interpretable image-based anomaly detection system tailored for smart manufacturing pipelines. Utilizing a curated image dataset from an industry-focused rocket assembly pipeline, we address the challenge of imbalanced image data and demonstrate the importance of image-based methods in anomaly detection. Our primary contributions include deriving an image dataset, fine-tuning an object detection model YOLO-FF, and implementing a custom anomaly detection model for assembly pipelines. The proposed approach leverages domain knowledge in data preparation, model development and reasoning. We implement several anomaly detection models on the derived image dataset, including a Convolutional Neural Network, Vision Transformer (ViT), and pre-trained versions of these models. Additionally, we incorporate explainability techniques at both user and model levels, utilizing ontology for user-level explanations and SCORE-CAM for in-depth feature and model analysis. Finally, the best-performing anomaly detection model and YOLO-FF are deployed in a real-time setting. Our results include ablation studies on the baselines and a comprehensive evaluation of the proposed system. This work highlights the broader impact of advanced image-based anomaly detection in enhancing the reliability and efficiency of smart manufacturing processes. The image dataset, codes to reproduce the results and additional experiments are available at https://github.com/renjithk4/AssemAI. △ Less

Submitted 16 October, 2024; v1 submitted 4 August, 2024; originally announced August 2024.

Comments: 8 Pages, 6 Figures, 4 Tables, Predictive Models in Engineering Applications special session (MLPMEA )at International Conference on Machine Learning and Applications (ICMLA) 2024

Journal ref: Predictive Models in Engineering Applications special session (MLPMEA) at International Conference on Machine Learning and Applications (ICMLA) 2024

Quantitative Group Testing and Pooled Data in the Linear Regime with Sublinear Tests

Authors: Nelvin Tan, Pablo Pascual Cobo, Ramji Venkataramanan

Abstract: In the pooled data problem, the goal is to identify the categories associated with a large collection of items via a sequence of pooled tests. Each pooled test reveals the number of items in the pool belonging to each category. A prominent special case is quantitative group testing (QGT), which is the case of pooled data with two categories. We consider these problems in the non-adaptive and linea… ▽ More In the pooled data problem, the goal is to identify the categories associated with a large collection of items via a sequence of pooled tests. Each pooled test reveals the number of items in the pool belonging to each category. A prominent special case is quantitative group testing (QGT), which is the case of pooled data with two categories. We consider these problems in the non-adaptive and linear regime, where the fraction of items in each category is of constant order. We propose a scheme with a spatially coupled Bernoulli test matrix and an efficient approximate message passing (AMP) algorithm for recovery. We rigorously characterize its asymptotic performance in both the noiseless and noisy settings, and prove that in the noiseless case, the AMP algorithm achieves almost-exact recovery with a number of tests sublinear in the number of items. For both QGT and pooled data, this is the first efficient scheme that provably achieves recovery in the linear regime with a sublinear number of tests, with performance degrading gracefully in the presence of noise. Numerical simulations illustrate the benefits of the spatially coupled scheme at finite dimensions, showing that it outperforms i.i.d. test designs as well as other recovery algorithms based on convex programming. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Comments: 49 pages, 6 figures

Inferring Change Points in High-Dimensional Regression via Approximate Message Passing

Authors: Gabriel Arpino, Xiaoqi Liu, Julia Gontarek, Ramji Venkataramanan

Abstract: We consider the problem of localizing change points in a generalized linear model (GLM), a model that covers many widely studied problems in statistical learning including linear, logistic, and rectified linear regression. We propose a novel and computationally efficient Approximate Message Passing (AMP) algorithm for estimating both the signals and the change point locations, and rigorously chara… ▽ More We consider the problem of localizing change points in a generalized linear model (GLM), a model that covers many widely studied problems in statistical learning including linear, logistic, and rectified linear regression. We propose a novel and computationally efficient Approximate Message Passing (AMP) algorithm for estimating both the signals and the change point locations, and rigorously characterize its performance in the high-dimensional limit where the number of parameters $p$ is proportional to the number of samples $n$. This characterization is in terms of a state evolution recursion, which allows us to precisely compute performance measures such as the asymptotic Hausdorff error of our change point estimates, and allows us to tailor the algorithm to take advantage of any prior structural information on the signals and change points. Moreover, we show how our AMP iterates can be used to efficiently compute a Bayesian posterior distribution over the change point locations in the high-dimensional limit. We validate our theory via numerical experiments, and demonstrate the favorable performance of our estimators on both synthetic and real data in the settings of linear, logistic, and rectified linear regression. △ Less

Submitted 18 October, 2024; v1 submitted 11 April, 2024; originally announced April 2024.

Comments: 43 pages, 9 figures. A preliminary version of this paper appeared in ICML 2024

Coded Many-User Multiple Access via Approximate Message Passing

Authors: Xiaoqi Liu, Kuan Hsieh, Ramji Venkataramanan

Abstract: We consider communication over the Gaussian multiple-access channel in the regime where the number of users grows linearly with the codelength. In this regime, schemes based on sparse superposition coding can achieve a near-optimal tradeoff between spectral efficiency and signal-to-noise ratio. However, these schemes are feasible only for small values of user payload. This paper investigates effic… ▽ More We consider communication over the Gaussian multiple-access channel in the regime where the number of users grows linearly with the codelength. In this regime, schemes based on sparse superposition coding can achieve a near-optimal tradeoff between spectral efficiency and signal-to-noise ratio. However, these schemes are feasible only for small values of user payload. This paper investigates efficient schemes for larger user payloads, focusing on coded CDMA schemes where each user's information is encoded via a linear code before being modulated with a signature sequence. We propose an efficient approximate message passing (AMP) decoder that can be tailored to the structure of the linear code, and provide an exact asymptotic characterization of its performance. Based on this result, we consider a decoder that integrates AMP and belief propagation and characterize its tradeoff between spectral efficiency and signal-to-noise ratio, for a given target error rate. Simulation results show that the decoder achieves state-of-the-art performance at finite lengths, with a coded CDMA scheme defined using LDPC codes and a spatially coupled matrix of signature sequences. △ Less

Submitted 3 June, 2025; v1 submitted 8 February, 2024; originally announced February 2024.

Comments: 26 pages, 8 figures. To appear in Information Theory, Probability and Statistical Learning: A Festschrift in Honor of Andrew Barron. A shorter version appeared in the Proceedings of IEEE ISIT 2024

Approximate Message Passing with Rigorous Guarantees for Pooled Data and Quantitative Group Testing

Authors: Nelvin Tan, Pablo Pascual Cobo, Jonathan Scarlett, Ramji Venkataramanan

Abstract: In the pooled data problem, the goal is to identify the categories associated with a large collection of items via a sequence of pooled tests. Each pooled test reveals the number of items of each category within the pool. We study an approximate message passing (AMP) algorithm for estimating the categories and rigorously characterize its performance, in both the noiseless and noisy settings. For t… ▽ More In the pooled data problem, the goal is to identify the categories associated with a large collection of items via a sequence of pooled tests. Each pooled test reveals the number of items of each category within the pool. We study an approximate message passing (AMP) algorithm for estimating the categories and rigorously characterize its performance, in both the noiseless and noisy settings. For the noiseless setting, we show that the AMP algorithm is equivalent to one recently proposed by El Alaoui et al. Our results provide a rigorous version of their performance guarantees, previously obtained via non-rigorous techniques. For the case of pooled data with two categories, known as quantitative group testing (QGT), we use the AMP guarantees to compute precise limiting values of the false positive rate and the false negative rate. Though the pooled data problem and QGT are both instances of estimation in a linear model, existing AMP theory cannot be directly applied since the design matrices are binary valued. The key technical ingredient in our analysis is a rigorous asymptotic characterization of AMP for generalized linear models defined via generalized white noise design matrices. This result, established using a recent universality result of Wang et al., is of independent interest. Our theoretical results are validated by numerical simulations. For comparison, we propose estimators based on convex relaxation and iterative thresholding, without providing theoretical guarantees. The simulations indicate that AMP consistently outperforms these estimators. △ Less

Submitted 24 October, 2024; v1 submitted 27 September, 2023; originally announced September 2023.

Comments: 62 pages, 11 figures, appeared in SIAM Journal on Mathematics of Data Science. The simulation results here use a slightly different metric from the journal version; see Remark 4.2

Journal ref: SIAM Journal on Mathematics of Data Science, vol. 6, no. 4, pp. 1027-1054, 2024

Bayes-Optimal Estimation in Generalized Linear Models via Spatial Coupling

Authors: Pablo Pascual Cobo, Kuan Hsieh, Ramji Venkataramanan

Abstract: We consider the problem of signal estimation in a generalized linear model (GLM). GLMs include many canonical problems in statistical estimation, such as linear regression, phase retrieval, and 1-bit compressed sensing. Recent work has precisely characterized the asymptotic minimum mean-squared error (MMSE) for GLMs with i.i.d. Gaussian sensing matrices. However, in many models there is a signific… ▽ More We consider the problem of signal estimation in a generalized linear model (GLM). GLMs include many canonical problems in statistical estimation, such as linear regression, phase retrieval, and 1-bit compressed sensing. Recent work has precisely characterized the asymptotic minimum mean-squared error (MMSE) for GLMs with i.i.d. Gaussian sensing matrices. However, in many models there is a significant gap between the MMSE and the performance of the best known feasible estimators. In this work, we address this issue by considering GLMs defined via spatially coupled sensing matrices. We propose an efficient approximate message passing (AMP) algorithm for estimation and prove that with a simple choice of spatially coupled design, the MSE of a carefully tuned AMP estimator approaches the asymptotic MMSE in the high-dimensional limit. To prove the result, we first rigorously characterize the asymptotic performance of AMP for a GLM with a generic spatially coupled design. This characterization is in terms of a deterministic recursion (`state evolution') that depends on the parameters defining the spatial coupling. Then, using a simple spatially coupled design and a judicious choice of functions for the AMP algorithm, we analyze the fixed points of the resulting state evolution and show that it achieves the asymptotic MMSE. Numerical results for phase retrieval and rectified linear regression show that spatially coupled designs can yield substantially lower MSE than i.i.d. Gaussian designs at finite dimensions when used with AMP algorithms. △ Less

Submitted 28 October, 2024; v1 submitted 15 September, 2023; originally announced September 2023.

Comments: 41 pages, 4 figures. Appeared in the IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 70, no. 11, pp. 8343-8363, November 2024

Spectral Estimators for Structured Generalized Linear Models via Approximate Message Passing

Authors: Yihan Zhang, Hong Chang Ji, Ramji Venkataramanan, Marco Mondelli

Abstract: We consider the problem of parameter estimation in a high-dimensional generalized linear model. Spectral methods obtained via the principal eigenvector of a suitable data-dependent matrix provide a simple yet surprisingly effective solution. However, despite their wide use, a rigorous performance characterization, as well as a principled way to preprocess the data, are available only for unstructu… ▽ More We consider the problem of parameter estimation in a high-dimensional generalized linear model. Spectral methods obtained via the principal eigenvector of a suitable data-dependent matrix provide a simple yet surprisingly effective solution. However, despite their wide use, a rigorous performance characterization, as well as a principled way to preprocess the data, are available only for unstructured (i.i.d.\ Gaussian and Haar orthogonal) designs. In contrast, real-world data matrices are highly structured and exhibit non-trivial correlations. To address the problem, we consider correlated Gaussian designs capturing the anisotropic nature of the features via a covariance matrix $Σ$. Our main result is a precise asymptotic characterization of the performance of spectral estimators. This allows us to identify the optimal preprocessing that minimizes the number of samples needed for parameter estimation. Surprisingly, such preprocessing is universal across a broad set of designs, which partly addresses a conjecture on optimal spectral estimators for rotationally invariant models. Our principled approach vastly improves upon previous heuristic methods, including for designs common in computational imaging and genetics. The proposed methodology, based on approximate message passing, is broadly applicable and opens the way to the precise characterization of spiked matrices and of the corresponding spectral methods in a variety of settings. △ Less

Submitted 3 July, 2024; v1 submitted 28 August, 2023; originally announced August 2023.

Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes

Authors: Revathy Venkataramanan, Kaushik Roy, Kanak Raj, Renjith Prasad, Yuxin Zi, Vignesh Narayanan, Amit Sheth

Abstract: As people become more aware of their food choices, food computation models have become increasingly popular in assisting people in maintaining healthy eating habits. For example, food recommendation systems analyze recipe instructions to assess nutritional contents and provide recipe recommendations. The recent and remarkable successes of generative AI methods, such as auto-regressive large langua… ▽ More As people become more aware of their food choices, food computation models have become increasingly popular in assisting people in maintaining healthy eating habits. For example, food recommendation systems analyze recipe instructions to assess nutritional contents and provide recipe recommendations. The recent and remarkable successes of generative AI methods, such as auto-regressive large language models, can lead to robust methods for a more comprehensive understanding of recipes for healthy food recommendations beyond surface-level nutrition content assessments. In this study, we explore the use of generative AI methods to extend current food computation models, primarily involving the analysis of nutrition and ingredients, to also incorporate cooking actions (e.g., add salt, fry the meat, boil the vegetables, etc.). Cooking actions are notoriously hard to model using statistical learning methods due to irregular data patterns - significantly varying natural language descriptions for the same action (e.g., marinate the meat vs. marinate the meat and leave overnight) and infrequently occurring patterns (e.g., add salt occurs far more frequently than marinating the meat). The prototypical approach to handling irregular data patterns is to increase the volume of data that the model ingests by orders of magnitude. Unfortunately, in the cooking domain, these problems are further compounded with larger data volumes presenting a unique challenge that is not easily handled by simply scaling up. In this work, we propose novel aggregation-based generative AI methods, Cook-Gen, that reliably generate cooking actions from recipes, despite difficulties with irregular data patterns, while also outperforming Large Language Models and other strong baselines. △ Less

Submitted 1 June, 2023; originally announced June 2023.

Mixed Regression via Approximate Message Passing

Authors: Nelvin Tan, Ramji Venkataramanan

Abstract: We study the problem of regression in a generalized linear model (GLM) with multiple signals and latent variables. This model, which we call a matrix GLM, covers many widely studied problems in statistical learning, including mixed linear regression, max-affine regression, and mixture-of-experts. In mixed linear regression, each observation comes from one of $L$ signal vectors (regressors), but we… ▽ More We study the problem of regression in a generalized linear model (GLM) with multiple signals and latent variables. This model, which we call a matrix GLM, covers many widely studied problems in statistical learning, including mixed linear regression, max-affine regression, and mixture-of-experts. In mixed linear regression, each observation comes from one of $L$ signal vectors (regressors), but we do not know which one; in max-affine regression, each observation comes from the maximum of $L$ affine functions, each defined via a different signal vector. The goal in all these problems is to estimate the signals, and possibly some of the latent variables, from the observations. We propose a novel approximate message passing (AMP) algorithm for estimation in a matrix GLM and rigorously characterize its performance in the high-dimensional limit. This characterization is in terms of a state evolution recursion, which allows us to precisely compute performance measures such as the asymptotic mean-squared error. The state evolution characterization can be used to tailor the AMP algorithm to take advantage of any structural information known about the signals. Using state evolution, we derive an optimal choice of AMP `denoising' functions that minimizes the estimation error in each iteration. The theoretical results are validated by numerical simulations for mixed linear regression, max-affine regression, and mixture-of-experts. For max-affine regression, we propose an algorithm that combines AMP with expectation-maximization to estimate intercepts of the model along with the signals. The numerical results show that AMP significantly outperforms other estimators for mixed linear regression and max-affine regression in most parameter regimes. △ Less

Submitted 15 August, 2023; v1 submitted 5 April, 2023; originally announced April 2023.

Comments: 44 pages. To appear in the Journal of Machine Learning Research. A shorter version of this paper appeared in the proceedings of AISTATS 2023

Journal ref: Journal of Machine Learning Research, vol. 24, no. 317, pp. 1-44, 2023

Statistical-Computational Tradeoffs in Mixed Sparse Linear Regression

Authors: Gabriel Arpino, Ramji Venkataramanan

Abstract: We consider the problem of mixed sparse linear regression with two components, where two real $k$-sparse signals $β_1, β_2$ are to be recovered from $n$ unlabelled noisy linear measurements. The sparsity is allowed to be sublinear in the dimension, and additive noise is assumed to be independent Gaussian with variance $σ^2$. Prior work has shown that the problem suffers from a $\frac{k}{SNR^2}$-to… ▽ More We consider the problem of mixed sparse linear regression with two components, where two real $k$-sparse signals $β_1, β_2$ are to be recovered from $n$ unlabelled noisy linear measurements. The sparsity is allowed to be sublinear in the dimension, and additive noise is assumed to be independent Gaussian with variance $σ^2$. Prior work has shown that the problem suffers from a $\frac{k}{SNR^2}$-to-$\frac{k^2}{SNR^2}$ statistical-to-computational gap, resembling other computationally challenging high-dimensional inference problems such as Sparse PCA and Robust Sparse Mean Estimation; here $SNR$ is the signal-to-noise ratio. We establish the existence of a more extensive computational barrier for this problem through the method of low-degree polynomials, but show that the problem is computationally hard only in a very narrow symmetric parameter regime. We identify a smooth information-computation tradeoff between the sample complexity $n$ and runtime for any randomized algorithm in this hard regime. Via a simple reduction, this provides novel rigorous evidence for the existence of a computational barrier to solving exact support recovery in sparse phase retrieval with sample complexity $n = \tilde{o}(k^2)$. Our second contribution is to analyze a simple thresholding algorithm which, outside of the narrow regime where the problem is hard, solves the associated mixed regression detection problem in $O(np)$ time with square-root the number of samples and matches the sample complexity required for (non-mixed) sparse linear regression; this allows the recovery problem to be subsequently solved by state-of-the-art techniques from the dense case. As a special case of our results, we show that this simple algorithm is order-optimal among a large family of algorithms in solving exact signed support recovery in sparse linear regression. △ Less

Submitted 6 July, 2023; v1 submitted 3 March, 2023; originally announced March 2023.

Comments: To appear in Conference on Learning Theory (COLT) 2023

Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models

Authors: Yihan Zhang, Marco Mondelli, Ramji Venkataramanan

Abstract: In a mixed generalized linear model, the objective is to learn multiple signals from unlabeled observations: each sample comes from exactly one signal, but it is not known which one. We consider the prototypical problem of estimating two statistically independent signals in a mixed generalized linear model with Gaussian covariates. Spectral methods are a popular class of estimators which output th… ▽ More In a mixed generalized linear model, the objective is to learn multiple signals from unlabeled observations: each sample comes from exactly one signal, but it is not known which one. We consider the prototypical problem of estimating two statistically independent signals in a mixed generalized linear model with Gaussian covariates. Spectral methods are a popular class of estimators which output the top two eigenvectors of a suitable data-dependent matrix. However, despite the wide applicability, their design is still obtained via heuristic considerations, and the number of samples $n$ needed to guarantee recovery is super-linear in the signal dimension $d$. In this paper, we develop exact asymptotics on spectral methods in the challenging proportional regime in which $n, d$ grow large and their ratio converges to a finite constant. By doing so, we are able to optimize the design of the spectral method, and combine it with a simple linear estimator, in order to minimize the estimation error. Our characterization exploits a mix of tools from random matrices, free probability and the theory of approximate message passing algorithms. Numerical simulations for mixed linear regression and phase retrieval demonstrate the advantage enabled by our analysis over existing designs of spectral methods. △ Less

Submitted 18 April, 2024; v1 submitted 21 November, 2022; originally announced November 2022.

Sketching sparse low-rank matrices with near-optimal sample- and time-complexity using message passing

Authors: Xiaoqi Liu, Ramji Venkataramanan

Abstract: We consider the problem of recovering an $n_1 \times n_2$ low-rank matrix with $k$-sparse singular vectors from a small number of linear measurements (sketch). We propose a sketching scheme and an algorithm that can recover the singular vectors with high probability, with a sample complexity and running time that both depend only on $k$ and not on the ambient dimensions $n_1$ and $n_2$. Our sketch… ▽ More We consider the problem of recovering an $n_1 \times n_2$ low-rank matrix with $k$-sparse singular vectors from a small number of linear measurements (sketch). We propose a sketching scheme and an algorithm that can recover the singular vectors with high probability, with a sample complexity and running time that both depend only on $k$ and not on the ambient dimensions $n_1$ and $n_2$. Our sketching operator, based on a scheme for compressed sensing by Li et al. and Bakshi et al., uses a combination of a sparse parity check matrix and a partial DFT matrix. Our main contribution is the design and analysis of a two-stage iterative algorithm which recovers the singular vectors by exploiting the simultaneously sparse and low-rank structure of the matrix. We derive a nonasymptotic bound on the probability of exact recovery, which holds for any $n_1\times n_2 $ sparse, low-rank matrix. We also show how the scheme can be adapted to tackle matrices that are approximately sparse and low-rank. The theoretical results are validated by numerical simulations and comparisons with existing schemes that use convex programming for recovery. △ Less

Submitted 6 November, 2022; v1 submitted 12 May, 2022; originally announced May 2022.

Comments: 46 pages, 14 figures. A shorter version appeared at the 2022 IEEE International Symposium on Information Theory

ACM Class: E.4

A Rich Recipe Representation as Plan to Support Expressive Multi Modal Queries on Recipe Content and Preparation Process

Authors: Vishal Pallagani, Priyadharsini Ramamurthy, Vedant Khandelwal, Revathy Venkataramanan, Kausik Lakkaraju, Sathyanarayanan N. Aakur, Biplav Srivastava

Abstract: Food is not only a basic human necessity but also a key factor driving a society's health and economic well-being. As a result, the cooking domain is a popular use-case to demonstrate decision-support (AI) capabilities in service of benefits like precision health with tools ranging from information retrieval interfaces to task-oriented chatbots. An AI here should understand concepts in the food do… ▽ More Food is not only a basic human necessity but also a key factor driving a society's health and economic well-being. As a result, the cooking domain is a popular use-case to demonstrate decision-support (AI) capabilities in service of benefits like precision health with tools ranging from information retrieval interfaces to task-oriented chatbots. An AI here should understand concepts in the food domain (e.g., recipes, ingredients), be tolerant to failures encountered while cooking (e.g., browning of butter), handle allergy-based substitutions, and work with multiple data modalities (e.g. text and images). However, the recipes today are handled as textual documents which makes it difficult for machines to read, reason and handle ambiguity. This demands a need for better representation of the recipes, overcoming the ambiguity and sparseness that exists in the current textual documents. In this paper, we discuss the construction of a machine-understandable rich recipe representation (R3), in the form of plans, from the recipes available in natural language. R3 is infused with additional knowledge such as information about allergens and images of ingredients, possible failures and tips for each atomic cooking step. To show the benefits of R3, we also present TREAT, a tool for recipe retrieval which uses R3 to perform multi-modal reasoning on the recipe's content (plan objects - ingredients and cooking tools), food preparation process (plan actions and time), and media type (image, text). R3 leads to improved retrieval efficiency and new capabilities that were hither-to not possible in textual representation. △ Less

Submitted 31 March, 2022; originally announced March 2022.

Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing

Authors: Ramji Venkataramanan, Kevin Kögler, Marco Mondelli

Abstract: We consider the problem of signal estimation in generalized linear models defined via rotationally invariant design matrices. Since these matrices can have an arbitrary spectral distribution, this model is well suited for capturing complex correlation structures which often arise in applications. We propose a novel family of approximate message passing (AMP) algorithms for signal estimation, and r… ▽ More We consider the problem of signal estimation in generalized linear models defined via rotationally invariant design matrices. Since these matrices can have an arbitrary spectral distribution, this model is well suited for capturing complex correlation structures which often arise in applications. We propose a novel family of approximate message passing (AMP) algorithms for signal estimation, and rigorously characterize their performance in the high-dimensional limit via a state evolution recursion. Our rotationally invariant AMP has complexity of the same order as the existing AMP derived under the restrictive assumption of a Gaussian design; our algorithm also recovers this existing AMP as a special case. Numerical results showcase a performance close to Vector AMP (which is conjectured to be Bayes-optimal in some settings), but obtained with a much lower complexity, as the proposed algorithm does not require a computationally expensive singular value decomposition. △ Less

Submitted 9 June, 2022; v1 submitted 8 December, 2021; originally announced December 2021.

Comments: 35 pages, 8 figures, to appear in International Conference on Machine Learning (ICML), 2022

PCA Initialization for Approximate Message Passing in Rotationally Invariant Models

Authors: Marco Mondelli, Ramji Venkataramanan

Abstract: We study the problem of estimating a rank-$1$ signal in the presence of rotationally invariant noise-a class of perturbations more general than Gaussian noise. Principal Component Analysis (PCA) provides a natural estimator, and sharp results on its performance have been obtained in the high-dimensional regime. Recently, an Approximate Message Passing (AMP) algorithm has been proposed as an altern… ▽ More We study the problem of estimating a rank-$1$ signal in the presence of rotationally invariant noise-a class of perturbations more general than Gaussian noise. Principal Component Analysis (PCA) provides a natural estimator, and sharp results on its performance have been obtained in the high-dimensional regime. Recently, an Approximate Message Passing (AMP) algorithm has been proposed as an alternative estimator with the potential to improve the accuracy of PCA. However, the existing analysis of AMP requires an initialization that is both correlated with the signal and independent of the noise, which is often unrealistic in practice. In this work, we combine the two methods, and propose to initialize AMP with PCA. Our main result is a rigorous asymptotic characterization of the performance of this estimator. Both the AMP algorithm and its analysis differ from those previously derived in the Gaussian setting: at every iteration, our AMP algorithm requires a specific term to account for PCA initialization, while in the Gaussian case, PCA initialization affects only the first iteration of AMP. The proof is based on a two-phase artificial AMP that first approximates the PCA estimator and then mimics the true AMP. Our numerical simulations show an excellent agreement between AMP results and theoretical predictions, and suggest an interesting open direction on achieving Bayes-optimal performance. △ Less

Submitted 14 October, 2021; v1 submitted 4 June, 2021; originally announced June 2021.

Comments: 72 pages, 2 figures, appeared in Neural Information Processing Systems (NeurIPS), 2021

A unifying tutorial on Approximate Message Passing

Authors: Oliver Y. Feng, Ramji Venkataramanan, Cynthia Rush, Richard J. Samworth

Abstract: Over the last decade or so, Approximate Message Passing (AMP) algorithms have become extremely popular in various structured high-dimensional statistical problems. The fact that the origins of these techniques can be traced back to notions of belief propagation in the statistical physics literature lends a certain mystique to the area for many statisticians. Our goal in this work is to present the… ▽ More Over the last decade or so, Approximate Message Passing (AMP) algorithms have become extremely popular in various structured high-dimensional statistical problems. The fact that the origins of these techniques can be traced back to notions of belief propagation in the statistical physics literature lends a certain mystique to the area for many statisticians. Our goal in this work is to present the main ideas of AMP from a statistical perspective, to illustrate the power and flexibility of the AMP framework. Along the way, we strengthen and unify many of the results in the existing literature. △ Less

Submitted 5 May, 2021; originally announced May 2021.

Comments: 99 pages, 2 figures

Near-Optimal Coding for Many-user Multiple Access Channels

Authors: Kuan Hsieh, Cynthia Rush, Ramji Venkataramanan

Abstract: This paper considers the Gaussian multiple-access channel (MAC) in the asymptotic regime where the number of users grows linearly with the code length. We propose efficient coding schemes based on random linear models with approximate message passing (AMP) decoding and derive the asymptotic error rate achieved for a given user density, user payload (in bits), and user energy. The tradeoff between… ▽ More This paper considers the Gaussian multiple-access channel (MAC) in the asymptotic regime where the number of users grows linearly with the code length. We propose efficient coding schemes based on random linear models with approximate message passing (AMP) decoding and derive the asymptotic error rate achieved for a given user density, user payload (in bits), and user energy. The tradeoff between energy-per-bit and achievable user density (for a fixed user payload and target error rate) is studied, and it is demonstrated that in the large system limit, a spatially coupled coding scheme with AMP decoding achieves near-optimal tradeoffs for a wide range of user densities. Furthermore, in the regime where the user payload is large, we also study the tradeoff between energy-per-bit and spectral efficiency and discuss methods to reduce decoding complexity. △ Less

Submitted 9 March, 2022; v1 submitted 9 February, 2021; originally announced February 2021.

Comments: 15 pages, 4 figures. To appear in IEEE Journal on Selected Areas in Information Theory

Journal ref: IEEE Journal on Selected Areas in Information Theory, vol. 3, no. 1, pp. 21-36, March 2022

Approximate Message Passing with Spectral Initialization for Generalized Linear Models

Authors: Marco Mondelli, Ramji Venkataramanan

Abstract: We consider the problem of estimating a signal from measurements obtained via a generalized linear model. We focus on estimators based on approximate message passing (AMP), a family of iterative algorithms with many appealing features: the performance of AMP in the high-dimensional limit can be succinctly characterized under suitable model assumptions; AMP can also be tailored to the empirical dis… ▽ More We consider the problem of estimating a signal from measurements obtained via a generalized linear model. We focus on estimators based on approximate message passing (AMP), a family of iterative algorithms with many appealing features: the performance of AMP in the high-dimensional limit can be succinctly characterized under suitable model assumptions; AMP can also be tailored to the empirical distribution of the signal entries, and for a wide class of estimation problems, AMP is conjectured to be optimal among all polynomial-time algorithms. However, a major issue of AMP is that in many models (such as phase retrieval), it requires an initialization correlated with the ground-truth signal and independent from the measurement matrix. Assuming that such an initialization is available is typically not realistic. In this paper, we solve this problem by proposing an AMP algorithm initialized with a spectral estimator. With such an initialization, the standard AMP analysis fails since the spectral estimator depends in a complicated way on the design matrix. Our main contribution is a rigorous characterization of the performance of AMP with spectral initialization in the high-dimensional limit. The key technical idea is to define and analyze a two-phase artificial AMP algorithm that first produces the spectral estimator, and then closely approximates the iterates of the true AMP. We also provide numerical results that demonstrate the validity of the proposed approach. △ Less

Submitted 17 February, 2021; v1 submitted 7 October, 2020; originally announced October 2020.

Comments: 38 pages, 5 figures, AISTATS 2021

Optimal Combination of Linear and Spectral Estimators for Generalized Linear Models

Authors: Marco Mondelli, Christos Thrampoulidis, Ramji Venkataramanan

Abstract: We study the problem of recovering an unknown signal $\boldsymbol x$ given measurements obtained from a generalized linear model with a Gaussian sensing matrix. Two popular solutions are based on a linear estimator $\hat{\boldsymbol x}^{\rm L}$ and a spectral estimator $\hat{\boldsymbol x}^{\rm s}$. The former is a data-dependent linear combination of the columns of the measurement matrix, and its… ▽ More We study the problem of recovering an unknown signal $\boldsymbol x$ given measurements obtained from a generalized linear model with a Gaussian sensing matrix. Two popular solutions are based on a linear estimator $\hat{\boldsymbol x}^{\rm L}$ and a spectral estimator $\hat{\boldsymbol x}^{\rm s}$. The former is a data-dependent linear combination of the columns of the measurement matrix, and its analysis is quite simple. The latter is the principal eigenvector of a data-dependent matrix, and a recent line of work has studied its performance. In this paper, we show how to optimally combine $\hat{\boldsymbol x}^{\rm L}$ and $\hat{\boldsymbol x}^{\rm s}$. At the heart of our analysis is the exact characterization of the joint empirical distribution of $(\boldsymbol x, \hat{\boldsymbol x}^{\rm L}, \hat{\boldsymbol x}^{\rm s})$ in the high-dimensional limit. This allows us to compute the Bayes-optimal combination of $\hat{\boldsymbol x}^{\rm L}$ and $\hat{\boldsymbol x}^{\rm s}$, given the limiting distribution of the signal $\boldsymbol x$. When the distribution of the signal is Gaussian, then the Bayes-optimal combination has the form $θ\hat{\boldsymbol x}^{\rm L}+\hat{\boldsymbol x}^{\rm s}$ and we derive the optimal combination coefficient. In order to establish the limiting distribution of $(\boldsymbol x, \hat{\boldsymbol x}^{\rm L}, \hat{\boldsymbol x}^{\rm s})$, we design and analyze an Approximate Message Passing (AMP) algorithm whose iterates give $\hat{\boldsymbol x}^{\rm L}$ and approach $\hat{\boldsymbol x}^{\rm s}$. Numerical simulations demonstrate the improvement of the proposed combination with respect to the two methods considered separately. △ Less

Submitted 25 June, 2021; v1 submitted 7 August, 2020; originally announced August 2020.

Comments: 49 pages, 6 figures

Modulated Sparse Superposition Codes for the Complex AWGN Channel

Authors: Kuan Hsieh, Ramji Venkataramanan

Abstract: This paper studies a generalization of sparse superposition codes (SPARCs) for communication over the complex additive white Gaussian noise (AWGN) channel. In a SPARC, the codebook is defined in terms of a design matrix, and each codeword is a generated by multiplying the design matrix with a sparse message vector. In the standard SPARC construction, information is encoded in the locations of the… ▽ More This paper studies a generalization of sparse superposition codes (SPARCs) for communication over the complex additive white Gaussian noise (AWGN) channel. In a SPARC, the codebook is defined in terms of a design matrix, and each codeword is a generated by multiplying the design matrix with a sparse message vector. In the standard SPARC construction, information is encoded in the locations of the non-zero entries of the message vector. In this paper we generalize the construction and consider modulated SPARCs, where information in encoded in both the locations and the values of the non-zero entries of the message vector. We focus on the case where the non-zero entries take values from a phase-shift keying (PSK) constellation. We propose a computationally efficient approximate message passing (AMP) decoder, and obtain analytical bounds on the state evolution parameters which predict the error performance of the decoder. Using these bounds we show that PSK-modulated SPARCs are asymptotically capacity achieving for the complex AWGN channel, with either spatial coupling or power allocation. We also provide numerical simulation results to demonstrate the error performance at finite code lengths. These results show that introducing modulation to the SPARC design can significantly reduce decoding complexity without sacrificing error performance. △ Less

Submitted 11 May, 2021; v1 submitted 20 April, 2020; originally announced April 2020.

Comments: 20 pages, 6 figures. To appear in IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 67, no. 7, pp. 4385-4404, July 2021

Capacity-achieving Spatially Coupled Sparse Superposition Codes with AMP Decoding

Authors: Cynthia Rush, Kuan Hsieh, Ramji Venkataramanan

Abstract: Sparse superposition codes, also called sparse regression codes (SPARCs), are a class of codes for efficient communication over the AWGN channel at rates approaching the channel capacity. In a standard SPARC, codewords are sparse linear combinations of columns of an i.i.d. Gaussian design matrix, while in a spatially coupled SPARC the design matrix has a block-wise structure, where the variance of… ▽ More Sparse superposition codes, also called sparse regression codes (SPARCs), are a class of codes for efficient communication over the AWGN channel at rates approaching the channel capacity. In a standard SPARC, codewords are sparse linear combinations of columns of an i.i.d. Gaussian design matrix, while in a spatially coupled SPARC the design matrix has a block-wise structure, where the variance of the Gaussian entries can be varied across blocks. A well-designed spatial coupling structure can significantly enhance the error performance of iterative decoding algorithms such as Approximate Message Passing (AMP). In this paper, we obtain a non-asymptotic bound on the probability of error of spatially coupled SPARCs with AMP decoding. Applying this bound to a simple band-diagonal design matrix, we prove that spatially coupled SPARCs with AMP decoding achieve the capacity of the AWGN channel. The bound also highlights how the decay of error probability depends on each design parameter of the spatially coupled SPARC. An attractive feature of AMP decoding is that its asymptotic mean squared error (MSE) can be predicted via a deterministic recursion called state evolution. Our result provides the first proof that the MSE concentrates on the state evolution prediction for spatially coupled designs. Combined with the state evolution prediction, this result implies that spatially coupled SPARCs with the proposed band-diagonal design are capacity-achieving. Using the proof technique used to establish the main result, we also obtain a concentration inequality for the MSE of AMP applied to compressed sensing with spatially coupled design matrices. Finally we provide numerical simulation results that demonstrate the finite length error performance of spatially coupled SPARCs. The performance is compared with coded modulation schemes that use LDPC codes from the DVB-S2 standard. △ Less

Submitted 8 May, 2021; v1 submitted 18 February, 2020; originally announced February 2020.

Comments: To appear in IEEE Transactions on Information Theory. This version contains proofs of two technical lemmas that were omitted in the journal version

Journal ref: IEEE Transactions on Information Theory, vol. 67, no. 7, pp. 4446-4484, July 2021

Sparse Regression Codes

Authors: Ramji Venkataramanan, Sekhar Tatikonda, Andrew Barron

Abstract: Developing computationally-efficient codes that approach the Shannon-theoretic limits for communication and compression has long been one of the major goals of information and coding theory. There have been significant advances towards this goal in the last couple of decades, with the emergence of turbo codes, sparse-graph codes, and polar codes. These codes are designed primarily for discrete-alp… ▽ More Developing computationally-efficient codes that approach the Shannon-theoretic limits for communication and compression has long been one of the major goals of information and coding theory. There have been significant advances towards this goal in the last couple of decades, with the emergence of turbo codes, sparse-graph codes, and polar codes. These codes are designed primarily for discrete-alphabet channels and sources. For Gaussian channels and sources, where the alphabet is inherently continuous, Sparse Superposition Codes or Sparse Regression Codes (SPARCs) are a promising class of codes for achieving the Shannon limits. This survey provides a unified and comprehensive overview of sparse regression codes, covering theory, algorithms, and practical implementation aspects. The first part of the monograph focuses on SPARCs for AWGN channel coding, and the second part on SPARCs for lossy compression (with squared error distortion criterion). In the third part, SPARCs are used to construct codes for Gaussian multi-terminal channel and source coding models such as broadcast channels, multiple-access channels, and source and channel coding with side information. The survey concludes with a discussion of open problems and directions for future work. △ Less

Submitted 2 November, 2019; originally announced November 2019.

Comments: Published in Foundations and Trends in Communications and Information Theory, 2019

Journal ref: Foundations and Trends in Communications and Information Theory, vol. 15, no. 1-2, pp. 1-195, 2019

Coding for Deletion Channels with Multiple Traces

Authors: Mahed Abroshan, Ramji Venkataramanan, Lara Dolecek, Albert Guillén i Fàbregas

Abstract: Motivated by the sequence reconstruction problem from traces in DNA-based storage, we consider the problem of designing codes for the deletion channel when multiple observations (or traces) are available to the decoder. We propose simple binary and non-binary codes based on Varshamov-Tenengolts (VT) codes. The proposed codes split the codeword in blocks and employ a VT code in each block. The avai… ▽ More Motivated by the sequence reconstruction problem from traces in DNA-based storage, we consider the problem of designing codes for the deletion channel when multiple observations (or traces) are available to the decoder. We propose simple binary and non-binary codes based on Varshamov-Tenengolts (VT) codes. The proposed codes split the codeword in blocks and employ a VT code in each block. The availability of multiple traces helps the decoder to identify deletion-free copies of a block, and to avoid mis-synchronization while decoding. The encoding complexity of the proposed scheme is linear in the codeword length; the decoding complexity is linear in the codeword length, and quadratic in the number of deletions and the number of traces. The proposed scheme offers an explicit low-complexity technique for correcting deletions using multiple traces. △ Less

Submitted 24 June, 2019; v1 submitted 20 May, 2019; originally announced May 2019.

Comments: This paper will be presented at ISIT 2019

Boolean Functions with Biased Inputs: Approximation and Noise Sensitivity

Authors: Mohsen Heidari, S. Sandeep Pradhan, Ramji Venkataramanan

Abstract: This paper considers the problem of approximating a Boolean function $f$ using another Boolean function from a specified class. Two classes of approximating functions are considered: $k$-juntas, and linear Boolean functions. The $n$ input bits of the function are assumed to be independently drawn from a distribution that may be biased. The quality of approximation is measured by the mismatch proba… ▽ More This paper considers the problem of approximating a Boolean function $f$ using another Boolean function from a specified class. Two classes of approximating functions are considered: $k$-juntas, and linear Boolean functions. The $n$ input bits of the function are assumed to be independently drawn from a distribution that may be biased. The quality of approximation is measured by the mismatch probability between $f$ and the approximating function $g$. For each class, the optimal approximation and the associated mismatch probability is characterized in terms of the biased Fourier expansion of $f$. The technique used to analyze the mismatch probability also yields an expression for the noise sensitivity of $f$ in terms of the biased Fourier coefficients, under a general i.i.d. input perturbation model. △ Less

Submitted 8 July, 2019; v1 submitted 29 January, 2019; originally announced January 2019.

Comments: 5 pages, 2 figures, To appear in IEEE ISIT 2018

Determination of Personalized Asthma Triggers from Evidence based on Multimodal Sensing and Mobile Application

Authors: Revathy Venkataramanan, Dipesh Kadariya, Hong Yung Yip, Utkarshani Jamini, Krishnaprasad Thirunarayan, Maninder Kalra, Amit Sheth

Abstract: Objective: Asthma is a chronic pulmonary disease with multiple triggers manifesting as symptoms with various intensities. This paper evaluates the suitability of long-term monitoring of pediatric asthma using diverse data to qualify and quantify triggers that contribute to the asthma symptoms and control to enable a personalized management plan. Materials and Methods: Asthma condition, environme… ▽ More Objective: Asthma is a chronic pulmonary disease with multiple triggers manifesting as symptoms with various intensities. This paper evaluates the suitability of long-term monitoring of pediatric asthma using diverse data to qualify and quantify triggers that contribute to the asthma symptoms and control to enable a personalized management plan. Materials and Methods: Asthma condition, environment, and adherence to the prescribed care plan were continuously tracked for 97 pediatric patients using kHealth-Asthma technology for one or three months. Result: At the cohort level, among 21% of the patients deployed in spring, 63% and 19% indicated pollen and Particulate Matter (PM2.5), respectively, as the major asthma contributors. Of the 18% of the patients deployed in fall, 29% and 21% found pollen and PM2.5 respectively, to be the contributors. For the 28% of the patients deployed in winter, PM2.5 was identified as the major contributor for 80% of them. One patient across each season has been chosen to explain the determination of personalized triggers by observing correlations between triggers and asthma symptoms gathered from anecdotal evidence. Discussion and Conclusion: Both public and personal health signals including compliance to prescribed care plan have been captured through continuous monitoring using the kHealth-Asthma technology which generated insights on causes of asthma symptoms across different seasons. Collectively, they can form the underlying basis for personalized management plan and intervention. KEYWORDS: Personalized Digital Health, Medical Internet of Things, Pediatric Asthma Management, Patient Generated Health Data, Personalized Triggers, Telehealth, △ Less

Submitted 25 November, 2018; originally announced November 2018.

Comments: contact author: Amit Sheth [email protected]

Spatially Coupled Sparse Regression Codes: Design and State Evolution Analysis

Authors: Kuan Hsieh, Cynthia Rush, Ramji Venkataramanan

Abstract: We consider the design and analysis of spatially coupled sparse regression codes (SC-SPARCs), which were recently introduced by Barbier et al. for efficient communication over the additive white Gaussian noise channel. SC-SPARCs can be efficiently decoded using an Approximate Message Passing (AMP) decoder, whose performance in each iteration can be predicted via a set of equations called state evo… ▽ More We consider the design and analysis of spatially coupled sparse regression codes (SC-SPARCs), which were recently introduced by Barbier et al. for efficient communication over the additive white Gaussian noise channel. SC-SPARCs can be efficiently decoded using an Approximate Message Passing (AMP) decoder, whose performance in each iteration can be predicted via a set of equations called state evolution. In this paper, we give an asymptotic characterization of the state evolution equations for SC-SPARCs. For any given base matrix (that defines the coupling structure of the SC-SPARC) and rate, this characterization can be used to predict whether or not AMP decoding will succeed in the large system limit. We then consider a simple base matrix defined by two parameters $(ω, Λ)$, and show that AMP decoding succeeds in the large system limit for all rates $R < \mathcal{C}$. The asymptotic result also indicates how the parameters of the base matrix affect the decoding progression. Simulation results are presented to evaluate the performance of SC-SPARCs defined with the proposed base matrix. △ Less

Submitted 26 April, 2018; v1 submitted 5 January, 2018; originally announced January 2018.

Comments: 8 pages, 6 figures. A shorter version of this paper to appear in ISIT 2018

The Error Probability of Sparse Superposition Codes with Approximate Message Passing Decoding

Authors: Cynthia Rush, Ramji Venkataramanan

Abstract: Sparse superposition codes, or sparse regression codes (SPARCs), are a recent class of codes for reliable communication over the AWGN channel at rates approaching the channel capacity. Approximate message passing (AMP) decoding, a computationally efficient technique for decoding SPARCs, has been proven to be asymptotically capacity-achieving for the AWGN channel. In this paper, we refine the asymp… ▽ More Sparse superposition codes, or sparse regression codes (SPARCs), are a recent class of codes for reliable communication over the AWGN channel at rates approaching the channel capacity. Approximate message passing (AMP) decoding, a computationally efficient technique for decoding SPARCs, has been proven to be asymptotically capacity-achieving for the AWGN channel. In this paper, we refine the asymptotic result by deriving a large deviations bound on the probability of AMP decoding error. This bound gives insight into the error performance of the AMP decoder for large but finite problem sizes, giving an error exponent as well as guidance on how the code parameters should be chosen at finite block lengths. For an appropriate choice of code parameters, we show that for any fixed rate less than the channel capacity, the decoding error probability decays exponentially in $n/(\log n)^{2T}$, where $T$, the number of AMP iterations required for successful decoding, is bounded in terms of the gap from capacity. △ Less

Submitted 22 April, 2019; v1 submitted 19 December, 2017; originally announced December 2017.

Comments: 27 pages. IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 65, no.5, pp. 3278-3303, May 2019

Efficient Systematic Encoding of Non-binary VT Codes

Authors: Mahed Abroshan, Ramji Venkataramanan, Albert Guillen i Fabregas

Abstract: Varshamov-Tenengolts (VT) codes are a class of codes which can correct a single deletion or insertion with a linear-time decoder. This paper addresses the problem of efficient encoding of non-binary VT codes, defined over an alphabet of size $q >2$. We propose a simple linear-time encoding method to systematically map binary message sequences onto VT codewords. The method provides a new lower boun… ▽ More Varshamov-Tenengolts (VT) codes are a class of codes which can correct a single deletion or insertion with a linear-time decoder. This paper addresses the problem of efficient encoding of non-binary VT codes, defined over an alphabet of size $q >2$. We propose a simple linear-time encoding method to systematically map binary message sequences onto VT codewords. The method provides a new lower bound on the size of $q$-ary VT codes of length $n$. △ Less

Submitted 27 April, 2018; v1 submitted 14 August, 2017; originally announced August 2017.

Comments: This paper will appear in the proceedings of ISIT 2018

Empirical Bayes Estimators for High-Dimensional Sparse Vectors

Authors: Pavan Srinath, Ramji Venkataramanan

Abstract: The problem of estimating a high-dimensional sparse vector $\boldsymbolθ \in \mathbb{R}^n$ from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the well-known soft-thresholding estimator. We obtain concentration inequalities fo… ▽ More The problem of estimating a high-dimensional sparse vector $\boldsymbolθ \in \mathbb{R}^n$ from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the well-known soft-thresholding estimator. We obtain concentration inequalities for the Stein's unbiased risk estimate and the loss function of both estimators. The results show that for large $n$, both the risk estimate and the loss function concentrate on deterministic values close to the true risk. Depending on the underlying $\boldsymbolθ$, either the proposed empirical Bayes (eBayes) estimator or soft-thresholding may have smaller loss. We consider a hybrid estimator that attempts to pick the better of the soft-thresholding estimator and the eBayes estimator by comparing their risk estimates. It is shown that: i) the loss of the hybrid estimator concentrates on the minimum of the losses of the two competing estimators, and ii) the risk of the hybrid estimator is within order $\frac{1}{\sqrt{n}}$ of the minimum of the two risks. Simulation results are provided to support the theoretical results. Finally, we use the eBayes and hybrid estimators as denoisers in the approximate message passing (AMP) algorithm for compressed sensing, and show that their performance is superior to the soft-thresholding denoiser in a wide range of settings. △ Less

Submitted 27 December, 2018; v1 submitted 28 July, 2017; originally announced July 2017.

Comments: 35 pages, to appear in Information and Inference: A Journal of the IMA

A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation

Authors: Ramji Venkataramanan, Oliver Johnson

Abstract: In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically does not give a sharp converse result (error lower bound) for channel coding proble… ▽ More In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically does not give a sharp converse result (error lower bound) for channel coding problems. Moreover, recent work has shown that an argument based on binary hypothesis testing gives tighter results. We adapt this technique to the statistical setting, and argue that Fano's inequality can always be replaced by this approach to obtain tighter lower bounds that can be easily computed and are asymptotically sharp. We illustrate our technique in three applications: density estimation, active learning of a binary classifier, and compressed sensing, obtaining tighter risk lower bounds in each case. △ Less

Submitted 4 April, 2018; v1 submitted 14 June, 2017; originally announced June 2017.

Comments: In the latest version, the value of $λ$ in the statements of Lemma 4.1 and Proposition 4.2 is restricted to the interval $(0,1]$. This is the correct condition, rather than $λ>0$ stated in the journal version below

Journal ref: Electronic Journal of Statistics, Vol. 12, No. 1, pp. 1126-1149, 2018

Multilayer Codes for Synchronization from Deletions and Insertions

Authors: Mahed Abroshan, Ramji Venkataramanan, Albert Guillen i Fabregas

Abstract: Consider two remote nodes (encoder and decoder), each with a binary sequence. The encoder's sequence $X$ differs from the decoder's sequence $Y$ by a small number of edits (deletions and insertions). The goal is to construct a message $M$, to be sent via a one-way error free link, such that the decoder can reconstruct $X$ using $M$ and $Y$. In this paper, we devise a coding scheme for this one-way… ▽ More Consider two remote nodes (encoder and decoder), each with a binary sequence. The encoder's sequence $X$ differs from the decoder's sequence $Y$ by a small number of edits (deletions and insertions). The goal is to construct a message $M$, to be sent via a one-way error free link, such that the decoder can reconstruct $X$ using $M$ and $Y$. In this paper, we devise a coding scheme for this one-way synchronization model. The scheme is based on multiple layers of Varshamov-Tenengolts (VT) codes combined with off-the-shelf linear error-correcting codes, and uses a list decoder. We bound the expected list size of the decoder under certain assumptions, and validate its performance via numerical simulations. We also consider an alternative decoder that uses only the constraints from the VT codes (i.e., does not require a linear code), and has a smaller redundancy at the expense of a slightly larger average list size. △ Less

Submitted 27 October, 2020; v1 submitted 18 May, 2017; originally announced May 2017.

Comments: This paper is accepted for publication in the IEEE Transactions on Information Theory

Techniques for improving the finite length performance of sparse superposition codes

Authors: Adam Greig, Ramji Venkataramanan

Abstract: Sparse superposition codes are a recent class of codes introduced by Barron and Joseph for efficient communication over the AWGN channel. With an appropriate power allocation, these codes have been shown to be asymptotically capacity-achieving with computationally feasible decoding. However, a direct implementation of the capacity-achieving construction does not give good finite length error perfo… ▽ More Sparse superposition codes are a recent class of codes introduced by Barron and Joseph for efficient communication over the AWGN channel. With an appropriate power allocation, these codes have been shown to be asymptotically capacity-achieving with computationally feasible decoding. However, a direct implementation of the capacity-achieving construction does not give good finite length error performance. In this paper, we consider sparse superposition codes with approximate message passing (AMP) decoding, and describe a variety of techniques to improve their finite length performance. These include an iterative algorithm for SPARC power allocation, guidelines for choosing codebook parameters, and estimating a critical decoding parameter online instead of pre-computation. We also show how partial outer codes can be used in conjunction with AMP decoding to obtain a steep waterfall in the error performance curves. We compare the error performance of AMP-decoded sparse superposition codes with coded modulation using LDPC codes from the WiMAX standard. △ Less

Submitted 19 November, 2017; v1 submitted 5 May, 2017; originally announced May 2017.

Comments: 13 pages, 16 figures. To appear in IEEE Transactions on Communications

Journal ref: IEEE Transactions on Communications, vol. 66, no. 3, pp. 905-917, March 2018

Coding for Segmented Edit Channels

Authors: Mahed Abroshan, Ramji Venkataramanan, Albert Guillen i Fabregas

Abstract: This paper considers insertion and deletion channels with the additional assumption that the channel input sequence is implicitly divided into segments such that at most one edit can occur within a segment. No segment markers are available in the received sequence. We propose code constructions for the segmented deletion, segmented insertion, and segmented insertion-deletion channels based on subs… ▽ More This paper considers insertion and deletion channels with the additional assumption that the channel input sequence is implicitly divided into segments such that at most one edit can occur within a segment. No segment markers are available in the received sequence. We propose code constructions for the segmented deletion, segmented insertion, and segmented insertion-deletion channels based on subsets of Varshamov-Tenengolts codes chosen with pre-determined prefixes and/or suffixes. The proposed codes, constructed for any finite alphabet, are zero-error and can be decoded segment-by-segment. We also derive an upper bound on the rate of any zero-error code for the segmented edit channel, in terms of the segment length. This upper bound shows that the rate scaling of the proposed codes as the segment length increases is the same as that of the maximal code. △ Less

Submitted 16 March, 2018; v1 submitted 23 January, 2017; originally announced January 2017.

Comments: Appeared in IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 64, no.4, pp. 3086-3098, April 2018

Finite Sample Analysis of Approximate Message Passing Algorithms

Authors: Cynthia Rush, Ramji Venkataramanan

Abstract: Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate… ▽ More Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector $β_0$ from a noisy measurement $y=A β_0 + w$. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix $A$, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with i.i.d. Gaussian measurement matrices with finite size $n \times N$. The result shows that the probability of deviation from the state evolution prediction falls exponentially in $n$. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations $t$ can grow no faster than order $\frac{\log n}{\log \log n}$ for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation. △ Less

Submitted 16 March, 2018; v1 submitted 6 June, 2016; originally announced June 2016.

Comments: To appear in IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 64, no. 11, pp. 7264-7286, November 2018

Cluster-Seeking James-Stein Estimators

Authors: K. Pavan Srinath, Ramji Venkataramanan

Abstract: This paper considers the problem of estimating a high-dimensional vector of parameters $\boldsymbolθ \in \mathbb{R}^n$ from a noisy observation. The noise vector is i.i.d. Gaussian with known variance. For a squared-error loss function, the James-Stein (JS) estimator is known to dominate the simple maximum-likelihood (ML) estimator when the dimension $n$ exceeds two. The JS-estimator shrinks the o… ▽ More This paper considers the problem of estimating a high-dimensional vector of parameters $\boldsymbolθ \in \mathbb{R}^n$ from a noisy observation. The noise vector is i.i.d. Gaussian with known variance. For a squared-error loss function, the James-Stein (JS) estimator is known to dominate the simple maximum-likelihood (ML) estimator when the dimension $n$ exceeds two. The JS-estimator shrinks the observed vector towards the origin, and the risk reduction over the ML-estimator is greatest for $\boldsymbolθ$ that lie close to the origin. JS-estimators can be generalized to shrink the data towards any target subspace. Such estimators also dominate the ML-estimator, but the risk reduction is significant only when $\boldsymbolθ$ lies close to the subspace. This leads to the question: in the absence of prior information about $\boldsymbolθ$, how do we design estimators that give significant risk reduction over the ML-estimator for a wide range of $\boldsymbolθ$? In this paper, we propose shrinkage estimators that attempt to infer the structure of $\boldsymbolθ$ from the observed data in order to construct a good attracting subspace. In particular, the components of the observed vector are separated into clusters, and the elements in each cluster shrunk towards a common attractor. The number of clusters and the attractor for each cluster are determined from the observed vector. We provide concentration results for the squared-error loss and convergence results for the risk of the proposed estimators. The results show that the estimators give significant risk reduction over the ML-estimator for a wide range of $\boldsymbolθ$, particularly for large $n$. Simulation results are provided to support the theoretical claims. △ Less

Submitted 16 March, 2018; v1 submitted 1 February, 2016; originally announced February 2016.

Comments: Appeared in IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 64, no. 2, pp. 853-874, February 2018

Capacity-achieving Sparse Superposition Codes via Approximate Message Passing Decoding

Authors: Cynthia Rush, Adam Greig, Ramji Venkataramanan

Abstract: Sparse superposition codes were recently introduced by Barron and Joseph for reliable communication over the AWGN channel at rates approaching the channel capacity. The codebook is defined in terms of a Gaussian design matrix, and codewords are sparse linear combinations of columns of the matrix. In this paper, we propose an approximate message passing decoder for sparse superposition codes, whose… ▽ More Sparse superposition codes were recently introduced by Barron and Joseph for reliable communication over the AWGN channel at rates approaching the channel capacity. The codebook is defined in terms of a Gaussian design matrix, and codewords are sparse linear combinations of columns of the matrix. In this paper, we propose an approximate message passing decoder for sparse superposition codes, whose decoding complexity scales linearly with the size of the design matrix. The performance of the decoder is rigorously analyzed and it is shown to asymptotically achieve the AWGN capacity with an appropriate power allocation. Simulation results are provided to demonstrate the performance of the decoder at finite blocklengths. We introduce a power allocation scheme to improve the empirical performance, and demonstrate how the decoding complexity can be significantly reduced by using Hadamard design matrices. △ Less

Submitted 11 March, 2017; v1 submitted 23 January, 2015; originally announced January 2015.

Comments: 25 pages, 4 figures. IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 63, no.3, pp. 1476-1500, March 2017

The Rate-Distortion Function and Excess-Distortion Exponent of Sparse Regression Codes with Optimal Encoding

Authors: Ramji Venkataramanan, Sekhar Tatikonda

Abstract: This paper studies the performance of sparse regression codes for lossy compression with the squared-error distortion criterion. In a sparse regression code, codewords are linear combinations of subsets of columns of a design matrix. It is shown that with minimum-distance encoding, sparse regression codes achieve the Shannon rate-distortion function for i.i.d. Gaussian sources $R^*(D)$ as well as… ▽ More This paper studies the performance of sparse regression codes for lossy compression with the squared-error distortion criterion. In a sparse regression code, codewords are linear combinations of subsets of columns of a design matrix. It is shown that with minimum-distance encoding, sparse regression codes achieve the Shannon rate-distortion function for i.i.d. Gaussian sources $R^*(D)$ as well as the optimal excess-distortion exponent. This completes a previous result which showed that $R^*(D)$ and the optimal exponent were achievable for distortions below a certain threshold. The proof of the rate-distortion result is based on the second moment method, a popular technique to show that a non-negative random variable $X$ is strictly positive with high probability. In our context, $X$ is the number of codewords within target distortion $D$ of the source sequence. We first identify the reason behind the failure of the standard second moment method for certain distortions, and illustrate the different failure modes via a stylized example. We then use a refinement of the second moment method to show that $R^*(D)$ is achievable for all distortion values. Finally, the refinement technique is applied to Suen's correlation inequality to prove the achievability of the optimal Gaussian excess-distortion exponent. △ Less

Submitted 19 June, 2017; v1 submitted 21 January, 2014; originally announced January 2014.

Comments: 16 pages. IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, Vol. 63, no. 8, pp. 5228-5243 (August 2017)

Low-Complexity Interactive Algorithms for Synchronization from Deletions, Insertions, and Substitutions

Authors: Ramji Venkataramanan, Vasuki Narasimha Swamy, Kannan Ramchandran

Abstract: Consider two remote nodes having binary sequences $X$ and $Y$, respectively. $Y$ is an edited version of ${X}$, where the editing involves random deletions, insertions, and substitutions, possibly in bursts. The goal is for the node with $Y$ to reconstruct $X$ with minimal exchange of information over a noiseless link. The communication is measured in terms of both the total number of bits exchang… ▽ More Consider two remote nodes having binary sequences $X$ and $Y$, respectively. $Y$ is an edited version of ${X}$, where the editing involves random deletions, insertions, and substitutions, possibly in bursts. The goal is for the node with $Y$ to reconstruct $X$ with minimal exchange of information over a noiseless link. The communication is measured in terms of both the total number of bits exchanged and the number of interactive rounds of communication. This paper focuses on the setting where the number of edits is $o(\tfrac{n}{\log n})$, where $n$ is the length of $X$. We first consider the case where the edits are a mixture of insertions and deletions (indels), and propose an interactive synchronization algorithm with near-optimal communication rate and average computational complexity of $O(n)$ arithmetic operations. The algorithm uses interaction to efficiently split the source sequence into substrings containing exactly one deletion or insertion. Each of these substrings is then synchronized using an optimal one-way synchronization code based on the single-deletion correcting channel codes of Varshamov and Tenengolts (VT codes). We then build on this synchronization algorithm in three different ways. First, it is modified to work with a single round of interaction. The reduction in the number of rounds comes at the expense of higher communication, which is quantified. Next, we present an extension to the practically important case where the insertions and deletions may occur in (potentially large) bursts. Finally, we show how to synchronize the sources to within a target Hamming distance. This feature can be used to differentiate between substitution and indel edits. In addition to theoretical performance bounds, we provide several validating simulation results for the proposed algorithms. △ Less

Submitted 12 September, 2015; v1 submitted 8 October, 2013; originally announced October 2013.

Journal ref: IEEE Transactions on Information Theory, vol. 61, no. 10, pp. 5670-5689, October 2015

Sparse Regression Codes for Multi-terminal Source and Channel Coding

Authors: Ramji Venkataramanan, Sekhar Tatikonda

Abstract: We study a new class of codes for Gaussian multi-terminal source and channel coding. These codes are designed using the statistical framework of high-dimensional linear regression and are called Sparse Superposition or Sparse Regression codes. Codewords are linear combinations of subsets of columns of a design matrix. These codes were recently introduced by Barron and Joseph and shown to achieve t… ▽ More We study a new class of codes for Gaussian multi-terminal source and channel coding. These codes are designed using the statistical framework of high-dimensional linear regression and are called Sparse Superposition or Sparse Regression codes. Codewords are linear combinations of subsets of columns of a design matrix. These codes were recently introduced by Barron and Joseph and shown to achieve the channel capacity of AWGN channels with computationally feasible decoding. They have also recently been shown to achieve the optimal rate-distortion function for Gaussian sources. In this paper, we demonstrate how to implement random binning and superposition coding using sparse regression codes. In particular, with minimum-distance encoding/decoding it is shown that sparse regression codes attain the optimal information-theoretic limits for a variety of multi-terminal source and channel coding problems. △ Less

Submitted 10 December, 2012; originally announced December 2012.

Comments: 9 pages, appeared in the Proceedings of the 50th Annual Allerton Conference on Communication, Control, and Computing - 2012

Lossy Compression via Sparse Linear Regression: Computationally Efficient Encoding and Decoding

Authors: Ramji Venkataramanan, Tuhin Sarkar, Sekhar Tatikonda

Abstract: We propose computationally efficient encoders and decoders for lossy compression using a Sparse Regression Code. The codebook is defined by a design matrix and codewords are structured linear combinations of columns of this matrix. The proposed encoding algorithm sequentially chooses columns of the design matrix to successively approximate the source sequence. It is shown to achieve the optimal di… ▽ More We propose computationally efficient encoders and decoders for lossy compression using a Sparse Regression Code. The codebook is defined by a design matrix and codewords are structured linear combinations of columns of this matrix. The proposed encoding algorithm sequentially chooses columns of the design matrix to successively approximate the source sequence. It is shown to achieve the optimal distortion-rate function for i.i.d Gaussian sources under the squared-error distortion criterion. For a given rate, the parameters of the design matrix can be varied to trade off distortion performance with encoding complexity. An example of such a trade-off as a function of the block length n is the following. With computational resource (space or time) per source sample of O((n/\log n)^2), for a fixed distortion-level above the Gaussian distortion-rate function, the probability of excess distortion decays exponentially in n. The Sparse Regression Code is robust in the following sense: for any ergodic source, the proposed encoder achieves the optimal distortion-rate function of an i.i.d Gaussian source with the same variance. Simulations show that the encoder has good empirical performance, especially at low and moderate rates. △ Less

Submitted 28 March, 2014; v1 submitted 7 December, 2012; originally announced December 2012.

Comments: 14 pages, to appear in IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 60, no. 6, pp. 3265-3278, June 2014

Rewritable storage channels with hidden state

Authors: Ramji Venkataramanan, Sekhar Tatikonda, Luis Lastras, Michele Franceschini

Abstract: Many storage channels admit reading and rewriting of the content at a given cost. We consider rewritable channels with a hidden state which models the unknown characteristics of the memory cell. In addition to mitigating the effect of the write noise, rewrites can help the write controller obtain a better estimate of the hidden state. The paper has two contributions. The first is a lower bound on… ▽ More Many storage channels admit reading and rewriting of the content at a given cost. We consider rewritable channels with a hidden state which models the unknown characteristics of the memory cell. In addition to mitigating the effect of the write noise, rewrites can help the write controller obtain a better estimate of the hidden state. The paper has two contributions. The first is a lower bound on the capacity of a general rewritable channel with hidden state. The lower bound is obtained using a coding scheme that combines Gelfand-Pinsker coding with superposition coding. The rewritable AWGN channel is discussed as an example. The second contribution is a simple coding scheme for a rewritable channel where the write noise and hidden state are both uniformly distributed. It is shown that this scheme is asymptotically optimal as the number of rewrites gets large. △ Less

Submitted 3 June, 2013; v1 submitted 12 June, 2012; originally announced June 2012.

Comments: 10 pages. Part of the paper appeared in the proceedings of the 2012 IEEE International Symposium on Information Theory

Journal ref: IEEE Journal on Selected Areas in Communications, vol. 32, no. 5, pp. 815-824, May 2014

Lossy Compression via Sparse Linear Regression: Performance under Minimum-distance Encoding

Authors: Ramji Venkataramanan, Antony Joseph, Sekhar Tatikonda

Abstract: We study a new class of codes for lossy compression with the squared-error distortion criterion, designed using the statistical framework of high-dimensional linear regression. Codewords are linear combinations of subsets of columns of a design matrix. Called a Sparse Superposition or Sparse Regression codebook, this structure is motivated by an analogous construction proposed recently by Barron a… ▽ More We study a new class of codes for lossy compression with the squared-error distortion criterion, designed using the statistical framework of high-dimensional linear regression. Codewords are linear combinations of subsets of columns of a design matrix. Called a Sparse Superposition or Sparse Regression codebook, this structure is motivated by an analogous construction proposed recently by Barron and Joseph for communication over an AWGN channel. For i.i.d Gaussian sources and minimum-distance encoding, we show that such a code can attain the Shannon rate-distortion function with the optimal error exponent, for all distortions below a specified value. It is also shown that sparse regression codes are robust in the following sense: a codebook designed to compress an i.i.d Gaussian source of variance $σ^2$ with (squared-error) distortion $D$ can compress any ergodic source of variance less than $σ^2$ to within distortion $D$. Thus the sparse regression ensemble retains many of the good covering properties of the i.i.d random Gaussian ensemble, while having having a compact representation in terms of a matrix whose size is a low-order polynomial in the block-length. △ Less

Submitted 18 December, 2015; v1 submitted 3 February, 2012; originally announced February 2012.

Comments: This version corrects a typo in the statement of Theorem 2 of the published paper

Journal ref: IEEE Transactions on Information Theory, vol. 60, no. 6, pp. 3254-3264, June 2014

An Achievable Rate Region for the Broadcast Channel with Feedback

Authors: Ramji Venkataramanan, S. Sandeep Pradhan

Abstract: A single-letter achievable rate region is proposed for the two-receiver discrete memoryless broadcast channel with generalized feedback. The coding strategy involves block-Markov superposition coding, using Marton's coding scheme for the broadcast channel without feedback as the starting point. If the message rates in the Marton scheme are too high to be decoded at the end of a block, each receive… ▽ More A single-letter achievable rate region is proposed for the two-receiver discrete memoryless broadcast channel with generalized feedback. The coding strategy involves block-Markov superposition coding, using Marton's coding scheme for the broadcast channel without feedback as the starting point. If the message rates in the Marton scheme are too high to be decoded at the end of a block, each receiver is left with a list of messages compatible with its output. Resolution information is sent in the following block to enable each receiver to resolve its list. The key observation is that the resolution information of the first receiver is correlated with that of the second. This correlated information is efficiently transmitted via joint source-channel coding, using ideas similar to the Han-Costa coding scheme. Using the result, we obtain an achievable rate region for the stochastically degraded AWGN broadcast channel with noisy feedback from only one receiver. It is shown that this region is strictly larger than the no-feedback capacity region. △ Less

Submitted 19 July, 2013; v1 submitted 11 May, 2011; originally announced May 2011.

Comments: To appear in IEEE Transactions on Information Theory. Contains example of AWGN Broadcast Channel with noisy feedback

Journal ref: IEEE Transactions on Information Theory, vol. 59, no.10, pp. 6175 - 6191, Oct. 2013

Achievable Rates for Channels with Deletions and Insertions

Authors: Ramji Venkataramanan, Sekhar Tatikonda, Kannan Ramchandran

Abstract: This paper considers a binary channel with deletions and insertions, where each input bit is transformed in one of the following ways: it is deleted with probability d, or an extra bit is added after it with probability i, or it is transmitted unmodified with probability 1-d-i. A computable lower bound on the capacity of this channel is derived. The transformation of the input sequence by the chan… ▽ More This paper considers a binary channel with deletions and insertions, where each input bit is transformed in one of the following ways: it is deleted with probability d, or an extra bit is added after it with probability i, or it is transmitted unmodified with probability 1-d-i. A computable lower bound on the capacity of this channel is derived. The transformation of the input sequence by the channel may be viewed in terms of runs as follows: some runs of the input sequence get shorter/longer, some runs get deleted, and some new runs are added. It is difficult for the decoder to synchronize the channel output sequence to the transmitted codeword mainly due to deleted runs and new inserted runs. The main idea is a mutual information decomposition in terms of the rate achieved by a sub-optimal decoder that determines the positions of the deleted and inserted runs in addition to decoding the transmitted codeword. The mutual information between the channel input and output sequences is expressed as the sum of the rate achieved by this decoder and the rate loss due to its sub-optimality. Obtaining computable lower bounds on each of these quantities yields a lower bound on the capacity. The bounds proposed in this paper provide the first characterization of achievable rates for channels with general insertions, and for channels with both deletions and insertions. For the special case of the deletion channel, the proposed bound improves on the previous best lower bound for deletion probabilities up to 0.3. △ Less

Submitted 19 July, 2013; v1 submitted 24 February, 2011; originally announced February 2011.

Comments: To appear in IEEE Transactions on Information Theory. For the deletion channel, the new capacity lower bound improves on the previous best bound for deletion probabilities up to 0.3

Journal ref: IEEE Transactions on Information Theory, vol. 59, no.11, pp. 6990-7013, Nov. 2013

Typicality Graphs:Large Deviation Analysis

Authors: Ali Nazari, Ramji Venkataramanan, Dinesh Krithivasan, S. Sandeep Pradhan, Achilleas Anastasopoulos

Abstract: Let $\mathcal{X}$ and $\mathcal{Y}$ be finite alphabets and $P_{XY}$ a joint distribution over them, with $P_X$ and $P_Y$ representing the marginals. For any $ε> 0$, the set of $n$-length sequences $x^n$ and $y^n$ that are jointly typical \cite{ckbook} according to $P_{XY}$ can be represented on a bipartite graph. We present a formal definition of such a graph, known as a \emph{typicality} graph,… ▽ More Let $\mathcal{X}$ and $\mathcal{Y}$ be finite alphabets and $P_{XY}$ a joint distribution over them, with $P_X$ and $P_Y$ representing the marginals. For any $ε> 0$, the set of $n$-length sequences $x^n$ and $y^n$ that are jointly typical \cite{ckbook} according to $P_{XY}$ can be represented on a bipartite graph. We present a formal definition of such a graph, known as a \emph{typicality} graph, and study some of its properties. △ Less

Submitted 8 October, 2010; v1 submitted 6 October, 2010; originally announced October 2010.

A New Achievable Rate Region for the Discrete Memoryless Multiple-Access Channel with Noiseless Feedback

Authors: Ramji Venkataramanan, S. Sandeep Pradhan

Abstract: A new single-letter achievable rate region is proposed for the two-user discrete memoryless multiple-access channel(MAC) with noiseless feedback. The proposed region includes the Cover-Leung rate region [1], and it is shown that the inclusion is strict. The proof uses a block-Markov superposition strategy based on the observation that the messages of the two users are correlated given the feedback… ▽ More A new single-letter achievable rate region is proposed for the two-user discrete memoryless multiple-access channel(MAC) with noiseless feedback. The proposed region includes the Cover-Leung rate region [1], and it is shown that the inclusion is strict. The proof uses a block-Markov superposition strategy based on the observation that the messages of the two users are correlated given the feedback. The rates of transmission are too high for each encoder to decode the other's message directly using the feedback, so they transmit correlated information in the next block to learn the message of one another. They then cooperate in the following block to resolve the residual uncertainty of the decoder. The coding scheme may be viewed as a natural generalization of the Cover-Leung scheme with a delay of one extra block and a pair of additional auxiliary random variables. We compute the proposed rate region for two different MACs and compare the results with other known rate regions for the MAC with feedback. Finally, we show how the coding scheme can be extended to obtain larger rate regions with more auxiliary random variables. △ Less

Submitted 28 March, 2014; v1 submitted 13 January, 2009; originally announced January 2009.

Comments: appeared in IEEE Transactions on Information Theory

Journal ref: IEEE Transactions on Information Theory, vol. 57, no.12, pp. 8038-8054, Dec. 2011