-
Combining Open-box Simulation and Importance Sampling for Tuning Large-Scale Recommenders
Authors:
Kaushal Paneri,
Michael Munje,
Kailash Singh Maurya,
Adith Swaminathan,
Yifan Shi
Abstract:
Growing scale of recommender systems require extensive tuning to respond to market dynamics and system changes. We address the challenge of tuning a large-scale ads recommendation platform with multiple continuous parameters influencing key performance indicators (KPIs). Traditional methods like open-box Monte Carlo simulators, while accurate, are computationally expensive due to the high cost of…
▽ More
Growing scale of recommender systems require extensive tuning to respond to market dynamics and system changes. We address the challenge of tuning a large-scale ads recommendation platform with multiple continuous parameters influencing key performance indicators (KPIs). Traditional methods like open-box Monte Carlo simulators, while accurate, are computationally expensive due to the high cost of evaluating numerous parameter settings. To mitigate this, we propose a hybrid approach Simulator-Guided Importance Sampling (SGIS) that combines open-box simulation with importance sampling (IS). SGIS leverages the strengths of both techniques: it performs a coarse enumeration over the parameter space to identify promising initial settings and then uses IS to iteratively refine these settings. This approach significantly reduces computational costs while maintaining high accuracy in KPI estimation. We demonstrate the effectiveness of SGIS through simulations as well as real-world experiments, showing that it achieves substantial improvements in KPIs with lower computational overhead compared to traditional methods.
△ Less
Submitted 23 September, 2024;
originally announced October 2024.
-
Adaptive Mixture Importance Sampling for Automated Ads Auction Tuning
Authors:
Yimeng Jia,
Kaushal Paneri,
Rong Huang,
Kailash Singh Maurya,
Pavan Mallapragada,
Yifan Shi
Abstract:
This paper introduces Adaptive Mixture Importance Sampling (AMIS) as a novel approach for optimizing key performance indicators (KPIs) in large-scale recommender systems, such as online ad auctions. Traditional importance sampling (IS) methods face challenges in dynamic environments, particularly in navigating through complexities of multi-modal landscapes and avoiding entrapment in local optima f…
▽ More
This paper introduces Adaptive Mixture Importance Sampling (AMIS) as a novel approach for optimizing key performance indicators (KPIs) in large-scale recommender systems, such as online ad auctions. Traditional importance sampling (IS) methods face challenges in dynamic environments, particularly in navigating through complexities of multi-modal landscapes and avoiding entrapment in local optima for the optimization task. Instead of updating importance weights and mixing samples across iterations, as in canonical adaptive IS and multiple IS, our AMIS framework leverages a mixture distribution as the proposal distribution and dynamically adjusts both the mixture parameters and their mixing rates at each iteration, thereby enhancing search diversity and efficiency.
Through extensive offline simulations, we demonstrate that AMIS significantly outperforms simple Gaussian Importance Sampling (GIS), particularly in noisy environments. Moreover, our approach is validated in real-world scenarios through online A/B experiments on a major search engine, where AMIS consistently identifies optimal tuning points that are more likely to be adopted as mainstream configurations. These findings indicate that AMIS enhances convergence in noisy environments, leading to more accurate and reliable decision-making in the context of importance sampling off-policy estimators.
△ Less
Submitted 20 September, 2024;
originally announced September 2024.
-
Leveraging Structured Biological Knowledge for Counterfactual Inference: a Case Study of Viral Pathogenesis
Authors:
Jeremy Zucker,
Kaushal Paneri,
Sara Mohammad-Taheri,
Somya Bhargava,
Pallavi Kolambkar,
Craig Bakker,
Jeremy Teuton,
Charles Tapley Hoyt,
Kristie Oxford,
Robert Ness,
Olga Vitek
Abstract:
Counterfactual inference is a useful tool for comparing outcomes of interventions on complex systems. It requires us to represent the system in form of a structural causal model, complete with a causal diagram, probabilistic assumptions on exogenous variables, and functional assignments. Specifying such models can be extremely difficult in practice. The process requires substantial domain expertis…
▽ More
Counterfactual inference is a useful tool for comparing outcomes of interventions on complex systems. It requires us to represent the system in form of a structural causal model, complete with a causal diagram, probabilistic assumptions on exogenous variables, and functional assignments. Specifying such models can be extremely difficult in practice. The process requires substantial domain expertise, and does not scale easily to large systems, multiple systems, or novel system modifications. At the same time, many application domains, such as molecular biology, are rich in structured causal knowledge that is qualitative in nature. This manuscript proposes a general approach for querying a causal biological knowledge graph, and converting the qualitative result into a quantitative structural causal model that can learn from data to answer the question. We demonstrate the feasibility, accuracy and versatility of this approach using two case studies in systems biology. The first demonstrates the appropriateness of the underlying assumptions and the accuracy of the results. The second demonstrates the versatility of the approach by querying a knowledge base for the molecular determinants of a severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)-induced cytokine storm, and performing counterfactual inference to estimate the causal effect of medical countermeasures for severely ill patients.
△ Less
Submitted 13 January, 2021;
originally announced January 2021.
-
Integrating Markov processes with structural causal modeling enables counterfactual inference in complex systems
Authors:
Robert Osazuwa Ness,
Kaushal Paneri,
Olga Vitek
Abstract:
This manuscript contributes a general and practical framework for casting a Markov process model of a system at equilibrium as a structural causal model, and carrying out counterfactual inference. Markov processes mathematically describe the mechanisms in the system, and predict the system's equilibrium behavior upon intervention, but do not support counterfactual inference. In contrast, structura…
▽ More
This manuscript contributes a general and practical framework for casting a Markov process model of a system at equilibrium as a structural causal model, and carrying out counterfactual inference. Markov processes mathematically describe the mechanisms in the system, and predict the system's equilibrium behavior upon intervention, but do not support counterfactual inference. In contrast, structural causal models support counterfactual inference, but do not identify the mechanisms. This manuscript leverages the benefits of both approaches. We define the structural causal models in terms of the parameters and the equilibrium dynamics of the Markov process models, and counterfactual inference flows from these settings. The proposed approach alleviates the identifiability drawback of the structural causal models, in that the counterfactual inference is consistent with the counterfactual trajectories simulated from the Markov process model. We showcase the benefits of this framework in case studies of complex biomolecular systems with nonlinear dynamics. We illustrate that, in presence of Markov process model misspecification, counterfactual inference leverages prior data, and therefore estimates the outcome of an intervention more accurately than a direct simulation.
△ Less
Submitted 5 November, 2019;
originally announced November 2019.
-
Crop Planning using Stochastic Visual Optimization
Authors:
Gunjan Sehgal,
Bindu Gupta,
Kaushal Paneri,
Karamjit Singh,
Geetika Sharma,
Gautam Shroff
Abstract:
As the world population increases and arable land decreases, it becomes vital to improve the productivity of the agricultural land available. Given the weather and soil properties, farmers need to take critical decisions such as which seed variety to plant and in what proportion, in order to maximize productivity. These decisions are irreversible and any unusual behavior of external factors, such…
▽ More
As the world population increases and arable land decreases, it becomes vital to improve the productivity of the agricultural land available. Given the weather and soil properties, farmers need to take critical decisions such as which seed variety to plant and in what proportion, in order to maximize productivity. These decisions are irreversible and any unusual behavior of external factors, such as weather, can have catastrophic impact on the productivity of crop. A variety which is highly desirable to a farmer might be unavailable or in short supply, therefore, it is very critical to evaluate which variety or varieties are more likely to be chosen by farmers from a growing region in order to meet demand. In this paper, we present our visual analytics tool, ViSeed, showcased on the data given in Syngenta 2016 crop data challenge 1 . This tool helps to predict optimal soybean seed variety or mix of varieties in appropriate proportions which is more likely to be chosen by farmers from a growing region. It also allows to analyse solutions generated from our approach and helps in the decision making process by providing insightful visualizations
△ Less
Submitted 25 October, 2017;
originally announced October 2017.
