-
Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Training
Authors:
Shriram M S,
Xinyue Hao,
Shihao Hou,
Yang Lu,
Laura Sevilla-Lara,
Anurag Arnab,
Shreyank N Gowda
Abstract:
The success of the machine learning field has reliably depended on training on large datasets. While effective, this trend comes at an extraordinary cost. This is due to two deeply intertwined factors: the size of models and the size of datasets. While promising research efforts focus on reducing the size of models, the other half of the equation remains fairly mysterious. Indeed, it is surprising…
▽ More
The success of the machine learning field has reliably depended on training on large datasets. While effective, this trend comes at an extraordinary cost. This is due to two deeply intertwined factors: the size of models and the size of datasets. While promising research efforts focus on reducing the size of models, the other half of the equation remains fairly mysterious. Indeed, it is surprising that the standard approach to training remains to iterate over and over, uniformly sampling the training dataset. In this paper we explore a series of alternative training paradigms that leverage insights from hard-data-mining and dropout, simple enough to implement and use that can become the new training standard. The proposed Progressive Data Dropout reduces the number of effective epochs to as little as 12.4% of the baseline. This savings actually do not come at any cost for accuracy. Surprisingly, the proposed method improves accuracy by up to 4.82%. Our approach requires no changes to model architecture or optimizer, and can be applied across standard training pipelines, thus posing an excellent opportunity for wide adoption. Code can be found here: https://github.com/bazyagami/LearningWithRevision
△ Less
Submitted 6 June, 2025; v1 submitted 28 May, 2025;
originally announced May 2025.
-
Can Structured Data Reduce Epistemic Uncertainty?
Authors:
Shriram M S,
Sushmitha S,
Gayathri K S,
Shahina A
Abstract:
In this work, we present a framework that utilizes ontology alignment to improve the learning process of deep learning models. With this approach we show that models fine-tuned using ontologies learn a downstream task at a higher rate with better performance on a sequential classification task compared to the native version of the model. Additionally, we extend our work to showcase how subsumption…
▽ More
In this work, we present a framework that utilizes ontology alignment to improve the learning process of deep learning models. With this approach we show that models fine-tuned using ontologies learn a downstream task at a higher rate with better performance on a sequential classification task compared to the native version of the model. Additionally, we extend our work to showcase how subsumption mappings retrieved during the process of ontology alignment can help enhance Retrieval-Augmented Generation in Large Language Models. The results show that the responses obtained by using subsumption mappings show an increase of 8.97% in contextual similarity and a 1% increase in factual accuracy. We also use these scores to define our Hallucination Index and show that this approach reduces hallucination in LLMs by 4.847%.
△ Less
Submitted 14 October, 2024;
originally announced October 2024.
-
Fisher Mask Nodes for Language Model Merging
Authors:
Thennal D K,
Ganesh Nathan,
Suchithra M S
Abstract:
Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growi…
▽ More
Fine-tuning pre-trained models provides significant advantages in downstream performance. The ubiquitous nature of pre-trained models such as BERT and its derivatives in natural language processing has also led to a proliferation of task-specific fine-tuned models. As these models typically only perform one task well, additional training or ensembling is required in multi-task scenarios. The growing field of model merging provides a solution, dealing with the challenge of combining multiple task-specific models into a single multi-task model. In this study, we introduce a novel model merging method for Transformers, combining insights from previous work in Fisher-weighted averaging and the use of Fisher information in model pruning. Utilizing the Fisher information of mask nodes within the Transformer architecture, we devise a computationally efficient weighted-averaging scheme. Our method exhibits a regular and significant performance increase across various models in the BERT family, outperforming full-scale Fisher-weighted averaging in a fraction of the computational cost, with baseline performance improvements of up to +6.5 and a speedup between 57.4x and 321.7x across models. Our results prove the potential of our method in current multi-task learning environments and suggest its scalability and adaptability to new model architectures and learning scenarios.
△ Less
Submitted 3 May, 2024; v1 submitted 14 March, 2024;
originally announced March 2024.
-
PyRelationAL: a python library for active learning research and development
Authors:
Paul Scherer,
Alison Pouplin,
Alice Del Vecchio,
Suraj M S,
Oliver Bolton,
Jyothish Soman,
Jake P. Taylor-King,
Lindsay Edwards,
Thomas Gaudelet
Abstract:
Active learning (AL) is a sub-field of ML focused on the development of methods to iteratively and economically acquire data by strategically querying new data points that are the most useful for a particular task. Here, we introduce PyRelationAL, an open source library for AL research. We describe a modular toolkit based around a two step design methodology for composing pool-based active learnin…
▽ More
Active learning (AL) is a sub-field of ML focused on the development of methods to iteratively and economically acquire data by strategically querying new data points that are the most useful for a particular task. Here, we introduce PyRelationAL, an open source library for AL research. We describe a modular toolkit based around a two step design methodology for composing pool-based active learning strategies applicable to both single-acquisition and batch-acquisition strategies. This framework allows for the mathematical and practical specification of a broad number of existing and novel strategies under a consistent programming model and abstraction. Furthermore, we incorporate datasets and active learning tasks applicable to them to simplify comparative evaluation and benchmarking, along with an initial group of benchmarks across datasets included in this library. The toolkit is compatible with existing ML frameworks. PyRelationAL is maintained using modern software engineering practices -- with an inclusive contributor code of conduct -- to promote long term library quality and utilisation. PyRelationAL is available under a permissive Apache licence on PyPi and at https://github.com/RelationRx/pyrelational.
△ Less
Submitted 11 November, 2024; v1 submitted 23 May, 2022;
originally announced May 2022.
-
RECOVER: sequential model optimization platform for combination drug repurposing identifies novel synergistic compounds in vitro
Authors:
Paul Bertin,
Jarrid Rector-Brooks,
Deepak Sharma,
Thomas Gaudelet,
Andrew Anighoro,
Torsten Gross,
Francisco Martinez-Pena,
Eileen L. Tang,
Suraj M S,
Cristian Regep,
Jeremy Hayter,
Maksym Korablyov,
Nicholas Valiante,
Almer van der Sloot,
Mike Tyers,
Charles Roberts,
Michael M. Bronstein,
Luke L. Lairson,
Jake P. Taylor-King,
Yoshua Bengio
Abstract:
For large libraries of small molecules, exhaustive combinatorial chemical screens become infeasible to perform when considering a range of disease models, assay conditions, and dose ranges. Deep learning models have achieved state of the art results in silico for the prediction of synergy scores. However, databases of drug combinations are biased towards synergistic agents and these results do not…
▽ More
For large libraries of small molecules, exhaustive combinatorial chemical screens become infeasible to perform when considering a range of disease models, assay conditions, and dose ranges. Deep learning models have achieved state of the art results in silico for the prediction of synergy scores. However, databases of drug combinations are biased towards synergistic agents and these results do not necessarily generalise out of distribution. We employ a sequential model optimization search utilising a deep learning model to quickly discover synergistic drug combinations active against a cancer cell line, requiring substantially less screening than an exhaustive evaluation. Our small scale wet lab experiments only account for evaluation of ~5% of the total search space. After only 3 rounds of ML-guided in vitro experimentation (including a calibration round), we find that the set of drug pairs queried is enriched for highly synergistic combinations; two additional rounds of ML-guided experiments were performed to ensure reproducibility of trends. Remarkably, we rediscover drug combinations later confirmed to be under study within clinical trials. Moreover, we find that drug embeddings generated using only structural information begin to reflect mechanisms of action. Prior in silico benchmarking suggests we can enrich search queries by a factor of ~5-10x for highly synergistic drug combinations by using sequential rounds of evaluation when compared to random selection, or by a factor of >3x when using a pretrained model selecting all drug combinations at a single time point.
△ Less
Submitted 2 March, 2023; v1 submitted 6 February, 2022;
originally announced February 2022.
-
A Study on various state of the art of the Art Face Recognition System using Deep Learning Techniques
Authors:
Sukhada Chokkadi,
Sannidhan M S,
Sudeepa K B,
Abhir Bhandary
Abstract:
Considering the existence of very large amount of available data repositories and reach to the very advanced system of hardware, systems meant for facial identification ave evolved enormously over the past few decades. Sketch recognition is one of the most important areas that have evolved as an integral component adopted by the agencies of law administration in current trends of forensic science.…
▽ More
Considering the existence of very large amount of available data repositories and reach to the very advanced system of hardware, systems meant for facial identification ave evolved enormously over the past few decades. Sketch recognition is one of the most important areas that have evolved as an integral component adopted by the agencies of law administration in current trends of forensic science. Matching of derived sketches to photo images of face is also a difficult assignment as the considered sketches are produced upon the verbal explanation depicted by the eye witness of the crime scene and may have scarcity of sensitive elements that exist in the photograph as one can accurately depict due to the natural human error. Substantial amount of the novel research work carried out in this area up late used recognition system through traditional extraction and classification models. But very recently, few researches work focused on using deep learning techniques to take an advantage of learning models for the feature extraction and classification to rule out potential domain challenges. The first part of this review paper basically focuses on deep learning techniques used in face recognition and matching which as improved the accuracy of face recognition technique with training of huge sets of data. This paper also includes a survey on different techniques used to match composite sketches to human images which includes component-based representation approach, automatic composite sketch recognition technique etc.
△ Less
Submitted 19 November, 2019;
originally announced November 2019.
-
Architectural Pattern of Health Care System Using GSM Networks
Authors:
Meiappane. A,
Dr. V. Prasanna Venkatesan,
Selva Murugan. S,
Arun. A,
Ramachandran. A
Abstract:
Large-scale networked environments, such as the Internet, possess the characteristics of centralised data, centralised access and centralised control; this gives the user a powerful mechanism for building and integrating large repositories of centralised information from diverse resources set. However, a centralised network system with GSM Networks development for a hospital information systems or…
▽ More
Large-scale networked environments, such as the Internet, possess the characteristics of centralised data, centralised access and centralised control; this gives the user a powerful mechanism for building and integrating large repositories of centralised information from diverse resources set. However, a centralised network system with GSM Networks development for a hospital information systems or a health care information portal is still in its infancy. The shortcomings of the currently available tools have made the use of mobile devices more appealing. In mobile computing, the issues such as low bandwidth, high latency wireless Networks, loss or degradation of wireless connections, and network errors or failures need to be dealt with. Other issues to be addressed include system adaptability, reliability, robustness, extensibility, flexibility, and maintainability. GSM approach has emerged as the most viable approach for development of intelligent software applications for wireless mobile devices in a centralized environment, which gives higher band width of 900 MHz for transmission. The e-healthcare system that we have developed provides support for physicians, nurses, pharmacists and other healthcare professionals, as well as for patients and medical devices used to monitor patients. In this paper, we present the architecture and the demonstration prototype.
△ Less
Submitted 9 December, 2013;
originally announced December 2013.
