-
The Quest of Finding the Antidote to Sparse Double Descent
Authors:
Victor Quétu,
Marta Milovanović
Abstract:
In energy-efficient schemes, finding the optimal size of deep learning models is very important and has a broad impact. Meanwhile, recent studies have reported an unexpected phenomenon, the sparse double descent: as the model's sparsity increases, the performance first worsens, then improves, and finally deteriorates. Such a non-monotonic behavior raises serious questions about the optimal model's…
▽ More
In energy-efficient schemes, finding the optimal size of deep learning models is very important and has a broad impact. Meanwhile, recent studies have reported an unexpected phenomenon, the sparse double descent: as the model's sparsity increases, the performance first worsens, then improves, and finally deteriorates. Such a non-monotonic behavior raises serious questions about the optimal model's size to maintain high performance: the model needs to be sufficiently over-parametrized, but having too many parameters wastes training resources.
In this paper, we aim to find the best trade-off efficiently. More precisely, we tackle the occurrence of the sparse double descent and present some solutions to avoid it. Firstly, we show that a simple $\ell_2$ regularization method can help to mitigate this phenomenon but sacrifices the performance/sparsity compromise. To overcome this problem, we then introduce a learning scheme in which distilling knowledge regularizes the student model. Supported by experimental results achieved using typical image classification setups, we show that this approach leads to the avoidance of such a phenomenon.
△ Less
Submitted 31 August, 2023;
originally announced August 2023.
-
Sparse Double Descent in Vision Transformers: real or phantom threat?
Authors:
Victor Quétu,
Marta Milovanovic,
Enzo Tartaglione
Abstract:
Vision transformers (ViT) have been of broad interest in recent theoretical and empirical works. They are state-of-the-art thanks to their attention-based approach, which boosts the identification of key features and patterns within images thanks to the capability of avoiding inductive bias, resulting in highly accurate image analysis. Meanwhile, neoteric studies have reported a ``sparse double de…
▽ More
Vision transformers (ViT) have been of broad interest in recent theoretical and empirical works. They are state-of-the-art thanks to their attention-based approach, which boosts the identification of key features and patterns within images thanks to the capability of avoiding inductive bias, resulting in highly accurate image analysis. Meanwhile, neoteric studies have reported a ``sparse double descent'' phenomenon that can occur in modern deep-learning models, where extremely over-parametrized models can generalize well. This raises practical questions about the optimal size of the model and the quest over finding the best trade-off between sparsity and performance is launched: are Vision Transformers also prone to sparse double descent? Can we find a way to avoid such a phenomenon? Our work tackles the occurrence of sparse double descent on ViTs. Despite some works that have shown that traditional architectures, like Resnet, are condemned to the sparse double descent phenomenon, for ViTs we observe that an optimally-tuned $\ell_2$ regularization relieves such a phenomenon. However, everything comes at a cost: optimal lambda will sacrifice the potential compression of the ViT.
△ Less
Submitted 26 July, 2023;
originally announced July 2023.
-
Learn how to Prune Pixels for Multi-view Neural Image-based Synthesis
Authors:
Marta Milovanović,
Enzo Tartaglione,
Marco Cagnazzo,
Félix Henry
Abstract:
Image-based rendering techniques stand at the core of an immersive experience for the user, as they generate novel views given a set of multiple input images. Since they have shown good performance in terms of objective and subjective quality, the research community devotes great effort to their improvement. However, the large volume of data necessary to render at the receiver's side hinders appli…
▽ More
Image-based rendering techniques stand at the core of an immersive experience for the user, as they generate novel views given a set of multiple input images. Since they have shown good performance in terms of objective and subjective quality, the research community devotes great effort to their improvement. However, the large volume of data necessary to render at the receiver's side hinders applications in limited bandwidth environments or prevents their employment in real-time applications. We present LeHoPP, a method for input pixel pruning, where we examine the importance of each input pixel concerning the rendered view, and we avoid the use of irrelevant pixels. Even without retraining the image-based rendering network, our approach shows a good trade-off between synthesis quality and pixel rate. When tested in the general neural rendering framework, compared to other pruning baselines, LeHoPP gains between $0.9$ dB and $3.6$ dB on average.
△ Less
Submitted 5 May, 2023;
originally announced May 2023.
-
The Artists who Forged Themselves: Detecting Creativity in Art
Authors:
Milan Rajković,
Miloš Milovanović
Abstract:
Creativity and the understanding of cognitive processes involved in the creative process are relevant to all of human activities. Comprehension of creativity in the arts is of special interest due to the involvement of many scientific and non scientific disciplines. Using digital representation of paintings, we show that creative process in painting art may be objectively recognized within the mat…
▽ More
Creativity and the understanding of cognitive processes involved in the creative process are relevant to all of human activities. Comprehension of creativity in the arts is of special interest due to the involvement of many scientific and non scientific disciplines. Using digital representation of paintings, we show that creative process in painting art may be objectively recognized within the mathematical framework of self organization, a process characteristic of nonlinear dynamic systems and occurring in natural and social sciences. Unlike the artist identification process or the recognition of forgery, which presupposes the knowledge of the original work, our method requires no prior knowledge on the originality of the work of art. The original paintings are recognized as realizations of the creative process which, in general, is shown to correspond to self-organization of texture features which determine the aesthetic complexity of the painting. The method consists of the wavelet based statistical digital image processing and the measure of statistical complexity which represents the minimal (average) information necessary for optimal prediction. The statistical complexity is based on the properly defined causal states with optimal predictive properties. Two different time concepts related to the works of art are introduced: the internal time and the artistic time. The internal time of the artwork is determined by the span of causal dependencies between wavelet coefficients while the artistic time refers to the internal time during which complexity increases where complexity refers to compositional, aesthetic and structural arrangement of texture features. The method is illustrated by recognizing the original paintings from the copies made by the artists themselves, including the works of the famous surrealist painter René Magritte.
△ Less
Submitted 14 June, 2015;
originally announced June 2015.
-
Array Requirements for Scientific Applications and an Implementation for Microsoft SQL Server
Authors:
László Dobos,
Alexander Szalay,
José Blakeley,
Tamás Budavári,
István Csabai,
Dragan Tomic,
Milos Milovanovic,
Marko Tintor,
Andrija Jovanovic
Abstract:
This paper outlines certain scenarios from the fields of astrophysics and fluid dynamics simulations which require high performance data warehouses that support array data type. A common feature of all these use cases is that subsetting and preprocessing the data on the server side (as far as possible inside the database server process) is necessary to avoid the client-server overhead and to minim…
▽ More
This paper outlines certain scenarios from the fields of astrophysics and fluid dynamics simulations which require high performance data warehouses that support array data type. A common feature of all these use cases is that subsetting and preprocessing the data on the server side (as far as possible inside the database server process) is necessary to avoid the client-server overhead and to minimize IO utilization. Analyzing and summarizing the requirements of the various fields help software engineers to come up with a comprehensive design of an array extension to relational database systems that covers a wide range of scientific applications. We also present a working implementation of an array data type for Microsoft SQL Server 2008 to support large-scale scientific applications. We introduce the design of the array type, results from a performance evaluation, and discuss the lessons learned from this implementation. The library can be downloaded from our website at http://voservices.net/sqlarray/
△ Less
Submitted 8 October, 2011;
originally announced October 2011.
