A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition
Authors:
Dewan Tauhid Rahman,
Yeahia Sarker,
Antar Mazumder,
Md. Shamim Anower
Abstract:
This paper presents a novel knowledge distillation neural architecture leveraging efficient transformer networks for effective image classification. Natural images display intricate arrangements encompassing numerous extraneous elements. Vision transformers utilize localized patches to compute attention. However, exclusive dependence on patch segmentation proves inadequate in sufficiently encompas…
▽ More
This paper presents a novel knowledge distillation neural architecture leveraging efficient transformer networks for effective image classification. Natural images display intricate arrangements encompassing numerous extraneous elements. Vision transformers utilize localized patches to compute attention. However, exclusive dependence on patch segmentation proves inadequate in sufficiently encompassing the comprehensive nature of the image. To address this issue, we have proposed an inner-outer transformer-based architecture, which gives attention to the global and local aspects of the image. Moreover, The training of transformer models poses significant challenges due to their demanding resource, time, and data requirements. To tackle this, we integrate knowledge distillation into the architecture, enabling efficient learning. Leveraging insights from a larger teacher model, our approach enhances learning efficiency and effectiveness. Significantly, the transformer-in-transformer network acquires lightweight characteristics by means of distillation conducted within the feature extraction layer. Our featured network's robustness is established through substantial experimentation on the MNIST, CIFAR10, and CIFAR100 datasets, demonstrating commendable top-1 and top-5 accuracy. The conducted ablative analysis comprehensively validates the effectiveness of the chosen parameters and settings, showcasing their superiority against contemporary methodologies. Remarkably, the proposed Transformer-in-Transformer Network (TITN) model achieves impressive performance milestones across various datasets: securing the highest top-1 accuracy of 74.71% and a top-5 accuracy of 92.28% for the CIFAR100 dataset, attaining an unparalleled top-1 accuracy of 92.03% and top-5 accuracy of 99.80% for the CIFAR-10 dataset, and registering an exceptional top-1 accuracy of 99.56% for the MNIST dataset.
△ Less
Submitted 23 February, 2025;
originally announced February 2025.
Secrecy Outage Analysis of Energy Harvesting Relay-based Mixed UOWC-RF Network with Multiple Eavesdroppers
Authors:
Moloy Kumar Ghosh,
Milton Kumar Kundu,
Md Ibrahim,
A. S. M. Badrudduza,
Md. Shamim Anower,
Imran Shafique Ansari,
Ali A. Shaikhi,
Mohammed A. Mohandes
Abstract:
This work deals with the physical layer security performance of a dual-hop underwater optical communication (UOWC)-radio frequency (RF) network under the intruding attempts of multiple eavesdroppers via RF links. The intermediate decode and forward relay node between the underwater source and the destination transforms the optical signal into electrical form and re-transmits it to the destination…
▽ More
This work deals with the physical layer security performance of a dual-hop underwater optical communication (UOWC)-radio frequency (RF) network under the intruding attempts of multiple eavesdroppers via RF links. The intermediate decode and forward relay node between the underwater source and the destination transforms the optical signal into electrical form and re-transmits it to the destination node with the help of harvested energy by the relay from an integrated power beacon within the system. The source-to-relay link (UOWC) follows a mixture exponential generalized Gamma turbulence with pointing error impairments whereas all the remaining links (RF) undergo $κ-μ$ shadowed fading. With regards to the types of intruders, herein two scenarios are considered, i.e., colluding (\textit{Scenario-I}) and non-colluding (\textit{Scenario-II}) eavesdroppers and the analytical expressions of secure outage probability, probability of strictly positive secrecy capacity, and effective secrecy throughput are derived in closed form for each scenario. Furthermore, the impacts of UOWC and RF channel parameters as well as detection techniques on secrecy capacity are demonstrated, and following this a comparison between the two considered scenarios is demonstrated that reveals the collusion between the eavesdroppers imposes the most harmful threat on secrecy throughput but a better secrecy level can be attained adopting diversity at the destination and power beacon nodes along with heterodyne detection rather than intensity modulation and direct detection technique. Finally, all the derived expressions are corroborated via Monte Carlo simulations.
△ Less
Submitted 20 February, 2023;
originally announced February 2023.
