Showing 1–25 of 25 results for author: Ivanov, N

Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX

Authors: Nikita Martynov, Anastasia Mordasheva, Dmitriy Gorbetskiy, Danil Astafurov, Ulyana Isaeva, Elina Basyrova, Sergey Skachkov, Victoria Berestova, Nikolay Ivanov, Valeriia Zanina, Alena Fenogenova

Abstract: We introduce POLLUX, a comprehensive open-source benchmark designed to evaluate the generative capabilities of large language models (LLMs) in Russian. Our main contribution is a novel evaluation methodology that enhances the interpretability of LLM assessment. For each task type, we define a set of detailed criteria and develop a scoring protocol where models evaluate responses and provide justif… ▽ More We introduce POLLUX, a comprehensive open-source benchmark designed to evaluate the generative capabilities of large language models (LLMs) in Russian. Our main contribution is a novel evaluation methodology that enhances the interpretability of LLM assessment. For each task type, we define a set of detailed criteria and develop a scoring protocol where models evaluate responses and provide justifications for their ratings. This enables transparent, criteria-driven evaluation beyond traditional resource-consuming, side-by-side human comparisons. POLLUX includes a detailed, fine-grained taxonomy of 35 task types covering diverse generative domains such as code generation, creative writing, and practical assistant use cases, totaling 2,100 manually crafted and professionally authored prompts. Each task is categorized by difficulty (easy/medium/hard), with experts constructing the dataset entirely from scratch. We also release a family of LLM-as-a-Judge (7B and 32B) evaluators trained for nuanced assessment of generative outputs. This approach provides scalable, interpretable evaluation and annotation tools for model development, effectively replacing costly and less precise human judgments. △ Less

Submitted 27 June, 2025; v1 submitted 30 May, 2025; originally announced May 2025.

Comments: 178 pages

Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA

Authors: Sergey Pletenev, Maria Marina, Nikolay Ivanov, Daria Galimzianova, Nikita Krayko, Mikhail Salnikov, Vasily Konovalov, Alexander Panchenko, Viktor Moskvoretskii

Abstract: Large Language Models (LLMs) often hallucinate in question answering (QA) tasks. A key yet underexplored factor contributing to this is the temporality of questions -- whether they are evergreen (answers remain stable over time) or mutable (answers change). In this work, we introduce EverGreenQA, the first multilingual QA dataset with evergreen labels, supporting both evaluation and training. Usin… ▽ More Large Language Models (LLMs) often hallucinate in question answering (QA) tasks. A key yet underexplored factor contributing to this is the temporality of questions -- whether they are evergreen (answers remain stable over time) or mutable (answers change). In this work, we introduce EverGreenQA, the first multilingual QA dataset with evergreen labels, supporting both evaluation and training. Using EverGreenQA, we benchmark 12 modern LLMs to assess whether they encode question temporality explicitly (via verbalized judgments) or implicitly (via uncertainty signals). We also train EG-E5, a lightweight multilingual classifier that achieves SoTA performance on this task. Finally, we demonstrate the practical utility of evergreen classification across three applications: improving self-knowledge estimation, filtering QA datasets, and explaining GPT-4o retrieval behavior. △ Less

Submitted 27 May, 2025; originally announced May 2025.

Track Anything Annotate: Video annotation and dataset generation of computer vision models

Authors: Nikita Ivanov, Mark Klimov, Dmitry Glukhikh, Tatiana Chernysheva, Igor Glukhikh

Abstract: Modern machine learning methods require significant amounts of labelled data, making the preparation process time-consuming and resource-intensive. In this paper, we propose to consider the process of prototyping a tool for annotating and generating training datasets based on video tracking and segmentation. We examine different approaches to solving this problem, from technology selection through… ▽ More Modern machine learning methods require significant amounts of labelled data, making the preparation process time-consuming and resource-intensive. In this paper, we propose to consider the process of prototyping a tool for annotating and generating training datasets based on video tracking and segmentation. We examine different approaches to solving this problem, from technology selection through to final implementation. The developed prototype significantly accelerates dataset generation compared to manual annotation. All resources are available at https://github.com/lnikioffic/track-anything-annotate △ Less

Submitted 23 May, 2025; originally announced May 2025.

Comments: 9 pages, 11 figures

LLM-Independent Adaptive RAG: Let the Question Speak for Itself

Authors: Maria Marina, Nikolay Ivanov, Sergey Pletenev, Mikhail Salnikov, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Alexander Panchenko, Viktor Moskvoretskii

Abstract: Large Language Models~(LLMs) are prone to hallucinations, and Retrieval-Augmented Generation (RAG) helps mitigate this, but at a high computational cost while risking misinformation. Adaptive retrieval aims to retrieve only when necessary, but existing approaches rely on LLM-based uncertainty estimation, which remain inefficient and impractical. In this study, we introduce lightweight LLM-independ… ▽ More Large Language Models~(LLMs) are prone to hallucinations, and Retrieval-Augmented Generation (RAG) helps mitigate this, but at a high computational cost while risking misinformation. Adaptive retrieval aims to retrieve only when necessary, but existing approaches rely on LLM-based uncertainty estimation, which remain inefficient and impractical. In this study, we introduce lightweight LLM-independent adaptive retrieval methods based on external information. We investigated 27 features, organized into 7 groups, and their hybrid combinations. We evaluated these methods on 6 QA datasets, assessing the QA performance and efficiency. The results show that our approach matches the performance of complex LLM-based methods while achieving significant efficiency gains, demonstrating the potential of external information for adaptive retrieval. △ Less

Submitted 7 May, 2025; originally announced May 2025.

Comments: 11 pages, 5 figures, 2 tables

Disjunctive Complexity

Authors: Nikita Ivanov, Alexander Rubtsov, Michael Vyalyi

Abstract: A recently introduced measure of Boolean functions complexity--disjunc\-tive complexity (DC)--is compared with other complexity measures: the space complexity of streaming algorithms and the complexity of nondeterministic branching programs (NBP). We show that DC is incomparable with NBP. Specifically, we present a function that has low NBP but has subexponential DC. Conversely, we provide argumen… ▽ More A recently introduced measure of Boolean functions complexity--disjunc\-tive complexity (DC)--is compared with other complexity measures: the space complexity of streaming algorithms and the complexity of nondeterministic branching programs (NBP). We show that DC is incomparable with NBP. Specifically, we present a function that has low NBP but has subexponential DC. Conversely, we provide arguments based on computational complexity conjectures to show that DC can superpolynomially exceed NBP in certain cases. Additionally, we prove that the monotone version of NBP complexity is strictly weaker than DC. We prove that the space complexity of one-pass streaming algorithms is strictly weaker than DC. Furthermore, we introduce a generalization of streaming algorithms that captures the full power of DC. This generalization can be expressed in terms of nondeterministic algorithms that irreversibly write 1s to entries of a Boolean vector (i.e., changes from 1 to 0 are not allowed). Finally, we discuss an unusual phenomenon in disjunctive complexity: the existence of uniformly hard functions. These functions exhibit the property that their disjunctive complexity is maximized, and this property extends to all functions dominated by them. △ Less

Submitted 29 March, 2025; originally announced March 2025.

Comments: Submitted to DCFS 2025

ACM Class: F.1.3

Uncertainty Quantification for Data-Driven Machine Learning Models in Nuclear Engineering Applications: Where We Are and What Do We Need?

Authors: Xu Wu, Lesego E. Moloko, Pavel M. Bokov, Gregory K. Delipei, Joshua Kaizer, Kostadin N. Ivanov

Abstract: Machine learning (ML) has been leveraged to tackle a diverse range of tasks in almost all branches of nuclear engineering. Many of the successes in ML applications can be attributed to the recent performance breakthroughs in deep learning, the growing availability of computational power, data, and easy-to-use ML libraries. However, these empirical successes have often outpaced our formal understan… ▽ More Machine learning (ML) has been leveraged to tackle a diverse range of tasks in almost all branches of nuclear engineering. Many of the successes in ML applications can be attributed to the recent performance breakthroughs in deep learning, the growing availability of computational power, data, and easy-to-use ML libraries. However, these empirical successes have often outpaced our formal understanding of the ML algorithms. An important but under-rated area is uncertainty quantification (UQ) of ML. ML-based models are subject to approximation uncertainty when they are used to make predictions, due to sources including but not limited to, data noise, data coverage, extrapolation, imperfect model architecture and the stochastic training process. The goal of this paper is to clearly explain and illustrate the importance of UQ of ML. We will elucidate the differences in the basic concepts of UQ of physics-based models and data-driven ML models. Various sources of uncertainties in physical modeling and data-driven modeling will be discussed, demonstrated, and compared. We will also present and demonstrate a few techniques to quantify the ML prediction uncertainties. Finally, we will discuss the need for building a verification, validation and UQ framework to establish ML credibility. △ Less

Submitted 16 March, 2025; originally announced March 2025.

Comments: 34 pages, 13 figures, invited journal article from BEPU-2024 conference

Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home

Authors: Viktor Moskvoretskii, Maria Lysyuk, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, Alexander Panchenko

Abstract: Retrieval Augmented Generation (RAG) improves correctness of Question Answering (QA) and addresses hallucinations in Large Language Models (LLMs), yet greatly increase computational costs. Besides, RAG is not always needed as may introduce irrelevant information. Recent adaptive retrieval methods integrate LLMs' intrinsic knowledge with external information appealing to LLM self-knowledge, but the… ▽ More Retrieval Augmented Generation (RAG) improves correctness of Question Answering (QA) and addresses hallucinations in Large Language Models (LLMs), yet greatly increase computational costs. Besides, RAG is not always needed as may introduce irrelevant information. Recent adaptive retrieval methods integrate LLMs' intrinsic knowledge with external information appealing to LLM self-knowledge, but they often neglect efficiency evaluations and comparisons with uncertainty estimation techniques. We bridge this gap by conducting a comprehensive analysis of 35 adaptive retrieval methods, including 8 recent approaches and 27 uncertainty estimation techniques, across 6 datasets using 10 metrics for QA performance, self-knowledge, and efficiency. Our findings show that uncertainty estimation techniques often outperform complex pipelines in terms of efficiency and self-knowledge, while maintaining comparable QA performance. △ Less

Submitted 21 February, 2025; v1 submitted 22 January, 2025; originally announced January 2025.

Comments: The code and data are at https://github.com/s-nlp/AdaRAGUE

GigaPevt: Multimodal Medical Assistant

Authors: Pavel Blinov, Konstantin Egorov, Ivan Sviridov, Nikolay Ivanov, Stepan Botman, Evgeniy Tagin, Stepan Kudin, Galina Zubkova, Andrey Savchenko

Abstract: Building an intelligent and efficient medical assistant is still a challenging AI problem. The major limitation comes from the data modality scarceness, which reduces comprehensive patient perception. This demo paper presents the GigaPevt, the first multimodal medical assistant that combines the dialog capabilities of large language models with specialized medical models. Such an approach shows im… ▽ More Building an intelligent and efficient medical assistant is still a challenging AI problem. The major limitation comes from the data modality scarceness, which reduces comprehensive patient perception. This demo paper presents the GigaPevt, the first multimodal medical assistant that combines the dialog capabilities of large language models with specialized medical models. Such an approach shows immediate advantages in dialog quality and metric performance, with a 1.18% accuracy improvement in the question-answering task. △ Less

Submitted 30 July, 2024; v1 submitted 26 February, 2024; originally announced February 2024.

Comments: IJCAI 2024, 4 pages, 2 figures, 2 tables

MSC Class: 68T07 ACM Class: I.2.1

Journal ref: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI) Demo Track, 2024, pp. 8614-8618

Co-orchestration of Multiple Instruments to Uncover Structure-Property Relationships in Combinatorial Libraries

Authors: Boris N. Slautin, Utkarsh Pratiush, Ilia N. Ivanov, Yongtao Liu, Rohit Pant, Xiaohang Zhang, Ichiro Takeuchi, Maxim A. Ziatdinov, Sergei V. Kalinin

Abstract: The rapid growth of automated and autonomous instrumentations brings forth an opportunity for the co-orchestration of multimodal tools, equipped with multiple sequential detection methods, or several characterization tools to explore identical samples. This can be exemplified by the combinatorial libraries that can be explored in multiple locations by multiple tools simultaneously, or downstream c… ▽ More The rapid growth of automated and autonomous instrumentations brings forth an opportunity for the co-orchestration of multimodal tools, equipped with multiple sequential detection methods, or several characterization tools to explore identical samples. This can be exemplified by the combinatorial libraries that can be explored in multiple locations by multiple tools simultaneously, or downstream characterization in automated synthesis systems. In the co-orchestration approaches, information gained in one modality should accelerate the discovery of other modalities. Correspondingly, the orchestrating agent should select the measurement modality based on the anticipated knowledge gain and measurement cost. Here, we propose and implement a co-orchestration approach for conducting measurements with complex observables such as spectra or images. The method relies on combining dimensionality reduction by variational autoencoders with representation learning for control over the latent space structure, and integrated into iterative workflow via multi-task Gaussian Processes (GP). This approach further allows for the native incorporation of the system's physics via a probabilistic model as a mean function of the GP. We illustrated this method for different modalities of piezoresponse force microscopy and micro-Raman on combinatorial $Sm-BiFeO_3$ library. However, the proposed framework is general and can be extended to multiple measurement modalities and arbitrary dimensionality of measured signals. The analysis code that supports the funding is publicly available at https://github.com/Slautin/2024_Co-orchestration. △ Less

Submitted 17 March, 2024; v1 submitted 3 February, 2024; originally announced February 2024.

Comments: 22 pages, 9 figures

A Practical Survey on Emerging Threats from AI-driven Voice Attacks: How Vulnerable are Commercial Voice Control Systems?

Authors: Yuanda Wang, Qiben Yan, Nikolay Ivanov, Xun Chen

Abstract: The emergence of Artificial Intelligence (AI)-driven audio attacks has revealed new security vulnerabilities in voice control systems. While researchers have introduced a multitude of attack strategies targeting voice control systems (VCS), the continual advancements of VCS have diminished the impact of many such attacks. Recognizing this dynamic landscape, our study endeavors to comprehensively a… ▽ More The emergence of Artificial Intelligence (AI)-driven audio attacks has revealed new security vulnerabilities in voice control systems. While researchers have introduced a multitude of attack strategies targeting voice control systems (VCS), the continual advancements of VCS have diminished the impact of many such attacks. Recognizing this dynamic landscape, our study endeavors to comprehensively assess the resilience of commercial voice control systems against a spectrum of malicious audio attacks. Through extensive experimentation, we evaluate six prominent attack techniques across a collection of voice control interfaces and devices. Contrary to prevailing narratives, our results suggest that commercial voice control systems exhibit enhanced resistance to existing threats. Particularly, our research highlights the ineffectiveness of white-box attacks in black-box scenarios. Furthermore, the adversaries encounter substantial obstacles in obtaining precise gradient estimations during query-based interactions with commercial systems, such as Apple Siri and Samsung Bixby. Meanwhile, we find that current defense strategies are not completely immune to advanced attacks. Our findings contribute valuable insights for enhancing defense mechanisms in VCS. Through this survey, we aim to raise awareness within the academic community about the security concerns of VCS and advocate for continued research in this crucial area. △ Less

Submitted 4 January, 2024; v1 submitted 10 December, 2023; originally announced December 2023.

Comments: 14 pages

DynamicFL: Balancing Communication Dynamics and Client Manipulation for Federated Learning

Authors: Bocheng Chen, Nikolay Ivanov, Guangjing Wang, Qiben Yan

Abstract: Federated Learning (FL) is a distributed machine learning (ML) paradigm, aiming to train a global model by exploiting the decentralized data across millions of edge devices. Compared with centralized learning, FL preserves the clients' privacy by refraining from explicitly downloading their data. However, given the geo-distributed edge devices (e.g., mobile, car, train, or subway) with highly dyna… ▽ More Federated Learning (FL) is a distributed machine learning (ML) paradigm, aiming to train a global model by exploiting the decentralized data across millions of edge devices. Compared with centralized learning, FL preserves the clients' privacy by refraining from explicitly downloading their data. However, given the geo-distributed edge devices (e.g., mobile, car, train, or subway) with highly dynamic networks in the wild, aggregating all the model updates from those participating devices will result in inevitable long-tail delays in FL. This will significantly degrade the efficiency of the training process. To resolve the high system heterogeneity in time-sensitive FL scenarios, we propose a novel FL framework, DynamicFL, by considering the communication dynamics and data quality across massive edge devices with a specially designed client manipulation strategy. \ours actively selects clients for model updating based on the network prediction from its dynamic network conditions and the quality of its training data. Additionally, our long-term greedy strategy in client selection tackles the problem of system performance degradation caused by short-term scheduling in a dynamic network. Lastly, to balance the trade-off between client performance evaluation and client manipulation granularity, we dynamically adjust the length of the observation window in the training process to optimize the long-term system efficiency. Compared with the state-of-the-art client selection scheme in FL, \ours can achieve a better model accuracy while consuming only 18.9\% -- 84.0\% of the wall-clock time. Our component-wise and sensitivity studies further demonstrate the robustness of \ours under various real-life scenarios. △ Less

Submitted 16 July, 2023; originally announced August 2023.

Journal ref: 2023 SECON

Security Defense For Smart Contracts: A Comprehensive Survey

Authors: Nikolay Ivanov, Chenning Li, Qiben Yan, Zhiyuan Sun, Zhichao Cao, Xiapu Luo

Abstract: The blockchain technology has been used for recording state transitions of smart contracts - decentralized applications that can be invoked through external transactions. Smart contracts gained popularity and accrued hundreds of billions of dollars in market capitalization in recent years. Unfortunately, like all other programs, smart contracts are prone to security vulnerabilities that have incur… ▽ More The blockchain technology has been used for recording state transitions of smart contracts - decentralized applications that can be invoked through external transactions. Smart contracts gained popularity and accrued hundreds of billions of dollars in market capitalization in recent years. Unfortunately, like all other programs, smart contracts are prone to security vulnerabilities that have incurred multimillion-dollar damages over the past decade. As a result, many automated threat mitigation solutions have been proposed to counter the security issues of smart contracts. These threat mitigation solutions include various tools and methods that are challenging to compare. This survey develops a comprehensive classification taxonomy of smart contract threat mitigation solutions within five orthogonal dimensions: defense modality, core method, targeted contracts, input-output data mapping, and threat model. We classify 133 existing threat mitigation solutions using our taxonomy and confirm that the proposed five dimensions allow us to concisely and accurately describe any smart contract threat mitigation solution. In addition to learning what the threat mitigation solutions do, we also show how these solutions work by synthesizing their actual designs into a set of uniform workflows corresponding to the eight existing defense core methods. We further create an integrated coverage map for the known smart contract vulnerabilities by the existing threat mitigation solutions. Finally, we perform the evidence-based evolutionary analysis, in which we identify trends and future perspectives of threat mitigation in smart contracts and pinpoint major weaknesses of the existing methodologies. For the convenience of smart contract security developers, auditors, users, and researchers, we deploy a regularly updated comprehensive open-source online registry of threat mitigation solutions. △ Less

Submitted 9 May, 2023; v1 submitted 14 February, 2023; originally announced February 2023.

Comments: ACM Computing Surveys (CSUR)

TxT: Real-time Transaction Encapsulation for Ethereum Smart Contracts

Authors: Nikolay Ivanov, Qiben Yan, Anurag Kompalli

Abstract: Ethereum is a permissionless blockchain ecosystem that supports execution of smart contracts, the key enablers of decentralized finance (DeFi) and non-fungible tokens (NFT). However, the expressiveness of Ethereum smart contracts is a double-edged sword: while it enables blockchain programmability, it also introduces security vulnerabilities, i.e., the exploitable discrepancies between expected an… ▽ More Ethereum is a permissionless blockchain ecosystem that supports execution of smart contracts, the key enablers of decentralized finance (DeFi) and non-fungible tokens (NFT). However, the expressiveness of Ethereum smart contracts is a double-edged sword: while it enables blockchain programmability, it also introduces security vulnerabilities, i.e., the exploitable discrepancies between expected and actual behaviors of the contract code. To address these discrepancies and increase the vulnerability coverage, we propose a new smart contract security testing approach called transaction encapsulation. The core idea lies in the local execution of transactions on a fully-synchronized yet isolated Ethereum node, which creates a preview of outcomes of transaction sequences on the current state of blockchain. This approach poses a critical technical challenge -- the well-known time-of-check/time-of-use (TOCTOU) problem, i.e., the assurance that the final transactions will exhibit the same execution paths as the encapsulated test transactions. In this work, we determine the exact conditions for guaranteed execution path replicability of the tested transactions, and implement a transaction testing tool, TxT, which reveals the actual outcomes of Ethereum transactions. To ensure the correctness of testing, TxT deterministically verifies whether a given sequence of transactions ensues an identical execution path on the current state of blockchain. We analyze over 1.3 billion Ethereum transactions and determine that 96.5% of them can be verified by TxT. We further show that TxT successfully reveals the suspicious behaviors associated with 31 out of 37 vulnerabilities (83.8% coverage) in the smart contract weakness classification (SWC) registry. In comparison, the vulnerability coverage of all the existing defense approaches combined only reaches 40.5%. △ Less

Submitted 20 January, 2023; originally announced January 2023.

Comments: To appear in IEEE Transactions on Information Forensics and Security

Prediction and Uncertainty Quantification of SAFARI-1 Axial Neutron Flux Profiles with Neural Networks

Authors: Lesego E. Moloko, Pavel M. Bokov, Xu Wu, Kostadin N. Ivanov

Abstract: Artificial Neural Networks (ANNs) have been successfully used in various nuclear engineering applications, such as predicting reactor physics parameters within reasonable time and with a high level of accuracy. Despite this success, they cannot provide information about the model prediction uncertainties, making it difficult to assess ANN prediction credibility, especially in extrapolated domains.… ▽ More Artificial Neural Networks (ANNs) have been successfully used in various nuclear engineering applications, such as predicting reactor physics parameters within reasonable time and with a high level of accuracy. Despite this success, they cannot provide information about the model prediction uncertainties, making it difficult to assess ANN prediction credibility, especially in extrapolated domains. In this study, Deep Neural Networks (DNNs) are used to predict the assembly axial neutron flux profiles in the SAFARI-1 research reactor, with quantified uncertainties in the ANN predictions and extrapolation to cycles not used in the training process. The training dataset consists of copper-wire activation measurements, the axial measurement locations and the measured control bank positions obtained from the reactor's historical cycles. Uncertainty Quantification of the regular DNN models' predictions is performed using Monte Carlo Dropout (MCD) and Bayesian Neural Networks solved by Variational Inference (BNN VI). The regular DNNs, DNNs solved with MCD and BNN VI results agree very well among each other as well as with the new measured dataset not used in the training process, thus indicating good prediction and generalization capability. The uncertainty bands produced by MCD and BNN VI agree very well, and in general, they can fully envelop the noisy measurement data points. The developed ANNs are useful in supporting the experimental measurements campaign and neutronics code Verification and Validation (V&V). △ Less

Submitted 15 November, 2022; originally announced November 2022.

Comments: 34 pages, 12 figures

Is Rust C++-fast? Benchmarking System Languages on Everyday Routines

Authors: Nikolay Ivanov

Abstract: Rust is a relatively new system programming language that has been experiencing a rapid adoption in the past 10 years. Rust incorporates a memory ownership model enforced at a compile time. Since this model involves zero runtime overhead, programs written in Rust are not only memory-safe but also fast, leading to performance comparable to C and C++. Multiple existing benchmarks comparing the perfo… ▽ More Rust is a relatively new system programming language that has been experiencing a rapid adoption in the past 10 years. Rust incorporates a memory ownership model enforced at a compile time. Since this model involves zero runtime overhead, programs written in Rust are not only memory-safe but also fast, leading to performance comparable to C and C++. Multiple existing benchmarks comparing the performance of Rust with other languages focus on rarely used superficial algorithms, leading to somewhat inconclusive results. In this work, we conduct a comparative performance benchmark of Rust and C++ using commonly used algorithms and data structures rather than exotic ones. Our evaluation shows that the overall performance of Rust is similar to C++, with only minor disadvantage. We also demonstrate that in some Rust routines are slightly faster than the ones of C++. △ Less

Submitted 19 September, 2022; originally announced September 2022.

Comments: Michigan State University

Decentralization Paradox: A Study of Hegemonic and Risky ERC-20 Tokens

Authors: Nikolay Ivanov, Qiben Yan

Abstract: In this work, we explore the class of Ethereum smart contracts called the administrated ERC20 tokens. We demonstrate that these contracts are more owner-controlled and less safe than the services they try to disrupt, such as banks and centralized online payment systems. We develop a binary classifier for identification of administrated ERC20 tokens, and conduct extensive data analysis, which revea… ▽ More In this work, we explore the class of Ethereum smart contracts called the administrated ERC20 tokens. We demonstrate that these contracts are more owner-controlled and less safe than the services they try to disrupt, such as banks and centralized online payment systems. We develop a binary classifier for identification of administrated ERC20 tokens, and conduct extensive data analysis, which reveals that nearly 9 out of 10 ERC20 tokens on Ethereum are administrated, and thereby unsafe to engage with even under the assumption of trust towards their owners. We design and implement SafelyAdministrated - a Solidity abstract class that safeguards users of administrated ERC20 tokens from adversarial attacks or frivolous behavior of the tokens' owners. △ Less

Submitted 17 September, 2022; originally announced September 2022.

Comments: 2022 Engineering Graduate Research Symposium (EGRS)

Et tu, Blockchain? Outsmarting Smart Contracts via Social Engineering

Authors: Nikolay Ivanov, Qiben Yan

Abstract: We reveal six zero-day social engineering attacks in Ethereum, and subdivide them into two classes: Address Manipulation and Homograph. We demonstrate the attacks by embedding them in source codes of five popular smart contracts with combined market capitalization of over \$29 billion, and show that the attacks have the ability to remain dormant during the testing phase and activate only after pro… ▽ More We reveal six zero-day social engineering attacks in Ethereum, and subdivide them into two classes: Address Manipulation and Homograph. We demonstrate the attacks by embedding them in source codes of five popular smart contracts with combined market capitalization of over \$29 billion, and show that the attacks have the ability to remain dormant during the testing phase and activate only after production deployment. We analyze 85,656 open source smart contracts and find 1,027 contracts that can be directly used for performing social engineering attacks. For responsible disclosure, we contact seven smart contract security firms. In the spirit of open research, we make the source codes of the attack benchmark, tools, and datasets available to the public. △ Less

Submitted 17 September, 2022; originally announced September 2022.

Comments: 14th annual Graduate Academic Conference (GAC). arXiv admin note: text overlap with arXiv:2105.00132

SuperVoice: Text-Independent Speaker Verification Using Ultrasound Energy in Human Speech

Authors: Hanqing Guo, Qiben Yan, Nikolay Ivanov, Ying Zhu, Li Xiao, Eric J. Hunter

Abstract: Voice-activated systems are integrated into a variety of desktop, mobile, and Internet-of-Things (IoT) devices. However, voice spoofing attacks, such as impersonation and replay attacks, in which malicious attackers synthesize the voice of a victim or simply replay it, have brought growing security concerns. Existing speaker verification techniques distinguish individual speakers via the spectrogr… ▽ More Voice-activated systems are integrated into a variety of desktop, mobile, and Internet-of-Things (IoT) devices. However, voice spoofing attacks, such as impersonation and replay attacks, in which malicious attackers synthesize the voice of a victim or simply replay it, have brought growing security concerns. Existing speaker verification techniques distinguish individual speakers via the spectrographic features extracted from an audible frequency range of voice commands. However, they often have high error rates and/or long delays. In this paper, we explore a new direction of human voice research by scrutinizing the unique characteristics of human speech at the ultrasound frequency band. Our research indicates that the high-frequency ultrasound components (e.g. speech fricatives) from 20 to 48 kHz can significantly enhance the security and accuracy of speaker verification. We propose a speaker verification system, SUPERVOICE that uses a two-stream DNN architecture with a feature fusion mechanism to generate distinctive speaker models. To test the system, we create a speech dataset with 12 hours of audio (8,950 voice samples) from 127 participants. In addition, we create a second spoofed voice dataset to evaluate its security. In order to balance between controlled recordings and real-world applications, the audio recordings are collected from two quiet rooms by 8 different recording devices, including 7 smartphones and an ultrasound microphone. Our evaluation shows that SUPERVOICE achieves 0.58% equal error rate in the speaker verification task, it only takes 120 ms for testing an incoming utterance, outperforming all existing speaker verification systems. Moreover, within 91 ms processing time, SUPERVOICE achieves 0% equal error rate in detecting replay attacks launched by 5 different loudspeakers. △ Less

Submitted 28 May, 2022; originally announced May 2022.

EthClipper: A Clipboard Meddling Attack on Hardware Wallets with Address Verification Evasion

Authors: Nikolay Ivanov, Qiben Yan

Abstract: Hardware wallets are designed to withstand malware attacks by isolating their private keys from the cyberspace, but they are vulnerable to the attacks that fake an address stored in a clipboard. To prevent such attacks, a hardware wallet asks the user to verify the recipient address shown on the wallet display. Since crypto addresses are long sequences of random symbols, their manual verification… ▽ More Hardware wallets are designed to withstand malware attacks by isolating their private keys from the cyberspace, but they are vulnerable to the attacks that fake an address stored in a clipboard. To prevent such attacks, a hardware wallet asks the user to verify the recipient address shown on the wallet display. Since crypto addresses are long sequences of random symbols, their manual verification becomes a difficult task. Consequently, many users of hardware wallets elect to verify only a few symbols in the address, and this can be exploited by an attacker. In this work, we introduce EthClipper, an attack that targets owners of hardware wallets on the Ethereum platform. EthClipper malware queries a distributed database of pre-mined accounts in order to select the address with maximum visual similarity to the original one. We design and implement a EthClipper malware, which we test on Trezor, Ledger, and KeepKey wallets. To deliver computation and storage resources for the attack, we implement a distributed service, ClipperCloud, and test it on different deployment environments. Our evaluation shows that with off-the-shelf PCs and NAS storage, an attacker would be able to mine a database capable of matching 25% of the digits in an address to achieve a 50% chance of finding a fitting fake address. For responsible disclosure, we have contacted the manufactures of the hardware wallets used in the attack evaluation, and they all confirm the danger of EthClipper. △ Less

Submitted 31 August, 2021; originally announced August 2021.

Comments: IEEE Conference on Communications and Network Security (CNS 2021)

Rectifying Administrated ERC20 Tokens

Authors: Nikolay Ivanov, Hanqing Guo, Qiben Yan

Abstract: The developers of Ethereum smart contracts often implement administrating patterns, such as censoring certain users, creating or destroying balances on demand, destroying smart contracts, or injecting arbitrary code. These routines turn an ERC20 token into an administrated token - the type of Ethereum smart contract that we scrutinize in this research. We discover that many smart contracts are adm… ▽ More The developers of Ethereum smart contracts often implement administrating patterns, such as censoring certain users, creating or destroying balances on demand, destroying smart contracts, or injecting arbitrary code. These routines turn an ERC20 token into an administrated token - the type of Ethereum smart contract that we scrutinize in this research. We discover that many smart contracts are administrated, and the owners of these tokens carry lesser social and legal responsibilities compared to the traditional centralized actors that those tokens intend to disrupt. This entails two major problems: a) the owners of the tokens have the ability to quickly steal all the funds and disappear from the market; and b) if the private key of the owner's account is stolen, all the assets might immediately turn into the property of the attacker. We develop a pattern recognition framework based on 9 syntactic features characterizing administrated ERC20 tokens, which we use to analyze existing smart contracts deployed on Ethereum Mainnet. Our analysis of 84,062 unique Ethereum smart contracts reveals that nearly 58% of them are administrated ERC20 tokens, which accounts for almost 90% of all ERC20 tokens deployed on Ethereum. To protect users from the frivolousness of unregulated token owners without depriving the ability of these owners to properly manage their tokens, we introduce SafelyAdministrated - a library that enforces a responsible ownership and management of ERC20 tokens. The library introduces three mechanisms: deferred maintenance, board of trustees and safe pause. We implement and test SafelyAdministrated in the form of Solidity abstract contract, which is ready to be used by the next generation of safely administrated ERC20 tokens. △ Less

Submitted 17 July, 2021; originally announced July 2021.

Comments: 23rd International Conference on Information and Communications Security (ICICS 2021)

System-Wide Security for Offline Payment Terminals

Authors: Nikolay Ivanov, Qiben Yan

Abstract: Most self-service payment terminals require network connectivity for processing electronic payments. The necessity to maintain network connectivity increases costs, introduces cybersecurity risks, and significantly limits the number of places where the terminals can be installed. Leading payment service providers have proposed offline payment solutions that rely on algorithmically generated paymen… ▽ More Most self-service payment terminals require network connectivity for processing electronic payments. The necessity to maintain network connectivity increases costs, introduces cybersecurity risks, and significantly limits the number of places where the terminals can be installed. Leading payment service providers have proposed offline payment solutions that rely on algorithmically generated payment tokens. Existing payment token solutions, however, require complex mechanisms for authentication, transaction management, and most importantly, security risk management. In this paper, we present VolgaPay, a blockchain-based system that allows merchants to deploy secure offline payment terminal infrastructure that does not require collection and storage of any sensitive data. We design a novel payment protocol which mitigates security threats for all the participants of VolgaPay, such that the maximum loss from gaining full access to any component by an adversary incurs only a limited scope of harm. We achieve significant enhancements in security, operation efficiency, and cost reduction via a combination of polynomial multi-hash chain micropayment channels and blockchain grafting for off-chain channel state transition. We implement the VolgaPay payment system, and with thorough evaluation and security analysis, we demonstrate that VolgaPay is capable of delivering a fast, secure, and cost-efficient solution for offline payment terminals. △ Less

Submitted 18 July, 2021; originally announced July 2021.

Comments: 17th EAI International Conference on Security and Privacy in Communication Networks (SecureComm 2021)

Blockumulus: A Scalable Framework for Smart Contracts on the Cloud

Authors: Nikolay Ivanov, Qiben Yan, Qingyang Wang

Abstract: Public blockchains have spurred the growing popularity of decentralized transactions and smart contracts, but they exhibit limitations on the transaction throughput, storage, and computation. To avoid transaction gridlock, public blockchains impose large fees and per-block resource limits, making it difficult to accommodate the ever-growing transaction demand. Previous research endeavors to improv… ▽ More Public blockchains have spurred the growing popularity of decentralized transactions and smart contracts, but they exhibit limitations on the transaction throughput, storage, and computation. To avoid transaction gridlock, public blockchains impose large fees and per-block resource limits, making it difficult to accommodate the ever-growing transaction demand. Previous research endeavors to improve the scalability of blockchain through various technologies, such as side-chaining, sharding, secured off-chain computation, communication network optimizations, and efficient consensus protocols. However, these approaches have not attained a widespread adoption due to their inability in delivering a cloud-like performance, in terms of the scalability in transaction throughput, storage, and compute capacity. In this work, we determine that the major obstacle to public blockchain scalability is their underlying unstructured P2P networks. We further show that a centralized network can support the deployment of decentralized smart contracts. We propose a novel approach for achieving scalable decentralization: instead of trying to make blockchain scalable, we deliver decentralization to already scalable cloud by using an Ethereum smart contract. We introduce Blockumulus, a framework that can deploy decentralized cloud smart contract environments using a novel technique called overlay consensus. Through experiments, we demonstrate that Blockumulus is scalable in all three dimensions: computation, data storage, and transaction throughput. Besides eliminating the current code execution and storage restrictions, Blockumulus delivers a transaction latency between 2 and 5 seconds under normal load. Moreover, the stress test of our prototype reveals the ability to execute 20,000 simultaneous transactions under 26 seconds, which is on par with the average throughput of worldwide credit card transactions. △ Less

Submitted 10 July, 2021; originally announced July 2021.

Comments: 41st IEEE International Conference on Distributed Computing Systems (ICDCS 2021)

Constraint-Based Inference of Heuristics for Foreign Exchange Trade Model Optimization

Authors: Nikolay Ivanov, Qiben Yan

Abstract: The Foreign Exchange (Forex) is a large decentralized market, on which trading analysis and algorithmic trading are popular. Research efforts have been focusing on proof of efficiency of certain technical indicators. We demonstrate, however, that the values of indicator functions are not reproducible and often reduce the number of trade opportunities, compared to price-action trading. In this wo… ▽ More The Foreign Exchange (Forex) is a large decentralized market, on which trading analysis and algorithmic trading are popular. Research efforts have been focusing on proof of efficiency of certain technical indicators. We demonstrate, however, that the values of indicator functions are not reproducible and often reduce the number of trade opportunities, compared to price-action trading. In this work, we develop two dataset-agnostic Forex trading heuristic templates with high rate of trading signals. In order to determine most optimal parameters for the given heuristic prototypes, we perform a machine learning simulation of 10 years of Forex price data over three low-margin instruments and 6 different OHLC granularities. As a result, we develop a specific and reproducible list of most optimal trade parameters found for each instrument-granularity pair, with 118 pips of average daily profit for the optimized configuration. △ Less

Submitted 10 May, 2021; originally announced May 2021.

Comments: 8 pages

Targeting the Weakest Link: Social Engineering Attacks in Ethereum Smart Contracts

Authors: Nikolay Ivanov, Jianzhi Lou, Ting Chen, Jin Li, Qiben Yan

Abstract: Ethereum holds multiple billions of U.S. dollars in the form of Ether cryptocurrency and ERC-20 tokens, with millions of deployed smart contracts algorithmically operating these funds. Unsurprisingly, the security of Ethereum smart contracts has been under rigorous scrutiny. In recent years, numerous defense tools have been developed to detect different types of smart contract code vulnerabilities… ▽ More Ethereum holds multiple billions of U.S. dollars in the form of Ether cryptocurrency and ERC-20 tokens, with millions of deployed smart contracts algorithmically operating these funds. Unsurprisingly, the security of Ethereum smart contracts has been under rigorous scrutiny. In recent years, numerous defense tools have been developed to detect different types of smart contract code vulnerabilities. When opportunities for exploiting code vulnerabilities diminish, the attackers start resorting to social engineering attacks, which aim to influence humans -- often the weakest link in the system. The only known class of social engineering attacks in Ethereum are honeypots, which plant hidden traps for attackers attempting to exploit existing vulnerabilities, thereby targeting only a small population of potential victims. In this work, we explore the possibility and existence of new social engineering attacks beyond smart contract honeypots. We present two novel classes of Ethereum social engineering attacks - Address Manipulation and Homograph - and develop six zero-day social engineering attacks. To show how the attacks can be used in popular programming patterns, we conduct a case study of five popular smart contracts with combined market capitalization exceeding $29 billion, and integrate our attack patterns in their source codes without altering their existing functionality. Moreover, we show that these attacks remain dormant during the test phase but activate their malicious logic only at the final production deployment. We further analyze 85,656 open-source smart contracts, and discover that 1,027 of them can be used for the proposed social engineering attacks. We conduct a professional opinion survey with experts from seven smart contract auditing firms, corroborating that the exposed social engineering attacks bring a major threat to the smart contract systems. △ Less

Submitted 29 May, 2021; v1 submitted 30 April, 2021; originally announced May 2021.

Comments: ACM ASIA Conference on Computer and Communications Security 2021, 15 pages

A system of relational syllogistic incorporating full Boolean reasoning

Authors: Nikolay Ivanov, Dimiter Vakarelov

Abstract: We present a system of relational syllogistic, based on classical propositional logic, having primitives of the following form: Some A are R-related to some B; Some A are R-related to all B; All A are R-related to some B; All A are R-related to all B. Such primitives formalize sentences from natural language like `All students read some textbooks'. Here A and B denote arbitrary sets (of… ▽ More We present a system of relational syllogistic, based on classical propositional logic, having primitives of the following form: Some A are R-related to some B; Some A are R-related to all B; All A are R-related to some B; All A are R-related to all B. Such primitives formalize sentences from natural language like `All students read some textbooks'. Here A and B denote arbitrary sets (of objects), and R denotes an arbitrary binary relation between objects. The language of the logic contains only variables denoting sets, determining the class of set terms, and variables denoting binary relations between objects, determining the class of relational terms. Both classes of terms are closed under the standard Boolean operations. The set of relational terms is also closed under taking the converse of a relation. The results of the paper are the completeness theorem with respect to the intended semantics and the computational complexity of the satisfiability problem. △ Less

Submitted 23 January, 2013; v1 submitted 22 February, 2011; originally announced February 2011.

Comments: Available at http://link.springer.com/article/10.1007/s10849-012-9165-1

MSC Class: 03B65

Journal ref: Journal of Logic, Language and Information, 21 (2012) 433-459