-
ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries
Authors:
Kishan Maharaj,
Vitobha Munigala,
Srikanth G. Tamilselvam,
Prince Kumar,
Sayandeep Sen,
Palani Kodeswaran,
Abhijit Mishra,
Pushpak Bhattacharyya
Abstract:
Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarization. However, LLMs are prone to hallucination-outputs that stray from intended meanings. Detecting hallucinations in code summarization is especially difficult due to the com…
▽ More
Recent advancements in large language models (LLMs) have significantly enhanced their ability to understand both natural language and code, driving their use in tasks like natural language-to-code (NL2Code) and code summarization. However, LLMs are prone to hallucination-outputs that stray from intended meanings. Detecting hallucinations in code summarization is especially difficult due to the complex interplay between programming and natural languages. We introduce a first-of-its-kind dataset with $\sim$10K samples, curated specifically for hallucination detection in code summarization. We further propose a novel Entity Tracing Framework (ETF) that a) utilizes static program analysis to identify code entities from the program and b) uses LLMs to map and verify these entities and their intents within generated code summaries. Our experimental analysis demonstrates the effectiveness of the framework, leading to a 0.73 F1 score. This approach provides an interpretable method for detecting hallucinations by grounding entities, allowing us to evaluate summary accuracy.
△ Less
Submitted 18 December, 2024; v1 submitted 17 October, 2024;
originally announced October 2024.
-
Benchmarking tunnel and encryption methodologies in cloud environments
Authors:
Pravein Govindan Kannan,
Brent Salisbury,
Palanivel Kodeswaran,
Sayandeep Sen
Abstract:
The recent past has seen the adoption of multi-cloud deployments by enterprises due to availability, features, and regulatory requirements. A typical deployment involves parts of an application/workloads running inside a private cloud with the other parts spread across multiple on-prem/public clouds. Typical cluster-to-cluster networking in such deployments involve the establishment of site-to-sit…
▽ More
The recent past has seen the adoption of multi-cloud deployments by enterprises due to availability, features, and regulatory requirements. A typical deployment involves parts of an application/workloads running inside a private cloud with the other parts spread across multiple on-prem/public clouds. Typical cluster-to-cluster networking in such deployments involve the establishment of site-to-site encrypted tunnels to connect the workloads.
In this report, we benchmark the performance of various tunneling and encryption technologies to provide directions on their use in multi-cloud deployments. Based on the various experiments conducted on three different testbeds, we present quantifiable data which can be leveraged by operators and cloud providers tasked with design and development decisions of multi-cloud network connectivity and orchestration.
△ Less
Submitted 4 March, 2022;
originally announced March 2022.
-
Trusted Data Notifications from Private Blockchains
Authors:
Dushyant Behl,
Palanivel Kodeswaran,
Venkatraman Ramakrishna,
Sayandeep Sen,
Dhinakaran Vinayagamurthy
Abstract:
Private blockchain networks are used by enterprises to manage decentralized processes without trusted mediators and without exposing their assets publicly on an open network like Ethereum. Yet external parties that cannot join such networks may have a compelling need to be informed about certain data items on their shared ledgers along with certifications of data authenticity; e.g., a mortgage ban…
▽ More
Private blockchain networks are used by enterprises to manage decentralized processes without trusted mediators and without exposing their assets publicly on an open network like Ethereum. Yet external parties that cannot join such networks may have a compelling need to be informed about certain data items on their shared ledgers along with certifications of data authenticity; e.g., a mortgage bank may need to know about the sale of a mortgaged property from a network managing property deeds. These parties are willing to compensate the networks in exchange for privately sharing information with proof of authenticity and authorization for external use. We have devised a novel and cryptographically secure protocol to effect a fair exchange between rational network members and information recipients using a public blockchain and atomic swap techniques. Using our protocol, any member of a private blockchain can atomically reveal private blockchain data with proofs in exchange for a monetary reward to an external party if and only if the external party is a valid recipient. The protocol preserves confidentiality of data for the recipient, and in addition, allows it to mount a challenge if the data turns out to be inauthentic. We also formally analyze the security and privacy of this protocol, which can be used in a wide array of practical scenarios
△ Less
Submitted 23 January, 2021;
originally announced January 2021.
-
Verifiable and Auditable Digital Interchange Framework
Authors:
Prabal Banerjee,
Dushyant Behl,
Palanivel Kodeswaran,
Chaitanya Kumar,
Sushmita Ruj,
Sayandeep Sen
Abstract:
We address the problem of fairness and transparency in online marketplaces selling digital content, where all parties are not actively participating in the trade. We present the design, implementation and evaluation of VADER, a highly scalable solution for multi-party fair digital exchange that combines the trusted execution of blockchains with intelligent protocol design and incentivization schem…
▽ More
We address the problem of fairness and transparency in online marketplaces selling digital content, where all parties are not actively participating in the trade. We present the design, implementation and evaluation of VADER, a highly scalable solution for multi-party fair digital exchange that combines the trusted execution of blockchains with intelligent protocol design and incentivization schemes. We prototype VADER on Hyperledger Fabric and extensively evaluate our system on a realistic testbed spanning five public cloud datacenters, spread across four continents. Our results demonstrate that VADER adds only minimal overhead of 16% in median case compared to a baseline solution, while significantly outperforming a naive blockchain based solution that adds an overhead of 764%.
△ Less
Submitted 27 January, 2020; v1 submitted 11 January, 2020;
originally announced January 2020.
-
Blockchain Enabled Trustless API Marketplace
Authors:
Vijay Arya,
Sayandeep Sen,
Palani Kodeswaran
Abstract:
There has been an unprecedented surge in the number of service providers offering a wide range of machine learning prediction APIs for tasks such as image classification, language translation, etc. thereby monetizing the underlying data and trained models. Typically, a data owner (API provider) develops a model, often over proprietary data, and leverages the infrastructure services of a cloud vend…
▽ More
There has been an unprecedented surge in the number of service providers offering a wide range of machine learning prediction APIs for tasks such as image classification, language translation, etc. thereby monetizing the underlying data and trained models. Typically, a data owner (API provider) develops a model, often over proprietary data, and leverages the infrastructure services of a cloud vendor for hosting and serving API requests. Clearly, this model assumes complete trust between the API Provider and cloud vendor. On the other hand, a malicious/buggy cloud vendor may copy the APIs and offer an identical service, under-report model usage metrics, or unfairly discriminate between different API providers by offering them a nominal share of the revenue. In this work, we present the design of a blockchain based decentralized trustless API marketplace that enables all the stakeholders in the API ecosystem to audit the behavior of the parties without having to trust a single centralized entity. In particular, our system divides an AI model into multiple pieces and deploys them among multiple cloud vendors who then collaboratively execute the APIs. Our design ensures that cloud vendors cannot collude with each other to steal the combined model, while individual cloud vendors and clients cannot repudiate their input or model executions.
△ Less
Submitted 5 December, 2018;
originally announced December 2018.
