Showing 1–50 of 180 results for author: Barnett, S

Is diamagnetism really acausal?

Authors: Niclas Westerberg, Stephen M. Barnett

Abstract: Diamagnetism, in which the magnetisation in a medium opposes the direction of an applied magnetic field, is a weak but familiar effect in a wide class of materials. Being weak it is also a linear response to any applied field. The problem is that the existence of diamagnetism is in direct conflict with the requirements of causality as embodied in the familiar Kramers-Kronig relations. Nature does… ▽ More Diamagnetism, in which the magnetisation in a medium opposes the direction of an applied magnetic field, is a weak but familiar effect in a wide class of materials. Being weak it is also a linear response to any applied field. The problem is that the existence of diamagnetism is in direct conflict with the requirements of causality as embodied in the familiar Kramers-Kronig relations. Nature does not care about our confusion and diamagnetism exists and physics is constrained by the requirements of causality (that effect cannot precede its cause). This puzzle has received intermittent attention from time to time, with a variety of arguments made to resolve the paradox. None of these, no matter how plausible, reveal the mechanism that resolves the existence of diamagnetism without sacrificing causality. The full resolution is presented in this letter. △ Less

Submitted 15 May, 2025; originally announced May 2025.

Comments: 4 pages, partner to 2501.13098. Comments welcome

Stokes and skyrmion tensors with application to non-Cartesian coordinates

Authors: Stephen M. Barnett, Sonja Franke-Arnold, Fiona C. Speirits

Abstract: We replace the familiar Stokes vector by a tensor. This enables us to derive the skyrmion field as a tensor quantity. The advantage we obtain is the freedom to work with non-Cartesian coordinates, including polar coordinates. An expression of optical skyrmions in terms of cylindrical coordinates should simplify the descriptions of their propagation and focussing. We reveal a fundamental electromag… ▽ More We replace the familiar Stokes vector by a tensor. This enables us to derive the skyrmion field as a tensor quantity. The advantage we obtain is the freedom to work with non-Cartesian coordinates, including polar coordinates. An expression of optical skyrmions in terms of cylindrical coordinates should simplify the descriptions of their propagation and focussing. We reveal a fundamental electromagnetic source of a skyrmion field in the form of radiation by an oscillating dipole. △ Less

Submitted 17 March, 2025; originally announced March 2025.

How to Entangle or Disentangle Gaussian States with Hong-Ou-Mandel Interference

Authors: Fabian Schlue, Patrick Folge, Takefumi Nomura, Philip Held, Federico Pegoraro, Michael Stefszky, Benjamin Brecht, Stephen M. Barnett, Christine Silberhorn

Abstract: The interference of two indistinguishable photons from a photon pair on a balanced beam splitter, where each photon enters in one of the input ports, yields a maximally entangled N00N-state shared between the outputs of the beam splitter through Hong-Ou-Mandel interference. At the same time, interference between the two modes of a two-mode squeezed state - the generalized output of a parametric ph… ▽ More The interference of two indistinguishable photons from a photon pair on a balanced beam splitter, where each photon enters in one of the input ports, yields a maximally entangled N00N-state shared between the outputs of the beam splitter through Hong-Ou-Mandel interference. At the same time, interference between the two modes of a two-mode squeezed state - the generalized output of a parametric photon-pair source - yields two independent, hence separable, single-mode squeezed states. Notably, both experiments use the exact same experimental setup. We resolve this seeming contradiction by measuring photon-number resolved quantum interference of the two modes of a two-mode squeezed state from an engineered parametric down-conversion source and show that the notion of inseparability is a result of post-selection on the presence of photons. Indeed, the complete joint photon statistics including the vacuum component are separable, as expected from textbook calculations. △ Less

Submitted 10 February, 2025; originally announced February 2025.

The M-factor: A Novel Metric for Evaluating Neural Architecture Search in Resource-Constrained Environments

Authors: Srikanth Thudumu, Hy Nguyen, Hung Du, Nhat Duong, Zafaryab Rasool, Rena Logothetis, Scott Barnett, Rajesh Vasa, Kon Mouzakis

Abstract: Neural Architecture Search (NAS) aims to automate the design of deep neural networks. However, existing NAS techniques often focus on maximising accuracy, neglecting model efficiency. This limitation restricts their use in resource-constrained environments like mobile devices and edge computing systems. Moreover, current evaluation metrics prioritise performance over efficiency, lacking a balanced… ▽ More Neural Architecture Search (NAS) aims to automate the design of deep neural networks. However, existing NAS techniques often focus on maximising accuracy, neglecting model efficiency. This limitation restricts their use in resource-constrained environments like mobile devices and edge computing systems. Moreover, current evaluation metrics prioritise performance over efficiency, lacking a balanced approach for assessing architectures suitable for constrained scenarios. To address these challenges, this paper introduces the M-factor, a novel metric combining model accuracy and size. Four diverse NAS techniques are compared: Policy-Based Reinforcement Learning, Regularised Evolution, Tree-structured Parzen Estimator (TPE), and Multi-trial Random Search. These techniques represent different NAS paradigms, providing a comprehensive evaluation of the M-factor. The study analyses ResNet configurations on the CIFAR-10 dataset, with a search space of 19,683 configurations. Experiments reveal that Policy-Based Reinforcement Learning and Regularised Evolution achieved M-factor values of 0.84 and 0.82, respectively, while Multi-trial Random Search attained 0.75, and TPE reached 0.67. Policy-Based Reinforcement Learning exhibited performance changes after 39 trials, while Regularised Evolution optimised within 20 trials. The research investigates the optimisation dynamics and trade-offs between accuracy and model size for each strategy. Findings indicate that, in some cases, random search performed comparably to more complex algorithms when assessed using the M-factor. These results highlight how the M-factor addresses the limitations of existing metrics by guiding NAS towards balanced architectures, offering valuable insights for selecting strategies in scenarios requiring both performance and efficiency. △ Less

Submitted 28 January, 2025; originally announced January 2025.

Is there a conflict between causality and diamagnetism?

Authors: Niclas Westerberg, Stephen M. Barnett

Abstract: There is a long-standing apparent conflict between the existence of diamagnetism and causality as expressed through the Kramers-Kronig relations. In essence, using causality arguments, along with a small number of seemingly well-justified assumptions, one can show that diamagnetism is impossible. Nature naturally does not mind this, and experiments show diamagnetic responses from magnetic media. W… ▽ More There is a long-standing apparent conflict between the existence of diamagnetism and causality as expressed through the Kramers-Kronig relations. In essence, using causality arguments, along with a small number of seemingly well-justified assumptions, one can show that diamagnetism is impossible. Nature naturally does not mind this, and experiments show diamagnetic responses from magnetic media. We present a resolution to this issue, which also explains the absence of observed dia-electric responses in media. In the process, we expose some of the short-comings in earlier analyses that have kept the paradox alive. △ Less

Submitted 15 May, 2025; v1 submitted 22 January, 2025; originally announced January 2025.

Comments: 12 pages + appendices, 2 figures. Updated with expanded discussions

RAGProbe: An Automated Approach for Evaluating RAG Applications

Authors: Shangeetha Sivasothy, Scott Barnett, Stefanus Kurniawan, Zafaryab Rasool, Rajesh Vasa

Abstract: Retrieval Augmented Generation (RAG) is increasingly being used when building Generative AI applications. Evaluating these applications and RAG pipelines is mostly done manually, via a trial and error process. Automating evaluation of RAG pipelines requires overcoming challenges such as context misunderstanding, wrong format, incorrect specificity, and missing content. Prior works therefore focuse… ▽ More Retrieval Augmented Generation (RAG) is increasingly being used when building Generative AI applications. Evaluating these applications and RAG pipelines is mostly done manually, via a trial and error process. Automating evaluation of RAG pipelines requires overcoming challenges such as context misunderstanding, wrong format, incorrect specificity, and missing content. Prior works therefore focused on improving evaluation metrics as well as enhancing components within the pipeline using available question and answer datasets. However, they have not focused on 1) providing a schema for capturing different types of question-answer pairs or 2) creating a set of templates for generating question-answer pairs that can support automation of RAG pipeline evaluation. In this paper, we present a technique for generating variations in question-answer pairs to trigger failures in RAG pipelines. We validate 5 open-source RAG pipelines using 3 datasets. Our approach revealed the highest failure rates when prompts combine multiple questions: 91% for questions when spanning multiple documents and 78% for questions from a single document; indicating a need for developers to prioritise handling these combined questions. 60% failure rate was observed in academic domain dataset and 53% and 62% failure rates were observed in open-domain datasets. Our automated approach outperforms the existing state-of-the-art methods, by increasing the failure rate by 51% on average per dataset. Our work presents an automated approach for continuously monitoring the health of RAG pipelines, which can be integrated into existing CI/CD pipelines, allowing for improved quality. △ Less

Submitted 24 September, 2024; originally announced September 2024.

Comments: 11 pages, 5 figures, 9 tables

Minimising changes to audit when updating decision trees

Authors: Anj Simmons, Scott Barnett, Anupam Chaudhuri, Sankhya Singh, Shangeetha Sivasothy

Abstract: Interpretable models are important, but what happens when the model is updated on new training data? We propose an algorithm for updating a decision tree while minimising the number of changes to the tree that a human would need to audit. We achieve this via a greedy approach that incorporates the number of changes to the tree as part of the objective function. We compare our algorithm to existing… ▽ More Interpretable models are important, but what happens when the model is updated on new training data? We propose an algorithm for updating a decision tree while minimising the number of changes to the tree that a human would need to audit. We achieve this via a greedy approach that incorporates the number of changes to the tree as part of the objective function. We compare our algorithm to existing methods and show that it sits in a sweet spot between final accuracy and number of changes to audit. △ Less

Submitted 29 August, 2024; originally announced August 2024.

Comments: 12 pages

Monotonic warpings for additive and deep Gaussian processes

Authors: Steven D. Barnett, Lauren J. Beesley, Annie S. Booth, Robert B. Gramacy, Dave Osthus

Abstract: Gaussian processes (GPs) are canonical as surrogates for computer experiments because they enjoy a degree of analytic tractability. But that breaks when the response surface is constrained, say to be monotonic. Here, we provide a mono-GP construction for a single input that is highly efficient even though the calculations are non-analytic. Key ingredients include transformation of a reference proc… ▽ More Gaussian processes (GPs) are canonical as surrogates for computer experiments because they enjoy a degree of analytic tractability. But that breaks when the response surface is constrained, say to be monotonic. Here, we provide a mono-GP construction for a single input that is highly efficient even though the calculations are non-analytic. Key ingredients include transformation of a reference process and elliptical slice sampling. We then show how mono-GP may be deployed effectively in two ways. One is additive, extending monotonicity to more inputs; the other is as a prior on injective latent warping variables in a deep Gaussian process for (non-monotonic, multi-input) non-stationary surrogate modeling. We provide illustrative and benchmarking examples throughout, showing that our methods yield improved performance over the state-of-the-art on examples from those two classes of problems. △ Less

Submitted 10 March, 2025; v1 submitted 2 August, 2024; originally announced August 2024.

Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models

Authors: Scott Barnett, Zac Brannelly, Stefanus Kurniawan, Sheng Wong

Abstract: Large Language Models (LLMs) have the unique capability to understand and generate human-like text from input queries. When fine-tuned, these models show enhanced performance on domain-specific queries. OpenAI highlights the process of fine-tuning, stating: "To fine-tune a model, you are required to provide at least 10 examples. We typically see clear improvements from fine-tuning on 50 to 100 tra… ▽ More Large Language Models (LLMs) have the unique capability to understand and generate human-like text from input queries. When fine-tuned, these models show enhanced performance on domain-specific queries. OpenAI highlights the process of fine-tuning, stating: "To fine-tune a model, you are required to provide at least 10 examples. We typically see clear improvements from fine-tuning on 50 to 100 training examples, but the right number varies greatly based on the exact use case." This study extends this concept to the integration of LLMs within Retrieval-Augmented Generation (RAG) pipelines, which aim to improve accuracy and relevance by leveraging external corpus data for information retrieval. However, RAG's promise of delivering optimal responses often falls short in complex query scenarios. This study aims to specifically examine the effects of fine-tuning LLMs on their ability to extract and integrate contextual data to enhance the performance of RAG systems across multiple domains. We evaluate the impact of fine-tuning on the LLMs' capacity for data extraction and contextual understanding by comparing the accuracy and completeness of fine-tuned models against baseline performances across datasets from multiple domains. Our findings indicate that fine-tuning resulted in a decline in performance compared to the baseline models, contrary to the improvements observed in standalone LLM applications as suggested by OpenAI. This study highlights the need for vigorous investigation and validation of fine-tuned models for domain-specific tasks. △ Less

Submitted 30 June, 2024; v1 submitted 17 June, 2024; originally announced June 2024.

Comments: 8 pages, 4 figures

Large language models for generating rules, yay or nay?

Authors: Shangeetha Sivasothy, Scott Barnett, Rena Logothetis, Mohamed Abdelrazek, Zafaryab Rasool, Srikanth Thudumu, Zac Brannelly

Abstract: Engineering safety-critical systems such as medical devices and digital health intervention systems is complex, where long-term engagement with subject-matter experts (SMEs) is needed to capture the systems' expected behaviour. In this paper, we present a novel approach that leverages Large Language Models (LLMs), such as GPT-3.5 and GPT-4, as a potential world model to accelerate the engineering… ▽ More Engineering safety-critical systems such as medical devices and digital health intervention systems is complex, where long-term engagement with subject-matter experts (SMEs) is needed to capture the systems' expected behaviour. In this paper, we present a novel approach that leverages Large Language Models (LLMs), such as GPT-3.5 and GPT-4, as a potential world model to accelerate the engineering of software systems. This approach involves using LLMs to generate logic rules, which can then be reviewed and informed by SMEs before deployment. We evaluate our approach using a medical rule set, created from the pandemic intervention monitoring system in collaboration with medical professionals during COVID-19. Our experiments show that 1) LLMs have a world model that bootstraps implementation, 2) LLMs generated less number of rules compared to experts, and 3) LLMs do not have the capacity to generate thresholds for each rule. Our work shows how LLMs augment the requirements' elicitation process by providing access to a world model for domains. △ Less

Submitted 10 June, 2024; originally announced June 2024.

Comments: 5 pages, 1 figure

Both orbital and spin torques originate from r x g

Authors: Frances Crimin, Stephen M. Barnett

Abstract: Does the cross-product of the position and the electromagnetic momentum density, g, include the optical spin momentum? We answer this long-standing question in the affirmative by evaluating, explicitly, the torque exerted on a particle by a beam of light carrying angular momentum. Does the cross-product of the position and the electromagnetic momentum density, g, include the optical spin momentum? We answer this long-standing question in the affirmative by evaluating, explicitly, the torque exerted on a particle by a beam of light carrying angular momentum. △ Less

Submitted 7 June, 2024; originally announced June 2024.

A frustrated antipolar phase analogous to classical spin liquids

Authors: G. Bastien, D. Repček, A. Eliáš, A. Kancko, Q. Courtade, T. Haidamak, M. Savinov, V. Bovtun, M. Kempa, K. Carva, M. Vališka, P. Doležal, M. Kratochvílová, S. A. Barnett, P. Proschek, J. Prokleška, C. Kadlec, P. Kužel, R. H. Colman, S. Kamba

Abstract: The study of magnetic frustration in classical spin systems was motivated by the prediction and discovery of classical spin liquid states. These uncommon magnetic phases are characterized by a massive degeneracy of their ground state implying a finite magnetic entropy at zero temperature. While the classical spin liquid state was originally predicted in the Ising triangular lattice antiferromagnet… ▽ More The study of magnetic frustration in classical spin systems was motivated by the prediction and discovery of classical spin liquid states. These uncommon magnetic phases are characterized by a massive degeneracy of their ground state implying a finite magnetic entropy at zero temperature. While the classical spin liquid state was originally predicted in the Ising triangular lattice antiferromagnet in 1950, this state has never been experimentally observed in any triangular magnets. We report here the discovery of an electric analogue of classical spin liquids on a triangular lattice of uniaxial electric dipoles in EuAl12O19 . This new type of frustrated antipolar phase is characterized by a highly degenerate state at low temperature implying an absence of long-range antiferroelectric order, despite short-range antipolar correlations. Its dynamics are governed by a thermally activated process, slowing down upon cooling towards a complete freezing at zero temperature. △ Less

Submitted 15 October, 2024; v1 submitted 30 May, 2024; originally announced May 2024.

Journal ref: Adv. Mater. 2024, 2410282

Quantised helicity in optical media

Authors: Neel Mackinnon, Jörg B. Götte, Stephen M. Barnett, Niclas Westerberg

Abstract: Optical helicity quantifies the handedness of light, and plays a central role in the description of interactions between light and chiral matter. In free space, it is related to the duality symmetry of the electromagnetic field, a continuous symmetry encapsulating the invariance of Maxwell's equations under the interchange of electric and magnetic fields. However, in materials the situation is not… ▽ More Optical helicity quantifies the handedness of light, and plays a central role in the description of interactions between light and chiral matter. In free space, it is related to the duality symmetry of the electromagnetic field, a continuous symmetry encapsulating the invariance of Maxwell's equations under the interchange of electric and magnetic fields. However, in materials the situation is not so straightforward, as the free space transformation must be extended to encompass mixing of both the $\mathbf{E}$/$\mathbf{H}$ and $\mathbf{D}$/$\mathbf{B}$ field pairs. The simultaneous direct interchange of $\mathbf{E}$/$\mathbf{H}$ and of $\mathbf{D}$/$\mathbf{B}$ is incompatible with the presence of linear constitutive relations. In this work, we extend the duality transform in a way that resolves this incompatibility, and use this to define the optical helicity in a general medium, which may be dispersive, lossy, chiral or nonreciprocal. We find that the helicity density must contain an explicit contribution associated with the polarisation and magnetisation of the matter, and we show that the form of this matter contribution is independent of the details of the medium. We also show that the in-medium helicity can be naturally expressed in terms of the elementary quantised excitations of the system. △ Less

Submitted 13 May, 2024; originally announced May 2024.

Comments: 4 pages + supplemental. 2 figures. Comments welcome

Dressing the Lorentz atom

Authors: Stephen M. Barnett, James D. Cresser, Sarah Croke

Abstract: We investigate the effects of the electromagnetic vacuum field on a harmonically bound electron. We show that in the electric-dipole approximation the model atom couples only to an effective one-dimensional electric field. In a simplified form, in which the problem is reduced to a single spatial dimension, we determine, analytically, the form of the ground state and discuss the significance of thi… ▽ More We investigate the effects of the electromagnetic vacuum field on a harmonically bound electron. We show that in the electric-dipole approximation the model atom couples only to an effective one-dimensional electric field. In a simplified form, in which the problem is reduced to a single spatial dimension, we determine, analytically, the form of the ground state and discuss the significance of this. △ Less

Submitted 19 April, 2024; originally announced April 2024.

Comments: 11 pages, 2 figures. This article is a contribution to the special issue of Physics, "Matter-Radiation Interactions -- In Memory of Professor Francesco Saverio Persico"

Theory of paraxial optical Skyrmions

Authors: Z. Ye, S. M. Barnett, S. Franke-Arnold, J. B. Götte, A. McWilliam, F. C. Speirits, C. M. Cisowski

Abstract: Vector light beams, characterised by a spatially varying polarisation, can exhibit localised structures reminiscent of the Skyrmions familiar from the study of magnetic media. We present a theory of such Skyrmions within paraxial optics, exploiting mathematical analogies with the study of superfluids, especially the A phase of superfluid $\textrm{He}^3$. The key feature is the Skyrmion field which… ▽ More Vector light beams, characterised by a spatially varying polarisation, can exhibit localised structures reminiscent of the Skyrmions familiar from the study of magnetic media. We present a theory of such Skyrmions within paraxial optics, exploiting mathematical analogies with the study of superfluids, especially the A phase of superfluid $\textrm{He}^3$. The key feature is the Skyrmion field which, together with the underlying Skyrmion vector potential, determines the properties of the Skyrmions and, more generally, the polarisation structure of every paraxial vector beam. In addition to structures with integer Skyrmion number we find polarisation patterns with non-integer Skyrmion number; these seem to have no analogue in other fields of physics. △ Less

Submitted 17 April, 2024; originally announced April 2024.

Diagonalizing the Jaynes-Cummings Hamiltonian and Jaynes-Cummings coherent states

Authors: Stephen M. Barnett, Bryan J. Dalton

Abstract: We determine the form of the unitary transformation that diagonalizes the Jaynes-Cummings Hamiltonian. This leads to operators the action of which has a simple interpretation in terms of the dressed states, the energy eigenstates. This suggests a set of coherent states and spin coherent states based on the dressed states. We determine the form of the unitary transformation that diagonalizes the Jaynes-Cummings Hamiltonian. This leads to operators the action of which has a simple interpretation in terms of the dressed states, the energy eigenstates. This suggests a set of coherent states and spin coherent states based on the dressed states. △ Less

Submitted 17 April, 2024; originally announced April 2024.

Comments: 5 pages, no figures

LLMs for Test Input Generation for Semantic Caches

Authors: Zafaryab Rasool, Scott Barnett, David Willie, Stefanus Kurniawan, Sherwin Balugo, Srikanth Thudumu, Mohamed Abdelrazek

Abstract: Large language models (LLMs) enable state-of-the-art semantic capabilities to be added to software systems such as semantic search of unstructured documents and text generation. However, these models are computationally expensive. At scale, the cost of serving thousands of users increases massively affecting also user experience. To address this problem, semantic caches are used to check for answe… ▽ More Large language models (LLMs) enable state-of-the-art semantic capabilities to be added to software systems such as semantic search of unstructured documents and text generation. However, these models are computationally expensive. At scale, the cost of serving thousands of users increases massively affecting also user experience. To address this problem, semantic caches are used to check for answers to similar queries (that may have been phrased differently) without hitting the LLM service. Due to the nature of these semantic cache techniques that rely on query embeddings, there is a high chance of errors impacting user confidence in the system. Adopting semantic cache techniques usually requires testing the effectiveness of a semantic cache (accurate cache hits and misses) which requires a labelled test set of similar queries and responses which is often unavailable. In this paper, we present VaryGen, an approach for using LLMs for test input generation that produces similar questions from unstructured text documents. Our novel approach uses the reasoning capabilities of LLMs to 1) adapt queries to the domain, 2) synthesise subtle variations to queries, and 3) evaluate the synthesised test dataset. We evaluated our approach in the domain of a student question and answer system by qualitatively analysing 100 generated queries and result pairs, and conducting an empirical case study with an open source semantic cache. Our results show that query pairs satisfy human expectations of similarity and our generated data demonstrates failure cases of a semantic cache. Additionally, we also evaluate our approach on Qasper dataset. This work is an important first step into test input generation for semantic applications and presents considerations for practitioners when calibrating a semantic cache. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Comments: Accepted in International Conference on AI Engineering Software Engineering (CAIN 2024)

ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study

Authors: Hala Abdelkader, Mohamed Abdelrazek, Scott Barnett, Jean-Guy Schneider, Priya Rani, Rajesh Vasa

Abstract: Machine learning (ML), especially with the emergence of large language models (LLMs), has significantly transformed various industries. However, the transition from ML model prototyping to production use within software systems presents several challenges. These challenges primarily revolve around ensuring safety, security, and transparency, subsequently influencing the overall robustness and trus… ▽ More Machine learning (ML), especially with the emergence of large language models (LLMs), has significantly transformed various industries. However, the transition from ML model prototyping to production use within software systems presents several challenges. These challenges primarily revolve around ensuring safety, security, and transparency, subsequently influencing the overall robustness and trustworthiness of ML models. In this paper, we introduce ML-On-Rails, a protocol designed to safeguard ML models, establish a well-defined endpoint interface for different ML tasks, and clear communication between ML providers and ML consumers (software engineers). ML-On-Rails enhances the robustness of ML models via incorporating detection capabilities to identify unique challenges specific to production ML. We evaluated the ML-On-Rails protocol through a real-world case study of the MoveReminder application. Through this evaluation, we emphasize the importance of safeguarding ML models in production. △ Less

Submitted 12 January, 2024; originally announced January 2024.

Seven Failure Points When Engineering a Retrieval Augmented Generation System

Authors: Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, Mohamed Abdelrazek

Abstract: Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinat… ▽ More Software engineers are increasingly adding semantic search capabilities to applications using a strategy known as Retrieval Augmented Generation (RAG). A RAG system involves finding documents that semantically match a query and then passing the documents to a large language model (LLM) such as ChatGPT to extract the right answer using an LLM. RAG systems aim to: a) reduce the problem of hallucinated responses from LLMs, b) link sources/references to generated responses, and c) remove the need for annotating documents with meta-data. However, RAG systems suffer from limitations inherent to information retrieval systems and from reliance on LLMs. In this paper, we present an experience report on the failure points of RAG systems from three case studies from separate domains: research, education, and biomedical. We share the lessons learned and present 7 failure points to consider when designing a RAG system. The two key takeaways arising from our work are: 1) validation of a RAG system is only feasible during operation, and 2) the robustness of a RAG system evolves rather than designed in at the start. We conclude with a list of potential research directions on RAG systems for the software engineering community. △ Less

Submitted 11 January, 2024; originally announced January 2024.

Evaluating LLMs on Document-Based QA: Exact Answer Selection and Numerical Extraction using Cogtale dataset

Authors: Zafaryab Rasool, Stefanus Kurniawan, Sherwin Balugo, Scott Barnett, Rajesh Vasa, Courtney Chesser, Benjamin M. Hampstead, Sylvie Belleville, Kon Mouzakis, Alex Bahar-Fuchs

Abstract: Document-based Question-Answering (QA) tasks are crucial for precise information retrieval. While some existing work focus on evaluating large language models performance on retrieving and answering questions from documents, assessing the LLMs performance on QA types that require exact answer selection from predefined options and numerical extraction is yet to be fully assessed. In this paper, we… ▽ More Document-based Question-Answering (QA) tasks are crucial for precise information retrieval. While some existing work focus on evaluating large language models performance on retrieving and answering questions from documents, assessing the LLMs performance on QA types that require exact answer selection from predefined options and numerical extraction is yet to be fully assessed. In this paper, we specifically focus on this underexplored context and conduct empirical analysis of LLMs (GPT-4 and GPT-3.5) on question types, including single-choice, yes-no, multiple-choice, and number extraction questions from documents in zero-shot setting. We use the CogTale dataset for evaluation, which provide human expert-tagged responses, offering a robust benchmark for precision and factual grounding. We found that LLMs, particularly GPT-4, can precisely answer many single-choice and yes-no questions given relevant context, demonstrating their efficacy in information retrieval tasks. However, their performance diminishes when confronted with multiple-choice and number extraction formats, lowering the overall performance of the model on this task, indicating that these models may not yet be sufficiently reliable for the task. This limits the applications of LLMs on applications demanding precise information extraction from documents, such as meta-analysis tasks. These findings hinge on the assumption that the retrievers furnish pertinent context necessary for accurate responses, emphasizing the need for further research. Our work offers a framework for ongoing dataset evaluation, ensuring that LLM applications for information retrieval and document analysis continue to meet evolving standards. △ Less

Submitted 3 January, 2024; v1 submitted 13 November, 2023; originally announced November 2023.

Comments: 10 pages, 1 figure

MLGuard: Defend Your Machine Learning Model!

Authors: Sheng Wong, Scott Barnett, Jessica Rivera-Villicana, Anj Simmons, Hala Abdelkader, Jean-Guy Schneider, Rajesh Vasa

Abstract: Machine Learning (ML) is used in critical highly regulated and high-stakes fields such as finance, medicine, and transportation. The correctness of these ML applications is important for human safety and economic benefit. Progress has been made on improving ML testing and monitoring of ML. However, these approaches do not provide i) pre/post conditions to handle uncertainty, ii) defining correctiv… ▽ More Machine Learning (ML) is used in critical highly regulated and high-stakes fields such as finance, medicine, and transportation. The correctness of these ML applications is important for human safety and economic benefit. Progress has been made on improving ML testing and monitoring of ML. However, these approaches do not provide i) pre/post conditions to handle uncertainty, ii) defining corrective actions based on probabilistic outcomes, or iii) continual verification during system operation. In this paper, we propose MLGuard, a new approach to specify contracts for ML applications. Our approach consists of a) an ML contract specification defining pre/post conditions, invariants, and altering behaviours, b) generated validation models to determine the probability of contract violation, and c) an ML wrapper generator to enforce the contract and respond to violations. Our work is intended to provide the overarching framework required for building ML applications and monitoring their safety. △ Less

Submitted 4 September, 2023; originally announced September 2023.

Comments: Accepted in SE4SafeML'23

Revisiting the damped quantum harmonic oscillator

Authors: Stephen M. Barnett, James D. Cresser, Sarah Croke

Abstract: We reanalyse the quantum damped harmonic oscillator, introducing three less than common features. These are (i) the use of a continuum model of the reservoir rather than an ensemble of discrete oscillators, (ii) an exact diagonalisation of the Hamiltonian by adapting a technique pioneered by Fano, and (iii) the use of the thermofield technique for describing a finite temperature reservoir. We reco… ▽ More We reanalyse the quantum damped harmonic oscillator, introducing three less than common features. These are (i) the use of a continuum model of the reservoir rather than an ensemble of discrete oscillators, (ii) an exact diagonalisation of the Hamiltonian by adapting a technique pioneered by Fano, and (iii) the use of the thermofield technique for describing a finite temperature reservoir. We recover in this way a number of well-known and some, perhaps, less familiar results. An example of the latter is an ab initio proof that the oscillator relaxes to the mean-force Gibbs state. We find that special care is necessary when comparing the damped oscillator with its undamped counterpart as the former has two distinct natural frequencies, one associated with short time evolution and the other with longer times. △ Less

Submitted 26 June, 2023; originally announced June 2023.

Comments: 38 pages, 6 figures. This article, part review and part new results, is a contribution to the special issue of Physica Scripta "Challenges in quantum dynamics'', in honour of Professor Igor Jex on the occasion of his 60th birthday. arXiv admin note: substantial text overlap with arXiv:1508.02442

Green Runner: A tool for efficient model selection from model repositories

Authors: Jai Kannan, Scott Barnett, Anj Simmons, Taylan Selvi, Luis Cruz

Abstract: Deep learning models have become essential in software engineering, enabling intelligent features like image captioning and document generation. However, their popularity raises concerns about environmental impact and inefficient model selection. This paper introduces GreenRunnerGPT, a novel tool for efficiently selecting deep learning models based on specific use cases. It employs a large languag… ▽ More Deep learning models have become essential in software engineering, enabling intelligent features like image captioning and document generation. However, their popularity raises concerns about environmental impact and inefficient model selection. This paper introduces GreenRunnerGPT, a novel tool for efficiently selecting deep learning models based on specific use cases. It employs a large language model to suggest weights for quality indicators, optimizing resource utilization. The tool utilizes a multi-armed bandit framework to evaluate models against target datasets, considering tradeoffs. We demonstrate that GreenRunnerGPT is able to identify a model suited to a target use case without wasteful computations that would occur under a brute-force approach to model selection. △ Less

Submitted 26 May, 2023; originally announced May 2023.

On lines of constant polarisation in structured light beams

Authors: Stephen M. Barnett, Fiona C. Speirits, Joerg B. Goette

Abstract: We show that Skyrmion field lines, constructed from the local Stokes parameters, trace out lines of constant optical polarisation. We show that Skyrmion field lines, constructed from the local Stokes parameters, trace out lines of constant optical polarisation. △ Less

Submitted 4 August, 2023; v1 submitted 19 April, 2023; originally announced April 2023.

Comments: Updated version published in Europhysics Letters. Only minor changes

Glauber-Sudarshan P-representations for fermions

Authors: Stephen M. Barnett, Bryan J. Dalton

Abstract: The Glauber-Sudarshan P-representation is well-known within quantum optics, and is widely applied to problems involving photon statistics. Less familiar, perhaps, is its fermionic counterpart. We present a derivation of both the bosonic and fermionic distributions and, in doing so, demonstrate the reason for the existence of two distinct fermionic forms and the relationship between these. We consi… ▽ More The Glauber-Sudarshan P-representation is well-known within quantum optics, and is widely applied to problems involving photon statistics. Less familiar, perhaps, is its fermionic counterpart. We present a derivation of both the bosonic and fermionic distributions and, in doing so, demonstrate the reason for the existence of two distinct fermionic forms and the relationship between these. We consider both single mode systems and also multiparticle systems with many modes. For simplicity only one type of boson or fermion will be considered. △ Less

Submitted 5 April, 2023; v1 submitted 21 October, 2022; originally announced October 2022.

Comments: To appear in Physica Scripta (2023)

Comparative analysis of real bugs in open-source Machine Learning projects -- A Registered Report

Authors: Tuan Dung Lai, Anj Simmons, Scott Barnett, Jean-Guy Schneider, Rajesh Vasa

Abstract: Background: Machine Learning (ML) systems rely on data to make predictions, the systems have many added components compared to traditional software systems such as the data processing pipeline, serving pipeline, and model training. Existing research on software maintenance has studied the issue-reporting needs and resolution process for different types of issues, such as performance and security i… ▽ More Background: Machine Learning (ML) systems rely on data to make predictions, the systems have many added components compared to traditional software systems such as the data processing pipeline, serving pipeline, and model training. Existing research on software maintenance has studied the issue-reporting needs and resolution process for different types of issues, such as performance and security issues. However, ML systems have specific classes of faults, and reporting ML issues requires domain-specific information. Because of the different characteristics between ML and traditional Software Engineering systems, we do not know to what extent the reporting needs are different, and to what extent these differences impact the issue resolution process. Objective: Our objective is to investigate whether there is a discrepancy in the distribution of resolution time between ML and non-ML issues and whether certain categories of ML issues require a longer time to resolve based on real issue reports in open-source applied ML projects. We further investigate the size of fix of ML issues and non-ML issues. Method: We extract issues reports, pull requests and code files in recent active applied ML projects from Github, and use an automatic approach to filter ML and non-ML issues. We manually label the issues using a known taxonomy of deep learning bugs. We measure the resolution time and size of fix of ML and non-ML issues on a controlled sample and compare the distributions for each category of issue. △ Less

Submitted 20 September, 2022; originally announced September 2022.

Comments: 6 pages, 3 figures, registered report, conference paper, accepted at ESEM -- the International Symposium on Empirical Software Engineering and Measurement

Topological approach of characterizing optical Skyrmions and Skyrmion lattices

Authors: A. McWilliam, C. M. Cisowski, Z. Ye, F. C. Speirits, J. B. Götte, S. M. Barnett, S. Franke-Arnold

Abstract: The Skyrmion number of paraxial optical Skyrmions can be defined solely via their polarization singularities and associated winding numbers, using a mathematical derivation that exploits Stokes's theorem. It is demonstrated that this definition provides a robust way to extract the Skyrmion number from experimental data, as illustrated for a variety of optical (Néel-type) Skyrmions and bimerons, an… ▽ More The Skyrmion number of paraxial optical Skyrmions can be defined solely via their polarization singularities and associated winding numbers, using a mathematical derivation that exploits Stokes's theorem. It is demonstrated that this definition provides a robust way to extract the Skyrmion number from experimental data, as illustrated for a variety of optical (Néel-type) Skyrmions and bimerons, and their corresponding lattices. This method generates not only an increase in accuracy, but also provides an intuitive geometrical approach to understanding the topology of such quasi-particles of light, and their robustness against smooth transformations. △ Less

Submitted 30 May, 2023; v1 submitted 14 September, 2022; originally announced September 2022.

On single-photon and classical interference

Authors: Stephen M. Barnett

Abstract: It has often been remarked that single-photon interference experiments, however complicated, seem to behave very much in the same way as those performed in the classical regime, using the field generated by a laser. This observation has the status of being `well-known to those who know it', but perhaps mysterious to others. We discuss the reasons underlying the similarity and also some of the limi… ▽ More It has often been remarked that single-photon interference experiments, however complicated, seem to behave very much in the same way as those performed in the classical regime, using the field generated by a laser. This observation has the status of being `well-known to those who know it', but perhaps mysterious to others. We discuss the reasons underlying the similarity and also some of the limitations of this simple idea. △ Less

Submitted 29 July, 2022; originally announced July 2022.

Neuro-Nav: A Library for Neurally-Plausible Reinforcement Learning

Authors: Arthur Juliani, Samuel Barnett, Brandon Davis, Margaret Sereno, Ida Momennejad

Abstract: In this work we propose Neuro-Nav, an open-source library for neurally plausible reinforcement learning (RL). RL is among the most common modeling frameworks for studying decision making, learning, and navigation in biological organisms. In utilizing RL, cognitive scientists often handcraft environments and agents to meet the needs of their particular studies. On the other hand, artificial intelli… ▽ More In this work we propose Neuro-Nav, an open-source library for neurally plausible reinforcement learning (RL). RL is among the most common modeling frameworks for studying decision making, learning, and navigation in biological organisms. In utilizing RL, cognitive scientists often handcraft environments and agents to meet the needs of their particular studies. On the other hand, artificial intelligence researchers often struggle to find benchmarks for neurally and biologically plausible representation and behavior (e.g., in decision making or navigation). In order to streamline this process across both fields with transparency and reproducibility, Neuro-Nav offers a set of standardized environments and RL algorithms drawn from canonical behavioral and neural studies in rodents and humans. We demonstrate that the toolkit replicates relevant findings from a number of studies across both cognitive science and RL literatures. We furthermore describe ways in which the library can be extended with novel algorithms (including deep RL) and environments to address future research needs of the field. △ Less

Submitted 6 June, 2022; originally announced June 2022.

MLSmellHound: A Context-Aware Code Analysis Tool

Authors: Jai Kannan, Scott Barnett, Luís Cruz, Anj Simmons, Akash Agarwal

Abstract: Meeting the rise of industry demand to incorporate machine learning (ML) components into software systems requires interdisciplinary teams contributing to a shared code base. To maintain consistency, reduce defects and ensure maintainability, developers use code analysis tools to aid them in identifying defects and maintaining standards. With the inclusion of machine learning, tools must account f… ▽ More Meeting the rise of industry demand to incorporate machine learning (ML) components into software systems requires interdisciplinary teams contributing to a shared code base. To maintain consistency, reduce defects and ensure maintainability, developers use code analysis tools to aid them in identifying defects and maintaining standards. With the inclusion of machine learning, tools must account for the cultural differences within the teams which manifests as multiple programming languages, and conflicting definitions and objectives. Existing tools fail to identify these cultural differences and are geared towards software engineering which reduces their adoption in ML projects. In our approach we attempt to resolve this problem by exploring the use of context which includes i) purpose of the source code, ii) technical domain, iii) problem domain, iv) team norms, v) operational environment, and vi) development lifecycle stage to provide contextualised error reporting for code analysis. To demonstrate our approach, we adapt Pylint as an example and apply a set of contextual transformations to the linting results based on the domain of individual project files under analysis. This allows for contextualised and meaningful error reporting for the end-user. △ Less

Submitted 8 May, 2022; originally announced May 2022.

Comments: ICSE 2022 NIER track

Optical angular momentum in atomic transitions: a paradox

Authors: Stephen M. Barnett, Fiona C. Speirits, Mohamed Babiker

Abstract: Stated simply the paradox is as follows: it is clear that the orbital angular momentum of a light beam in its direction of propagation is an intrinsic quantity, and therefore has the same value everywhere in the beam. How then can a Gaussian beam, with precisely zero orbital angular momentum, drive a (single-photon) quadrupole transition which requires the transfer of angular momentum 2$\hbar$ to… ▽ More Stated simply the paradox is as follows: it is clear that the orbital angular momentum of a light beam in its direction of propagation is an intrinsic quantity, and therefore has the same value everywhere in the beam. How then can a Gaussian beam, with precisely zero orbital angular momentum, drive a (single-photon) quadrupole transition which requires the transfer of angular momentum 2$\hbar$ to an absorbing atom? △ Less

Submitted 7 March, 2022; originally announced March 2022.

Comments: Submitted to J.Phys.A. Special Issue "Claritons and the Asymptotics of Ideas: the Physics of Michael Berry"

Generally applicable holographic torque measurement for optically trapped particles

Authors: Franziska Strasser, Stephen M. Barnett, Monika Ritsch-Marte, Gregor Thalhammer

Abstract: We present a method to measure the optical torque applied to particles of arbitrary shape held in an optical trap, inferred from the change of angular momentum of light induced by the particle. All torque components can be determined from a single interference pattern recorded by a camera in the back focal plane of a high-NA condenser collecting all forward scattered light. We derive explicit expr… ▽ More We present a method to measure the optical torque applied to particles of arbitrary shape held in an optical trap, inferred from the change of angular momentum of light induced by the particle. All torque components can be determined from a single interference pattern recorded by a camera in the back focal plane of a high-NA condenser collecting all forward scattered light. We derive explicit expressions mapping the measured complex field in this plane to the torque components. The required phase is retrieved by an iterative algorithm, using the known position of the optical traps as constraints. The torque pertaining to individual particles is accessible, as well as separate spin or orbital parts of the total torque. △ Less

Submitted 22 February, 2022; v1 submitted 18 February, 2022; originally announced February 2022.

Comments: V2, with updated figures

Duality, decay rates and local-field models in macroscopic QED

Authors: Niclas Westerberg, Anette Messinger, Stephen M. Barnett

Abstract: Any treatment of magnetic interactions between atoms, molecules and optical media must start at the form of the interaction energy. This forms the base on which predictions about any number of magnetic atom-light properties stands -- spontaneous decay rates and forces included. As is well-known, the Heaviside-Larmor duality symmetry of Maxwell's equations, where electric and magnetic quantities ar… ▽ More Any treatment of magnetic interactions between atoms, molecules and optical media must start at the form of the interaction energy. This forms the base on which predictions about any number of magnetic atom-light properties stands -- spontaneous decay rates and forces included. As is well-known, the Heaviside-Larmor duality symmetry of Maxwell's equations, where electric and magnetic quantities are exchanges, is broken by the usual form of the magnetic interaction energy. We argue that this symmetry can be restored by including general local-field effects, and that local fields should be treated as a necessity for correctly translating between the microscopic world of the dipole and the macroscopic world of the measured fields. This may additionally aid in resolving a long standing debate over the form of the force on a dipole in a medium. Finally, we compute the magnetic dipole decay rate in a magneto-dielectric with local-field effects taken into account, and show that macroscopic quantum electrodynamics can be made to be dual symmetric at an operator level, instead of only for expectation values. △ Less

Submitted 10 December, 2021; originally announced December 2021.

Comments: 9 pages, 4 figures, plus appendices. Comments welcome

A pragmatic account of the weak evidence effect

Authors: Samuel A. Barnett, Thomas L. Griffiths, Robert D. Hawkins

Abstract: Language is not only used to transmit neutral information; we often seek to persuade by arguing in favor of a particular view. Persuasion raises a number of challenges for classical accounts of belief updating, as information cannot be taken at face value. How should listeners account for a speaker's "hidden agenda" when incorporating new information? Here, we extend recent probabilistic models of… ▽ More Language is not only used to transmit neutral information; we often seek to persuade by arguing in favor of a particular view. Persuasion raises a number of challenges for classical accounts of belief updating, as information cannot be taken at face value. How should listeners account for a speaker's "hidden agenda" when incorporating new information? Here, we extend recent probabilistic models of recursive social reasoning to allow for persuasive goals and show that our model provides a pragmatic account for why weakly favorable arguments may backfire, a phenomenon known as the weak evidence effect. Critically, this model predicts a systematic relationship between belief updates and expectations about the information source: weak evidence should only backfire when speakers are expected to act under persuasive goals and prefer the strongest evidence. We introduce a simple experimental paradigm called the Stick Contest to measure the extent to which the weak evidence effect depends on speaker expectations, and show that a pragmatic listener model accounts for the empirical data better than alternative models. Our findings suggest further avenues for rational models of social reasoning to illuminate classical decision-making phenomena. △ Less

Submitted 13 September, 2022; v1 submitted 7 December, 2021; originally announced December 2021.

Comments: in press at Open Mind

Probing Strong Coupling between a Microwave Cavity and a Spin Ensemble with Raman Heterodyne Spectroscopy

Authors: Gavin G. G. King, Peter S. Barnett, John G. Bartholomew, Andrei Faraon, Jevon J. Longdell

Abstract: Raman heterodyne spectroscopy is a powerful tool for characterizing the energy and dynamics of spins. The technique uses an optical pump to transfer coherence from a spin transition to an optical transition where the coherent emission is more easily detected. Here Raman heterodyne spectroscopy is used to probe an isotopically purified ensemble of erbium dopants, in a yttrium orthosilicate (Y… ▽ More Raman heterodyne spectroscopy is a powerful tool for characterizing the energy and dynamics of spins. The technique uses an optical pump to transfer coherence from a spin transition to an optical transition where the coherent emission is more easily detected. Here Raman heterodyne spectroscopy is used to probe an isotopically purified ensemble of erbium dopants, in a yttrium orthosilicate (Y$_2$SiO$_5$) crystal coupled to a microwave cavity. Because the erbium electron spin transition is strongly coupled to the microwave cavity, we observed Raman heterodyne signals at the resonant frequencies of the hybrid spin-cavity modes (polaritons) rather than the bare erbium spin transition frequency. Using the coupled system, we made saturation recovery measurements of the ground state spin relaxation time T$_1$ = 10$\pm$3 seconds, and also observed Raman heterodyne signals using an excited state spin transition. We discuss the implications of these results for efforts towards converting microwave quantum states to optical quantum states. △ Less

Submitted 11 May, 2021; originally announced May 2021.

Comments: 9 pages, 5 figures

Journal ref: Phys. Rev. B 103, 214305 (2021)

Quantum retrodiction: foundations and controversies

Authors: Stephen M. Barnett, John Jeffers, David T. Pegg

Abstract: Prediction is the making of statements, usually probabilistic, about future events based on current information. Retrodiction is the making of statements about past events based on current information. We present the foundations of quantum retrodiction and highlight its intimate connection with the Bayesian interpretation of probability. The close link with Bayesian methods enables us to explore c… ▽ More Prediction is the making of statements, usually probabilistic, about future events based on current information. Retrodiction is the making of statements about past events based on current information. We present the foundations of quantum retrodiction and highlight its intimate connection with the Bayesian interpretation of probability. The close link with Bayesian methods enables us to explore controversies and misunderstandings about retrodiction that have appeared in the literature. To be clear, quantum retrodiction is universally applicable and draws its validity directly from conventional predictive quantum theory coupled with Bayes' theorem. △ Less

Submitted 26 March, 2021; v1 submitted 10 March, 2021; originally announced March 2021.

Comments: New version, accepted by journal: Symmetry

Theory of Microwave-Optical Conversion Using Rare-Earth Ion Dopants

Authors: Peter S. Barnett, Jevon J. Longdell

Abstract: We develop a theoretical description of a device for coherent conversion of microwave to optical photons. For the device, dopant ions in a crystal are used as three-level systems, and interact with the fields inside overlapping microwave and optical cavities. We develop a model for the cavity fields interacting with an ensemble of ions, and model the ions using an open quantum systems approach, wh… ▽ More We develop a theoretical description of a device for coherent conversion of microwave to optical photons. For the device, dopant ions in a crystal are used as three-level systems, and interact with the fields inside overlapping microwave and optical cavities. We develop a model for the cavity fields interacting with an ensemble of ions, and model the ions using an open quantum systems approach, while accounting for the effect of inhomogeneous broadening. Numerical methods are developed to allow us to accurately simulate the device. We also further develop a simplified model, applicable in the case of small cavity fields which is relevant to quantum information applications. This simplified model is used to predict the maximum conversion efficiency of the device. We investigate the effect of various parameters, and predict that conversion efficiency of above 80% should be possible with currently existing experimental setups inside a dilution refrigerator. △ Less

Submitted 25 August, 2020; originally announced August 2020.

Comments: 10 pages, 7 figures

Journal ref: Phys. Rev. A 102, 063718 (2020)

Threshy: Supporting Safe Usage of Intelligent Web Services

Authors: Alex Cummaudo, Scott Barnett, Rajesh Vasa, John Grundy

Abstract: Increased popularity of `intelligent' web services provides end-users with machine-learnt functionality at little effort to developers. However, these services require a decision threshold to be set which is dependent on problem-specific data. Developers lack a systematic approach for evaluating intelligent services and existing evaluation tools are predominantly targeted at data scientists for pr… ▽ More Increased popularity of `intelligent' web services provides end-users with machine-learnt functionality at little effort to developers. However, these services require a decision threshold to be set which is dependent on problem-specific data. Developers lack a systematic approach for evaluating intelligent services and existing evaluation tools are predominantly targeted at data scientists for pre-development evaluation. This paper presents a workflow and supporting tool, Threshy, to help software developers select a decision threshold suited to their problem domain. Unlike existing tools, Threshy is designed to operate in multiple workflows including pre-development, pre-release, and support. Threshy is designed for tuning the confidence scores returned by intelligent web services and does not deal with hyper-parameter optimisation used in ML models. Additionally, it considers the financial impacts of false positives. Threshold configuration files exported by Threshy can be integrated into client applications and monitoring infrastructure. Demo: https://bit.ly/2YKeYhE. △ Less

Submitted 19 August, 2020; originally announced August 2020.

A large-scale comparative analysis of Coding Standard conformance in Open-Source Data Science projects

Authors: Andrew J. Simmons, Scott Barnett, Jessica Rivera-Villicana, Akshat Bajaj, Rajesh Vasa

Abstract: Background: Meeting the growing industry demand for Data Science requires cross-disciplinary teams that can translate machine learning research into production-ready code. Software engineering teams value adherence to coding standards as an indication of code readability, maintainability, and developer expertise. However, there are no large-scale empirical studies of coding standards focused speci… ▽ More Background: Meeting the growing industry demand for Data Science requires cross-disciplinary teams that can translate machine learning research into production-ready code. Software engineering teams value adherence to coding standards as an indication of code readability, maintainability, and developer expertise. However, there are no large-scale empirical studies of coding standards focused specifically on Data Science projects. Aims: This study investigates the extent to which Data Science projects follow code standards. In particular, which standards are followed, which are ignored, and how does this differ to traditional software projects? Method: We compare a corpus of 1048 Open-Source Data Science projects to a reference group of 1099 non-Data Science projects with a similar level of quality and maturity. Results: Data Science projects suffer from a significantly higher rate of functions that use an excessive numbers of parameters and local variables. Data Science projects also follow different variable naming conventions to non-Data Science projects. Conclusions: The differences indicate that Data Science codebases are distinct from traditional software codebases and do not follow traditional software engineering conventions. Our conjecture is that this may be because traditional software engineering conventions are inappropriate in the context of Data Science projects. △ Less

Submitted 28 July, 2020; v1 submitted 17 July, 2020; originally announced July 2020.

Comments: 11 pages, 7 figures. To appear in ESEM 2020. Updated based on peer review

The oxygen partial pressure in solid oxide electrolysis cells with two layer electrolytes

Authors: Qian Zhang, Qinyuan Liu, Beom-Kyeong Park, Scott Barnett, Peter Voorhees

Abstract: A number of degradation mechanisms have been observed during the long-term operation of solid oxide electrolysis cells (SOEC). Using an electrolyte charge carrier transport model, we quantify the oxygen potentials across the electrolyte and thereby provide insights into these degradation mechanisms. Our model describes the transport of charge carriers in the electrolyte when the oxygen partial pre… ▽ More A number of degradation mechanisms have been observed during the long-term operation of solid oxide electrolysis cells (SOEC). Using an electrolyte charge carrier transport model, we quantify the oxygen potentials across the electrolyte and thereby provide insights into these degradation mechanisms. Our model describes the transport of charge carriers in the electrolyte when the oxygen partial pressure is extremely low by accounting for the spatial variation of the concentration of oxygen vacancies in the electrolyte. Moreover, we identify four quantities that characterize the distribution of oxygen partial pressure in the electrolyte, which are directly related to the degradation mechanisms in the electrolyte as well: the two oxygen partial pressures at the interfaces of the electrodes and the electrolyte, the oxygen partial pressure at the interface of YSZ/GDC, and the position of the abrupt change in oxygen potential near the p-n junction that develops in YSZ when one side of the cell is exposed to fuel (low oxygen potential, n-type conduction) and the other side is exposed to oxidant (high oxygen potential, p-type conduction). We give analytical estimates for all of these quantities. These analytical expressions provide guidance on the parameters that need to be controlled to suppress the degradation observed in the electrolyte. In addition, the effects of operating conditions, particularly current density and operating temperature, on degradation are discussed. △ Less

Submitted 3 June, 2020; originally announced June 2020.

Beware the evolving 'intelligent' web service! An integration architecture tactic to guard AI-first components

Authors: Alex Cummaudo, Scott Barnett, Rajesh Vasa, John Grundy, Mohamed Abdelrazek

Abstract: Intelligent services provide the power of AI to developers via simple RESTful API endpoints, abstracting away many complexities of machine learning. However, most of these intelligent services-such as computer vision-continually learn with time. When the internals within the abstracted 'black box' become hidden and evolve, pitfalls emerge in the robustness of applications that depend on these evol… ▽ More Intelligent services provide the power of AI to developers via simple RESTful API endpoints, abstracting away many complexities of machine learning. However, most of these intelligent services-such as computer vision-continually learn with time. When the internals within the abstracted 'black box' become hidden and evolve, pitfalls emerge in the robustness of applications that depend on these evolving services. Without adapting the way developers plan and construct projects reliant on intelligent services, significant gaps and risks result in both project planning and development. Therefore, how can software engineers best mitigate software evolution risk moving forward, thereby ensuring that their own applications maintain quality? Our proposal is an architectural tactic designed to improve intelligent service-dependent software robustness. The tactic involves creating an application-specific benchmark dataset baselined against an intelligent service, enabling evolutionary behaviour changes to be mitigated. A technical evaluation of our implementation of this architecture demonstrates how the tactic can identify 1,054 cases of substantial confidence evolution and 2,461 cases of substantial changes to response label sets using a dataset consisting of 331 images that evolve when sent to a service. △ Less

Submitted 27 May, 2020; originally announced May 2020.

Programmable coherent linear quantum operations with high-dimensional optical spatial modes

Authors: Shikang Li, Shan Zhang Xue Feng, Stephen M. Barnett, Wei Zhang, Kaiyu Cui, Fang Liu, Yidong Huang

Abstract: A simple and flexible scheme for high-dimensional linear quantum operations on optical transverse spatial modes is demonstrated. The quantum Fourier transformation (QFT) and quantum state tomography (QST) via symmetric informationally complete positive operator-valued measures (SIC POVMs) are implemented with dimensionality of 15. The matrix fidelity of QFT is 0.85, while the statistical fidelity… ▽ More A simple and flexible scheme for high-dimensional linear quantum operations on optical transverse spatial modes is demonstrated. The quantum Fourier transformation (QFT) and quantum state tomography (QST) via symmetric informationally complete positive operator-valued measures (SIC POVMs) are implemented with dimensionality of 15. The matrix fidelity of QFT is 0.85, while the statistical fidelity of SIC POVMs and fidelity of QST are ~0.97 and up to 0.853, respectively. We believe that our device has the potential for further exploration of high-dimensional spatial entanglement provided by spontaneous parametric down conversion in nonlinear crystals. △ Less

Submitted 14 May, 2020; originally announced May 2020.

Journal ref: Phys. Rev. Applied 14, 024027 (2020)

Spontaneous emission in anisotropic dielectrics

Authors: Anette Messinger, Niclas Westerberg, Stephen M. Barnett

Abstract: The emission properties of atoms lie at the foundations of both quantum theory and light-matter interactions. In the context of macroscopic media, exact knowledge thereof is important both in current quantum technologies as well as in fundamental studies. While for isotropic media, this is a very well-studied problem, there are still big gaps in the theory of anisotropic media. In particular, to t… ▽ More The emission properties of atoms lie at the foundations of both quantum theory and light-matter interactions. In the context of macroscopic media, exact knowledge thereof is important both in current quantum technologies as well as in fundamental studies. While for isotropic media, this is a very well-studied problem, there are still big gaps in the theory of anisotropic media. In particular, to the best of our knowledge, an explicit expression for the spontaneous emission rate in general anisotropic media has not been presented. In this work, we first derive the quantised electromagnetic field operators to calculate the emission rate in uniaxial media. For the more general case of biaxial media we propose an approximate expression based on interpolation between the limiting cases of uniaxial media. We support our model with numerical simulations which are in strong agreement for typical media configurations, and furthermore show how local field effects can be taken into account in the model. △ Less

Submitted 22 July, 2020; v1 submitted 23 April, 2020; originally announced April 2020.

Comments: 11 pages, 3 figures

Journal ref: Phys. Rev. A 102, 013721 (2020)

Ranking Computer Vision Service Issues using Emotion

Authors: Maheswaree K Curumsing, Alex Cummaudo, Ulrike Maria Graetsch, Scott Barnett, Rajesh Vasa

Abstract: Software developers are increasingly using machine learning APIs to implement 'intelligent' features. Studies show that incorporating machine learning into an application increases technical debt, creates data dependencies, and introduces uncertainty due to non-deterministic behaviour. However, we know very little about the emotional state of software developers who deal with such issues. In this… ▽ More Software developers are increasingly using machine learning APIs to implement 'intelligent' features. Studies show that incorporating machine learning into an application increases technical debt, creates data dependencies, and introduces uncertainty due to non-deterministic behaviour. However, we know very little about the emotional state of software developers who deal with such issues. In this paper, we do a landscape analysis of emotion found in 1,245 Stack Overflow posts about computer vision APIs. We investigate the application of an existing emotion classifier EmoTxt and manually verify our results. We found that the emotion profile varies for different question categories. △ Less

Submitted 27 May, 2020; v1 submitted 7 April, 2020; originally announced April 2020.

Interpreting Cloud Computer Vision Pain-Points: A Mining Study of Stack Overflow

Authors: Alex Cummaudo, Rajesh Vasa, Scott Barnett, John Grundy, Mohamed Abdelrazek

Abstract: Intelligent services are becoming increasingly more pervasive; application developers want to leverage the latest advances in areas such as computer vision to provide new services and products to users, and large technology firms enable this via RESTful APIs. While such APIs promise an easy-to-integrate on-demand machine intelligence, their current design, documentation and developer interface hid… ▽ More Intelligent services are becoming increasingly more pervasive; application developers want to leverage the latest advances in areas such as computer vision to provide new services and products to users, and large technology firms enable this via RESTful APIs. While such APIs promise an easy-to-integrate on-demand machine intelligence, their current design, documentation and developer interface hides much of the underlying machine learning techniques that power them. Such APIs look and feel like conventional APIs but abstract away data-driven probabilistic behaviour - the implications of a developer treating these APIs in the same way as other, traditional cloud services, such as cloud storage, is of concern. The objective of this study is to determine the various pain-points developers face when implementing systems that rely on the most mature of these intelligent services, specifically those that provide computer vision. We use Stack Overflow to mine indications of the frustrations that developers appear to face when using computer vision services, classifying their questions against two recent classification taxonomies (documentation-related and general questions). We find that, unlike mature fields like mobile development, there is a contrast in the types of questions asked by developers. These indicate a shallow understanding of the underlying technology that empower such systems. We discuss several implications of these findings via the lens of learning taxonomies to suggest how the software engineering community can improve these services and comment on the nature by which developers use them. △ Less

Submitted 27 January, 2020; originally announced January 2020.

Dynamics of Hot Bose-Einstein Condensates: stochastic Ehrenfest relations for number and energy damping

Authors: Rob G. McDonald, Peter S. Barnett, Fradom Atayee, Ashton S. Bradley

Abstract: Describing partially-condensed Bose gases poses a long-standing theoretical challenge. We present exact stochastic Ehrenfest relations for the stochastic projected Gross-Pitaevskii equation, including both number and energy damping mechanisms, and all projector terms that arise from the energy cutoff separating system from reservoir. We test the theory by applying it to the centre of mass fluctuat… ▽ More Describing partially-condensed Bose gases poses a long-standing theoretical challenge. We present exact stochastic Ehrenfest relations for the stochastic projected Gross-Pitaevskii equation, including both number and energy damping mechanisms, and all projector terms that arise from the energy cutoff separating system from reservoir. We test the theory by applying it to the centre of mass fluctuations of a harmonically trapped prolate system, finding close agreement between c-field simulations and analytical results. The formalism lays the foundation to analytically explore experimentally accessible hot Bose-Einstein condensates. △ Less

Submitted 12 December, 2019; v1 submitted 15 August, 2019; originally announced August 2019.

Comments: Revised discussion of reservoir properties, centre of mass motion, and their relevance to experimental systems

Journal ref: SciPost Phys. 8, 029 (2020)

On the conservation of helicity in a chiral medium

Authors: Frances Crimin, Neel Mackinnon, Jörg B. Götte, Stephen M. Barnett

Abstract: We consider the energy and helicity densities of circularly polarised light within a lossless chiral medium, characterised by the chirality parameter $β$. A form for the helicity density is introduced, valid to first order in $β$, that produces a helicity of $\pm\hbar$ per photon for right and left circular polarisation, respectively. This is in contrast to the result obtained if we use the form o… ▽ More We consider the energy and helicity densities of circularly polarised light within a lossless chiral medium, characterised by the chirality parameter $β$. A form for the helicity density is introduced, valid to first order in $β$, that produces a helicity of $\pm\hbar$ per photon for right and left circular polarisation, respectively. This is in contrast to the result obtained if we use the form of the helicity density employed for linear media. We examine the helicity continuity equation, and show that this modified form of the helicity density is required for consistency with the dual symmetry condition of a chiral medium with a constant value of $ε/μ$. Extending the results to arbitrary order in $β$ establishes an exact relationship between the energy and helicity densities in a chiral medium. △ Less

Submitted 23 July, 2019; originally announced July 2019.

Iodide-methylammonium interaction is responsible for ferroelectricity in CH3NH3PbI3

Authors: Joachim Breternitz, Frederike Lehmann, Sarah A. Barnett, Harriott Nowell, Susan Schorr

Abstract: Excellent conversion efficiencies of over 20 % and facile cell production have placed hybrid perovskites at the forefront of novel solar cell materials with CH3NH3PbI3 being its archetypal compound. The question why CH3NH3PbI3 has such extraordinary characteristics, particularly a hugely efficient light absorption, is hotly debated with ferroelectricity being a promising candidate. This does, howe… ▽ More Excellent conversion efficiencies of over 20 % and facile cell production have placed hybrid perovskites at the forefront of novel solar cell materials with CH3NH3PbI3 being its archetypal compound. The question why CH3NH3PbI3 has such extraordinary characteristics, particularly a hugely efficient light absorption, is hotly debated with ferroelectricity being a promising candidate. This does, however, afford the crystal structure to be non-centrosymmetric and we herein present crystallographic evidence as to how the symmetry breaking occurs on a crystallographic, and therefore long-scale, level. While the molecular cation CH3NH3+ is intrinsically polar, it is heavily disordered and cannot be the sole reason for ferroelectricity. We show that it, nonetheless, plays an important role as it distorts the neighboring iodide positions from their centrosymmetric positions. △ Less

Submitted 15 July, 2019; originally announced July 2019.

Multiple-copy state discrimination of noisy qubits

Authors: Kieran Flatt, Stephen M. Barnett, Sarah Croke

Abstract: Multiple-copy state discrimination is a fundamental task in quantum information processing. If there are two, pure, non-orthogonal states then both local and collective schemes are known to reach the Helstrom bound, the maximum probability of successful discrimination allowed by quantum theory. For mixed states, it is known that only collective schemes can perform optimally, so it might be expecte… ▽ More Multiple-copy state discrimination is a fundamental task in quantum information processing. If there are two, pure, non-orthogonal states then both local and collective schemes are known to reach the Helstrom bound, the maximum probability of successful discrimination allowed by quantum theory. For mixed states, it is known that only collective schemes can perform optimally, so it might be expected that these schemes are more resilient to preparation noise. We calculate the probability of success for two schemes, one local and one collective, in the regime of imperfect preparation fidelity. We find two surprising results. Firstly, both schemes converge upon the same many-copy limit, which is less than unity. Secondly, the local scheme performs better in all cases. This highlights the point that one should take into account noise when designing state discrimination schemes. △ Less

Submitted 26 June, 2019; originally announced June 2019.

Journal ref: Phys. Rev. A 100, 032122 (2019)

Paraxial Skyrmionic beams

Authors: Sijia Gao, Jörg B. Götte, Fiona C. Speirits, Francesco Castellucci, Sonja Franke-Arnold, Stephen M. Barnett

Abstract: Vector vortex beams possess a topological property that derives both from the spatially varying amplitude of the field and also from its varying polarization. This property arises as a consequence of the inherent Skyrmionic nature of such beams and is quantified by the associated Skyrmion number, which embodies a topological property of the beam. We illustrate this idea for some of the simplest ve… ▽ More Vector vortex beams possess a topological property that derives both from the spatially varying amplitude of the field and also from its varying polarization. This property arises as a consequence of the inherent Skyrmionic nature of such beams and is quantified by the associated Skyrmion number, which embodies a topological property of the beam. We illustrate this idea for some of the simplest vector beams and discuss the physical significance of the Skyrmion number in this context. △ Less

Submitted 24 June, 2020; v1 submitted 12 June, 2019; originally announced June 2019.

Comments: 6 pages, 6 figures

Journal ref: Phys. Rev. A 102, 053513 (2020)