-
Metadata practices for simulation workflows
Authors:
José Villamar,
Matthias Kelbling,
Heather L. More,
Michael Denker,
Tom Tetzlaff,
Johanna Senk,
Stephan Thober
Abstract:
Computer simulations are an essential pillar of knowledge generation in science. Exploring, understanding, reproducing, and sharing the results of simulations relies on tracking and organizing the metadata describing the numerical experiments. The models used to understand real-world systems, and the computational machinery required to simulate them, are typically complex, and produce large amount…
▽ More
Computer simulations are an essential pillar of knowledge generation in science. Exploring, understanding, reproducing, and sharing the results of simulations relies on tracking and organizing the metadata describing the numerical experiments. The models used to understand real-world systems, and the computational machinery required to simulate them, are typically complex, and produce large amounts of heterogeneous metadata. Here, we present general practices for acquiring and handling metadata that are agnostic to software and hardware, and highly flexible for the user. These consist of two steps: 1) recording and storing raw metadata, and 2) selecting and structuring metadata. As a proof of concept, we develop the Archivist, a Python tool to help with the second step, and use it to apply our practices to distinct high-performance computing use cases from neuroscience and hydrology. Our practices and the Archivist can readily be applied to existing workflows without the need for substantial restructuring. They support sustainable numerical workflows, fostering replicability, reproducibility, data exploration, and data sharing in simulation-based research.
△ Less
Submitted 12 June, 2025; v1 submitted 30 August, 2024;
originally announced August 2024.
-
Towards a Smart Data Processing and Storage Model
Authors:
Ronie Salgado,
Marcus Denker,
Stéphane Ducasse,
Anne Etien,
Vincent Aranega
Abstract:
In several domains it is crucial to store and manipulate data whose origin needs to be completely traceable to guarantee the consistency, trustworthiness and reliability on the data itself typically for ethical and legal reasons. It is also important to guarantee that such properties are also carried further when such data is composed and processed into new data. In this article we present the mai…
▽ More
In several domains it is crucial to store and manipulate data whose origin needs to be completely traceable to guarantee the consistency, trustworthiness and reliability on the data itself typically for ethical and legal reasons. It is also important to guarantee that such properties are also carried further when such data is composed and processed into new data. In this article we present the main requirements and theorethical problems that arise by the design of a system supporting data with such capabilities. We present an architecture for implementing a system as well as a prototype developed in Pharo.
△ Less
Submitted 7 January, 2021;
originally announced January 2021.
-
Sub-method, partial behavioral reflection with Reflectivity: Looking back on 10 years of use
Authors:
Steven Costiou,
Vincent Aranega,
Marcus Denker
Abstract:
Context. Refining or altering existing behavior is the daily work of every developer, but that cannot be always anticipated, and software sometimes cannot be stopped. In such cases, unanticipated adaptation of running systems is of interest for many scenarios, ranging from functional upgrades to on-the-fly debugging or monitoring of critical applications.
Inquiry. A way of altering software at r…
▽ More
Context. Refining or altering existing behavior is the daily work of every developer, but that cannot be always anticipated, and software sometimes cannot be stopped. In such cases, unanticipated adaptation of running systems is of interest for many scenarios, ranging from functional upgrades to on-the-fly debugging or monitoring of critical applications.
Inquiry. A way of altering software at run time is using behavioral reflection, which is particularly well-suited for unanticipated adaptation of real-world systems. Partial behavioral reflection is not a new idea, and for years many efforts have been made to propose a practical way of expressing it. All these efforts resulted in practical solutions, but which introduced a semantic gap between the code that requires adaptation and the expression of the partial behavior. For example, in Aspect-Oriented Programming, a pointcut description is expressed in another language, which introduces a new distance between the behavior expression (the Advice) and the source code in itself.
Approach. Ten years ago, the idea of closing the gap between the code and the expression of the partial behavior led to the implementation of the Reflectivity framework. Using Reflectivity, developers annotate Abstract Syntax Tree (AST) nodes with meta-behavior which is taken into account by the compiler to produce behavioral variations. In this paper, we present Reflectivity, its API, its implementation and its usage in Pharo. We reflect on ten years of use of Reflectivity, and show how it has been used as a basic building block of many innovative ideas.
Knowledge. Reflectivity brings a practical way of working at the AST level, which is a high-level representation of the source code manipulated by software developers. It enables a powerful way of dynamically add and modify behavior. Reflectivity is also a flexible mean to bridge the gap between the expression of the meta-behavior and the source code. This ability to apply unanticipated adaptation and to provide behavioral reflection led to many experiments and projects during this last decade by external users. Existing work use Reflectivity to implement reflective libraries or languages extensions, featherweight code instrumentation, dynamic software update, debugging tools and visualization and software analysis tools.
Grounding. Reflectivity is actively used in research projects. During the past ten years, it served as a support, either for implementation or as a fundamental base, for many research work including PhD theses, conference, journal and workshop papers. Reflectivity is now an important library of the Pharo language, and is integrated at the heart of the platform.
Importance. Reflectivity exposes powerful abstractions to deal with partial behavioral adaptation, while providing a mature framework for unanticipated, non-intrusive and partial behavioral reflection based on AST annotation. Furthermore, even if Reflectivity found its home inside Pharo, it is not a pure Smalltalk-oriented solution. As validation over the practical use of Reflectivity in dynamic object-oriented languages, the API has been ported to Python. Finally, the AST annotation feature of Reflectivity opens new experimentation opportunities about the control that developers could gain on the behavior of their own software.
△ Less
Submitted 14 February, 2020;
originally announced February 2020.
-
Ghost: A Uniform and General-Purpose Proxy Implementation
Authors:
Mariano Martinez Peck,
Noury Bouraqadi,
Stéphane Ducasse,
Luc Fabresse,
Marcus Denker
Abstract:
A proxy object is a surrogate or placeholder that controls access to another target object. Proxy objects are a widely used solution for different scenarios such as remote method invocation, future objects, behavioral reflection, object databases, inter-languages communications and bindings, access control, lazy or parallel evaluation, security, among others. Most proxy implementations support pro…
▽ More
A proxy object is a surrogate or placeholder that controls access to another target object. Proxy objects are a widely used solution for different scenarios such as remote method invocation, future objects, behavioral reflection, object databases, inter-languages communications and bindings, access control, lazy or parallel evaluation, security, among others. Most proxy implementations support proxies for regular objects but are unable to create proxies for objects with an important role in the runtime infrastructure such as classes or methods. Proxies can be complex to install, they can have a significant overhead, they can be limited to certain kind of classes, etc. Moreover, proxy implementations are often not stratified and they do not have a clear separation between proxies (the objects intercepting messages) and handlers (the objects handling interceptions). In this paper, we present Ghost: a uniform and general-purpose proxy implementation for the Pharo programming language. Ghost provides low memory consuming proxies for regular objects as well as for classes and methods. When a proxy takes the place of a class, it intercepts both the messages received by the class and the lookup of methods for messages received by its instances. Similarly, if a proxy takes the place of a method, then the method execution is intercepted too.
△ Less
Submitted 29 October, 2013;
originally announced October 2013.
-
Waterfall: Primitives Generation on the Fly
Authors:
Guido Chari,
Diego Garbervetsky,
Camillo Bruni,
Marcus Denker,
Stéphane Ducasse
Abstract:
Modern languages are typically supported by managed runtimes (Virtual Machines). Since VMs have to deal with many concepts such as memory management, abstract execution model and scheduling, they tend to be very complex. Additionally, VMs have to meet strong performance requirements. This demand of performance is one of the main reasons why many VMs are built statically. Thus, design decisions are…
▽ More
Modern languages are typically supported by managed runtimes (Virtual Machines). Since VMs have to deal with many concepts such as memory management, abstract execution model and scheduling, they tend to be very complex. Additionally, VMs have to meet strong performance requirements. This demand of performance is one of the main reasons why many VMs are built statically. Thus, design decisions are frozen at compile time preventing changes at runtime. One clear example is the impossibility to dynamically adapt or change primitives of the VM once it has been compiled. In this work we present a toolchain that allows for altering and configuring components such as primitives and plug-ins at runtime. The main contribution is Waterfall, a dynamic and reflective translator from Slang, a restricted subset of Smalltalk, to native code. Waterfall generates primitives on demand and executes them on the fly. We validate our approach by implementing dynamic primitive modification and runtime customization of VM plug-ins.
△ Less
Submitted 10 October, 2013;
originally announced October 2013.
-
Empowering Collections with Swarm Behavior
Authors:
Adrian Kuhn,
David Erni,
Marcus Denker
Abstract:
Often, when modelling a system there are properties and operations that are related to a group of objects rather than to a single object. In this paper we extend Java with Swarm Behavior, a new composition operator that associates behavior with a collection of instances. The lookup resolution of swarm behavior is based on the element type of a collection and is thus orthogonal to the collection hi…
▽ More
Often, when modelling a system there are properties and operations that are related to a group of objects rather than to a single object. In this paper we extend Java with Swarm Behavior, a new composition operator that associates behavior with a collection of instances. The lookup resolution of swarm behavior is based on the element type of a collection and is thus orthogonal to the collection hierarchy.
△ Less
Submitted 1 July, 2010;
originally announced July 2010.
