-
The OpenCPU System: Towards a Universal Interface for Scientific Computing through Separation of Concerns
Authors:
Jeroen Ooms
Abstract:
Applications integrating analysis components require a programmable interface which defines statistical operations independently of any programming language. By separating concerns of scientific computing from application and implementation details we can derive an interoperable API for data analysis. But what exactly are the concerns of scientific computing? To answer this question, the paper sta…
▽ More
Applications integrating analysis components require a programmable interface which defines statistical operations independently of any programming language. By separating concerns of scientific computing from application and implementation details we can derive an interoperable API for data analysis. But what exactly are the concerns of scientific computing? To answer this question, the paper starts with an exploration of the purpose, problems, characteristics, struggles, culture, and community of this unique branch of computing. By mapping out the domain logic, we try to unveil the fundamental principles and concepts behind statistical software. Along the way we highlight important problems and bottlenecks that need to be addressed by the system in order to facilitate reliable and scalable analysis units. Finally, the OpenCPU software is introduced as an example implementation that builds on HTTP and R to expose a simple, abstracted interface for scientific computing.
△ Less
Submitted 3 June, 2014;
originally announced June 2014.
-
The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects
Authors:
Jeroen Ooms
Abstract:
A naive realization of JSON data in R maps JSON arrays to an unnamed list, and JSON objects to a named list. However, in practice a list is an awkward, inefficient type to store and manipulate data. Most statistical applications work with (homogeneous) vectors, matrices or data frames. Therefore JSON packages in R typically define certain special cases of JSON structures which map to simpler R typ…
▽ More
A naive realization of JSON data in R maps JSON arrays to an unnamed list, and JSON objects to a named list. However, in practice a list is an awkward, inefficient type to store and manipulate data. Most statistical applications work with (homogeneous) vectors, matrices or data frames. Therefore JSON packages in R typically define certain special cases of JSON structures which map to simpler R types. Currently there exist no formal guidelines, or even consensus between implementations on how R data should be represented in JSON. Furthermore, upon closer inspection, even the most basic data structures in R actually do not perfectly map to their JSON counterparts and leave some ambiguity for edge cases. These problems have resulted in different behavior between implementations and can lead to unexpected output. This paper explicitly describes a mapping between R classes and JSON data, highlights potential problems, and proposes conventions that generalize the mapping to cover all common structures. We emphasize the importance of type consistency when using JSON to exchange dynamic data, and illustrate using examples and anecdotes. The jsonlite R package is used throughout the paper as a reference implementation.
△ Less
Submitted 12 March, 2014;
originally announced March 2014.
-
RProtoBuf: Efficient Cross-Language Data Serialization in R
Authors:
Dirk Eddelbuettel,
Murray Stokely,
Jeroen Ooms
Abstract:
Modern data collection and analysis pipelines often involve a sophisticated mix of applications written in general purpose and specialized programming languages. Many formats commonly used to import and export data between different programs or systems, such as CSV or JSON, are verbose, inefficient, not type-safe, or tied to a specific programming language. Protocol Buffers are a popular method of…
▽ More
Modern data collection and analysis pipelines often involve a sophisticated mix of applications written in general purpose and specialized programming languages. Many formats commonly used to import and export data between different programs or systems, such as CSV or JSON, are verbose, inefficient, not type-safe, or tied to a specific programming language. Protocol Buffers are a popular method of serializing structured data between applications - while remaining independent of programming languages or operating systems. They offer a unique combination of features, performance, and maturity that seems particularly well suited for data-driven applications and numerical computing. The RProtoBuf package provides a complete interface to Protocol Buffers from the R environment for statistical computing. This paper outlines the general class of data serialization requirements for statistical computing, describes the implementation of the RProtoBuf package, and illustrates its use with example applications in large-scale data collection pipelines and web services.
△ Less
Submitted 28 January, 2014;
originally announced January 2014.
-
The RAppArmor Package: Enforcing Security Policies in R Using Dynamic Sandboxing on Linux
Authors:
Jeroen Ooms
Abstract:
The increasing availability of cloud computing and scientific super computers brings great potential for making R accessible through public or shared resources. This allows us to efficiently run code requiring lots of cycles and memory, or embed R functionality into, e.g., systems and web services. However some important security concerns need to be addressed before this can be put in production.…
▽ More
The increasing availability of cloud computing and scientific super computers brings great potential for making R accessible through public or shared resources. This allows us to efficiently run code requiring lots of cycles and memory, or embed R functionality into, e.g., systems and web services. However some important security concerns need to be addressed before this can be put in production. The prime use case in the design of R has always been a single statistician running R on the local machine through the interactive console. Therefore the execution environment of R is entirely unrestricted, which could result in malicious behavior or excessive use of hardware resources in a shared environment. Properly securing an R process turns out to be a complex problem. We describe various approaches and illustrate potential issues using some of our personal experiences in hosting public web services. Finally we introduce the RAppArmor package: a Linux based reference implementation for dynamic sandboxing in R on the level of the operating system.
△ Less
Submitted 1 November, 2013; v1 submitted 19 March, 2013;
originally announced March 2013.
-
Possible Directions for Improving Dependency Versioning in R
Authors:
Jeroen Ooms
Abstract:
One of the most powerful features of R is its infrastructure for contributed code. The built-in package manager and complementary repositories provide a great system for development and exchange of code, and have played an important role in the growth of the platform towards the de-facto standard in statistical computing that it is today. However, the number of packages on CRAN and other repositor…
▽ More
One of the most powerful features of R is its infrastructure for contributed code. The built-in package manager and complementary repositories provide a great system for development and exchange of code, and have played an important role in the growth of the platform towards the de-facto standard in statistical computing that it is today. However, the number of packages on CRAN and other repositories has increased beyond what might have been foreseen, and is revealing some limitations of the current design. One such problem is the general lack of dependency versioning in the infrastructure. This paper explores this problem in greater detail, and suggests approaches taken by other open source communities that might work for R as well. Three use cases are defined that exemplify the issue, and illustrate how improving this aspect of package management could increase reliability while supporting further growth of the R community.
△ Less
Submitted 1 November, 2013; v1 submitted 8 March, 2013;
originally announced March 2013.
