Collaborators
Brown University
Portland State
Carnegie Mellon

About

Stream processing addresses the needs of real-time applications. Transaction processing addresses the coordination and safety of short atomic computations. In the past, these two modes of operation were found only in separate, stove-piped systems. However, with the creation of NewSQL OLTP systems, it becomes possible to perform scalable real-time operations without sacrificing transactional support. Enter S-Store, the world’s first transactional streaming database system.

Architecture

Architecture

Architecture

S-Store is built on H-Store, a distributed main-memory OLTP system, taking advantage of its ability to process transactions with low latency and high throughput. We add support for push-based processing, including two layers of triggering mechanisms. We also introduce native data structures for streams and windows.

Transaction Model

Transaction Model

Transaction Model

S-Store models its streaming workflow as DAGs of streaming transactions. A streaming transaction is considered to be a stored procedure (SP) operating on an input batch of tuples. Due to data and processing dependencies, transaction executions (TEs) cannot be performed in an arbitrary order, and additional isolation rules are necessary to preserve ACID guarantees.
Transaction Model

Applications

Applications

Applications

S-Store is designed to meet the needs of a number of transactional applications driven by high-velocity streaming inputs. As a demonstration, we have implemented a system that models a bicycle share program, known as BikeShare. S-Store is also applicable to systems such as online gaming, real-time marketing, Internet of Things, voting leaderboards, and real-time ETL frontends.

S-Store is integrated with BigDAWG, a reference implementation of a polystore database. S-Store has been demonstrated live within the context of BigDAWG, running a MIMIC II application for ICU research.

A second demo showing S-Store's role in BigDAWG, this time in a the context of analyzing ocean metagenomics.