CircuitFlow: : A Domain Specific Language for Dataflow Programming
Dataflow applications, such as machine learning algorithms, can run for days, making it desirable to have assurances that they will work correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable runtime errors.This paper presents a new declarative Haskell embedded DSL (eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder on data that models dependencies in the workflow, it has a strong mathematical basis, refocusing on how data flows through an application, resulting in a more expressive solution that not only catches errors statically, but also achieves competitive run-time performance. In our preliminary evaluation, CircuitFlow outperforms the industry-leading Luigi library of Spotify by scaling better with the number of inputs.
Tue 18 JanDisplayed time zone: Eastern Time (US & Canada) change
13:30 - 14:45
|Graph-based Interpretation of Normal Logic ProgramsRemote|
|Functional Programming on Top of SQL EnginesRemote|
|CircuitFlow: : A Domain Specific Language for Dataflow ProgrammingIn-Person|