Domain-Specific Visual Language for Data Engineering Quality
Data engineering pipelines process large amounts of information, and ensuring that the quality and integrity of the data is maintained throughout is critical for technical, business, and social reasons. Conventional data quality assurance approaches require a large amount of fine-grained testing code, which is laborious, easy to get out of sync, and inscrutable to non-technical stakeholders. An executable higher-level visual approach to expressing quality requirements can serve as a shared representation of these constraints and their implications for all parties, eliminating repetition while increasing accessibility and maintainability. We present a visual programming language for expressing data quality requirements within a pipeline declaratively, structured as a diagram of compositional data flow, transformation, and validation steps.
Mon 5 DecDisplayed time zone: Auckland, Wellington change
10:30 - 12:00 | |||
10:30 15mTalk | Integration testing can be reliable and low-effort in a projectional IDE through snapshots - DEMOVirtual PAINT Bastian Kruck itemis SECURE // Hasso Plattner Institute | ||
10:45 15mTalk | Towards a Python 3 IDE for Teaching Creative Programming PAINT Tristan Bunn Victoria University of Wellington, Craig Anslow Victoria University of Wellington, Karsten Lundqvist | ||
11:00 15mTalk | Conjecturing on a Fundamental Theorem of Computation and its Implications for a New Theory in Programmer Experience Design PAINT Gary Miller University of Technology Sydney | ||
11:15 15mTalk | Domain-Specific Visual Language for Data Engineering Quality PAINT DOI Pre-print | ||
11:30 15mTalk | Blocks, Blocks, and More Blocks-Based Programming PAINT Benjamin Selwyn-Smith Oracle Labs, Craig Anslow Victoria University of Wellington, Michael Homer Victoria University of Wellington DOI | ||
11:45 15mTalk | Interleaved 2D Notation for Concatenative Programming PAINT Michael Homer Victoria University of Wellington DOI Pre-print |