Data pipeline branching

Abstract

A workbook management system provides a master branch of a data pipeline comprising a pointer(s) to a snapshot(s) of an initial dataset(s), a first logic, and a pointer(s) to a snapshot(s) of a first derived dataset(s) resulting from applying the first logic to the initial dataset(s). Responsive to user input requesting a test branch corresponding to the master branch, the system creates the test branch comprising the pointer(s) to the snapshot(s) of the initial dataset(s) and a copy of the first logic. The system receives a request to modify the test branch comprising at least one change to the copy of the first logic, and modifies the test branch independently of the master branch to include second logic reflecting the at least one change to the copy of the first logic, the pointer(s) to the snapshot(s) of the initial dataset(s), and a pointer(s) to snapshot(s) of a second derived dataset(s) resulting from applying the second logic to the initial dataset(s). Responsive to user input requesting a merge of the modified test branch into the master branch, the system updates the master branch to replace the first logic with the second logic and to replace the pointer(s) to the snapshot(s) of the first derived dataset(s) with the pointer(s) to the snapshot(s) of the second derived dataset(s).