Distributed data processing using embedded hermetic and deterministic language

Abstract

In one embodiment, a computer system comprises one or more central processing units (CPUs) that are communicatively coupled to a system clock, one or more network interfaces, and one or more database interfaces; digital electronic main memory that is communicatively coupled to the one or more CPUs and storing one or more sequences of stored program instructions which, when executed using the one or more CPUs, cause the one or more CPUs to execute a plurality of different consumer services of a SaaS-based data analytics platform, each of the consumer services hosting an instance of a sandboxed runtime for a hermetic and deterministic programming language; user function storage that is communicatively coupled to one of the database interfaces and storing a plurality of different user functions, each of the user functions having been programmed using the programming language, each of the user functions being stored in association with a reference to a destination table of a destination database; each of the consumer services being programmed to initiate a data ingestion process; load a copy of a user function from the user function storage to the sandboxed runtime that is associated with the particular consumer service; using the sandboxed runtime local to the particular consumer service, execute the user function over records directed to the destination table identified in the reference; filter the records or write new records resulting from the function to the destination table.