Clustering at ingest time

Abstract

A system and method for optimizing data ingestion through clustering during the copy process. The system detects a data processing pipeline definition with a COPY command and determines storage location and destination table parameters. Upon detecting a COPY command, the system parses the command syntax to identify storage location and destination table parameters. The system monitors for available file batches at the storage location and executes the COPY command to transform these batches into formatted files compatible with the destination table format. The system then performs clustering operations on the formatted files, organizing data according to pre-defined clustering keys, before ingesting the clustered files into the destination table.