System and method for data storage, transfer, synchronization, and security using automated model monitoring and training with a load-adaptive cache

Abstract

A system and method for efficient data storage, transfer, synchronization, and security using automated model monitoring and training. The system analyzes test datasets to detect data drift, retraining encoding and decoding algorithms as needed. New data sourceblocks are created and assigned codewords, compiling an updated codebook for distribution to connected devices. A novel dyadic distribution subsystem simultaneously compresses and encrypts data by transforming input streams into a dyadic distribution. This process generates a compressed main data stream and a secondary stream of transformation information, which are combined into a secure output. The system includes a network device manager for optimizing codebook distribution based on device resource usage. Operating in both lossless and lossy modes, the system offers flexible, efficient, and secure data handling across various network configurations.