Atombeam Technologies Inc.

The University of North Carolina at Chapel Hill

University of South Carolina

Element Data, Inc.

Joshua Cooper

Charles Yeomans

Aliasghar Riahi

Mojgan Haddad

Razmin Riahi

Ryan Kourosh Riahi

Grant Fickes

Brian R Galvin

Gregory Caltabiano

Zhu Li

Leslie Victoria Parise

Noah Sciaky

Laila Elsherif

Keith A Harry

Erin Johnston

An adaptive transformation matrix generation is used simultaneous compression and encryption of data. The system continuously monitors input data streams to detect changes in distribution patterns, dynamically updating transformation matrices in response to these changes. Using performance evaluation criteria, the system selects and deploys optimal matrices that maintain both compression efficiency and cryptographic security as data characteristics evolve. The system implements configurable adaptation policies that govern when and how matrices are updated, ensuring system stability while maximizing performance. By employing sliding window analysis and distribution variance metrics, the system can quantify data drift and trigger appropriate adaptations. The system maintains compatibility with various operational modes including lossless, lossy, and modified lossless configurations, applying mode-specific optimization criteria to each scenario. This adaptive approach offers a resilient solution for data transmission and storage scenarios where both data reduction and security must be maintained across diverse and evolving data streams.

Dyadic distribution-based compression and encryption with adaptive transformation matrix generation

A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into a plurality of sourceblocks at all reasonable sourceblock lengths, and then counting how many times each sourceblock occurs in the dataset, resulting in a tally record of tokens and their count value. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent, in descending order of count value, to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

System and method for data compaction and encryption of anonymized data records

A system and method for filesystem data compression using codebooks, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams. Further, the system comprises a system for integrating the compression into a filesystem for both system-wide compression on a per-file or filegroup basis, and intrusion or alteration detection of files.

System and method for filesystem data compression using codebooks

A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by transforming the codebook into a hierarchical representation and then traversing the representation scanning for specific codewords associated with a data query request. In an embodiment, an estimator module is present and configured to utilize cardinality estimation to determine a starting codeword to begin searching the compacted data file for the data associated with the data query. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

System and method for random-access manipulation of compacted data files

A system and method for data compaction optimization which leverages a neural network to predict optimal block sizes for data encoding, enhancing efficiency and adaptability in various applications. It begins with data preprocessing, extracting features, and creating labeled datasets for training. The neural network architecture is carefully designed, allowing it to learn complex relationships between data characteristics and optimal block sizes. During training, the network is fine-tuned and optimized using appropriate loss functions and regularization techniques. Once deployed, it continuously monitors incoming data streams for shifts in data patterns and adapts predictions accordingly. By predicting multiple block sizes, the system accommodates diverse compression needs. This versatile system offers real-time adaptability, ensuring optimal encoding performance as data patterns evolve over time.

System and method for sourceblock length optimization for data compaction

A system and method for lossy precompression for data compaction using automated model monitoring and training, wherein statistical analyses of test datasets are used to determine if the probability distribution of two datasets are within a pre-determined range, and responsive to that determination new encoding and decoding algorithms may be retrained in order to produce new data sourceblocks, and pre-compression of data prior to processing and statistical analysis allows for the compaction of already compressed data into highly dense formats. The new data sourceblocks may then be processed and assigned new codewords which are compiled into an updated codebook which may be distributed back to encoding and decoding systems and devices.

Adaptive data processing system and method with dynamic optimization

Compression of medical imaging data using codebooks and entropy encoding. Medical imaging data such as tomosynthesis imagery data may be compressed using codewords based on frequency analysis. In an implementation sequential registration technique may be applied to the medical imaging data to create a plurality transformation matrices. The plurality of transformation matrices may be compressed using a matrix codebook. The compressed medical imaging data may be represented as an image codebook and the matrix codebook, providing secure storage and lossless compression of sensitive medical information.

Medical imaging data compression utilizing codebooks

A system and method for scriptable selective obfuscation of records comprising a data obfuscation module configured to identify and perform data anonymization on personal identifiable information (PII) contained within a plurality of records to create a partially-blurred dataset, and further comprising an encoder which receives the partially-blurred dataset and performs data compaction on the partially-blurred dataset before storing the compacted dataset in a data storage system. In some implementations, the data storage system is a blockchain database and the system functions as a clearinghouse to validate and monitor transactions involving data access rights between record owners and third-party entities. The system can further broker such transactions and direct payment form the third-party entity to the record owner when access rights have been purchased.

System and method for data compaction and security with extended functionality

A system and method for encrypted data compression with hardware management combines data compression techniques with real-time hardware optimization. The system receives an input data stream comprising data blocks, analyzes frequency distributions within the stream, and generates a conditioned data stream by identifying portions that deviate from target frequency distributions and applying conditioning rules. A hardware management layer continuously monitors the system resource utilization including processing latency, memory consumption, and field-programmable gate array (FPGA) resource usage during data processing operations. When resource utilization exceeds performance thresholds, the system generates hardware optimization plans and implements configuration changes including memory reallocation, processor adjustments, and FPGA resource management. The system applies Burrow's-Wheeler transforms (BWT) to improve data compressibility and creates error streams through XOR operations between original and conditioned data. The hardware management layer enables dynamic adaptation to changing processing demands while maintaining optimal system performance for data compression operations.

System and method for encrypted data compression with a hardware management layer

A system and methods for secure deduplication of compacted data comprising a data deconstruction engine, a data reconstruction engine, a library manager, a reference codebook, and a codeword storage which performs simultaneous compaction and deduplication of data sets. A data set may be comprised of one or more sourcepackets which may be optimally deconstructed into a plurality of sourceblocks and wherein each sourceblock may be compared against a reference codebook that contains key-value pairs of a sourceblock and its associated reference code in order to determine if a received sourceblock is a duplicate of data already stored within the reference codebook. Non-duplicate sourceblocks can have a reference code algorithmically created and stored in the reference codebook, thereby ensuring that when a duplicate sourceblock is received, it will not be stored as duplicated data.

System and methods for secure deduplication of compacted data

A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by accessing a reference codebook associated with the compacted data file, a frequency table used to construct the reference codebook, and data query details. A data read request is supported by random-access search capabilities that may enable the locating and decoding of the bits corresponding to data query details. A random-access engine facilitates data write processes. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

System and method for providing randomly-accessible compacted data

A system and method for codebook management is disclosed. Training datasets are obtained from various data sources. A similarity score is generated for each training dataset with reference to the other training datasets. In response to detecting a similarity score above a predetermined threshold for one or more of the other training datasets, a combined codebook is created based on training datasets that have a similarity score above a predetermined threshold. Based on the similarity score, multiple data sources are combined into a group, and the combined codebook is used for the data sources within the group. A mismatch performance metric can be computed for the combined codebook, and a revised combined codebook can be regenerated in response to the mismatch performance metric being above a predetermined threshold.

Codebook management based on data source grouping

A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into sourceblocks at reasonable intervals and tallying each sourceblock's frequency, creating a tally record of tokens and count values. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The data deconstruction engine and library manager may be distributed across multiple nodes or devices. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent descending order of count value to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

Data compaction using codebooks, that applies delta encoding methods to entropy encoding methods to improve data compaction of entropy encoding methods under certain conditions and when compacting data having certain characteristics. Delta encoding may be applied to entropy encoding methods to further compact data sets by reducing the number of sourceblocks included in a codebook to those most commonly encountered in data to be encoded and, where mismatches occur during encoding, using delta encoding of bit differences with existing sourceblocks in the codebook rather than adding new sourceblocks to the codebook.

Data compaction utilizing delta encoding

A system and method for file type identification involving extraction of a file-print of a file, the file-print being a unique or practically-unique representation of statistical characteristics associated with the distribution of bits in the binary contents of the file, similar to a fingerprint. The file-print is then passed to a machine learning algorithm that has been trained to recognize file types from their file-prints. The machine learning algorithm returns a predicted file type and, in some cases, a probability of correctness of the prediction. The file may then be encoded using an encoding algorithm chosen based on the predicted file type.

Growing community of inventors

Columbia, SC, United States of America

Joshua Cooper