Payload size reduction and reassembly of recursively parsable structures

Abstract

Here is message compression using schema inference for condensing semantic content by removal of syntactic structure in multiple kinds of content. Topological structures of different trees are generalized to generate a merged tree. Because compression discards redundant content and often only semantic content is retained, the signal-to-noise ratio is increased, which increases accuracy of downstream semantic analytics such as machine learning. Compression based on the merged tree removes redundant information from new messages that, without obscuring semantic content, decreases the data volume for downstream analytics or archiving. This compression extracts semantic values that can be assembled into a sequence of lexical tokens that is suitable for natural language processing (NLP), and the sequence of lexical tokens does not contain tokens that represent syntax or structure. Thus, compression provides fewer tokens to be processed by a downstream language model, which is suitable for efficient processing of a live data stream.