Fireeye, Inc.

Christopher Hayes Fauerbach

Dennis Lee Edwards

Randy I Caldejon

An indexing arrangement enables efficient search and retrieval of indexes persistently stored in a metadata repository and used to locate packets captured from a network and persistently stored in a data repository. The packets are captured at a packet capture and retrieval system having persistent storage devices organized as files of the metadata and data repositories. Search and retrieval of the indexes within the files of the metadata repository occur at substantially a same time as one or more other captured packets is written to one or more files of the data repository to realize a substantially high sustained packet transfer rate of the network.

Arrangement for efficient search and retrieval of indexes used to locate captured packets

A secondary indexing technique cooperates with primary indices of an indexing arrangement to enable efficient storage and access of metadata used to retrieve packets persistently stored in data files of a data repository. Efficient storage and access of the metadata used to retrieve the persistently stored packets may be based on a target value of the packets over a search time window. The metadata is illustratively organized as a metadata repository of primary index files that store the primary indices containing hash values of network flows of the packets, as well as offsets and paths to those packets stored in the data files. The technique includes one or more secondary indices having a plurality of present bits arranged in a binary format (i.e., a bit array) to indicate the presence of the target value in one or more packets stored in the data files over the search time window. Notably, the present bits may be used to reduce (i.e., 'prune') a relatively large search space of the stored packets (e.g., defined by the hash values) to a pruned search space of only those data files in which packets having the target value are stored.

Efficient access to sparse packets in large repositories of stored network traffic

A processing technique provides an improved indexing arrangement that enables storage, filtering and querying of metadata used to retrieve packets captured from a network and persistently stored in a data repository. A packet capture engine records the packets in packet capture (PCAP) formats from a network link at a substantially high packet transfer rate to persistent storage of the data repository in a sustained manner. Efficient filtering and querying of the metadata to retrieve the stored packets may be achieved, in part, by organizing the metadata as one or more metadata repositories. The processing technique uses the Berkeley Packet Filter (BPF) language as an interface of a BPF engine to search or index the stored packets in response to queries. The BPF engine processes BPF expressions used as precursors to the indexing arrangement to enable access to the repositories when searching and locating stored packets matching the expressions.

Growing community of inventors

Midlothian, VA, United States of America

Christopher Hayes Fauerbach