Storing network bidirectional flow data and metadata with efficient processing technique

Abstract

A processing technique provides an improved indexing arrangement that enables storage, filtering and querying of metadata used to retrieve packets captured from a network and persistently stored in a data repository. A packet capture engine records the packets in packet capture (PCAP) formats from a network link at a substantially high packet transfer rate to persistent storage of the data repository in a sustained manner. Efficient filtering and querying of the metadata to retrieve the stored packets may be achieved, in part, by organizing the metadata as one or more metadata repositories. The processing technique uses the Berkeley Packet Filter (BPF) language as an interface of a BPF engine to search or index the stored packets in response to queries. The BPF engine processes BPF expressions used as precursors to the indexing arrangement to enable access to the repositories when searching and locating stored packets matching the expressions.