Methods for intelligent load balancing and high speed intelligent network recorders

Abstract

A high speed intelligent network recorder for recording a plurality of flows of network data packets into and out of a computer network over a relevant data time window is disclosed. The high speed intelligent network recorder includes a printed circuit board; a high speed network switching device mounted to the printed circuit board; and an X column by Y row array of a plurality of intelligent hard drives with micro-computers mounted to the printed circuit board and coupled in parallel with the high speed network switching device. A method for network recording is disclosed. In one embodiment, the method includes the following: receiving a plurality of incoming packets, wherein each incoming packet belongs to a conversation flow; forming a capture stream of packet records for the incoming packets; and performing intelligent load balancing on the capture stream of packet records, the load balancing including reading the metadata for each packet record, determining a packet record is part of either a hot flow or a cold flow, selecting a destination node for each packet record based on the flow hash, and steering the packet record to one of a plurality of encapsulation buffers based on the destination node, wherein a cold flow tends to be maintained in a flow coherency at a node. The method may further include operations that include querying and back-testing in order to enable distributed analytics by using low cost, low band width nodes.