Splunk Inc.

Denis Vergnes

Daniel Ferstay

Zhimin Liang

Jeff Fan

This technology is directed to facilitating scalable and secure data collection. In particular, scalability of data collection is enabled in a secure manner by, among other things, abstracting a connector(s) to a pod(s) and/or container(s) that executes separate from other data-collecting functionality. For example, an execution manager can initiate deployment of a collect coordinator on a first pod associated with a first job and deployment of a first connector on a second pod associated with a second job separate from the first job of a container-managed platform. The collect coordinator can provide a data collection task to the first connector deployed on the second pod of the second job. The first connector can then obtain the set of data from the data source and provide the set of data to the collect coordinator for providing the set of data to a remote source.

Enhancing efficiency of data collection using a discover process

Systems and methods are described for balancing workloads and reliably delivering data to a plurality of indexing systems in a data intake and query system. A topic-based indexing system load balancer may receive event data from various data sources, each of which may be associated with a topic. The event data may be entirely unparsed, unparsed but divided into events, or parsed into events. The topic-based indexing system load balancer may distribute the received event data on a per-topic or per-event basis to a set of indexing systems, and may distribute topics and events based on the volume received. Unparsed data may be divided into portions, and the topic-based indexing system load balancer may ensure that portions data associated with the same topic are delivered to the same indexer so that events split between two portions may be recombined and indexed.

Load balancing, failover, and reliable delivery of data in a data intake and query system

Embodiments of the present invention are directed to facilitating efficient message queueing. In particular, embodiments herein describe, among other things, a redelivery monitor used to monitor when to redeliver messages, or tasks, for reprocessing based on expiration of a redelivery deadline. In this regard, markers indicating processing states for tasks being processed are read by the redelivery monitor. When the processing state indicates that processing is ongoing, the redelivery deadline is extended such that a message or task is not redelivered for processing while the message or task is being processed.

Growing community of inventors

Vancouver, Canada

Denis Vergnes