Enhanced instrumentation software in fault tolerant systems

Abstract

A monitor function is implemented to monitor and control service processes and other system entities that perform tasks on a distributed network. The monitor function tracks the demise and instantiation of processes and entities that either export or import instrumentation. Any service process or other system entity (driver, interrupt handler, system library procedure) can export instruments (indicators, controls, testpoints). Instrument updates are propagated automatically if they are significant. The importing process conveys the information to a management system so that a human operator, or automated system, can observe and control the operation of the network service. One aspect of the invention uses a backup exporter to take over the processing of an exporter that has become nonfunctional. Another aspect of the invention determines when a CPU has gone down and acts accordingly to identify service processes that were associated with an exporter in the down CPU. For each exporter in the down CPU, any importer that was monitoring with the exporter must perform a switchover to monitor instrumentation on the new exporter in a fault tolerant manner. Additional methods are disclosed which implement fault tolerant monitoring and control in a distributed network.