Monitoring cross-server services?

[Steven Grimm]

What's the accepted way to set up monitoring of distributed or peer-to-peer
services?  For example, I have a server app that connects to copies of
itself on other hosts to keep its data synchronized.  I'd ideally like to
make sure that each host is connected to all the peers it's supposed to
talk to.

The problem is, if hosts A, B, and C are all connected to each other, I
don't want the services on hosts A and C to show up as broken if host B
crashes, since I'll have already gotten a notification about host B and
the error indications on the other two hosts will just be distracting
clutter on the status web page.

On the other hand, if there's a connectivity problem between hosts A and
B but both hosts are healthy and reachable from the monitoring host, I
*do* want the error condition to be flagged.  (Application-level
connectivity, I mean, not necessarily IP connectivity.)

The best solution I've come up with is to wrap a local plugin around the
call to the plugin on the remote host that checks the status of the app;
the remote plugin reports back a list of peers that the app isn't
connected to, and the local plugin checks the Nagios status file to make
sure the app is listed as OK on all those peers.  If all the problem
peers are offline, the warning/critical result from the remote plugin
is suppressed.

Anyone have a better approach?

-Steve


-------------------------------------------------------
This SF.NET email is sponsored by: FREE  SSL Guide from Thawte
are you planning your Web Server Security? Click here to get a FREE
Thawte SSL guide and find the answers to all your  SSL security issues.
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0026en