Event Handlers are not runing or logging. (on WARNING or CRITICAL)
bruce
bruce at webfarm.co.nz
Sat Aug 28 15:15:13 CEST 2004
Hi,
Ive done a little more testing and it appears the event handlers ARE
running but only when the state changes to OK, which of course is no use
for fixing the problem.
Below is the nagios.log file from one of the live system (well result of:
egrep 'creeper.*Defun' var/nagios.log), freshclam seems
to be running on all the severs but the Defunct processes check does get
some results. The nagios configs are excatly the same for these also (the
command sends fixdefuncts.sh instead of restartFreshClam.sh and thats the
only difference.
-- 8<-- nagios.log
[1093669850] SERVICE ALERT: creeper;Defuncts;OK;HARD;1;OK - 5 processes
running with STATE = Z
[1093670146] SERVICE ALERT: creeper;Defuncts;WARNING;HARD;1;WARNING - 6
processes running with STATE = Z
[1093673451] SERVICE ALERT: creeper;Defuncts;WARNING;HARD;1;WARNING - 7
processes running with STATE = Z
[1093677052] SERVICE ALERT: creeper;Defuncts;WARNING;HARD;1;WARNING - 8
processes running with STATE = Z
[1093680652] SERVICE ALERT: creeper;Defuncts;WARNING;HARD;1;WARNING - 10
processes running with STATE = Z
[1093684251] SERVICE ALERT: creeper;Defuncts;WARNING;HARD;1;WARNING - 10
processes running with STATE = Z
[1093685900] SERVICE ALERT: creeper;Defuncts;CRITICAL;HARD;1;CRITICAL -
11 processes running with STATE = Z
[1093687852] SERVICE ALERT: creeper;Defuncts;CRITICAL;HARD;1;CRITICAL -
11 processes running with STATE = Z
[1093691451] SERVICE ALERT: creeper;Defuncts;CRITICAL;HARD;1;CRITICAL -
13 processes running with STATE = Z
[1093695059] SERVICE ALERT: creeper;Defuncts;CRITICAL;HARD;1;CRITICAL -
15 processes running with STATE = Z
[1093696438] SERVICE ALERT: creeper;Defuncts;OK;HARD;1;OK - 0 processes
running with STATE = Z
[1093696438] SERVICE EVENT HANDLER:
creeper;Defuncts;OK;HARD;1;allserver_defunct_fix
[1093696516] SERVICE ALERT: creeper;Defuncts;OK;HARD;1;OK - 0 processes
running with STATE = Z
[1093696624] SERVICE ALERT: creeper;Defuncts;OK;HARD;1;OK - 0 processes
running with STATE = Z
[1093696673] SERVICE ALERT: creeper;Defuncts;OK;HARD;1;OK - 0 processes
running with STATE = Z
[1093697080] SERVICE ALERT: creeper;Defuncts;OK;HARD;1;OK - 1 processes
running with STATE = Z
-- 8<-- End nagios.log
As you can see it goes through the motions, OK => WARNING => CRITICAL =>
OK (when we mannually restart the offending process on the server, yeah
the better fix would be to fix the process but we are still investigating
why it happens :( very weird, but different issue )
When changing from OK => WARNING it dosnt run the event handler, only when
it goes back to OK does it run.
If I change the event handlers args to be a static CIRITCAL the handler
logs in and does the restart, so everything is fine there.
Here are the related config sections just for reference of this command
and service:
define service {
use hosted
service_description Defuncts
check_command serv_check_zombie_procs
event_handler allserver_defunct_fix
event_handler_enabled 1
hostgroup_name shared
}
define command {
command_name allserver_defunct_fix
command_line $USER1$/fix-w-allserver.sh $HOSTADDRESS$ $SERVICESTATE$ $SERVICEATTEMPT$ defunctFix.sh
}
Any thoughts or suggestions?
Cheers,
--
+------------------------------------------+ \|||/
| Bruce at WebFarm.co.nz +64 06 7572881 | (o o)
| Systems Technician +---ooO-(_)-Ooo---+
| |
| WebFarm http://www.webfarm.co.nz |
| FreeParking http://www.freeparking.co.nz |
+------------------------------------------------------------+
... FreeParking - NZ's best value Domain, WebHosting and email accounts - bar none
... WebFarm - NZ's eCommerce specialists since 1997
On Fri, 27 Aug 2004, Balestra, Roberto wrote:
> Hi,
> if $USER1$ is right configured, if fix-w-allserver.sh has right permission, if commands into fix-w-allserver.sh are right defined, I think problem may be in service.
> Eventhandle running on the change of the service's state. If your service is always Critical, eventhandler never run.
> Try force service's state to change.
> Bye
> Roberto
>
> > -----Original Message-----
> > From: nagios-users-admin at lists.sourceforge.net
> > [mailto:nagios-users-admin at lists.sourceforge.net]On Behalf Of Bruce
> > Sent: venerdì 27 agosto 2004 8.27
> > To: Nagios Users
> > Subject: Re: [Nagios-users] Event Handlers are not runing or logging.
> >
> >
> > Hi,
> >
> > Sorry, my email client decided the the -- was the end of the
> > message and
> > didnt recopy the rest of the configuration when I followed it up, ive
> > included these again below,
> >
> > Joe suggested that I check the nagios.log file this was included also
> > and it dosnt show the messages at all, just the services getting the
> > errors and nothing else (It was running for about 30 minutes after it
> > detected the service problem, and all it done was repeat the services
> > were Critical)
> > If its a config issue, then please let me know what wrong (or if you
> > need more information) as it really has me stumped.
> >
> >
> > Here is the configs that I originally had posted, (Ive changed the --
> > File... to --8<-- so it dosnt get knocked off next time)
> >
> > --8<----8<-- Original post information --8<----8<--
> > --8<-- nagios.cfg event stuff --
> > [root etc]# grep event nagios.cfg
> > log_event_handlers=1
> > #global_host_event_handler=somecommand
> > #global_service_event_handler=somecommand
> > event_handler_timeout=30
> > enable_event_handlers=1
> > --8<--
> >
> > --8<-- checkcommands.cfg --
> > define command {
> > command_name allserver_freshclam_fix
> > command_line $USER1$/fix-w-allserver.sh
> > $HOSTADDRESS$
> > restartFreshClam.sh $SERVICESTATE$ $SERVICEATTEMPT$
> > }
> > --8<--
> >
> > --8<-- hosts.cfg --
> > # Nagios hosts config
> > # Autogenerated file, do not modify
> > # Modify main etc hosts.cfg file
> > #
> > # Generated: 13-Aug-2004 06 pm
> >
> > define host {
> > use basic
> > host_name creeper
> > alias creeper
> > address 192.168.56.32
> > }
> > --8<--
> >
> > --8<-- services.cfg --
> > # Nagios services config
> > # Autogenerated file, do not modify
> > # Modify main etc services.cfg file
> > #
> > # Generated: 13-Aug-2004 06 pm
> >
> > define service {
> > name generic-service
> > passive_checks_enabled 1
> > parallelize_check 1
> > obsess_over_service 1
> > check_freshness 1
> > notifications_enabled 1
> > event_handler_enabled 1
> > flap_detection_enabled 1
> > process_perf_data 1
> > retain_status_information 1
> > retain_nonstatus_information 1
> > register 0
> > }
> >
> > define service {
> > use generic-service
> > name hosted-basic
> > is_volatile 0
> > check_period 24x7
> > max_check_attempts 7
> > normal_check_interval 3
> > retry_check_interval 1
> > notification_interval 240
> > notification_period 24x7
> > notification_options w,u,c,r
> > register 0
> > }
> >
> > define service {
> > use hosted-basic
> > name hosted
> > register 0
> > host_name creeper
> > }
> >
> > define service {
> > use hosted
> > service_description FreshClam
> > check_command check_serv_freshclam!
> >
> > event_handler allserver_freshclam_fix
> > event_handler_enabled 1
> > }
> > --8<--
> >
> > --8<-- nagios.log --
> > [1092451070] Nagios 1.1 starting... (PID=28391)
> > [1092451070] Warning: Service 'FreshClam' on host 'creeper' has no
> > default contact group(s) defined!
> > [1092451070] Warning: Host 'creeper' is not a member of any
> > host groups!
> > [1092451070] Warning: Host 'localhost' has no services
> > associated with it!
> > [1092451070] Finished daemonizing... (New PID=28392)
> > [1092451345] HOST ALERT: creeper;DOWN;HARD;10;/bin/ping -n -c 1
> > 66.70.72.180
> > [1092451345] SERVICE ALERT:
> > creeper;FreshClam;CRITICAL;HARD;1;CRITICAL -
> > process freshclam is not running !
> > --8<--
> >
> > Running the command does work as the user nagios, but even if the
> > contents of the command is:
> > /bin/echo "it ran" >/tmp/nagios.event
> > it dosnt create the file /tmp/nagios.event (which ive ensured dosnt
> > already exist and the permissions are correct)
> >
> > Can anyone see any reason why nagios is not running (or even just
> > logging) the event? I did notice that the service alert says this is
> > HARD, since nagios has just come up and there was no status
> > file before
> > hand it should be SOFT.
> >
> > Any help would be appreciated,
> >
> >
> > --8<----8<-- End of original message --8<----8<--
> >
> > --
> > +------------------------------------------+ \|||/
> > | Bruce at WebFarm.co.nz +64 06 7572881 | (o o)
> > | Systems Technician +---ooO-(_)-Ooo---+
> > | |
> > | WebFarm http://www.webfarm.co.nz |
> > | FreeParking http://www.freeparking.co.nz |
> > +------------------------------------------------------------+
> >
> > ... FreeParking - NZ's best value Domain, WebHosting and
> > email accounts - bar none
> > ... WebFarm - NZ's eCommerce specialists since 1997
> >
> >
> >
> >
> > Balestra, Roberto wrote:
> >
> > >Hi,
> > >I see your command configuration 'allserver_freshclam_fix'
> > but I don't see the event that has to run it.
> > >Heve you defined any service's checks for your event command?
> > >Roberto
> > >
> > >
> > >
> > >>-----Original Message-----
> > >>From: nagios-users-admin at lists.sourceforge.net
> > >>[mailto:nagios-users-admin at lists.sourceforge.net]On Behalf Of Bruce
> > >>Sent: giovedì 26 agosto 2004 7.50
> > >>To: Nagios Users
> > >>Subject: Re: [Nagios-users] Event Handlers are not runing
> > or logging.
> > >>
> > >>
> > >>Hi,
> > >>
> > >>Has anyone had any thoughts on what the cause on this
> > problem (below)
> > >>maybe? Do I need to provide more information or is everyone stumped?
> > >>
> > >>Cheers,
> > >>
> > >>--
> > >>+------------------------------------------+ \|||/
> > >>| Bruce at WebFarm.co.nz +64 06 7572881 | (o o)
> > >>| Systems Technician +---ooO-(_)-Ooo---+
> > >>| |
> > >>| WebFarm http://www.webfarm.co.nz |
> > >>| FreeParking http://www.freeparking.co.nz |
> > >>+------------------------------------------------------------+
> > >>
> > >>... FreeParking - NZ's best value Domain, WebHosting and
> > >>email accounts
> > >>- bar none
> > >>... WebFarm - NZ's eCommerce specialists since 1997
> > >>
> > >>
> > >>
> > >>
> > >>Bruce wrote:
> > >>
> > >>
> > >>
> > >>>Hi,
> > >>>
> > >>>Ive been searching through the archives at sourceforge and
> > >>>
> > >>>
> > >>cannot seem
> > >>
> > >>
> > >>>to find what my problem here maybe (I searched for event,
> > >>>
> > >>>
> > >>event handler,
> > >>
> > >>
> > >>>handler and scroll through 10 pages about nothing even related, I
> > >>>could have missed it though..) there was one that was
> > close but was
> > >>>for hosts... so ill post my one and see if anyone can spot
> > >>>
> > >>>
> > >>whats wrong.
> > >>
> > >>
> > >>>We have been using Nagios for around 12 months now, and its exelent
> > >>>except for it keeps reporting that FreshClam or ClamD are
> > >>>
> > >>>
> > >>not running on
> > >>
> > >>
> > >>>a server, which is alright we login and restart it
> > >>>
> > >>>
> > >>manually. But to save
> > >>
> > >>
> > >>>time I thought I might set up an event handler to call our
> > >>>
> > >>>
> > >>existing root
> > >>
> > >>
> > >>>access module for the servers to do it.
> > >>>Im trying to get it ruining from the host that monitors and not the
> > >>>reporting host to avoid bandwidth usage, but even though it
> > >>>
> > >>>
> > >>is defined
> > >>
> > >>
> > >>>and everything looks right it will NOT run. and the logs
> > >>>
> > >>>
> > >>say nothing.
> > >>
> > >>
> > >>>Here is my dumbing down of the configs to see if it was
> > >>>
> > >>>
> > >>something else
> > >>
> > >>
> > >>>(which is wasnt): (Yes, its running nagios 1.1)
> > >>>
> > >>>-- nagios.cfg event stuff --
> > >>>[root etc]# grep event nagios.cfg
> > >>>log_event_handlers=1
> > >>>#global_host_event_handler=somecommand
> > >>>#global_service_event_handler=somecommand
> > >>>event_handler_timeout=30
> > >>>enable_event_handlers=1
> > >>>--
> > >>>
> > >>>-- checkcommands.cfg --
> > >>>define command {
> > >>>command_name allserver_freshclam_fix
> > >>>command_line $USER1$/fix-w-allserver.sh
> > >>>
> > >>>
> > >>$HOSTADDRESS$
> > >>
> > >>
> > >>>restartFreshClam.sh $SERVICESTATE$ $SERVICEATTEMPT$
> > >>>}
> > >>>--
> > >>>
> > >>>-- hosts.cfg --
> > >>># Nagios hosts config
> > >>># Autogenerated file, do not modify
> > >>># Modify main etc hosts.cfg file
> > >>>#
> > >>># Generated: 13-Aug-2004 06 pm
> > >>>
> > >>>define host {
> > >>> use basic
> > >>> host_name creeper
> > >>> alias creeper
> > >>> address 192.168.56.32
> > >>>}
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>-------------------------------------------------------
> > >>SF.Net email is sponsored by Shop4tech.com-Lowest price on
> > Blank Media
> > >>100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> > >>Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> > >>http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> > >>_______________________________________________
> > >>Nagios-users mailing list
> > >>Nagios-users at lists.sourceforge.net
> > >>https://lists.sourceforge.net/lists/listinfo/nagios-users
> > >>::: Please include Nagios version, plugin version (-v) and OS
> > >>when reporting any issue.
> > >>::: Messages without supporting info will risk being sent
> > to /dev/null
> > >>
> > >>
> > >>
> > >
> > >
> > >-------------------------------------------------------
> > >SF.Net email is sponsored by Shop4tech.com-Lowest price on
> > Blank Media
> > >100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> > >Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> > >http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> > >_______________________________________________
> > >Nagios-users mailing list
> > >Nagios-users at lists.sourceforge.net
> > >https://lists.sourceforge.net/lists/listinfo/nagios-users
> > >::: Please include Nagios version, plugin version (-v) and
> > OS when reporting any issue.
> > >::: Messages without supporting info will risk being sent to
> > /dev/null
> > >
> > >
> >
> >
> >
> > -------------------------------------------------------
> > SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> > 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> > Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> > http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> > _______________________________________________
> > Nagios-users mailing list
> > Nagios-users at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nagios-users
> > ::: Please include Nagios version, plugin version (-v) and OS
> > when reporting any issue.
> > ::: Messages without supporting info will risk being sent to /dev/null
> >
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
> 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
> Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
> http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>
-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_idP47&alloc_id808&op=click
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null
More information about the Users
mailing list