High latency on small installation

Assaf Flatto nagios at flatto.net
Tue Jul 20 18:36:09 CEST 2010


Hello All


I am having a problem with very high latency on my main nagios server 
(3.2.0 from source on SLES 10.3 x64).
I recompiled the core with the  embedded perl and that helped for a 
while to lower the latency but it keeps growing to times that are not 
reasonable for this size of a nagios installation .


The setup of this server is as the main nagios with two other server 
sending passive results to it , one server's time interval is 40 sec 
(doing mainly http queries ) and the other has a 1 minute interval , and 
monitors Switches ,firewalls and hosts.

The machine is a dual CPU Quad core , 6 GB ram and the load is about 3.3 .

I've attached the output of nagiostats below and the directives of 
nagios.cfg .

moving the result/ stats  to a ramfs is not an option at this time ( i 
know about this , but sadly not a viable option).

Thanks


CURRENT STATUS DATA
------------------------------------------------------
Status File:                            /usr/local/nagios/var/status.dat
Status File Age:                        0d 0h 0m 30s
Status File Version:                    3.2.0

Program Running Time:                   0d 1h 46m 11s
Nagios PID:                             27584
Used/High/Total Command Buffers:        8 / 141 / 4096

Total Services:                         2687
Services Checked:                       2687
Services Scheduled:                     2638
Services Actively Checked:              2638
Services Passively Checked:             49
Total Service State Change:             0.000 / 56.710 / 0.071 %
Active Service Latency:                 376.745 / 438.212 / 413.404 sec
Active Service Execution Time:          0.040 / 26.399 / 0.439 sec
Active Service State Change:            0.000 / 56.710 / 0.072 %
Active Services Last 1/5/15/60 min:     0 / 1088 / 2627 / 2638
Passive Service Latency:                9.664 / 47.661 / 30.084 sec
Passive Service State Change:           0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min:    10 / 49 / 49 / 49
Services Ok/Warn/Unk/Crit:              2673 / 11 / 1 / 2
Services Flapping:                      4
Services In Downtime:                   0

Total Hosts:                            244
Hosts Checked:                          244
Hosts Scheduled:                        219
Hosts Actively Checked:                 232
Host Passively Checked:                 12
Total Host State Change:                0.000 / 0.000 / 0.000 %
Active Host Latency:                    0.018 / 435.890 / 400.684 sec
Active Host Execution Time:             0.041 / 4.136 / 3.944 sec
Active Host State Change:               0.000 / 0.000 / 0.000 %
Active Hosts Last 1/5/15/60 min:        0 / 96 / 219 / 219
Passive Host Latency:                   0.184 / 67.224 / 18.642 sec
Passive Host State Change:              0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min:       0 / 0 / 0 / 0
Hosts Up/Down/Unreach:                  244 / 0 / 0
Hosts Flapping:                         0
Hosts In Downtime:                      0

Active Host Checks Last 1/5/15 min:     21 / 129 / 357
   Scheduled:                           20 / 124 / 334
   On-demand:                           1 / 5 / 23
   Parallel:                            20 / 124 / 335
   Serial:                              0 / 0 / 0
   Cached:                              1 / 5 / 22
Passive Host Checks Last 1/5/15 min:    0 / 0 / 0
Active Service Checks Last 1/5/15 min:  297 / 1481 / 4462
   Scheduled:                           297 / 1481 / 4462
   On-demand:                           0 / 0 / 0
   Cached:                              0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 21 / 37 / 37

External Commands Last 1/5/15 min:      77 / 350 / 1030


log_file=/usr/local/nagios/var/nagios.log
cfg_dir=/usr/local/nagios/etc/objects
cfg_dir=/usr/local/nagios/etc/LFI
cfg_dir=/usr/local/nagios/etc/GB
cfg_dir=/usr/local/nagios/etc/DE
cfg_dir=/usr/local/nagios/etc/SE
cfg_dir=/usr/local/nagios/etc/NOR-CLOUD
cfg_dir=/usr/local/nagios/etc/network
cfg_dir=/usr/local/nagios/etc/Leaf
cfg_dir=/usr/local/nagios/etc/sites
object_cache_file=/usr/local/nagios/var/objects.cache
precached_object_file=/usr/local/nagios/var/objects.precache
resource_file=/usr/local/nagios/etc/resource.cfg
status_file=/usr/local/nagios/var/status.dat
status_update_interval=10
nagios_user=nagios
nagios_group=nagios
check_external_commands=1
command_check_interval=-1
command_file=/var/spool/nagios/nagios.cmd
external_command_buffer_slots=4096
lock_file=/var/run/nagios/nagios.lock
temp_file=/usr/local/nagios/nagios.tmp
temp_path=/tmp
event_broker_options=-1
broker_module=/usr/local/nagios/bin/ndomod-3x.o 
config_file=/usr/local/nagios/etc/ndomod.cfg
log_rotation_method=d
log_archive_path=/usr/local/nagios/var/archives
use_syslog=0
log_notifications=1
log_service_retries=1
log_host_retries=1
log_event_handlers=1
log_initial_states=0
log_passive_checks=1
service_inter_check_delay_method=s
max_service_check_spread=30
service_interleave_factor=s
host_inter_check_delay_method=s
max_host_check_spread=30
max_concurrent_checks=0
check_result_reaper_frequency=5
max_check_result_reaper_time=30
check_result_path=/var/spool/nagios
max_check_result_file_age=3600
cached_host_check_horizon=15
cached_service_check_horizon=15
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
soft_state_dependencies=0
sleep_time=0.25
service_check_timeout=60
host_check_timeout=30
event_handler_timeout=60
notification_timeout=30
ocsp_timeout=5
perfdata_timeout=5
retain_state_information=1
state_retention_file=/usr/local/nagios/lib/retention.dat
retention_update_interval=60
use_retained_program_state=1
use_retained_scheduling_info=1
retained_host_attribute_mask=0
retained_service_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
interval_length=60
use_aggressive_host_checking=0
execute_service_checks=1
accept_passive_service_checks=1
execute_host_checks=1
accept_passive_host_checks=1
enable_notifications=1
enable_event_handlers=1
process_performance_data=1
host_perfdata_command=process-host-perfdata
service_perfdata_command=process-service-perfdata
obsess_over_services=0
obsess_over_hosts=0
translate_passive_host_checks=1
passive_host_checks_are_soft=0
check_for_orphaned_services=1
check_for_orphaned_hosts=1
check_service_freshness=1
service_freshness_check_interval=60
check_host_freshness=1
host_freshness_check_interval=120
additional_freshness_latency=10
enable_flap_detection=1
low_service_flap_threshold=5.0
high_service_flap_threshold=20.0
low_host_flap_threshold=5.0
high_host_flap_threshold=20.0
date_format=euro
p1_file=/usr/lib/nagios/p1.pl
enable_embedded_perl=1
use_embedded_perl_implicitly=1
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
illegal_macro_output_chars=`~$&|'"<>
use_regexp_matching=0
use_true_regexp_matching=0
admin_email=nagios at localhost
admin_pager=pagenagios at localhost
daemon_dumps_core=0
use_large_installation_tweaks=1
enable_environment_macros=1
free_child_process_memory=1
debug_level=0
debug_verbosity=0
debug_file=/usr/local/nagios/var/nagios.debug
max_debug_file_size=1000000

-- 
Never,Ever Cut A Deal With a Dragon 




------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null





More information about the Users mailing list