<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 11 (filtered medium)"> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"/>  <style>  </style> </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> I have a problem with my Nagios server constantly crashing. It keeps outputting on the screen Out of Memory errors which causes loss of access to the server. I can ping the box but I cannot SSH or web into it to view any information. This has been happening increasingly more lately. Now it is about every 2-3 days that this is occurring. We have been adding more and more devices to the servers and this problem has been increasing as this occurs. This is how I have it set up.<o:p></o:p> <o:p> </o:p> I have a Main Nagios server that is running the latest 2.0 (stable) Nagios release. It is monitoring about 6800 devices but it is not actively checking the devices. Its main role is to provide a web interface and receive passive polls from three other servers which do the polling. The main server also does email notifications when a device goes down. The server sends about 30-40 emails a day. I am using NSCA 2.5 between the server and the client Nagios servers. I am only monitoring one service for each device which is either TCP or ping depending on the device. Mostly all devices are monitored with TCP (roughly 6000). The rest are monitored with ping. The individual servers are pretty evenly spread with the number of devices. They are about 2000-2500 each.<o:p></o:p> <o:p> </o:p> All the servers are just basic computers, Dell Dimension 2400s with base hardware. The main server was upgraded to 2GB RAM while the other servers are running 512MB each. They are all running Celeron 2.4 GHz processors. The individual servers are not having out of memory problems and they are running the latest 2.0 (stable) release as well. They all run RedHat 9.0 with everything installed for the packages.<o:p></o:p> <o:p> </o:p> Can someone please help me in resolving this problem? Thanks.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> The TOP process does not appear like it is running out of memory. This is the normal output when the server has been running for a few hours.<o:p></o:p> 57 processes: 54 sleeping, 3 running, 0 zombie, 0 stopped<o:p></o:p> CPU states: 41.1% user 58.8% system 0.0% nice 0.0% iowait 0.0% idle<o:p></o:p> Mem: 2063556k av, 285940k used, 1777616k free, 0k shrd, 41056k buff<o:p></o:p> 177644k actv, 51688k in_d, 10892k in_c<o:p></o:p> Swap: 1044184k av, 0k used, 1044184k free 114208k cached<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Here is a sample configuration that I have on the devices on the main server:<o:p></o:p> <o:p> </o:p> hosts.cfg<o:p></o:p> define host {<o:p></o:p> name generic-host ; The name of this host template - referenced in other host definitions, used for template recursion/resolution<o:p></o:p> notifications_enabled 1 ; Host notifications are enabled<o:p></o:p> event_handler_enabled 0 ; Host event handler is enabled<o:p></o:p> flap_detection_enabled 1 ; Flap detection is enabled<o:p></o:p> process_perf_data 1 ; Process performance data<o:p></o:p> retain_status_information 1 ; Retain status information across program restarts<o:p></o:p> retain_nonstatus_information 1 ; Retain non-status information across program restarts<o:p></o:p> max_check_attempts 10<o:p></o:p> notification_interval 720<o:p></o:p> notification_period 24x7<o:p></o:p> obsess_over_host 0<o:p></o:p> notification_options d,u,r,f<o:p></o:p> register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!<o:p></o:p> }<o:p></o:p> define host {<o:p></o:p> use generic-host ; Name of host template to use<o:p></o:p> host_name DETAH-R1<o:p></o:p> alias DETAH-R1<o:p></o:p> address x.x.x.x<o:p></o:p> check_command check_ping!200,40%!10000,100%<o:p></o:p> contact_groups device-admins,DETAH-admins,router-admins<o:p></o:p> }<o:p></o:p> <o:p> </o:p> services.cfg<o:p></o:p> define service {<o:p></o:p> name generic-service ; The 'name' of this service template, referenced in other service definitions<o:p></o:p> active_checks_enabled 0 ; Active service checks are enabled<o:p></o:p> passive_checks_enabled 1 ; Passive service checks are enabled/accepted<o:p></o:p> parallelize_check 1 ; Active service checks should be parallelized (disabling this can lead to major performance problems)<o:p></o:p> obsess_over_service 0 ; We should obsess over this service (if necessary)<o:p></o:p> check_freshness 1 ; Default is to NOT check service 'freshness'<o:p></o:p> freshness_threshold 1800<o:p></o:p> notifications_enabled 1 ; Service notifications are enabled<o:p></o:p> event_handler_enabled 0 ; Service event handler is enabled<o:p></o:p> flap_detection_enabled 1 ; Flap detection is enabled<o:p></o:p> process_perf_data 1 ; Process performance data<o:p></o:p> retain_status_information 1 ; Retain status information across program restarts<o:p></o:p> retain_nonstatus_information 1 ; Retain non-status information across program restarts<o:p></o:p> is_volatile 0<o:p></o:p> check_period 24x7<o:p></o:p> max_check_attempts 6<o:p></o:p> normal_check_interval 20<o:p></o:p> retry_check_interval 5<o:p></o:p> notification_interval 720<o:p></o:p> notification_period 24x7<o:p></o:p> notification_options n<o:p></o:p> register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!<o:p></o:p> }<o:p></o:p> define service {<o:p></o:p> use generic-service ; Name of service template to use<o:p></o:p> host_name DETAH-R1<o:p></o:p> service_description <st1:place w:st="on">PING</st1:place><o:p></o:p> contact_groups device-admins,DETAH-admins,router-admins<o:p></o:p> check_command check_ping!200,40%!1000,100%<o:p></o:p> }<o:p></o:p> <o:p> </o:p> Here is a sample config on the individual server.<o:p></o:p> <o:p> </o:p> hosts.cfg<o:p></o:p> define host {<o:p></o:p> name generic-host ; The name of this host template - referenced in other host definitions, used for template recursion/resolution<o:p></o:p> notifications_enabled 1 ; Host notifications are enabled<o:p></o:p> event_handler_enabled 0 ; Host event handler is enabled<o:p></o:p> flap_detection_enabled 1 ; Flap detection is enabled<o:p></o:p> process_perf_data 1 ; Process performance data<o:p></o:p> retain_status_information 1 ; Retain status information across program restarts<o:p></o:p> retain_nonstatus_information 1 ; Retain non-status information across program restarts<o:p></o:p> max_check_attempts 10<o:p></o:p> notification_interval 720<o:p></o:p> notification_period 24x7<o:p></o:p> obsess_over_host 0<o:p></o:p> notification_options d,u,r,f<o:p></o:p> register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!<o:p></o:p> }<o:p></o:p> define host {<o:p></o:p> use generic-host ; Name of host template to use<o:p></o:p> host_name DETAH-R1<o:p></o:p> alias DETAH-R1<o:p></o:p> address x.x.x.x<o:p></o:p> check_command check_ping!200,40%!10000,100%<o:p></o:p> contact_groups device-admins,DETAH-admins,router-admins<o:p></o:p> }<o:p></o:p> <o:p> </o:p> services.cfg<o:p></o:p> define service {<o:p></o:p> name generic-service ; The 'name' of this service template, referenced in other service definitions<o:p></o:p> active_checks_enabled 1 ; Active service checks are enabled<o:p></o:p> passive_checks_enabled 1 ; Passive service checks are enabled/accepted<o:p></o:p> parallelize_check 1 ; Active service checks should be parallelized (disabling this can lead to major performance problems)<o:p></o:p> obsess_over_service 1 ; We should obsess over this service (if necessary)<o:p></o:p> check_freshness 1 ; Default is to NOT check service 'freshness'<o:p></o:p> freshness_threshold 1800<o:p></o:p> notifications_enabled 1 ; Service notifications are enabled<o:p></o:p> event_handler_enabled 0 ; Service event handler is enabled<o:p></o:p> flap_detection_enabled 1 ; Flap detection is enabled<o:p></o:p> process_perf_data 1 ; Process performance data<o:p></o:p> retain_status_information 1 ; Retain status information across program restarts<o:p></o:p> retain_nonstatus_information 1 ; Retain non-status information across program restarts<o:p></o:p> is_volatile 0<o:p></o:p> check_period 24x7<o:p></o:p> max_check_attempts 6<o:p></o:p> normal_check_interval 20<o:p></o:p> retry_check_interval 5<o:p></o:p> notification_interval 720<o:p></o:p> notification_period 24x7<o:p></o:p> notification_options n<o:p></o:p> register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!<o:p></o:p> }<o:p></o:p> define service {<o:p></o:p> use generic-service ; Name of service template to use<o:p></o:p> host_name DETAH-R1<o:p></o:p> service_description <st1:place w:st="on">PING</st1:place><o:p></o:p> contact_groups device-admins,DETAH-admins,router-admins<o:p></o:p> check_command check_ping!200,40%!1000,100%<o:p></o:p> }<o:p></o:p> <o:p> </o:p> Raffy<o:p></o:p> <o:p> </o:p> </div> </body> </html>