<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi!<div><br></div><div>We've seen situations where this appears in the nagios.log:</div><div><br></div><div>Max concurrent service checks (50) has been reached. Delaying further checks until previous checks are complete...</div><div><br></div><div>When switching on debugging, what we noticed is that services are invoked all around the same time. I guess this happens when you have selected a host and say "force check all services on this host".</div><div><br></div><div>What happens is that in the event code (base/events.c), it seems that if this max_concurrent_checks is reached, then the service is ignored and is rescheduled with a next check time based on the next regular check interval. But if you do that, then all the other services will still be invoked around the same time.</div><div><br></div><div><br></div><div><div> /* reschedule the check if we can't run it now */</div><div> if(run_event==FALSE){</div><div> /* remove the service check from the event queue and reschedule it for a later time */</div><div> /* 12/20/05 since event was not executed, it needs to be remove()'ed to maintain sync with event broker modules */</div><div> temp_event=event_list_low;</div><div> remove_event(temp_event,&event_list_low,&event_list_low_tail);</div><div> if(temp_service->state_type==SOFT_STATE && temp_service->current_state!=STATE_OK)</div><div> temp_service->next_check=(time_t)(temp_service->next_check+(temp_service->retry_interval*interval_length));</div><div> else</div><div> temp_service->next_check=(time_t)(temp_service->next_check+(temp_service->check_interval*interval_length));</div><div> temp_event->run_time=temp_service->next_check;</div><div> reschedule_event(temp_event,&event_list_low,&event_list_low_tail);</div><div> update_service_status(temp_service,FALSE);</div><div> run_event=FALSE;</div><div> }</div><div><br></div></div><div>I propose that instead of setting next_time = next_time + check_interval, that there is a random factor added, maybe something like:</div><div><br></div><div><span class="Apple-style-span" style="font-family: Arial; ">next_time = now + max(5, min(int(rand(15)), int(rand(retry_interval*interval_length))))</span></div><div><br></div><div>This means that the next check has been moved at least 5 seconds away from now (to overcome the temporary load due to the number of concurrent service checks), with a maximum of 15 seconds away (or less if the retry_interval is lower).</div><div><br></div><div>Thoughts?</div><div><br></div><div>Ton</div><div><br></div></body></html>