<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> Okay and keep us updated with your findings. <div class="moz-cite-prefix">On 08/09/2017 05:10 PM, Francesco Toffoli wrote: </div> <blockquote type="cite" cite="mid:7ilg5qgandtn8mtacf8eik3r.1502288419593@email.android.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <div>Hi Anders,</div> <div>I modified the log configuration as you suggested, but after the bischeckd daemon stop and start i didn't notice any particular warning or critical messages. So i decided to start waiting for a crash and then to proceed with the logs analisys. I'll keep you updated</div> <div>.Thanks</div> <div> </div> <div> </div> <div id="composer_signature"> <div style="font-size:85%;color:#575757" dir="auto">Inviato da smartphone Samsung Galaxy.</div> </div> <div> </div> <div style="font-size:100%;color:#000000"> <div>-------- Messaggio originale --------</div> <div>Da: Anders Håål <a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com"><anders.haal@ingby.com></a> </div> <div>Data: 09/08/17 08:16 (GMT+01:00) </div> <div>A: <a class="moz-txt-link-abbreviated" href="mailto:bischeck-users@monitoring-lists.org">bischeck-users@monitoring-lists.org</a> </div> <div>Oggetto: Re: bischeck suddenly stops working </div> <div> </div> </div> Francesco - any progress on the issue? <div class="moz-cite-prefix">On 07/26/2017 05:52 PM, Anders Håål wrote: </div> <blockquote type="cite" cite="mid:fd8d8556-683a-c18b-4d58-111afb2d0e4b@ingby.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> Thanks for the feedback. When bischeck "stop working" it would be interesting to understand if anything gets logged after it "stops" and also what is logged when you do a restart - but I suggest you do a stop and see what is logged before starting. I would suggest that you change the log level in logback.xml for all packages <root level="INFO"> <appender-ref ref="bischeck"/> </root> To avoid duplicates you should also add the additivity="false" on the other logger. Based on the standard logback.xml you can test this in your test environment first, have not tested it my self, and if it looks good deploy in in production according to your specific customization of paths, etc. logback.xml: <?xml version="1.0" encoding="UTF-8"?> <configuration> <jmxConfigurator /> <appender name="bischeck" class="ch.qos.logback.core.rolling.RollingFileAppender">  <File>/var/tmp/bischeck.log</File> <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS,Europe/Stockholm} ; %p ; %t ; %c ; %m%ex%n</pattern> </encoder> <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy"> <maxIndex>3</maxIndex> <FileNamePattern>/var/tmp/bischeck.log.%i</FileNamePattern> </rollingPolicy> <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> <MaxFileSize>1000KB</MaxFileSize> </triggeringPolicy> </appender> <logger name="com.ingby" level="INFO" additivity="false"> <appender-ref ref="bischeck"/> </logger> <logger name="com.ingby.socbox.bischeck.configuration.CachePurgeJob" level="DEBUG" additivity="false"> <appender-ref ref="bischeck"/> </logger> <logger name="com.ingby.socbox.bischeck.cache.provider.redis" level="DEBUG" additivity="false"> <appender-ref ref="bischeck"/> </logger> <logger name="org.quartz" level="INFO" additivity="false"> <appender-ref ref="bischeck"/> </logger> <root level="WARN"> <appender-ref ref="bischeck"/> </root> </configuration> The root section will secure that everything from any java packages with WARN or ERROR is logged to the bischeck appender. Regards Anders <div class="moz-cite-prefix">On 07/25/2017 09:55 AM, Francesco Giuseppe Toffoli wrote: </div> <blockquote type="cite" cite="mid:9f1a3b07-f520-e457-07af-f7b44ee42026@skylogic.it"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> Hi Anders, thanks for your reply. I'll answer you to the variuos questions: (1) the java version is: openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) and has not been updated recently. In our test environment, (where the problem does not occur), the version is nearly the same (1.8.0_121). The OS has not been updated, (CentOS release 6.6). (2) Redis has not been uptaded recently, (redis 2.8.23). At the moment we have more or less 13.000 keys used. (3) We usually add checks, maybe weekly. The issue started to occur some months ago, but it could happen that for 2 or 3 weeks everything is ok, then we have several crashes in a week. I'm not so inclined to give the guilt to some new checks, also because the testing server is aligned to the production one. (5) Yes, the restart is done via '/etc/init.d/bischeckd restart' and it solves the issue. Physical memory on the server is always OK, i don't think to a jvm out of memory. In the Bischeck logs i didn't notice any error. However, at the next crash i'll try have a deeper look at them. Could i have a look at some other logs maybe? Thanks, Francesco <div class="moz-cite-prefix">Il 24/07/2017 21:57, Anders Håål ha scritto: </div> <blockquote type="cite" cite="mid:b685ee8b-e1ce-d943-b6e8-88545cdc8fca@ingby.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> Hi Giuseppe, Sounds strange that it just stopped working after along time of stability if not something has change: - Anything change on the server you run bischeck on - OS, jdk version, ...... - Update redis version? Change in configuration? - Added any new bischeck check or changed something in the configuration? - Anything else you can think about that may have change? When you say restarting is it the normal /etc/init.d/bischeckd restart that fix the problem? The reason I ask is that the script just do a kill with TERM signal. If the jvm would be in a out of memory situation it may not be enough, but you should have seen that in the log I guess. Sure you do not have any ERROR or WARN entries in the log. /Anders <div class="moz-cite-prefix">On 07/24/2017 02:14 PM, Francesco Giuseppe Toffoli wrote: </div> <blockquote type="cite" cite="mid:95f4a4eb-1319-f8cb-acd0-1832252656d4@skylogic.it"> <meta http-equiv="content-type" content="text/html; charset=utf-8"> Hi, we are experiencing a critical problem with Bischeck. It's a couple of months it sometimes suddenly stops working: the daemon /etc/init.d/bicheckd is running but no check results are sent to Nagios. Restarting bischeck daemon fixes the issue. Unfortunately we can't find any clue about the root cause on bischeck logs, not even with DEBUG logging level enabled. Redis database seems working properly and no increasing of memory/cpu usage are reported on the server hosting bischeck while the issue occurs. Do you have any suggestion on how to deeply investigate this? Regards, Francesco <div class="moz-signature">-- <meta http-equiv="content-type" content="text/html; charset=utf-8"> <title></title> Francesco Giuseppe Toffoli Monitoring Engineer GSE Department Tel: +39 01127387488<o:p></o:p> Mobile: +39 349.800.60.35 Email: <a href="mailto:ftoffoli@skylogic.it" moz-do-not-send="true">ftoffoli@skylogic.it</a> <img shrinktofit="true" id="_x0000_i1025" src="imap://aha%40ingby%2Ecom@mail.messagingengine.com:993/fetch%3EUID%3E.INBOX.Drafts%3E5624?part=&filename=/home/franz/skylogic_logo.jpeg" moz-do-not-send="true" border="0" width="190" height="44"> Skylogic S. p. A. Strada Pianezza, 289 10151 Torino, Italy </div> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through this latter, any of the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate. </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com" moz-do-not-send="true"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org" moz-do-not-send="true"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com" moz-do-not-send="true">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com" moz-do-not-send="true"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com" moz-do-not-send="true">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/" moz-do-not-send="true"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </blockquote> <div class="moz-signature">-- <meta http-equiv="content-type" content="text/html; charset=utf-8"> <title></title> Francesco Giuseppe Toffoli Monitoring Engineer GSE Department Tel: +39 01127387488<o:p></o:p> Mobile: +39 349.800.60.35 Email: <a href="mailto:ftoffoli@skylogic.it" moz-do-not-send="true">ftoffoli@skylogic.it</a> <img shrinktofit="true" id="_x0000_i1025" src="imap://aha%40ingby%2Ecom@mail.messagingengine.com:993/fetch%3EUID%3E.INBOX.Drafts%3E5624?part=&filename=/home/franz/skylogic_logo.jpeg" moz-do-not-send="true" border="0" width="190" height="44"> Skylogic S. p. A. Strada Pianezza, 289 10151 Torino, Italy </div> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through this latter, any of the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate. </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com" moz-do-not-send="true"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org" moz-do-not-send="true"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com" moz-do-not-send="true">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com" moz-do-not-send="true"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com" moz-do-not-send="true">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/" moz-do-not-send="true"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com" moz-do-not-send="true"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org" moz-do-not-send="true"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com" moz-do-not-send="true">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com" moz-do-not-send="true"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com" moz-do-not-send="true">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/" moz-do-not-send="true"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. No employee or agent is authorized to conclude any binding agreement on behalf of this Company nor, through this latter, any of the Eutelsat Communication group with another party by email without express written confirmation by a duly authorized officer of the Company. The list of duly authorized officers and the scope of their powers is published on the Trade Register according to the national law of each affiliate. </blockquote> <pre class="moz-signature" cols="72">-- Ingby <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com"><http://www.ingby.com></a> bischeck - dynamic and adaptive monitoring for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </body> </html>