<html><head><style type="text/css"></style></head><body><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><DIV>Thank you, Ethan for your response. </DIV> <DIV>The CGI reads status.dat. </DIV> <DIV> </DIV> <DIV>Here are some lines from one of the blocks:</DIV> <DIV>servicestatus {<BR> host_name=wtf3a<BR> service_description=check_ses<BR> modified_attributes=0<BR> check_command=check_ddnfaults_ses<BR> check_period=24x7<BR> notification_period=24x7<BR> check_interval=5.000000<BR> retry_interval=1.000000<BR> event_handler=<BR> has_been_checked=1<BR> should_be_scheduled=1<BR> check_execution_time=46912587.078<BR> check_latency=0.399<BR> check_type=0<BR> current_state=0<BR> last_hard_state=0<BR> last_event_id=87848<BR> current_event_id=87862<BR> current_problem_id=0<BR> last_problem_id=43437<BR> current_attempt=1<BR> max_attempts=3<BR> state_type=1<BR> last_state_change=1280725323<BR> last_hard_state_change=1280082023<BR> last_time_ok=1281047224<BR> last_time_warning=0<BR> last_time_unknown=1280725253<BR> last_time_critical=1280081723<BR> plugin_output=CHECK_DDN_ENCLOSURE OK - No errors were found.<BR>***<BR><BR>Please notice the check_execution time as more than a year in seconds. </DIV> <DIV> </DIV> <DIV>I don't see anything time-change related in the logs. After I filter out host/service alerts/notifications, nothing but auto-saves and start/stop information remain as follows: </DIV> <DIV> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">[1281028702] Auto-save of retention data completed successfully.<BR>[1281028706] Caught SIGTERM, shutting down...<BR>[1281028706] Successfully shutdown... (PID=6509)<BR>[1281028706] Event broker module '/usr/local/nagios/modules/dnxServer.so' deinitialized successfully.<BR>[1281028727] Nagios 3.2.1 starting... (PID=28820)<BR>[1281028727] Local time is Thu Aug 05 10:18:47 PDT 2010<BR>[1281028727] LOG VERSION: 2.0<BR>[1281028727] Event broker module '/usr/local/nagios/modules/dnxServer.so' initialized successfully.<BR>[1281028728] Finished daemonizing... (New PID=28821)<BR>[1281028763] EXTERNAL COMMAND: SCHEDULE_FORCED_SVC_CHECK;nagios06;check_app_java_cluster;1281028760<BR>[1281029029] Auto-save of retention data completed successfully.</DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">Here is the detail from the <A href="https://nagios06.internal.shutterfly.com/nagios/cgi-bin/extinfo.cgi?type=4">https://nagios06.internal.shutterfly.com/nagios/cgi-bin/extinfo.cgi?type=4</A><BR></DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> <TABLE border=1 cellSpacing=0 cellPadding=0> <TBODY> <TR> <TD class=stateInfoTable2> <TABLE border=0> <TBODY> <TR class=data> <TH class=data>Metric</TH> <TH class=data>Min.</TH> <TH class=data>Max.</TH> <TH class=data>Average</TH></TR> <TR> <TD class=dataVar>Check Execution Time: </TD> <TD class=dataVal>0.00 sec</TD> <TD class=dataVal>46912714.32 sec</TD> <TD class=dataVal>26955114.505 sec</TD></TR> <TR> <TD class=dataVar>Check Latency:</TD> <TD class=dataVal>0.00 sec</TD> <TD class=dataVal>3.40 sec</TD> <TD class=dataVal>0.277 sec</TD></TR> <TR> <TD class=dataVar>Percent State Change:</TD> <TD class=dataVal>0.00%</TD> <TD class=dataVal>37.43%</TD> <TD class=dataVal>0.33%</TD></TR></TBODY></TABLE></TD></TR></TBODY></TABLE></DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">If I stop Nagios and remove retention.dat and status.dat and restart fresh, Nagios looks normal for about 2 minutes and then reports the 1.5 year execution time. </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">Any idea on how to investigate and fix this bug? </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">Thank you!</DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">-Larry Findley</DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">Sr. Systems Engineer </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">Shutterfly</DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"><A href="mailto:lfindley@shutterfly.com">lfindley@shutterfly.com</A> </DIV> <DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt"> </DIV> <DIV style="FONT-FAMILY: arial, helvetica, sans-serif; FONT-SIZE: 13px"><FONT size=2 face=Tahoma> <HR SIZE=1> <B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Ethan Galstad <egalstad@nagios.org><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> Nagios Developers List <nagios-devel@lists.sourceforge.net><BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Wed, August 4, 2010 6:22:10 PM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: [Nagios-devel] Bug in Performance Data<BR></FONT><BR>Are there any message in the Nagios log file that relate to detected <BR>time changes?<BR><BR>The (stated) execution time for these checks is approx 542 days, which <BR>is strange. Most time issues would show just a few hours offset, not <BR>almost 2 years time.<BR><BR>What times are reflected in the status.dat file? Are you sure your CGI <BR>script is reading/processing the correct values from that file?<BR><BR><BR>- Ethan Galstad<BR><BR><BR><BR>Lawrence Findley wrote:<BR>> Hello Folks,<BR>> This info is from a cgi script that we use to show execution times. <BR>> These are obviously incorrect. None of the checks actually take more <BR>> than a few seconds to complete.<BR>> Any ideas? Thank you.<BR>> -Larry Findley<BR>> <BR>> <BR>> <BR>> <BR>> Wed Aug 4 17:00:01 PDT 2010<BR>> <BR>> <BR>> Top 10 Service Check Execution Times<BR>> <BR>> HOST SERVICE TIME<BR>> im477 check_rdf_content 46912687.968<BR>> vividpics104 check_lab_min_procs 46912684.576<BR>> vividpics158b check-win-mem 46912683.695<BR>> vividpics110e check-win-cpu 46912683.695<BR>> grf133 check_all_local_disk 46912683.695<BR>> vividpics147c check-win-mem 46912683.695<BR>> vividpics162e check-win-disk 46912683.695<BR>> vividpics156d check-win-disk 46912683.576<BR>> vividpics144e check-win-disk 46912683.576<BR>> vividpics165b check-win-mem 46912683.576<BR>> <BR>> <BR>> ------------------------------------------------------------------------<BR>> *From:* Lawrence Findley <<A href="mailto:larryfindley@yahoo.com" ymailto="mailto:larryfindley@yahoo.com">larryfindley@yahoo.com</A>><BR>> *To:* Nagios Developers List <<A href="mailto:nagios-devel@lists.sourceforge.net" ymailto="mailto:nagios-devel@lists.sourceforge.net">nagios-devel@lists.sourceforge.net</A>><BR>> *Sent:* Wed, August 4, 2010 4:35:20 PM<BR>> *Subject:* Re: [Nagios-devel] Bug in Performance Data<BR>> <BR>> Yes, Benny,<BR>> we run ntp to keep everything correct.<BR>> We use 4 satellites with DNX and I also verified that the time is <BR>> correct on all of the satellites too.<BR>> <BR>> Wed Aug 4 16:26:13 PDT 2010<BR>> nagios@nagios06 <mailto:nagios@nagios06> /usr/local/nagios/etc/objects $<BR>> The Nagios server is not a VM.<BR>> <BR>> Thank you for taking a look at this!<BR>> -Larry Findley<BR>> <BR>> Shutterfly<BR>> <BR>> <BR>> <BR>> <BR>> ------------------------------------------------------------------------<BR>> *From:* C. Bensend <<A href="mailto:benny@bennyvision.com" ymailto="mailto:benny@bennyvision.com">benny@bennyvision.com</A>><BR>> *To:* <A href="mailto:nagios-devel@lists.sourceforge.net" ymailto="mailto:nagios-devel@lists.sourceforge.net">nagios-devel@lists.sourceforge.net</A><BR>> *Sent:* Wed, August 4, 2010 12:15:27 PM<BR>> *Subject:* Re: [Nagios-devel] Bug in Performance Data<BR>> <BR>> <BR>> Is the time synced properly on your Nagios host?<BR>> <BR>> Is this a VM?<BR>> <BR>> Benny<BR>> <BR>> <BR>> > I found a bug where the performance stats do not reflect execution times<BR>> > accurately.<BR>> ><BR>> > version 3.2.1 shows execution times to be millions of seconds.<BR>> > Any ideas?<BR>> > Thank you.<BR>> > -Larry Findley<BR>> ><BR>> > Monitoring Performance<BR>> > Service Check Execution Time: 0.00 / 46912714.32 / 30610220.908 sec<BR>> > Service Check Latency: 0.00 / 3.40 / 0.242 sec<BR>> > Host Check Execution Time: 0.01 / 8.02 / 1.077 sec<BR>> > Host Check Latency: 0.00 / 1.13 / 0.403 sec<BR>> > # Active Host / Service Checks: 2159 / 18342<BR>> > # Passive Host / Service Checks: 0 / 0<BR>> ><BR>> ><BR>> > ________________________________<BR>> > From: Lawrence Findley <<A href="mailto:larryfindley@yahoo.com" ymailto="mailto:larryfindley@yahoo.com">larryfindley@yahoo.com</A> <BR>> <mailto:<A href="mailto:larryfindley@yahoo.com" ymailto="mailto:larryfindley@yahoo.com">larryfindley@yahoo.com</A>>><BR>> > To: Nagios Developers List <<A href="mailto:nagios-devel@lists.sourceforge.net" ymailto="mailto:nagios-devel@lists.sourceforge.net">nagios-devel@lists.sourceforge.net</A> <BR>> <mailto:<A href="mailto:nagios-devel@lists.sourceforge.net" ymailto="mailto:nagios-devel@lists.sourceforge.net">nagios-devel@lists.sourceforge.net</A>>><BR>> > Sent: Tue, August 3, 2010 12:42:12 PM<BR>> > Subject: [Nagios-devel] 1.5 year execution time?<BR>> ><BR>> ><BR>> > Here is the text:<BR>> ><BR>> ><BR>> > Monitoring Performance<BR>> > Service Check Execution Time: 0.00 / 46912714.32 / 30610220.908 sec<BR>> > Service Check Latency: 0.00 / 3.40 / 0.242 sec<BR>> > Host Check Execution Time: 0.01 / 8.02 / 1.077 sec<BR>> > Host Check Latency: 0.00 / 1.13 / 0.403 sec<BR>> > # Active Host / Service Checks: 2159 / 18342<BR>> > # Passive Host / Service Checks: 0 / 0<BR>> > I have tried removing status.dat and retention files. Within a few<BR>> > minutes, it<BR>> > goes back to these numbers.<BR>> ><BR>> > Anyone with idea?<BR>> > Thank you.<BR>> > -Larry Findley<BR>> ><BR>> ><BR>> > <BR>> ------------------------------------------------------------------------------<BR>> > The Palm PDK Hot Apps Program offers developers who use the<BR>> > Plug-In Development Kit to bring their C/C++ apps to Palm for a share<BR>> > of $1 Million in cash or HP Products. Visit us here for more details:<BR>> > <BR>> http://p.sf.net/sfu/dev2dev-palm_______________________________________________<BR>> > Nagios-devel mailing list<BR>> > <A href="mailto:Nagios-devel@lists.sourceforge.net" ymailto="mailto:Nagios-devel@lists.sourceforge.net">Nagios-devel@lists.sourceforge.net</A> <BR>> <mailto:<A href="mailto:Nagios-devel@lists.sourceforge.net" ymailto="mailto:Nagios-devel@lists.sourceforge.net">Nagios-devel@lists.sourceforge.net</A>><BR>> > <A href="https://lists.sourceforge.net/lists/listinfo/nagios-devel" target=_blank>https://lists.sourceforge.net/lists/listinfo/nagios-devel</A><BR>> ><BR>> <BR>> <BR>> -- <BR>> "Something's going on in this house - last night, I saw a face!"<BR>> "Did it have a nose?"<BR>> "Yes!"<BR>> "That sounds like a face all right."<BR>> -- Scary Movie 4<BR>> <BR><BR>------------------------------------------------------------------------------<BR>The Palm PDK Hot Apps Program offers developers who use the<BR>Plug-In Development Kit to bring their C/C++ apps to Palm for a share<BR>of $1 Million in cash or HP Products. Visit us here for more details:<BR>http://p.sf.net/sfu/dev2dev-palm<BR>_______________________________________________<BR>Nagios-devel mailing list<BR><A href="mailto:Nagios-devel@lists.sourceforge.net" ymailto="mailto:Nagios-devel@lists.sourceforge.net">Nagios-devel@lists.sourceforge.net</A><BR><A href="https://lists.sourceforge.net/lists/listinfo/nagios-devel" target=_blank>https://lists.sourceforge.net/lists/listinfo/nagios-devel</A><BR></DIV></div><br> </body></html>