<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v = "urn:schemas-microsoft-com:vml" xmlns:o = "urn:schemas-microsoft-com:office:office" xmlns:w = "urn:schemas-microsoft-com:office:word" xmlns:m = "http://schemas.microsoft.com/office/2004/12/omml"><HEAD> <META http-equiv=Content-Type content="text/html; charset=us-ascii"> <META content="MSHTML 6.00.2900.6036" name=GENERATOR> <STYLE>@font-face { font-family: SimSun; } @font-face { font-family: Mangal; } @font-face { font-family: Cambria Math; } @font-face { font-family: Calibri; } @font-face { font-family: Tahoma; } @font-face { font-family: @SimSun; } @page WordSection1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; } P.MsoNormal { FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif" } LI.MsoNormal { FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif" } DIV.MsoNormal { FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif" } A:link { COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99 } SPAN.MsoHyperlink { COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99 } A:visited { COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99 } SPAN.MsoHyperlinkFollowed { COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99 } P { FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "Times New Roman","serif"; mso-style-priority: 99; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto } SPAN.EmailStyle18 { COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal } SPAN.EmailStyle19 { COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply } .MsoChpDefault { FONT-SIZE: 10pt; mso-style-type: export-only } DIV.WordSection1 { page: WordSection1 } </STYLE> </HEAD> <BODY lang=EN-US vLink=purple link=blue> <DIV dir=ltr align=left>Hi there --</DIV> <DIV dir=ltr align=left> </DIV> <DIV dir=ltr align=left>The output shown below shows the top processes on the server:</DIV> <DIV dir=ltr align=left> </DIV> <DIV dir=ltr align=left>439 processes: 438 sleeping, 1 running, 0 zombie, 0 stopped CPU0 states: 19.0% user, 9.4% system, 0.0% nice, 71.0% idle CPU1 states: 20.1% user, 13.0% system, 0.0% nice, 66.3% idle CPU2 states: 27.1% user, 17.3% system, 0.0% nice, 55.0% idle Mem: 2064324K av, 2013820K used, 50504K free, 0K shrd, 487764K buff Swap: 2096472K av, 12436K used, 2084036K free 976244K cached</DIV> <DIV> </DIV> <DIV dir=ltr align=left> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 2398 root 15 0 1280 1280 824 R 1.9 0.0 0:00 top 5648 root 22 0 1196 1196 1104 S 1.3 0.0 0:00 ASMProServer 1 root 15 0 488 484 448 S 0.0 0.0 2:28 init 2 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU0 3 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU1 4 root 0K 0 0 0 0 SW 0.0 0.0 0:00 migration_CPU2 5 root 15 0 0 0 0 SW 0.0 0.0 0:03 keventd 6 root 34 19 0 0 0 SWN 0.0 0.0 17:52 ksoftirqd_CPU0 7 root 34 19 0 0 0 SWN 0.0 0.0 16:39 ksoftirqd_CPU1 8 root 34 19 0 0 0 SWN 0.0 0.0 17:33 ksoftirqd_CPU2 9 root 15 0 0 0 0 SW 0.0 0.0 28:22 kswapd 10 root 15 0 0 0 0 SW 0.0 0.0 42:39 bdflush 11 root 15 0 0 0 0 SW 0.0 0.0 3:08 kupdated 12 root 25 0 0 0 0 SW 0.0 0.0 0:00 mdrecoveryd 18 root 16 0 0 0 0 SW 0.0 0.0 0:00 scsi_eh_0 21 root 15 0 0 0 0 SW 0.0 0.0 4:38 kjournald 101 root 15 0 0 0 0 SW 0.0 0.0 0:00 khubd 265 root 15 0 0 0 0 SW 0.0 0.0 0:03 kjournald 266 root 15 0 0 0 0 SW 0.0 0.0 3:43 kjournald 267 root 15 0 0 0 0 SW 0.0 0.0 0:04 kjournald 268 root 15 0 0 0 0 SW 0.0 0.0 0:01 kjournald 269 root 15 0 0 0 0 SW 0.0 0.0 0:11 kjournald 270 root 15 0 0 0 0 SW 0.0 0.0 4:34 kjournald 271 root 15 0 0 0 0 SW 0.0 0.0 4:28 kjournald 272 root 15 0 0 0 0 SW 0.0 0.0 0:08 kjournald 273 root 15 0 0 0 0 SW 0.0 0.0 0:14 kjournald 274 root 15 0 0 0 0 SW 0.0 0.0 0:07 kjournald 275 root 15 0 0 0 0 SW 0.0 0.0 1:14 kjournald 805 root 15 0 588 576 532 S 0.0 0.0 1:39 syslogd 810 root 15 0 448 432 432 S 0.0 0.0 0:00 klogd 830 rpc 15 0 596 572 508 S 0.0 0.0 0:04 portmap 858 rpcuser 19 0 708 608 608 S 0.0 0.0 0:00 rpc.statd 970 root 15 0 0 0 0 SW 0.0 0.0 0:21 rpciod 971 root 15 0 0 0 0 SW 0.0 0.0 0:00 lockd 999 ntp 15 0 1812 1812 1732 S 0.0 0.0 5:04 ntpd 1022 root 15 0 772 720 632 S 0.0 0.0 0:00 ypbind 1024 root 15 0 772 720 632 S 0.0 0.0 1:16 ypbind</DIV> <DIV> </DIV> <DIV>What caught my eye was the number of processes along with the number of sleeping processes.</DIV> <DIV>I tried running the kill command on the kjournald instances, but that did not appear to stop them.</DIV> <DIV> </DIV> <DIV>Aside from rebooting the server, which can be done if necessary, what other approach can I try?</DIV> <DIV> </DIV> <DIV> </DIV> <DIV dir=ltr align=left> </DIV> <DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left> <HR tabIndex=-1> From: Daniel Wittenberg [mailto:daniel.wittenberg.r0ko@statefarm.com] Sent: Tuesday, December 07, 2010 9:11 AM To: Nagios Users List Subject: Re: [Nagios-users] Determining what is causing a highloadreportedby check_load plugin </DIV> <DIV></DIV> <DIV class=WordSection1> So what are the first few processes listed in top? That should be what is causing your load then.<o:p></o:p> <o:p> </o:p> Dan<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <DIV> <DIV style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0in; PADDING-BOTTOM: 0in; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none"> From: Kaplan, Andrew H. [mailto:AHKAPLAN@PARTNERS.ORG] Sent: Tuesday, December 07, 2010 7:49 AM To: Nagios Users List Subject: Re: [Nagios-users] Determining what is causing a high loadreportedby check_load plugin<o:p></o:p></DIV></DIV> <o:p> </o:p> Hi there --<o:p></o:p> <o:p></o:p> The load values that are displayed in top match those for the check_load plugin. This is the case whether the plugin<o:p></o:p> is run either automatically or interactively. The output for the uptime command is shown below:<o:p></o:p> <o:p></o:p> <DIV> 8:48am up 153 days, 23:21, 1 user, load average: 73.36, 73.29, 73.21<o:p></o:p></DIV> <DIV> <o:p></o:p></DIV> <DIV> <o:p></o:p></DIV> <o:p></o:p> <o:p> </o:p> <DIV class=MsoNormal style="TEXT-ALIGN: center" align=center> <HR align=center width="100%" SIZE=2> </DIV> From: Daniel Wittenberg [mailto:daniel.wittenberg.r0ko@statefarm.com] Sent: Monday, December 06, 2010 4:40 PM To: Nagios Users List Subject: Re: [Nagios-users] Determining what is causing a high load reportedby check_load plugin<o:p></o:p> In top, does it show the same load values? The status of your memory shouldn’t cause the nagios plugin to report high cpu. What does the uptime command say? Try running the check_load script by hand on that host and verify it returns the same results.<o:p></o:p> Dan<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <DIV style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0in; PADDING-BOTTOM: 0in; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none"> From: Marc Powell [mailto:lists@xodus.org] Sent: Monday, December 06, 2010 3:26 PM To: Nagios Users List Subject: Re: [Nagios-users] Determining what is causing a high load reported by check_load plugin<o:p></o:p></DIV> <o:p> </o:p> <o:p> </o:p> <DIV> On Mon, Dec 6, 2010 at 1:50 PM, Kaplan, Andrew H. <<A href="mailto:AHKAPLAN@partners.org">AHKAPLAN@partners.org</A>> wrote:<o:p></o:p> <DIV> Hi there -- <o:p></o:p> We are running Nagios 3.1.2 server, and the client that is the subject of this e-mail is running version 2.6 of the nrpe client.<o:p></o:p> The check_load plugin, version 1.4, is indicating the past three readings are the following: <o:p></o:p> load average: 71.00, 71.00, 70.95 CRITICAL <o:p></o:p> The critical threshold of the plugin has been set to the 30, 25, 20 settings. <o:p></o:p> When I checked the client in question, the first thing I did was to run the top command. The results are shown below: <o:p></o:p> CPU0 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle CPU1 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle CPU2 states: 1.0% user, 4.0% system, 0.0% nice, 93.0% idle Mem: 2064324K av, 2032308K used, 32016K free, 0K shrd, 509924K buff Swap: 2096472K av, 21432K used, 2075040K free 1035592K cached <o:p></o:p> The one thing that I noticed was the amount of free memory was at thirty-two megabytes. I wanted to know if that was what was causing the critical status to occur, or if there is something(s) else that I should investigate.<o:p></o:p></DIV> <DIV> Memory is not a factor in the load calculation, only the number of processes running or waiting to run. For at least 15 minutes you had approximately 71 processes either running or ready to run and waiting on CPU resources. Running top/ps was the right thing to do but you really need to do it when the problem is occurring to see what's actually using all the CPU resources. There are far too many reasons why load could be high but it should be easy for someone familiar with your system to figure it out (at least generally) while in-the-act. -- Marc<o:p></o:p></DIV></DIV> The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.<o:p></o:p></DIV></BODY></HTML>