<html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">Sorry for the link - <a class="moz-txt-link-freetext" href="http://stats.stackexchange.com/questions/38001/detecting-outliers-using-standard-deviations">http://stats.stackexchange.com/questions/38001/detecting-outliers-using-standard-deviations</a> The problem is not to write the code, the problem is to find a logic to determine which numbers to remove from the data set. What is a deviation from the normal difference in the set? Googling a bit more I found these definitions that may be applicable using stdev for your use case: <meta http-equiv="content-type" content="text/html; charset=utf-8"> Mean and Standard Deviation Method For this outlier detection method, the mean and standard deviation of the residuals are calculated and compared. If a value is a certain number of standard deviations away from the mean, that data point is identified as an outlier. The specified number of standard deviations is called the threshold. The default value is 3. This method can fail to detect outliers because the outliers increase the standard deviation. The more extreme the outlier, the more the standard deviation is affected. Median and Median Absolute Deviation Method (MAD) For this outlier detection method, the median of the residuals is calculated. Then, the difference is calculated between each historical value and this median. These differences are expressed as their absolute values, and a new median is calculated and multiplied by an empirically derived constant to yield the median absolute deviation (MAD). If a value is a certain number of MAD away from the median of the residuals, that value is classified as an outlier. The default threshold is 3 MAD. This method is generally more effective than the mean and standard deviation method for detecting outliers, but it can be too aggressive in classifying values that are not really extremely different. Also, if more than 50% of the data points have the same value, MAD is computed to be 0, so any value different from the residual median is classified as an outlier. Median and Interquartile Deviation Method (IQD) For this outlier detection method, the median of the residuals is calculated, along with the 25th percentile and the 75th percentile. The difference between the 25th and 75th percentile is the interquartile deviation (IQD). Then, the difference is calculated between each historical value and the residual median. If the historical value is a certain number of MAD away from the median of the residuals, that value is classified as an outlier. The default threshold is 2.22, which is equivalent to 3 standard deviations or MADs. This method is somewhat susceptible to influence from extreme outliers, but less so than the mean and standard deviation method. Box plots are based on this approach. The median and interquartile deviation method can be used for both symmetric and asymmetric data. If you find a method that you think could work, we could implement it together and you can verify it with your data. Can you say anything about the data collected? Anders On 12/17/2014 09:25 PM, Rahul Amaram wrote: </div> <blockquote cite="mid:5491E6AE.5000901@vizury.com" type="cite">Hi Andre, So, I would like to remove the outlier and calculate the mean for the remaining elements. Any suggestion apart from writing my own custom math function? Also, I don't think that you have shared the link. Thanks, Rahul. On Thursday 18 December 2014 12:55 AM, Anders Håål wrote: <blockquote type="cite">Hi Rahul, Its possible, but the question is what algorithm to use. The second question would also be what would you do with the remaining set, calculate a mean? When it comes to exclude a deviant value it sound close to what is called a outlier, <a class="moz-txt-link-freetext" href="http://en.wikipedia.org/wiki/Outlier">http://en.wikipedia.org/wiki/Outlier</a>. There are a number of mathematical solutions to this problem, but not sure which would be applicable or correct. Check this link for a discussions on the topic where one approach is using standard deviation, but from the discussion it does not sound like a statistical correct approach. If you or anyone else on this list find an good approach, I more then happy to try it. In Bischeck its possible to plug in your own functions as described in <a class="moz-txt-link-freetext" href="http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-6.2">http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-6.2</a> so you can easily do your own testing. Using the cache browser cli <a class="moz-txt-link-freetext" href="http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-4.4">http://www.bischeck.org/wp-content/uploads/2014/06/Bischeck_installation_and_administration_guide.html#toc-Section-4.4</a> you can easily test your function. Anders On 12/17/2014 03:40 PM, Rahul Amaram wrote: <blockquote type="cite">Hi, I had a quick question. Let us say we calculate the threshold based on the values of the past six days, one value per day. Now let us say, out of 6 values, one of these values is way too deviant. Then is it possible to exclude this deviant value from calculating the threshold? Thanks, Rahul. </blockquote> </blockquote> </blockquote> <pre class="moz-signature" cols="72">-- Ingby<a class="moz-txt-link-rfc2396E" href="http://www.ingby.com"><http://www.ingby.com></a> IngbyForge<a class="moz-txt-link-rfc2396E" href="http://gforge.ingby.com"><http://gforge.ingby.com></a> bischeck - dynamic and adaptive thresholds for Nagios <a class="moz-txt-link-rfc2396E" href="http://www.bischeck.org"><http://www.bischeck.org></a> <a class="moz-txt-link-abbreviated" href="mailto:anders.haal@ingby.com">anders.haal@ingby.com</a><a class="moz-txt-link-rfc2396E" href="mailto:anders.haal@ingby.com"><mailto:anders.haal@ingby.com></a> Mjukvara genom ingenjörsmässig kreativitet och kompetens Ingenjörsbyn Box 531 101 30 Stockholm Sweden <a class="moz-txt-link-abbreviated" href="http://www.ingby.com">www.ingby.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.ingby.com/"><http://www.ingby.com/></a> Mobil: +46 70 575 35 46 Tele: +46 75 75 75 090 Fax: +46 75 75 75 091 </pre> </body> </html>