redis memory increasing linearly

[Anders Håål]

Great to here that you solved it. I was pretty sure that redis was not 
your problem, and glad that it was not bischeck. :)

/Anders


On 12/04/2017 12:26 PM, Francesco Giuseppe Toffoli wrote:
>
> Hi Handers,
> thanks for your reply, we found out the problem.
> We had a list on which we did an update on a value to reset a spike in 
> the value itself. Due to a syntax mistake the LSET command set the 
> field in a non standard format, (in the bischeck log we have found an 
> error stating the impossibility to perform the purge operation).
> For several lists (on which the max size is 1000 points) we had a much 
> higher number of points.
>
> We solved purging manually the problematic list with the LTRIM command.
> Thanks for your ime,
>
> Regards,
> Francesco
>
>
> Il 04/12/2017 11:24, Anders Håål ha scritto:
>>
>> It was not clear in your answer, but have you restarted bischeck? You 
>> say you have restarted redis, but what about bischeck?
>>
>> The redis config you restarted with, what did you configure differently?
>>
>> Anders
>>
>>
>> On 12/04/2017 11:12 AM, Francesco Giuseppe Toffoli wrote:
>>> Il 04/12/2017 10:43, Anders Håål ha scritto:
>>>>
>>>> Hi Francesco,
>>>> Redis memory will increase with the number of data points you 
>>>> collect and how long time you configured to keep them. So what was 
>>>> the memory before it started to grow? Was it steady on this level? 
>>>> How much is it growing? And not using the redis instance for 
>>>> anything else? (redis-cli info)
>>>>
>>>
>>> Hi Anders, thanks for your fast and really appreciated reply. The 
>>> memory has also been stable, around 8/9% of total server available 
>>> RAM. Now is around 61% and it grows more or less of +4% a day. So to 
>>> be optimist we have a week before everything will crash :-(
>>> Redis is used only by bischeck, except for a couple of script 
>>> reading datas from it (but are very old and light scripts, so I 
>>> exclude they can cause the issue).
>>> Here below a screenshot of memory growth:
>>> asdasd
>>>
>>>> It's a good sign that your memory is the same between restarts, 
>>>> since that means that the redis persistence works.
>>>>
>>>> Have you restarted bischeck after the 20 Nov?
>>>>
>>> Yes we have done some restarts, the last this morning to load a new 
>>> redis configuration ( I set logs to debug level).
>>>
>>>> When it comes to the memory config in redis there are a number of 
>>>> strategies to manage memory, but in the bischeck setup redis is 
>>>> used as a data store and not as a cache, so you would not like to 
>>>> have any eviction polices. Setting max-memory will just make any 
>>>> write fail when redis is at its max-memory.
>>>> I think its safe to upgrade redis to a later version, but please 
>>>> try it first. Currently I run bischeck on 4.0.2 without problem. At 
>>>> the same time I do not think this is a redis bug.
>>>> To track what goes on in the your redis instance you can check with 
>>>> redis-cli. Every servicedefinition becomes its own list in redis. 
>>>> You can check the size (llen) of the list and see if its relates to 
>>>> your purge configuration. You can also read the first and last 
>>>> entities in the list to check the time stamp (lrange). Depending on 
>>>> the bischeck config you may also have additional lists for the 
>>>> servicedefinition for aggregrations.
>>>> /Anders
>>>>
>>>
>>> Ok, i'm writing a script to get, from Redis server, the lenght and 
>>> the timestamp of every list writtend by Bischeck. I hope to find out 
>>> if there is something growing too much.
>>> I'll keep you updated.
>>> Thanks for your support.
>>>
>>> Francesco
>>>
>>>
>>>
>>>>
>>>> On 12/04/2017 10:11 AM, Francesco Giuseppe Toffoli wrote:
>>>>>
>>>>> Hi everybody,
>>>>> since a coupe of weeks the memory used by Redis in increasing 
>>>>> linearly, continuosly.
>>>>> We are a bit worried as also after a Redis server restart the 
>>>>> situation is the same (the RAM taken is equal before and after the 
>>>>> restart).
>>>>> I would exclude the issued has been caused by a configuration 
>>>>> change, the issue started on 20 November at 11:44PM when nobody 
>>>>> was working on bischeck.
>>>>> I'm enabling logs in order to see if there is something wrong.
>>>>>
>>>>> In the meanwhile does anyone have a suggestion on how to proceed?
>>>>> Upgrading to Redis 4.0 (stable) can be done maintaining the 
>>>>> retro-compatibility with Bischeck? (we have Redis 2.8.23)
>>>>>
>>>>>
>>>>> As a workaround I was thinking to enable the Redis option 
>>>>> 'maxmemory' with the 'maxmemory policy': about this last option, 
>>>>> as we need all keys to be maintained on database, does anyone know 
>>>>> as can be set? (I've read something about key eviction but i still 
>>>>> don't have any clear idea).
>>>>>
>>>>>
>>>>> Regards,
>>>>> Francesco
>>>>>
>>>>>
>>>>> -- 
>>>>>
>>>>> Francesco Giuseppe Toffoli
>>>>> Monitoring Engineer
>>>>>
>>>>> GSE Department
>>>>>
>>>>> Tel: +39 01127387488
>>>>>
>>>>> Mobile: +39 349.800.60.35
>>>>> Email: _ftoffoli at skylogic.it <mailto:ftoffoli at skylogic.it>_
>>>>> *
>>>>> **Skylogic S. p. A.*
>>>>> Strada Pianezza, 289
>>>>> 10151 Torino, Italy
>>>>>
>>>>> This message contains confidential information and is intended 
>>>>> only for the individual named. If you are not the named addressee 
>>>>> you should not disseminate, distribute or copy this e-mail. Please 
>>>>> notify the sender immediately by e-mail if you have received this 
>>>>> e-mail by mistake and delete this e-mail from your system. E-mail 
>>>>> transmission cannot be guaranteed to be secure or error-free as 
>>>>> information could be intercepted, corrupted, lost, destroyed, 
>>>>> arrive late or incomplete, or contain viruses. The sender 
>>>>> therefore does not accept liability for any errors or omissions in 
>>>>> the contents of this message, which arise as a result of e-mail 
>>>>> transmission. If verification is required please request a 
>>>>> hard-copy version. Please note that any views or opinions 
>>>>> presented in this email are solely those of the author and do not 
>>>>> necessarily represent those of the Company.
>>>>> No employee or agent is authorized to conclude any binding 
>>>>> agreement on behalf of this Company nor, through thi s latter, any 
>>>>> of the Eutelsat Communication group with another party by email 
>>>>> without express written confirmation by a duly authorized officer 
>>>>> of the Company. The list of duly authorized officers and the scope 
>>>>> of their powers is published on the Trade Register according to 
>>>>> the national law of each affiliate.
>>>>
>>>> -- 
>>>>
>>>>
>>>> Ingby<http://www.ingby.com>
>>>>
>>>> bischeck - dynamic and adaptive monitoring for Nagios<http://www.bischeck.org>
>>>>
>>>> anders.haal at ingby.com<mailto:anders.haal at ingby.com>
>>>>
>>>> Mjukvara genom ingenjörsmässig kreativitet och kompetens
>>>>
>>>> Ingenjörsbyn
>>>> Box 531
>>>> 101 30 Stockholm
>>>> Sweden
>>>> www.ingby.com  <http://www.ingby.com/>
>>>> Mobil: +46 70 575 35 46
>>>> Tele: +46 75 75 75 090
>>>> Fax:  +46 75 75 75 091
>>>
>>> -- 
>>>
>>> Francesco Giuseppe Toffoli
>>> Monitoring Engineer
>>>
>>> GSE Department
>>>
>>> Tel: +39 01127387488
>>>
>>> Mobile: +39 349.800.60.35
>>> Email: _ftoffoli at skylogic.it <mailto:ftoffoli at skylogic.it>_
>>> *
>>> **Skylogic S. p. A.*
>>> Strada Pianezza, 289
>>> 10151 Torino, Italy
>>>
>>> This message contains confidential information and is intended only 
>>> for the individual named. If you are not the named addressee you 
>>> should not disseminate, distribute or copy this e-mail. Please 
>>> notify the sender immediately by e-mail if you have received this 
>>> e-mail by mistake and delete this e-mail from your system. E-mail 
>>> transmission cannot be guaranteed to be secure or error-free as 
>>> information could be intercepted, corrupted, lost, destroyed, arrive 
>>> late or incomplete, or contain viruses. The sender therefore does 
>>> not accept liability for any errors or omissions in the contents of 
>>> this message, which arise as a result of e-mail transmission. If 
>>> verification is required please request a hard-copy version. Please 
>>> note that any views or opinions presented in this email are solely 
>>> those of the author and do not necessarily represent those of the 
>>> Company.
>>> No employee or agent is authorized to conclude any binding agreement 
>>> on behalf of this Company nor, through thi s latter, any of the 
>>> Eutelsat Communication group with another party by email without 
>>> express written confirmation by a duly authorized officer of the 
>>> Company. The list of duly authorized officers and the scope of their 
>>> powers is published on the Trade Register according to the national 
>>> law of each affiliate.
>>
>> -- 
>>
>>
>> Ingby<http://www.ingby.com>
>>
>> bischeck - dynamic and adaptive monitoring for Nagios<http://www.bischeck.org>
>>
>> anders.haal at ingby.com<mailto:anders.haal at ingby.com>
>>
>> Mjukvara genom ingenjörsmässig kreativitet och kompetens
>>
>> Ingenjörsbyn
>> Box 531
>> 101 30 Stockholm
>> Sweden
>> www.ingby.com  <http://www.ingby.com/>
>> Mobil: +46 70 575 35 46
>> Tele: +46 75 75 75 090
>> Fax:  +46 75 75 75 091
>
> -- 
>
> Francesco Giuseppe Toffoli
> Monitoring Engineer
>
> GSE Department
>
> Tel: +39 01127387488
>
> Mobile: +39 349.800.60.35
> Email: _ftoffoli at skylogic.it <mailto:ftoffoli at skylogic.it>_
> *
> **Skylogic S. p. A.*
> Strada Pianezza, 289
> 10151 Torino, Italy
>
> This message contains confidential information and is intended only 
> for the individual named. If you are not the named addressee you 
> should not disseminate, distribute or copy this e-mail. Please notify 
> the sender immediately by e-mail if you have received this e-mail by 
> mistake and delete this e-mail from your system. E-mail transmission 
> cannot be guaranteed to be secure or error-free as information could 
> be intercepted, corrupted, lost, destroyed, arrive late or incomplete, 
> or contain viruses. The sender therefore does not accept liability for 
> any errors or omissions in the contents of this message, which arise 
> as a result of e-mail transmission. If verification is required please 
> request a hard-copy version. Please note that any views or opinions 
> presented in this email are solely those of the author and do not 
> necessarily represent those of the Company.
> No employee or agent is authorized to conclude any binding agreement 
> on behalf of this Company nor, through thi s latter, any of the 
> Eutelsat Communication group with another party by email without 
> express written confirmation by a duly authorized officer of the 
> Company. The list of duly authorized officers and the scope of their 
> powers is published on the Trade Register according to the national 
> law of each affiliate.

-- 


Ingby <http://www.ingby.com>

bischeck - dynamic and adaptive monitoring for Nagios <http://www.bischeck.org>

anders.haal at ingby.com<mailto:anders.haal at ingby.com>

Mjukvara genom ingenjörsmässig kreativitet och kompetens

Ingenjörsbyn
Box 531
101 30 Stockholm
Sweden
www.ingby.com <http://www.ingby.com/>
Mobil: +46 70 575 35 46
Tele: +46 75 75 75 090
Fax:  +46 75 75 75 091

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/bischeck-users/attachments/20171204/70af16d2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bischeck_memory.png
Type: image/png
Size: 14993 bytes
Desc: not available
URL: <https://www.monitoring-lists.org/archive/bischeck-users/attachments/20171204/70af16d2/attachment-0001.png>