Memory usage climbs until the computer crashes

Are you sure this is the cause of it ?

I'm a Linux noob, but my undestanding of it was that Linux unlike windows makes use of all available memory, mine is curently sat at 99% with no calls going on, it never crashes.

Thanks for your answers. The hardware is an HP ML110 Server. Pentium D 3.0 with 512MB of Ram and 380GB of HD.

What causes the memory usage to climb. When the computer is reset the memory usage is at 28%. With no calls and no access going on the memory usage continues to go up. After resetting last night, this morning the usage has gone from 28% to 58%. Why would the Admin page of Trixbox redline the memory usage at 90% if this factor wasn't important?

This time I will let it run until crashing and see how long it takes.

Thanks

James

This box is pure IP. No cards and nothing else running on the box. I only use it for outbound IP calls and for calling extensions. I have various extensions in different parts of the US and Brasil. All incoming calls are IP. There are very few of these.

James

Its just what Linux does, heres a brief explanation I got from 5 seconds on Google.

Your computer is using all of your memory to optimize the performance of Linux. It caches or stores pieces of applications on your memory so that when you need them, it will not have to hunt around the hard drive to find the application. Now IF you bring up a program, and your memory is "full" Linux automatically kicks some of the cache out to make room.

I have the same thing with Trixbox 2.2.3 and a computer with similar specs. The "physical memory" bar on the status page starts off at what seems like a reasonabel range (30%) and ends up redlining at 90%, then stops answering or making calls. What is causing this usage to grow?

I think there may be a memory leak. I wasn't aware of a problem until people called my cell to tell me my landline stopped answering. When i checked, the memory usage was 100%. After a reboot all was normal again.

Is there a memory leak with the 2.2 version? I ran Tom King's updates - I don't know if that was a factor.

All computer systems use RAM for storing temporary volatile frequently accessed data for access and re-access so as to optimize access rate via the RAM since the HDD is the slowest part of any computer system. As the RAM is used up, the system calls for the Virtual Memory aka Pagefile (1.5 times the size of the installed RAM), resident on the HDD. And when such activities die down things return to normal i.e. RAM usage reduces and stays put until when in use again. In the case of Linux using up RAM without the system being in use, that is not a normal system behavior. So that is an abnormal system behavior...if such happens in Windows or Apple, then there is some sort of Malicious Software on such a system or the system is in use somehow perhaps via remote access, etc.

I am experiencing the same thing with TB CE 2.3.0.2 (didn't happen with 2.3.0.1 until after I applied some updates that not only caused this behavior but also knocked out bidirectional audio on all remote extensions)...it starts off at about 20% and now with no single call on it, it is at 85% RAM usage, and we are talking 1GB RAM capacity here. Last couple of weeks it got to 99% and I had to reboot it. This is very abnormal and does not make me confident to deploy such a system to clients...this is a huge problem. A lot of technology manufacturers (Cisco, 3Com, Linksys, etc) use Linux for appliances but none of them ever behaves like this. My Laptop running XP Pro has 2GB RAM, I opened Task Manager and clicked on the Performance tab and see that the RAM has been used up 2.01GB (due to millions of things running on my screen not to talk of about 50 websites going in IE and Firefox tabs, and other, SQL Server, etc) without any performance issues, so for the purpose of observation I started to close out some of the tons of applications I am running and I see the RAM usage decrease. That is how a normal, well coded system would behave...when in us it is fine to see RAM usage "gradually" climb but when not in use there is absolutely no reason it should use up RAM. Such a system is unstable. It is like saying you are not making any calls but the amount of SIP channels and Zap Lines used continue to climb...weird, even worse, your phone company/ITSP charges you for the unmade calls. Anyway, this issue definitely needs to be fixed on TB distros...Let's put it in the bug tracker guys....no room for stories or any explanation to clients at all. It either works or not and in this situation, it does not work.

The RH post is simply reiterating my point "The cache is reclaimed, not at the time of process exit (you might start up another process soon that needs the same data), but upon demand - i.e. When you start a process that needs a lot of memory to run, the Linux kernel will reclaim memory that had been storing cached data and give it to the new process". And the weird RH philosophy about hogging all the memory if possible.

When you exit a program, memory usage is supposed to drop unless upon that exit you open another program that may use the same amount of memory you just freed up. In TB where there are no new processes or calls initiated but the system keeps eating all memory until sometimes crashing does not make sense at all. Tell me if my system is telling me 85% memory usage when not a single call is going on, what would it then climb to when I have say 10 concurrent calls?...I think that it will crash even before it gets to 10 concurrent calls. The architecture should preserve as much memory as possible until required for calls and other critical processes.

techieg read it again.

You say 'When you exit a program, memory usage is supposed to drop unless upon that exit you open another program that may use the same amount of memory you just freed up.'

Redhat say it will be retained incase it is needed again, it will only be dropped to make way for new data. So while the actual data may be dropped only to be replaced, no where does it say it will be dropped and your memory usage will go down.

I don't know how many times I have personally responded this same query.

Linux has a sophisticated algorithm for memory usage. The objective is that the most used pieces of code are resident in main memory. Having free memory to be occupied for code to be run is a lot more inefficient that having the code already there. This explains why the more you use an application in Linux the faster it runs, also explains why the memory is fully used.

This is NORMAL behavior for Linux.

Having said so, what is not normal is that while having 90% main memory used you also have fully used your disk swap space. In a dedicated server such as a TB box very seldom you would occupy more than 30-40% of your swap space. Having 90% swap space used is a sign of some sort of problem.

Hope this helps

Regards

Your saying its bad behaviour because windows and macs do it differant, I think you'll find its the other way around, Linux does it right the others do it badly.

Your failing to see the way it works and the reason for it, read up on it and it will make sense.

Whatever this problem is needs to be resolved. I did not think it was a Linux problem earlier because as I said it did not happen in any of my previous TB installs including 2.3.0.1 until I installed some updates on 2.3.0.1. So I thought it had to do with the updates until I got a lecture from TDF that such memory hogging behavior is normal with Linux. So my point is that because it is normal Linux behavior does not make it a good one and be it normal behavior or not, it is hurting system performance and perhaps RH, CentOS, or TB developers, etc can make a change in that area so that we don't keep getting system crashes due to evaporated RAM. Even as I write there is not a single call going on my TB but RAM usage is RED at 98% of 1GB, so what use is it if one cannot use a TB installation because Linux decides to use up the RAM? I am also quite sure that no matter what RAM size is installed it will still eat it all up.

I have this problem as well on only one out of 5 trixboxes. 'top' shows that almost all memory is used, but doesn't list anything that is using it all. Maybe this is a driver issue. I'm using Sangoma analog and digital cards on all of my machines. What about the rest of you having this problem? Are any of you using Sangoma cards? This driver is sensitive to the kernel or zaptel software changing because its a kernel driver and it needs a special patched version of the zaptel software.

Other than that possibility, the things that are unique about my one errant machine are:

It's the only one with only 512 MB memory

It's the only one running a non SMP kernel

It has the oldest processor (2.4 GHz Pentium 4)

It's the only one running the cgi management interface for 'nut' (Network UPS Tools)

It's the only one I've customized FOP to show multiple locations and day/night status

Anyone spot any correlations here with their errant machines?

TechieG,

I understand your frustration with your problem, but I can assure you that while "memory hogging" is normal behavior for Linux, it is not your culprit. My box right now is at 99% physical memory (512 Meg) and 1% swap space. This is with no calls. The system has been as such for the last 43 days, and I only rebooted the box then because I was monkeying around with my network services and firewall. The system has 23 extentions and 4 trunks (2 ZAP and 2 SIP). Performance has been flawless.

I know that yum has caused issues with TB before (just look at 2.2.4, it fails to compile Zaptel properly on install!!). If you can, I'd restore your system to a pre-yum updated state and stay the heck away from it on your production system.

I know this doesn't necessarily fix your problem, but perhaps shed a little light on the issue here instead of it becoming a flame-war.

Good luck,
-Chris

mailboxetc, are you ssying there is a way to fix this? If so, how can that be done?

In the mean time you and techieg should contact Redhat and file a bug report. You'll probably win a prize for pointing out such a findamental flaw.

How about a system w/ 4Gb of memory
86% and increasing used by the cache -
swap at 0% , recording all calls 30 users and 15 cocurrent calls ( VoIP).

@drmessano,

Are you saying -WINMEM is a workable solution? If so, how is it used or applied?

Techieg,

I originally asked this question and after watching this thread maybe I can explain some of what is going on.

The original problem was on a machine that had a memory problem. When Linux (trixbox) occupied all of the memory for cache the computer crashed. I repeat, the box had a bad memory card.

I do not work with Linux. I only use this because of trixbox. I work with Windows XP, Vista and Server all the time. Yes Windows machines like to have lots of UNUSED memory to work. Linux also likes to have a lot of FREE memory. The difference is that Linux will occupy this FREE memory with CACHE data. If you delete a file off the hard drive the file doesn't really go away even if the drive says that is has more space. Linux does this with memory. If you unload a program, how is the OS going to know that you are going to load another program that will use the same data in the future. It has no way of knowing this, so it just keeps all the data in FREE memory as CACHE data. This is normal. If you look at the Trixbox Admin login page it will tell you that all of the memory is occupied. True. But if you need to load a different program it will sacrifice CACHE memory for Kernal/Application memory. Fine. As the other gentleman says if the Disk Swap is high you have a problem. Now, if you open the new FreePBX page it only tells you how much memory is USED. It doesn't say a thing about swap. If it makes you feel better used this value to go by.

As to why the memory usage climbs when the box isn't receiving or making calls, the OS, Asterisk and all of the other programs are FAR from being stopped patiently waiting for you to make a call. It is registering extension and trunks, creating logs, recording everything that you do. Take a look at TOP from the command like. I had never heard of this. The output reminds me of the process manager in windows. My system has nothing less that 70 processes in memory and I haven't made a call in 2 days. All of these programs are using CACHE memory to run faster. When you need to make a call this memory is freed immediately for your use.

As for the system that crashed after 95% memory full, I took it down (it was a Sempron 2600) and rebuilt Trixbox on an HP Proliant ML110 Pentium D 3.0GHz with 512MB of memory. I had the box up for 2 months with no more problems. Had to reboot recently to upgrade everything possible having to do with Trixbox and Asterisk. Thanks guys for the great job and the slick new interface.

I logged into my TB today and saw that my memory usage is down to 68% from 99%. I then checked the uptime, it says 2 days 10 hours. I have not restarted this box in over a month, so this means it must have crashed and rebooted about 2 days ago due to memory hogging issues as a lot of others have experienced. And during that time there were no calls going on at all...yes, I mean zero calls. SO it restarted and climbing back up again. This is definitely not something I am confident to deploy for clients. God forbid it terminates important business calls in that process. This issue is something the TB team should look to resolve and not wait for RHEL developers since they think it is a working cache solution they invented.

chill. Be Chill.

I think we've heard it enough times how Linux handles memory. It was very informative the first few times.

But keep in mind the original poster had a problem where his system _crashed_. In his particular case when his memory happened to be close to being "used up". Repeating the same old same old will not help solve his problem.

A good number of digium fixes are related to memory leaks. I'm not saying this to bash digium, but just to make some people aware that there are indeed memory related problems. Maybe not on your system. Or your particular combination of setup, hardware, colour of penguin, etc.

I'm all for educating people (with or without clue bat) if they seem to be off track. But none of this seems to be helping our poor chap with his _crashing_ system. If you have to reboot to keep the system up and running something is wrong. And it's not just misunderstanding how memory works under Linux.

-- J

I just assumed the op assumed the completely normal behaviour of memory being used up by cache memory over time was a problem. It may not be crashing or crashing for some other reason.

techieg just refuses to accept the normal behaviour of cache memory filling up and insists it's a Linux bug. I have resorted to referring him to Linus Torvalis or just sending him off on wild goose chases searching for non-existant kernel configurations.

So I guess your not getting it either then. If you run "top" and most of that 100% usage is in cache memory then you don't have a problem.

If your still not getting it then google for the -IDNTGTIT switch.

Keep searching. It can be hard to find.

The REAL problem here is more of an interaction issue. The visual indicator, by virtue of being red, is telling people there is a problem. There is no reason for it to be red. If the Linux OS is happy with 99.9% utilization, then frankly that bar should be green from start to finish, and only just an indicator of usage. In fact, one might argue that if it's NOT using enough memory then it is not optimizing itself for best performance, so heh, maybe the color scheme should be inverted!

But all kidding aside, it should be green all the way across, not yellow or red. Someone with skills better than mine should make that patch recommendation.

Its a Trixbox issue, if you look at FreePBX 2.3 that nows show your memory usage, but only shows app memory and swap and the way it looks wouldn't send anyone into a panic.

The Trixbox page has added buffers and cached sending the total into the red and looking like a problem if you don't know any better.

With an understanding of the cache config in Linux, there is still and issue of memory usage. If memory usage was climbing and TB had no problems, OK, no big deal, but; there are problems.
Our config: Dell 1950 dual DualCore 2G, 4G RAM, 70G RAID
TB 2.2.3, all SIP, no telephony cards, FreePBX 2.2.1
We currently have ~40 extensions, 12 routes (In/Out) , 1 trunk, a handfull of Misc. Destinations, and 8 parking slots. Call volume is low to moderate. We've found that after rebooting the system, memory usage will show at ~14% with very low cache usage. After three days, usage has risen to ~15% with cache and buffers occupying 10%. After about 2-3 weeks, memory usage will rise to ~85%, of which cache and buffers will be about ~25%. When we reach this point, we start to have "wacky" FreePBX issues, (extension don't function properly, BLF won't light, voicemails won't get delivered, new extension not functioning, etc). When we reboot the system, memory usage returns to ~14% and all functions return to expected behaviors.
This is classic memory leak behavior. We're testing FreePBX 2.3.1 in our test environment, but cannot simulate the load we see in production. Has anyone experienced this with FPBX 2.3.1 or plain Asterisk installations? I don't think it's asterisk itself, and top reports steady usage at .3-1% for the asterisk process...
Any other ideas or info from others with this problem would be helpful.
Matt W

Is there a command that I can run once a day to clear the cache memory? Our problem is that TB randomly crashes - could be after a week or a month and we never know when it is going to happen. But it does seem that there is a memory spike at that time (with only one call in process). If I could run a script each day that would clear the memory cache and see how long it works by doing that at least I can continue my process of elimination.

First of all and for about the millionth time cazimmy, If your....um....server is crashing because of memory it is not because of cache memory filling up which is 100% by design and perfectly normal and does not cause any problems with anything unless your memory has a problem that only shows up when it get's filled up!

However, since you asked, the "script" would be 'reboot' which you can add to cron.

compdoc, do the same sorts of things you are doing on System B on System A and it will start reserving memory as well. I wish Trixbox would change the GUI so we don't have to have these conversations about cache memory every few days.

install webmin and check the system stats on there to see what is using all the cpu and memory.

Are you by chance running iaxmodem? I've found that the version in the repo's will cause these symptoms.

I have small systems running just fine on 256Meg. The min. most systems would come with now a days is 512Meg which is plenty for anything less than 10 extensions IMHO.

cazimmy,
As long as people continue to demonstrate their lack of understanding of how memory is used in CentOS even thought they think they do know, they should expect attitude.

If they do know how memory works they should be able to trace it down to the daemon that is hogging all the memory using one of a few very simple commands. If they cannot get that far then they obviously don't know what they are talking about.

So are you saying that you did indeed have a memory problem or just that you had a problem and your memory was by coincidence doing exactly what it was supposed to do and due to a lack of understanding youve added up 1+1 and got three ?

I'm not rereading this sorry arsed thread, but from what I remember no one had any evidence of a memory problem, apart from the OP who infact had memory that had gone bad and needed replacing.

In my case the problem was the size of root mails in /var/spool/mail/root file. I`ve cleaned it
by typing in command line :
echo 0 > /var/spool/mail/root
It writes 0 (or something you like :) ) in file whitch contains root mail messages.
Before cleaning it was about 2GB. Now my problem solved without restarting which is critical for my company.

But if this harm your asterisk info page. you can simply edit the file /var/spool/mail/root
and delete the string you have written in (in previous examle 0). The asterisk info page will now work.

I`m not talking about hard drive space. May be some process is trying to read root messages, bacause my systems used physical memory before deleting root messages was 5.5 GB grom 6 GB, and now it`s only 300 MB.