Is this DDoS attack?
I am running a TB without any problem for more than a year. But since last four days, I could not get into the box because it seems like some processes are running permanently as the led is bleeping all the time. Therefore, I switched of the machine as it allows me to access for a few minutes and then stops responding to ssh at all. Any hints to overcome this problem? Thanks!
'top' shows all the swap and memory plus more than 50% of CPU without any transaction. The ps -aux looks like http://pastebin.ca/1017719 (thanks to Kodak for guiding, see below)
1) use iptables to turn off access to ssh from everywhere, then only allow known save IP addresses.
2) please use the code tags or post your output to http://pastebin.ca, this will avoid the mess that's above.
3) use "top" to find the top gluttons of CPU and memory. Use the "m" and "c" keys to switch back and forth. Report your findings.
If you need more detail, just ask.
Thanks Kodak.
1) ssh from outside is already disabled as it runs bastille-linux hardening.
2) I have pasted the output of dmesg of my gw-firewall at http://pastebin.ca/1017715 Please note that XX.YYY.ZZZ.AAA is my static public IP and 198.168.1.250 is the TB server.
3) The output of the 'ps -aux' is pasted at http://pastebin.ca/1017719
4) the top output was rendered garbled (http://pastebin.ca/1017724) and now the machine does not allow to ssh - allowed only from localnet to the machine (the TB machine is keyboardless and monitorless).
I hope this could help me to get some hint. The machine is completely locked now.
4) the top output was rendered garbled
Yes, that's why I said that you are going to have to use it to find the top resource hogs. You didn't state any of your findings. Top uses terminal escape codes to move the cursor around, so if you want a screen shot of it you're going to have to take a screen shot in the graphical sense. Or you can just tell us what seems to be using the most CPU and RAM. The top four or five contenders should be enough, as long as you give your impressions of order.
Also, if I'm understanding your second post correctly, you're using this as a firewall too? Bad idea. I think you may have just found out one of the reasons why.
Considering the ps you ran was basically at a random time, and it caught sendmail in the act of sending mail, that strikes me as kind of weird for a phone system. Unless you've set it up to be a mail server too (which you shouldn't do either) or this system takes a lot of voicemail, that probably shouldn't have happened. My gut feeling is that your box has been hijacked and it's now being used as a spam relay or sender. But that's my guess. A tail -f of /var/log/maillog should confirm this pretty quick.
I think you need to tell us a whole lot more about how this system is set up.
Oh, and do ps auxgwww and paste that output, it will show everything.
Thanks again KodaK.
I have pasted the output of top, tail -f /var/log/maillog and ps auxgwww at http://pastebin.ca/1018101 (after about 2 minutes of reboot, the resoureces are consumed 100% and the machine freezes without allowing to execute any commands or access)
Second, I have a different box for gw/firewall (the one with XXX.YYY.ZZZ.AAA static public IP is the machine) and the TB box (the box with local IP 192.168.1.250) are different physical entities.
The installation was done with TB1.8/2 (not sure) installer. I have upgraded to the TB latest and freepbx 2.4 only some 3 weeks back.
I hope this helps.
OK, sendmail is going crazy and it looks like munin is the culprit. I'm guessing from the information available to me that munin is going (and this is a technical term) "ape shit" and sending lots of messages through sendmail.
Stop munin from running to see if that fixes the problem:
service munin-node stop
If that is then you can dig a bit further to find out why. You probably have a lot of queued up mail, so you may want to clean it out.
You can do this pretty easily with mutt, but you'll have to install it:
yum install mutt
Then launch mutt and use the on-screen help for more information. You can delete messages by regex, so it'll make it easy to delete a lot of similar messages at once.
If you have more questions, go ahead and ask.
I rebooted and stopped the munin-node (also done chkconfig munin-node off), yet after two minutes the entire resources are consumed by unknown processes :( (the munin is something that I never used, so do not know whether that is accessible or not, I have a default installation of TB).
I found that it is the hudlite-server which was the culprit besides munin-node and sendmail. I stopped the services and disabled at boot time (chkconfig hudlite-server/munin-node/sendmail off).
Could anyone tell me how to disable munin-html and munin-graph?
About the mutt, I could not follow exactly how regex is used for group deletion?
Anyway thanks to KodaK for your help, he was immensely helpful. :)
I would suspect that there's some other underlying issue. You might want to keep digging.
You can just uninstall the munin stuff. Do:
rpm -qa | grep munin
and for each munin related package do:
rpm -e packagename
As for mutt:
once in mutt, type "?" and you'll get a help list. You'll see that one of the options is D for "delete messages matching a pattern". So, if you want to get rid of anything with "trixbox" in the subject, you would press "D" and then you'll get a message at the bottom that says "Delete messages matching:" and it will wait for input. Type in your pattern there and hit enter.
I removed both munin and munin-node packages. And I also removed all mails at /var/spool/mail/admin /var/spool/mail/asterisk using mutt. Then once I restart the sendmail service, the admin mailbox fills with hundreds of mails that reads like:
Date: Wed, 14 May 2008 22:35:21 +0200
From: root@asterisk1.local (Cron Daemon)
To: root@asterisk1.local
Subject: Cron
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
Cannot open /var/www/html/munin/localhost/localhost-cpu.html at /usr/share/munin/munin-html line 488.
Is it because munin and munin-node has been removed??
It's possible there's still a cron job trying to run.
If the file:
/etc/cron.d/munin
exists, delete it (or move it to somewhere else.)
However, your mail spool may be full too. Before starting sendmail, check /var/spool/mqueue and if there are any queue files and data files you might want to delete them out (beware of collateral damage such as emailed voicemail messages and such.)
Dear KodaK:
Thank you for your inputs. They are extremely helpful.
1) In the meantime, after I uninstalled munin, I could not call between one sip extension to another sip extension (gets "the extension is in use" message). But the extensions could call outside and even iax extension could call sip extensions.
2) Besides when I checked a mail to asterisk (mutt -f /var/spool/mail/asterisk) I do see a mail as below:
Date: Wed, 14 May 2008 17:00:01 +0200
From: root@asterisk1.local (Cron Daemon)
To: asterisk@asterisk1.local
Subject: Cron
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
X-Cron-Env:
[FATAL] database connection failure failed trying to connect to the configured database
I checked the relevant password and username for the amportal and they are alright.
Thanking you again for your invaluable inputs.
In the meantime, after I uninstalled munin, I could not call between one sip extension to another sip extension (gets "the extension is in use" message). But the extensions could call outside and even iax extension could call sip extensions.
Munin has nothing to do with the operation of Asterisk/FreePBX, so this is a coincidence. Please post a log of a failed call to http://pastebin.ca and we'll try to figure it out.
As for 2: I don't know what database it's trying to connect to, but if it's failing, then something is wrong (obviously). I'd have to trace through it to figure it out, but can you connect manually with the credentials given?

Member Since:
2006-06-22