If you have a junk motherboard...
Submitted by ethans on Wed, 08/15/2007 - 4:53pm.
We got into a bind where we had to push a system out fast (their voicemail module on their current Nortel PBX stopped working). All we had was an Intel D946GZIS motherboard (also referred to as D946GZISSL). It was paired with a Rhino R1T1 PRI card. The result? Horrible, horrible IRQ conflicts. This cause audio blips, echo, static, fax cutoffs, you name it. The solution is pretty involved, but brought our average zttest results up to 99.98556% from around 99.6% (ouch), which resolved all audio issues. We'll be promptly replacing the motherboard with a SuperMicro once it arrives, but for now at least the customer is up and running (with voicemail!) and happy.
If there is one lesson I can tell every single integrator out there it is to use a quality motherboard (SuperMicro), RAM (Crucial/Mushkin), power supply (Antec) and at least soft RAID 1 with 2 hard drives (which we actually prefer to hardware RAID). You'll spend a little more per system, but the customer experience pays off huge dividends over the long haul.
On to the good stuff, here's how the solution works (your results will vary - we had a dual core processor):
service stop irqbalance # this process dynamically allocates irqs to different processors
chkconfig irqbalace off # don't run this process at next system boot
lspci -vb
You'll see a lot of entries such as:
04:01.0 Network controller: Rhino Equiment Corp. Rhino R1T1
Flags: bus master, stepping, slow devsel, latency 32, IRQ 11
Memory at 50001000 (32-bit, non-prefetchable)
Make note of the following 04:01.0 on each of the following: your TDM card, hard drive(s), USB controllers, ethernet.
Run the following commands individually (or create a bash script) MAKE SURE TO REPLACE EACH ##:##.# ENTRY BELOW WITH THE SPECIFIC RESULTS YOU GOT FROM RUNNING lspci -vb! We just adjusted latency of all of our devices accordingly.
setpci -v -s 04:01.0 LATENCY_TIMER=ff # PRI
setpci -v -s 00:02.0 LATENCY_TIMER=4 # VGA
setpci -v -s 00:1d.0 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1d.1 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1d.2 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1d.3 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1f.2 LATENCY_TIMER=4 # IDE
setpci -v -s 04:08.0 LATENCY_TIMER=4 # Ethernet
Now, if you have a dual process or dual-core system, we can take this a step further and really make some difference.
Run: cat /proc/interrupts at the command line.
You'll see something like this:
0: 13716843 8000771 IO-APIC-edge timer
8: 1 0 IO-APIC-edge rtc
9: 0 0 IO-APIC-level acpi
14: 80458 87515 IO-APIC-edge ide0
15: 92937 69524 IO-APIC-edge ide1
169: 0 0 IO-APIC-level uhci_hcd
185: 0 0 IO-APIC-level uhci_hcd
193: 0 0 IO-APIC-level uhci_hcd
201: 0 0 IO-APIC-level uhci_hcd
209: 8533231 13196282 IO-APIC-level r1t1
217: 1265729 0 IO-APIC-level eth0
NMI: 0 0
LOC: 21197909 21208389
ERR: 0
MIS: 0
Make note of the first column and each of these devices: your TDM card, all ide devices, and ethernet devices.
Now run each of the following commands REPLACING THE ACTUAL IRQ # FROM cat /proc/interrupts
echo 1 > /proc/irq/201/smp_affinity #uhci_hcd
echo 1 > /proc/irq/217/smp_affinity #eth0
echo 2 > /proc/irq/209/smp_affinity #r1t1
echo 1 > /proc/irq/14/smp_affinity #ide0
echo 1 > /proc/irq/15/smp_affinity #ide1
What we just did was took the interrupt requests for the TDM card to one processor and everything else to a different (in our case cores of a processor). Essentially this allows your TDM card priority access to the processor core (or separate processor) it is assigned to. Run zttest for a minute or so and then do a Control-C to make sure you are above 99.98% on average.
Now nano /etc/rc.d/rc.local and put every setpci and echo command you ran into it so the next time you reboot they are sure to stick. Here is our final /etc/rc.d/rc.local file:
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
setpci -v -s 04:01.0 LATENCY_TIMER=ff # PRI
setpci -v -s 00:02.0 LATENCY_TIMER=4 # VGA
setpci -v -s 00:1d.0 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1d.1 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1d.2 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1d.3 LATENCY_TIMER=4 # USB Controller
setpci -v -s 00:1f.2 LATENCY_TIMER=4 # IDE
setpci -v -s 04:08.0 LATENCY_TIMER=4 # Ethernet
echo 1 > /proc/irq/201/smp_affinity #uhci_hcd
echo 1 > /proc/irq/217/smp_affinity #eth0
echo 2 > /proc/irq/209/smp_affinity #r1t1
echo 1 > /proc/irq/14/smp_affinity #ide0
echo 1 > /proc/irq/15/smp_affinity #ide1
/etc/trixbox/runonce
/usr/local/sbin/motd.sh > /etc/motd
/sbin/udevstart
/usr/sbin/ztcfg
/usr/sbin/fxotune -s
/usr/sbin/amportal start
Hope this helps someone!
P.S. with the Intel 946 chipset this will get Trixbox (if you are using Trixbox or any CentOS 4.5 distro) to install:
After you boot from the cd, you will have a prompt.
If you are going to do software Raid 1 with 2 drives:
linux ks=cdrom:/ks_sataraid.cfg pci=nommconf acpi=off irqbalance
For a single drive:
linux ks=cdrom:/ks.cfg pci=nommconf acpi=off irqbalance
Once your system is installed nano /boot/grub/menu.lst and replace acpi=off with pci=nommconf (you'll need acpi). If you miss this step, your system won't boot, but if you forget it, you can always type pci-nommconf at the boot prompt to get into your system to make the change.
Ethan Schroeder
Schmooze Communications
ethan dot schroeder at schmoozecom dot com
Ethan, we've done many
Thu, 08/16/2007 - 7:01am
Ethan, we've done many installs on Intel Desktop and Intel Server boards using Sangoma cards and have never had a problem. We do turn off all extraneous features like onboard audio, serial ports, floppy controller, etc.... Our results have been excellent and we've seen no need to use any other brand of boards. Intel also offers excellent customer support and a three year warranty on all boards. Server board support is 24/7 and they are replaced overnight if there is a problem. I just don't want your post to turn others off from what is in our company's opinion, an excellent manufacturer / vendor.
Motherboards
Thu, 08/16/2007 - 7:32am
The post by Ethan is targeted at the 946 chipset, a true JUNKER MB (a gaming PC motherboard).
Many folks out there do no really research the MB they buy, I have had to do as Ethan has done and find away to install on a "desktop" MB.
His posting will allow a few more folks to load TB on those $ 50.00 MB's
Rmember what those of us can do from a bare metal biox is not the same as what 90 % for the TBer's need / look for; they need the steps (as in Ethan's post) to install on some off the wall desktop MB.
Me; I just tell folks they need to buy to RIGHT MB, but Ethan put effort into help solve some issues.
Motherboards
Thu, 08/16/2007 - 6:36pm
Honestly, the issue here isn't Intel brand Mobos vs. Other Brand mobos. It is consumer grade vs. server grade. Can I get just about any motherboard to work well with an Asterisk install? Yes. But am I doing my customers justice by doing so? No. Also, what is my opportunity cost here? Endless hours of tweaking to get something to work that is going to save me $100? Come on. Remember, this isn't a SBS Exchange server, it is a phone system and your customers expect it to work every time all the time. If it doesn't, you are out of business real fast.
The "community" or whatever it is needs to realize that we aren't in competition with each other, we are in competition with the "big" brands out there: the Nortels, Avayas, NECs, Panasonics, etc, etc. They have it together. $100 in the grand scheme of things is *not* going to kill your ability to close sales, and if it does, well, you didn't really want that sale anyway (trust me).
Further, I made the decision recently to scrap all non-hardware-echo-canceling TDM cards over 4 ports. It just isn't worth the hassle and the degradation of the customer experience. 4 ports and under is licensed octasic echo cancellation. Everything else is hardware echo cancel. Add +15% to your cost, adjust your pricing accordingly and reap the benefits for years to come. This business is 25% yellow pages advertising and 75% referrals.
Us as high-level management were on-site at a customer install today for a new reseller to assist and train that reseller (and above all service that customer). 99% of your OMG "issues" will be training, the other 1% will be OMFG REAL issues,which will be directly related to the hardware you choose and your skill level in Linux and Asterisk. That 1%, however, has the ability to kill you. If you are going to make a living (and more) out of this stuff, you need to understand what is going on.
I have my opinions of but stay out of the politics of the "Trixbox Pro" crap, but what I'm really scared of is a bunch of people selling something that is so critical to businesses they have no understanding of and is basically unsupportable from an external perspective. If Trixbox Pro takes off like I think it will, I have to wait just a year or two to reap all the benefits of it. This stuff is not easy. You can't install Trixbox or Linux+Asterisk on anything and expect it to work well in an industry that demands 99.999% ("5 9's"). You have to know what you are doing.
Once you reach these conclusions, you will be able to go against Avaya on a $300,000 hospital install and win....and sleep well at night knowing you aren't going to kill someone.
Ethan
Schmooze Communications
ethan dot schroeder at schmoozecom dot com
Server class
Thu, 08/16/2007 - 6:49pm
We only use Supermicro boxes, I am looking at buying 5 for the Data center right now
The model we like is at EOL so I gotta stock up.
I see MANY of the first times using any ole box they can,
It really ticks me off to see that someone is soured on asterisk because of a 50.00 MB or some tdm echo issue.
And I am with you on those TDM cards I am so over the "echo / Lag"
It's definitely an easy land
Thu, 09/06/2007 - 9:37am
It's an easy land mine to step on in this business. Usually the stuff either works or it doesn't (at first) so the cheap stuff that just works can seem like such a bargain in the beginning. Long term it's almost never worth it. I've made the same mistakes in the IT business but no way I'm going to fall for it in the 5 9's PBX business.
15% margin may be considered ok in the IT business but you should be able to make MUCH more in Telco. On the prices they can check maybe 15-25% (like phones) but everything else I try for at least 50% and after installation cost you should be able to do well over that for the whole solution and still be a lot cheaper than the traditional guys.
It wouldn't be hard for
Thu, 09/06/2007 - 12:07pm
It wouldn't be hard for someone to build and market an Echo Canceller card Put it all in hardware and do 24+ channels on the PCI Bus, thus eliminating any issues with Echo. Asics are cheap enough to design these days. God knows it would be cheap enough to do and the added benefit is being able to run VOIP and TDM traffic through it. Timing would be the issue, but it could easily solved with a clock on the card to be used as a timing source.
Blah.. I'm just passionate about echo.
i like these small form factors from dell and hp
Mon, 12/24/2007 - 3:01pm
guys.
what do you think about HP DC7700 / 7800 SFF with SATA RAID? looks really neat. it has Q965 chipset which isnt bad and also great bios whereyou can chose IRQs for the ZAP card + core2due E4500 CPU and 2 GIGS of ram.
i like using them because they are small. you got parts for them everywhere. you can add hp onsite CarePack for around 150$ so you can send HP to replace a dead motherboard or HDD in case something goes wrong. hp even offers 4 hours Same business day packs for about 200$ extra.
i have about 5 boxes running with this configuration for about 6 month with no complains so far. i guess this isnt the best configuration for stability b/c it isnt a server but still i look at the stress trixbox puts on them and it isnt more than 10% of the cpu and anout 1% I/O.
could someone explain to me why i need a real server to run a 10% load on a linux based OS?
Regards
Isaac
The motherboard was not the problem
Mon, 12/24/2007 - 4:45pm
The problem that Ethan had was running a 2.6.9 kernel that didn't support the interrupt machanism on his board. The 2.6.18 kernels do support that routing machanism. When a "server class" board appears on the scene, you just can't use 2 year old kernels on them and expect to get your $$ worth out of them. That 945 / 965 chipset was not supported properly for a longer time than most. They had problems with documentation / BIOS seeing how it was jointly developed with Microsoft.
I say quality over quantity.
Mon, 12/24/2007 - 4:49pm
I say quality over quantity. Do it right with good hardware the first time instead of redundant cheap stuff lying around. Unless you and your customers don't mind the inevitable downtime and hardware swapping. That is not considered "normal" in traditional telco so gotta wonder what sort of long term reputation you will be able to build doing things that way. Just my opinion.
and hp isnt quality?
Mon, 12/24/2007 - 10:23pm
this is not some 250$ tower. it's a 500$ one. with a good ho mother board. i totaly agree about quality. we are after all in telco business not home pc's.
i could take a SC440 or ML110 servers. they cost double and i'm not sure they offer anything more than a tier1 workstation when it comes to trixbox. trixbox isnt heaby on I/O or CPU. its not SBS2003 server with SQL and Exchange 2003.
Isaac
When I speak of "quality"
Tue, 12/25/2007 - 4:40pm
Not at all...
Tue, 12/25/2007 - 5:07pm
Price is fair on the SC-440, but it is not why we use them. We have had consistently good results with the SC-440's with a Pentium-D (Dual Core) CPU, a gig of ECC RAM, and 2 Hard Drives in a Software Mirror - Most of our installs fall in the 15-30 phone range, and for that application, it is a perfect match - any bigger than that, and we jump up to a PowerEdge-840 with a Xeon and hardware RAID.
In operation with that many phones, the CPU never goes above 20% (only when someone is dialing and it is processing the Dial Plan does it even get close to that number) and usually idles at about 1-2% - we have customers that are recording 5-10 calls simultaneously with this box, and this is no problem also.
Finally, with the SC-440, I get predictable results - and we keep identical spares (although we have never had to use one) so that if one were to fail, we can just drive out there with an exact replacement and put it into service - MANY of the Clone boards get revved so often, they are a moving target.
Use what works for you - but there are MANY Trixboxes running WELL on SC-440's - It's even on the supported list for Pro, if that's the direction you are moving.
Greg
I haven't used one for a PBX
Tue, 12/25/2007 - 6:35pm
I haven't used one for a PBX so you would know better than I do. I'm just leary of cheap stuff in general. I've been burned too many times by these so called "good deals".
The lack of native Linux support for the Enet chip sucks. It's kind of a pain to deal with that in Linux. Not a show stopper once you get it all figured out I suppose.
I actually found a post from jahyde
Tue, 12/25/2007 - 10:39pm
saying just copy a compiled tg3.ko over to the box and reboot and sure enough - made a CD with the file on it compiled for SMP and Non-SMP and copied them over top of the files that were there and bingo - don't even have to compile the drivers - works first try.
Even better, the 2.4 tree finds it automatically, so there is nothing to load to get networking - also, the Sangoma driver suite in 2.4 is so easy it's scary.
We are in a market with pretty intense price pressure - and our bids are always being compared to Digital, so we have to be careful to price it where we still make money, but not price ourselves out of the market.
I really like SuperMicro and also had a great experience with HP last week, so we are open to different brands and models - I just don't want to get stuck with something that doesn't work and we have to eat it (I was stuck with 4 Digium TE110p's until a Digium guy took pity on me and traded them out for TE120p's) so I am a little paranoid - but I scour the boards looking for positive posts about what works - and I try to test things when the opportunity presents itself.
You are right about cheap though - when we were first stating out, we tried to use several different desktop boards (Asus and Gigabyte) and we paid dearly for that.
Greg
OS improvements ?
Wed, 12/26/2007 - 6:39am
"we keep identical spares so that if one were to fail, we can just drive out there with an exact replacement"
One item I've noticed with the change to CentOS 5.1 is that it seems to be more flexible from a hardware standpoint. The "production" server at work and my test system at home have completely different chipsets, processors, and networking but the 2.3.12 image from work came up at home with no complaints. That was not the case with 2.2.
Asus and Gigabyte
Wed, 12/26/2007 - 7:30am
I see from GSnover post that Asus and Gigabyte might not be a good choice, just purchased new equipment to update my box, but as luck will have it the MB didnt turn up due to the holidays. I did order a mini atx Gigabyte board to support a dual core processor but now thinking of changing this MB.
Any recommendation on a MB please.
Supermicro!I won't even use
Wed, 12/26/2007 - 12:25pm
Supermicro!
I won't even use Gigabyte for Desktop stuff. At least not their 'value' stuff. Their high end Gaming boards are ok........FOR GAMING!
Asus boards with Intel or Nvidia chipsets are ok for home or SoHo PBX's. Not for small business. Stay away from anything using SiS chipsets. I have mixed feelings about Via chipsets so in other words it's a non-starter for a PBX.
My recent experience with Nividia and Intel chipsets is that they both seem to work VERY well with CentOS 5.x
My 2 cents.
That's WAY Cool
Wed, 12/26/2007 - 1:17pm
"the 2.3.12 image from work came up at home with no complaints. That was not the case with 2.2."
That is really good to know, and should add a great bit of flexability as far as what you stock - we are anxiously awating the 2.4 final, and assuming there are no show-stoppers, we plan on migrating our customer base to it (about 37 systems at this point) because it seems so much more compatible with current hardware. Also, we are starting to use Bandwidth.com extensively, so Asterisk 1.4 is getting to be a requirement.
Greg
What's with Bandwidth.com and 1.4?
Wed, 12/26/2007 - 3:01pm
Greg-
What's with Bandwidth.com ? why do you need 1.4 to use Bandwidth's sip trunks?
and another questien i got:
I just got my hand on a new R1T1-EC its about 20% less than TE110 EC or A101D. is it worth trying? do you guys have experiance with RHINO cards? somehow they are less known here and if you think of it thre is only 1 post about rhino to 10 posts about digium / sangoma cards. should i be concerned about using these cards with production systems? what experiance do you have with this company?
i like to know about how keen they are to support and do replacments?
Happy new Year 2008
Bandwidth.com
Wed, 12/26/2007 - 3:54pm
No, 1.4 is not required yet but it will be shortly - right now, if you are using 1.2 and want to use Bandwidth.com trunks, they put your box on an old system for serving 1.2 clients - this is the same system that went down a few months ago.
Rhino? We have standardized on their Analog cards (4FXO and 8FXO) but I have not tried their Digital cards - I have been wanting to try them - several people have said they are excellent (their Analog cards sure are) - Post your results when you try one - I have only had to replace one Analog Rhino card, and VoIPSupply replaced it no questions asked.
I have had really good results with Digium, Sangoma, and Rhino with regards to Tech Support, so it's really hard to pick a favorite - maybe go round-robin and buy 1 from each when you need a card and support them all?
Greg
jfinstrom (James) belongs to
Wed, 12/26/2007 - 7:01pm
jfinstrom (James) belongs to this forum and works for Rhino, poor guy, we probably Sangoma him to death, but Rhino came a slightly late in the game for the echo cancelers, but they are great stuff, and the price is right.
and I have yet to find a nvidia chipset that works well on any voice card.
I only use Sangoma cards and
Thu, 12/27/2007 - 1:09pm
I only use Sangoma cards and haven't had any problems with Nvidia chipsets and the A200. My SOHO PBX runs on the ASUS M2MPV-VM which uses the Nforce 430 chipset and Athlon x2 5000.
I have another SOHO customer using an older Nvidia chipset and Sempron CPU with a Sangoma A200.
These are both running CentOS 5.x/Asterisk 1.4.x/FreePBX 2.3.x
they may work ok on small
Thu, 12/27/2007 - 5:57pm
Member Since:
2007-01-16