Regular static sounds
Submitted by Atamido on Wed, 03/19/2008 - 8:14am.
I've discovered a rather serious audio issue with a new phone system we are going to be using. Aside from being disruptive, it is killing our faxes. We are using Trixbox 2.2.9, with a Sangoma A102DE, connected to a single PRI on port 1. This is running in a Dell PowerEdge 2950.
It's a bit complex, so let me give the criteria.
1. There is a random bit of static for part of a second.
2. After it happens, the sound seems to happen again after about 42 seconds, and then again 10 seconds later, repeating every 42 and then 10 seconds.
3. It only happens with calls on the T1.
4. Have not heard the sound while making SIP to SIP calls. (phone to phone or phone to Exchange)
5. Only happens if sound is outgoing over T1. If I call in/out to a phone on Trixbox, and have the phone on Trixbox muted, then I don't hear the sound.
6. If I use a phone connected to Trixbox and call a DID through the PRI back to Trixbox, the issue still occurs.
7. The sound occurs using an outside phone and calling in to an auto attendant running on Trixbox.
I honestly have no idea if the issue is with the echo cancellation, drivers for the T1 card, the T1 card, the T1 circuit, or something in the telco's central office.
Sample audio file
Wed, 03/19/2008 - 8:31am
Here is a recorded call where you can hear the static sound. I called from a phone connected to the Trixbox, to a DID that routed out and back in over the PRI to the Trixbox, and am just listening to the auto attendant. You can hear the static at about 2 seconds and 11 seconds.
http://files.commo.de/static-clip.wav
[root@asterisk ~]#
Wed, 03/19/2008 - 8:37am
[root@asterisk ~]# lspci
00:00.0 Host bridge: Intel Corporation 5000X Chipset Memory Controller Hub (rev 12)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev 12)
00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev 12)
00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 (rev 12)
00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev 12)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 (rev 12)
00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev 12)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev 12)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev 12)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev 12)
00:1c.0 PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1 (rev 09)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09)
00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09)
00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09)
00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09)
01:00.0 PCI bridge: Intel Corporation 80333 Segment-A PCI Express-to-PCI Express Bridge
01:00.2 PCI bridge: Intel Corporation 80333 Segment-B PCI Express-to-PCI Express Bridge
02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller 5i
04:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
06:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01)
06:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01)
07:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01)
07:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01)
08:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
09:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
0e:00.0 PCI bridge: PLX Technology, Inc. PEX 8111 PCI Express-to-PCI Bridge (rev 21)
0f:04.0 Network controller: Sangoma Technologies Corp. A200/Remora FXO/FXS Analog AFT card
11:0d.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
[root@asterisk ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 403207461 403212283 403208310 403202626 IO-APIC-edge timer
1: 3 1 1 5 IO-APIC-edge i8042
8: 1366 1325 1364 1324 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-level acpi
12: 17 14 18 16 IO-APIC-edge i8042
14: 15 323 16 290 IO-APIC-edge ide0
169: 234891468 571730582 197999726 608609668 IO-APIC-level wanpipe1, wanpipe2
177: 3 3 6 6 IO-APIC-level ehci_hcd, uhci_hcd, uhci_hcd
185: 13 21 16 13 IO-APIC-level uhci_hcd
193: 64240 35114 463053 275953 IO-APIC-level megasas
217: 0 0 0 859070 PCI-MSI eth0
225: 0 983102 0 0 PCI-MSI eth1
NMI: 0 0 0 0
LOC: 1612906369 1612906986 1612906985 1612906990
ERR: 0
MIS: 0 The IRQ for your T1 card is
Wed, 03/19/2008 - 8:43am
The IRQ for your T1 card is 169 -- that's a "virtual" IRQ and will cause the problems you're having.
If you can, do the following:
1) Edit /boot/grub.conf and add "apci=off noapic" to the end of your default kernel line.
2) turn off all unneeded hardware in your bios to reduce used IRQs (serial ports, usb ports, everything you're not using.)
3) if possible, assign the Sangoma card an IRQ in the BIOS
4) try moving the Sangoma card to another PCI slot.
Your goal is to get the Sangoma card to an IRQ under 15, and make sure it's not shared. IRQs 15 and below are real, honest IRQs, and are what you need to have your T1 card running on.
Changed to this line in
Wed, 03/19/2008 - 11:44am
Changed to this line in /boot/grub/grub.conf
kernel /vmlinuz-2.6.9-34.0.2.ELsmp ro root=LABEL=/1 apci=off noapic
Which gave me this:
[root@asterisk ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 448813 0 0 0 XT-PIC timer
1: 9 0 0 0 XT-PIC i8042
2: 0 0 0 0 XT-PIC cascade
4: 455430 0 0 0 XT-PIC wanpipe1, wanpipe2
6: 8242 0 0 0 XT-PIC megasas
8: 4 0 0 0 XT-PIC rtc
9: 0 0 0 0 XT-PIC acpi
10: 63 0 0 0 XT-PIC uhci_hcd
11: 2 0 0 0 XT-PIC ehci_hcd, uhci_hcd
12: 65 0 0 0 XT-PIC i8042
14: 642 0 0 0 XT-PIC ide0
65: 0 0 0 99 PCI-MSI eth0
73: 0 0 14539 0 PCI-MSI eth1
NMI: 0 0 0 0
LOC: 448549 449230 449229 449228
ERR: 1
MIS: 0I also tried changing the PCIe slot it was plugged in to, disabling onboard ports and changing the interrupt of the card in the BIOS.
Unfortunately it still makes a small static sound at regular intervals. Any other ideas?
I also checked the w1g1
Wed, 03/19/2008 - 11:53am
I also checked the w1g1 interface for errors. It started with 35 overruns, but didn't increase despite hearing the static.
w1g1 Link encap:Point-to-Point Protocol
UP POINTOPOINT RUNNING NOARP MTU:8 Metric:1
RX packets:232267 errors:0 dropped:0 overruns:35 frame:35
TX packets:232267 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:8 (8.0 b) TX bytes:8 (8.0 b)
Interrupt:4 Memory:f89c0000-f89c1fff How do I check my firmware?
Wed, 03/19/2008 - 1:04pm
How do I check my firmware?
I called our telco and they said that there is an issue with the timing between us and their switch. I don't know what that means and the person on the other end of the line wasn't able to provide any additional information other than that this was most likely the result of a misconfiguration on our end.
We are using Time Warner in Texas, United States, North America. Do any of these config files look wrong?
/etc/zaptel.conf
# Autogenerated by /usr/local/sbin/sangoma/setup-sangoma -- do not hand edit # Zaptel Channels Configurations (zaptel.conf) # loadzone=us defaultzone=us #Sangoma A102 port 1 [slot:4 bus:15 span: 1] span=1,0,0,esf,b8zs bchan=1-23 dchan=24 #Sangoma A102 port 2 [slot:4 bus:15 span: 2] span=2,0,0,esf,b8zs bchan=25-47 dchan=48
/etc/asterisk/zapata-auto.conf
; Autogenerated by /usr/local/sbin/sangoma/setup-sangoma -- do not hand edit ; Zaptel Channels Configurations (zapata.conf) ; ; This is not intended to be a complete zapata.conf. Rather, it is intended ; to be #include-d by /etc/zapata.conf that will include the global settings ; callerid=asreceived ;Sangoma A102 port 1 [slot:4 bus:15 span: 1] switchtype=national context=from-zaptel group=0 signalling=pri_cpe channel => 1-23 ;Sangoma A102 port 2 [slot:4 bus:15 span: 2] switchtype=national context=from-zaptel group=0 signalling=pri_cpe channel => 25-47
/etc/wanpipe/wanpipe1.conf
#================================================ # WANPIPE1 Configuration File #================================================ # # Date: Wed Dec 6 20:29:03 UTC 2006 # # Note: This file was generated automatically # by /usr/local/sbin/setup-sangoma program. # # If you want to edit this file, it is # recommended that you use wancfg program # to do so. #================================================ # Sangoma Technologies Inc. #================================================ [devices] wanpipe1 = WAN_AFT_TE1, Comment [interfaces] w1g1 = wanpipe1, , TDM_VOICE, Comment [wanpipe1] CARD_TYPE = AFT S514CPU = A CommPort = PRI AUTO_PCISLOT = NO PCISLOT = 4 PCIBUS = 15 FE_MEDIA = T1 FE_LCODE = B8ZS FE_FRAME = ESF FE_LINE = 1 TE_CLOCK = MASTER TE_REF_CLOCK = 0 TE_HIGHIMPEDANCE = NO LBO = 0DB FE_TXTRISTATE = NO MTU = 1500 UDPPORT = 9000 TTL = 255 IGNORE_FRONT_END = NO TDMV_SPAN = 1 TDMV_DCHAN = 24 [w1g1] ACTIVE_CH = ALL TDMV_ECHO_OFF = NO TDMV_HWEC = YES
/etc/wanpipe/wanpipe2.conf
#================================================ # WANPIPE1 Configuration File #================================================ # # Date: Wed Dec 6 20:29:03 UTC 2006 # # Note: This file was generated automatically # by /usr/local/sbin/setup-sangoma program. # # If you want to edit this file, it is # recommended that you use wancfg program # to do so. #================================================ # Sangoma Technologies Inc. #================================================ [devices] wanpipe2 = WAN_AFT_TE1, Comment [interfaces] w2g1 = wanpipe2, , TDM_VOICE, Comment [wanpipe2] CARD_TYPE = AFT S514CPU = A CommPort = PRI AUTO_PCISLOT = NO PCISLOT = 4 PCIBUS = 15 FE_MEDIA = T1 FE_LCODE = B8ZS FE_FRAME = ESF FE_LINE = 2 TE_CLOCK = NORMAL TE_REF_CLOCK = 0 TE_HIGHIMPEDANCE = NO LBO = 0DB FE_TXTRISTATE = NO MTU = 1500 UDPPORT = 9000 TTL = 255 IGNORE_FRONT_END = NO TDMV_SPAN = 2 TDMV_DCHAN = 24 [w2g1] ACTIVE_CH = ALL TDMV_ECHO_OFF = NO TDMV_HWEC = YES
Make that span line
Wed, 03/19/2008 - 1:09pm
Make that span line say:
1,1,0
otherwise you're telling the system that your box is the timing source, and you want the timing source to be the telco.
See this page:
http://www.voip-info.org/wiki/index.php?page=Zaptel.conf+span+syn...
for more information.
Quote:
Use '1' if you want to use the circuit as your primary sync source. If '0' is used asterisk will try to provide timing to the span (say, if you were connecting to a legacy PBX). If Asterisk is connected directly to the telco you will want to use '1' to accept timing from them. If you have multiple spans, set them as 2, 3, 4, etc.
I
Thu, 03/20/2008 - 10:10am
I changed
span=1,0,0,esf,b8zs
in /etc/zaptel.conf to
span=1,1,0,esf,b8zs
Unfortunately it doesn't seem to have made any difference. I also tried the same for the second port and connected the cable to it, but it still had the same issue. Would any other settings be overriding this?
If I'm not mistaken, there's
Thu, 03/20/2008 - 10:25am
From your wanpipe1.conf
Thu, 03/20/2008 - 10:34am
Something recent to consider
Tue, 07/08/2008 - 8:17pm
I've been working on this problem for about five months and have the following to offer. I have two particular installations with very similar hardware, the same CLEC and LEC, and the same problem as illustrated in the audio sample above, earlier in this thread.
I have one Sangoma A104D-X at customer site A, and one A102D at customer site B.
Site A, shows the wanpipe drivers handled via virtual IRQ 185, which unfortunately is shared with the SCSI RAID (it is a DL380 G5 series box from HP) controller I believe:
185: 11737468 2712 2682 8679494 IO-APIC-level cciss0, uhci_hcd, wanpipe1, wanpipe2, wanpipe3, wanpipe4
and, Site B, shows the wanpipe drivers handled via hard IRQ 10, which unfortunately is shared with eth0 (it is a Dell PE 840):
10: 76644109 0 XT-PIC eth0, wanpipe1, wanpipe2
The most interesting piece of evidence is that Site B was running fine for about two years as shown above w/respect to IRQs, until one of the two PRIs went RED alarm at the Network Interface and the LEC was dispatched by the long distance carrier (CLEC also) to fix the problem with the local loop. The next day, the customer from Site B called several times to report audio distortion, similar to harmonic distortion on most outgoing calls. IOW, they never had the problem until the LEC "fixed" the circuit. No changes were made on their system for at least three months prior, maybe longer.
The customer at Site A, which has the four port version of the card, has had the problem since day 1 of installation, which was about nine months ago. It didn't happen very often, but under certain conditions the problem was really frequent. We have two different LD carriers, one of them on w1g1, and w2g1. The other carrier, where we see the biggest problem is on both w3g1, and w4g1. In terms of volume this site (Site A), handles about 11,000 calls per day across this A104D card. At the worst, about 1% or less of the calls had the harmonic distortion problem. But, one is enough if it happens to the wrong person!
I experienced the audio distortion myself several times at both Site A and Site B while doing testing over long periods of time. I spoke with Sangoma Tech Support and the first thing they said is that all of the A10xD (PCI-X or PCIe versions) series cards have only one chip handling echo cancellation, but it derives its timing source from the first use T1/PRI port on the card in ascending order. So, if you have your PRIs in ports 1, 2, and 3, your signalling source for the EC chip is port 1. The problem they said is that the LEC could have a very, very small difference in timing between each circuit or some of the PRI circuits, which will definetly cause the EC to distort the calls in the manner I described earlier. This distortion is consistent with the one sampled by someone earlier in this thread (so listen to if if you are curious). Our customers call it the "under-water effect".
Anyway, to make a long story short....I set TDMV_HWEC = NO, instead of the default of YES at site B, and the problem has completely disappeared. What I want to know is what did the LEC do at Site B on the day their "repaired" the PRI circuit? And two, what can I do long term to fix this problem at sites with multiple PRIs without having to disabled the hardware based EC I am paying extra for?
/etc/wanpipe/wanpipe1.conf from Site B after problem/effect went away:
[devices]
wanpipe1 = WAN_AFT_TE1, Comment
[interfaces]
w1g1 = wanpipe1, , TDM_VOICE, Comment
[wanpipe1]
CARD_TYPE = AFT
S514CPU = A
CommPort = PRI
AUTO_PCISLOT = NO
PCISLOT = 4
PCIBUS = 20
FE_MEDIA = T1
FE_LCODE = B8ZS
FE_FRAME = ESF
FE_LINE = 1
TE_CLOCK = NORMAL
TE_REF_CLOCK = 0
TE_HIGHIMPEDANCE = NO
LBO = 0DB
FE_TXTRISTATE = NO
MTU = 1500
UDPPORT = 9000
TTL = 255
IGNORE_FRONT_END = NO
TDMV_SPAN = 1
TDMV_DCHAN = 24
[w1g1]
ACTIVE_CH = ALL
TDMV_ECHO_OFF = NO
TDMV_HWEC = NO
Some of the other things we tried along the way, which helped Site A considerably, but not entirely was to increase the MTU size from 8 to 80, and to do this we had to make a change to the Zaptel source and recompile wanrouter and of course zaptel drivers. The load on the system's IRQ bus/processing went down by a factor of 10. There is an article on Sangoma's Wiki about this at Sangoma Wiki Reference and it explains the whole process and why you might want to consider doing this with larger installations like ours at Site A (96 trunks).
ifconfig output below:
w1g1 Link encap:Point-to-Point Protocol
UP POINTOPOINT RUNNING NOARP MTU:80 Metric:1
RX packets:18289235 errors:0 dropped:0 overruns:13 frame:13
TX packets:18289235 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:80 (80.0 b) TX bytes:80 (80.0 b)
Interrupt:185 Memory:f8aa0000-f8aa1fff
w2g1 Link encap:Point-to-Point Protocol
UP POINTOPOINT RUNNING NOARP MTU:80 Metric:1
RX packets:18288095 errors:0 dropped:0 overruns:10 frame:10
TX packets:18288095 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:80 (80.0 b) TX bytes:80 (80.0 b)
Interrupt:185 Memory:f8aa0000-f8aa1fff
w3g1 Link encap:Point-to-Point Protocol
UP POINTOPOINT RUNNING NOARP MTU:80 Metric:1
RX packets:18288091 errors:0 dropped:0 overruns:10 frame:10
TX packets:18288091 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:80 (80.0 b) TX bytes:80 (80.0 b)
Interrupt:185 Memory:f8aa0000-f8aa1fff
w4g1 Link encap:Point-to-Point Protocol
UP POINTOPOINT RUNNING NOARP MTU:80 Metric:1
RX packets:18287430 errors:0 dropped:0 overruns:6 frame:6
TX packets:18287430 errors:0 dropped:0 overruns:0 carrier:10
collisions:0 txqueuelen:100
RX bytes:160 (160.0 b) TX bytes:160 (160.0 b)
Interrupt:185 Memory:f8aa0000-f8aa1fff
The overruns you see above are very small in relation to the number of RX/TX packets over the last two days since the system was restarted, and not a source of their problem.
Ultimately, i am about to turn off the EC on all four ports for Site A and reboot and start testing again tomorrow morning and I'll post the results here in the next four weeks.
Comments and information
Tue, 07/08/2008 - 9:18pm
Very detailed post, please realize I have never used Sangoma or any other PRI cards in an Asterisk system we use external SIP gateways (Cisco) for may reasons, this however is not a cost effective option for most users.
With this in mind I do have quite a bit of experience with T1 lines. The statement you made of most interest to me is this:
Quote:
but it derives its timing source from the first use T1/PRI port on the card in ascending order.
I am assuming that the echo canceler timing is unrelated to the line timing. However the symptoms you are describing are exactly what would happen if timing was slowly slipping, you would eventually take frame errors of sufficient quantity to cause audible artifacts in the recovered audio.
I am making the assumption that all of the PRI's are served out of the same telco CO, however when they repaired the circuit your DXC assignment could have changed, this would alter the timing. You may have been 'getting away' with this master/slave timing arrangement because the PRI's where originally provisioned in a single group. The telco tech may have pulled history from the smart jack and seen the framing errors and performed a pair swap that resulted in the changes to the circuits timing characteristic.
Each T1 line must be independently timed and receive clocking from the network. The only time to generate clocking from the card would be in "back to back" arrangement with another DTE device.
Your post made me very curious and I found these statements on the Sangoma support site:
Quote:
* If the wanpipe T1/E1 port looses connection to the TELCO, it will automatically switch to its internal oscillator. In this case all slave ports will still be synchronized.
* If there is no clock coming from Telco (Port 1 in example), then the port using the referenced clock (Port 2 in example) will start using the internal oscillator. If your port 1 clock is unstable, this can cause noise and cracklings on the line as the clock source is being switched continuously between the referenced port and the internal oscillator.
This makes me believe that Wanpipe driver is very presumptuous about what to do with it's internal clocking reference in the event of timing issues.
If any of the symptoms appear I would keep my eye on wanpipemon. It looks like it has a wealth of T1 performance history (which makes sense, the card has an integrated CSU).
Best of luck on your troubleshooting and please keep us posted.
Scott
--
Scott
aka "Skyking"
Member Since:
2006-06-16