Haven’t been able to post in a while as my current project is keeping me very busy!
As the core multi-cluster deployment is now complete, we’ve moved on to site deployments, which has posed a significant challenge as I am attempting to streamline and automate the user and site configuration for about 3000 sites spread across 9 leaf clusters.
One of challenges was to define a working SRST template to account for some very specific business requirements, including differentiated Class of Restriction in SRST for older SCCP and newer SIP models. We also opted not to pursue SRST Manager, so a balance was struck between feature support and limiting configuration overhead, as the client’s support team is quite small for the scope of deployment, and simplicity was the key. This was all finalized and lab-tested and we were ready to proceed to site deployments and testing SRST on live sites.
However, at the first site test SRST failed! Problem description was:
- MGCP fallback was successful
- SCCP and SIP phones registered successfully
- SCCP devices were able to make/receive external calls
- SIP devices could make external calls, but could not receive external calls
- SIP devices could not make/receive internal calls
- When making a call from a SIP device, no ring-back was heard, and the remote party’s phone didn’t ring.
Based on the problem description, I was immediately able to determine the following:
- Problem was a SIP signalling issue
- No ring back – calling party was never receiving 18X
- No called party ringing – remote party never received, or wasn’t processing the INVITE from IOS gateway
Very strange, as this was working in the lab!
I looked at my basic voice service voip and voice register global configuration, and could not find anything a-miss:
!
voice service voip
allow-connections sip to sip
allow-connections h323 to h323
allow-connections sip to h323
allow-connections h323 to sip
supplementary-service sip refer
supplementary-service sip moved-temporarily
fax protocol t38 version 0 ls-redundancy 0 hs-redundancy 0 fallback pass-through g711alaw
sip
registrar server
bind all source <loopback-interface>
g729 annexb-all
asymmetric payload full
!!
voice register global
timeouts interdigit 5
system message Service Interruption
max-dn <#>
max-pool <#>
timezone 29
dialplan-pattern 1 0XXXXXX… extension-length 4 extension-pattern X… demote
!
So, time to look at the call flow signalling… I pulled debug ccsip messages from the gateway
Sample SIP Trace
005298: Oct 29 10:24:38.192: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
INVITE sip:0223333733@192.168.1.1;user=phone SIP/2.0
Via: SIP/2.0/UDP 10.1.2.3:5060;branch=z9hG4bK03efbc86
From: “Mr End User” <sip:0223333780@192.168.1.1>;tag=c47295a8075762c24ee36b42-7321fcdb
To: <sip:0223333733@192.168.1.1>
Call-ID: c47295a8-075701fc-0adbec71-182993ef@10.1.2.3
Max-Forwards: 70
Date: Thu, 29 Oct 2015 10:24:37 GMT
CSeq: 101 INVITE
User-Agent: Cisco-CP7841/10.3.1
Contact: <sip:d029fdee-4ea8-b62e-793a-8db0132ff40f@10.1.2.3:5060;transport=udp>
Expires: 180
Accept: application/sdp
Allow: ACK,BYE,CANCEL,INVITE,NOTIFY,OPTIONS,REFER,REGISTER,UPDATE,SUBSCRIBE,INFO
Remote-Party-ID: “Mr End User” <sip:0223333780@192.168.1.1>;party=calling;id-type=subscriber;privacy=off;screen=yes
Supported: replaces,join,sdp-anat,norefersub,resource-priority,X-cisco-srtp-fallback,X-cisco-xsi-8.5.1
Allow-Events: kpml,dialog
Content-Length: 350
Content-Type: application/sdp
Content-Disposition: session;handling=optionalv=0
o=Cisco-SIPUA 19428 0 IN IP4 10.1.2.3
s=SIP Call
t=0 0
m=audio 31580 RTP/AVP 9 0 8 116 18 101
c=IN IP4 10.1.2.3
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:116 iLBC/8000
a=fmtp:116 mode=20
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv005327: Oct 29 10:24:38.196: //65387/17A79D7C8B2A/SIP/Msg/ccsipDisplayMsg:
Sent:
SIP/2.0 100 Trying
Via: SIP/2.0/UDP 10.1.2.3:5060;branch=z9hG4bK03efbc86
From: “Mr End User” <sip:0223333780@192.168.1.1>;tag=c47295a8075762c24ee36b42-7321fcdb
To: <sip:0223333733@192.168.1.1>
Date: Thu, 29 Oct 2015 10:24:38 GMT
Call-ID: c47295a8-075701fc-0adbec71-182993ef@10.1.2.3
CSeq: 101 INVITE
Allow-Events: kpml, telephone-event
Server: Cisco-SIPGateway/IOS-15.4.3.M1
Content-Length: 0
005361: Oct 29 10:24:38.200: //65388/17A79D7C8B2A/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:0223333733@10.2.3.4:5060 SIP/2.0
Via: SIP/2.0/UDP 10.0.0.1:5060;branch=z9hG4bK4001264
Remote-Party-ID: “Mr End User” <sip:3780@10.0.0.1>;party=calling;screen=yes;privacy=off
From: “Mr End User” <sip:3780@10.0.0.1>;tag=A32FEE28-18F2
To: <sip:0223333733@10.2.3.4>
Date: Thu, 29 Oct 2015 10:24:38 GMT
Call-ID: 17A8D664-7D5E11E5-8B30FF52-8FAD66E0@10.0.0.1
Supported: 100rel,timer,resource-priority,replaces
Min-SE: 1800
Cisco-Guid: 0396860796-2103316965-2334850898-2410505952
User-Agent: Cisco-SIPGateway/IOS-15.4.3.M1
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 101 INVITE
Timestamp: 1446114278
Contact: <sip:3780@10.0.0.1:5060>
Expires: 180
Allow-Events: kpml, telephone-event
Max-Forwards: 69
Content-Type: application/sdp
Content-Disposition: session;handling=required
Content-Length: 324v=0
o=CiscoSystemsSIP-GW-UserAgent 7919 527 IN IP4 10.0.0.1
s=SIP Call
c=IN IP4 10.0.0.1
t=0 0
m=audio 24612 RTP/AVP 0 8 18 101
c=IN IP4 10.0.0.1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=yes
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20005363: Oct 29 10:24:38.700: //65388/17A79D7C8B2A/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:0223333733@10.2.3.4:5060 SIP/2.0
Via: SIP/2.0/UDP 10.0.0.1:5060;branch=z9hG4bK4001264
Remote-Party-ID: “Mr End User” <sip:3780@10.0.0.1>;party=calling;screen=yes;privacy=off
From: “Mr End User” <sip:3780@10.0.0.1>;tag=A32FEE28-18F2
To: <sip:0223333733@10.2.3.4>
Date: Thu, 29 Oct 2015 10:24:38 GMT
Call-ID: 17A8D664-7D5E11E5-8B30FF52-8FAD66E0@10.0.0.1
Supported: 100rel,timer,resource-priority,replaces
Min-SE: 1800
Cisco-Guid: 0396860796-2103316965-2334850898-2410505952
User-Agent: Cisco-SIPGateway/IOS-15.4.3.M1
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 101 INVITE
Timestamp: 1446114278
Contact: <sip:3780@10.0.0.1:5060>
Expires: 180
Allow-Events: kpml, telephone-event
Max-Forwards: 69
Content-Type: application/sdp
Content-Disposition: session;handling=required
Content-Length: 324v=0
o=CiscoSystemsSIP-GW-UserAgent 7919 527 IN IP4 10.0.0.1
s=SIP Call
c=IN IP4 10.0.0.1
t=0 0
m=audio 24612 RTP/AVP 0 8 18 101
c=IN IP4 10.0.0.1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=yes
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20
Suddenly the problem was immediately apparent, but made no sense! So, what is happening here?
- Side A phone IP : 10.1.2.3
- Side B phone IP : 10.2.3.4
- IOS Loopback IP : 192.168.1.1
- IOS Ethernet IP : 10.0.0.1
As we can see from the trace:
- Despite the Loopback binding, the IOS device was sourcing outbound SIP traffic from the outbound physical interface, instead of the loopback
- Outbound INVITE is sent to Side B, but is clearly never processed, as IOS never receives a 1XX Trying/Ringing. As a result, IOS never sends on a 18X Ringing to Side A, and ring back is never played to the calling party
- Multiple INVITEs are re-sent to Side B, which never responds to the call (call setup eventually ended with a CANCEL from Side A)
Why is this happening!? Well, the SRST reference in the phone was the Loopback IP, and as a result, the phones registered to the Loopback interface in SRST mode. As such, they are only going to respond to call setup signalling sourced from this IP – Side B is effectively ignoring the call setup request.
As I’m sure you can guess, this behaviour is not expected, and I immediately thought this was a bug. What had changed between lab testing and production? IOS version… The remote deployment teams had been deploying routers without an upgrade to the Cisco-recommended version that we had tested in the lab. To validate my conclusion, I looked on cisco.com Bug Search, and got a 1st time hit on the following:
https://tools.cisco.com/bugsearch/bug/CSCur95578
CME-SIP call fails when sip bind is applied under voice service voip
CSCur95578
Description
Symptom:
When a voice gateway falls back to the SRST mode, system message that is configured in the “voice register gloal” section is not displayed on SIP phones
Conditions:
– Voice gateway use the loopback IP address as CME source address
– SIP phones lost the communication to Cisco Unified Call Manager and fall back to SRST mode
Workaround:
Downgrade to an IOS without the fix for CSCup30934 Use a physical IP as CME source address.
We upgraded to the recommended IOS version and re-tested – with immediate positive results.
Conclusion:
This defect outlined a very important point regarding SRST/CME. In non-trivial deployments with multiple LAN interfaces (or if there are WAN-remote registration devices), always source traffic from a bound interface when traffic is destined to internal devices. In our case, I could have removed the bind, but the gateway was redundantly connected to 2 core switches with Layer 3 links, which was always going to pose a problem. Almost without fail, a loopback is the preferred option.
However… some may correctly comment that this may break ITSP call flows if a local SIP Trunking solution is in use – it will, and must be accounted for. In my base the remote site used ISDN technology, but when a SIP Trunking solution is in use, the ITSP-facing dial peers should be referenced with voice-class sip bind commands.
Great links on this topic here:
#dontcalltac
