Tag Archives: bug

VMware Tools 10.0 Update Failures Creates Outage – CSCux90747, CSCva09767, CSCuy84075

A friend hit this delightful defect this week:

 

 

There are several workarounds, but ultimately there’s a COP file for this – ciscocm.VMwareTools2016c.cop.sgn.  The attached defects cover several initial install and help-me-its-dead scenarios.

 

It’s a bit of a horror.  Watch out for those VMWare tools updates!

 

#dontcalltac

Tagged , , , , , , , , ,

CSCux67499 – SQL NullPointer Exception causes CUC Subscriber Sync Failures

The later version (11.0 – 11.2) of PCP have a fairly painful caveat when it comes to Unity Connection.  CUC is currently afflicted by a(nother) horrid Informix problem that will cause subscriber sync failures.

 

The defect is below:

 

Unfortunately, there’s no workaround available as per the bug description:

 

Symptom:
User Sync of Unity Connection fails on Prime Collaboration Tool (PCP).
NullPointerException exception is seen in Nice log (PCP).

System or internal error java.lang.NullPointerException
java.sql.SQLException: System or internal error java.lang.NullPointerException
at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:488)
at com.informix.jdbc.IfxSqli.a(IfxSqli.java:9547)
at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2634)
at com.informix.jdbc.IfxSqli.a(IfxSqli.java:1830)
at com.informix.jdbc.IfxSqli.executeStatementQuery(IfxSqli.java:1768)
at com.informix.jdbc.IfxSqli.executeStatementQuery(IfxSqli.java:1699)
at com.informix.jdbc.IfxResultSet.a(IfxResultSet.java:210)
at com.informix.jdbc.IfxStatement.executeQueryImpl(IfxStatement.java:1237)
at com.informix.jdbc.IfxPreparedStatement.executeQuery(IfxPreparedStatement.java:382)
at dfc.ipt.cisco.unity.util.UMPUtil.executeQuery(UMPUtil.java:247)

Conditions:
1) PCP User Sync in executed for Unity Connection
2) Having 5K or 20K End Users record in CUC11.x or 10.x

Workaround:
None

 

Please take note of the following, as this does affect planned PCP upgrades:

 

 

Nice work from the technical documentation team in listing out the caveat in the upgrade guide!

 

Before upgrading, you need to install an engineering special in Cisco Unity Connection(10.5.x & 11.0) for successful subscriber synchronization. Contact TAC to obtain the Cisco Unity Connection engineering special for the bug (CSCux67499). If you are using Cisco Unity Connection release below 10.5, then you must upgrade to Cisco Unity Connection version 10.5.x and then apply the engineering special.

 

A call to TAC to request an ES is going to be required (as of the date of this blog post).  Cross-reference the bug “Fixed In” details and check if this has been included in a later Unity Connection SU.  If in doubt, call TAC for  assistance.

 

 

Tagged , , ,

CSCup36072 – CUCM/CUC Patch Upgrade Fails

Hit another horror defect today…  Patch upgrades apparently do not work as of 10.5.x and up!

 

Selection_041

 

The defect is described below:

 

 

The bug details are below – only workaround is to install with a bootable image!

 

Symptom:
When doing a fresh install of a Call Manager node, there is an option to use an upgrade patch option.

For CUCM 10.5, this approach cannot be used. Not even by using the bootable image for 10.0 and upgrade file for 10.5

Conditions:
Installing CUCM 10.5 with bootable image of a lower version and upgrade file of CUCM 10.5

Workaround:
None -> Use Bootable image for CUCM 10.5 itself

 

To resolve, the only “supported” method is to request a bootable image from TAC.

 

Terrible.  Terrible.  Terrible.

Tagged , , , , ,

CSCug20588 – Auto-Reg Failure – “Could not insert new row – duplicate value in a UN”

Had an issue that’s affected multiple deployments that I’ve been working on.  At one of my clients in particular, this reached an escalation level.  Just sharing this to assist others, as we went through 4 TAC Engineers to get a root cause and eventually this was attributed to a hidden Cisco defect in 10.x and above…  The first engineer told us that we needed to rebuild an 18 node CUCM cluster from scratch to fix this!

Just sharing this to assist others, as we went through 4 TAC Engineers to get a root cause and eventually this was attributed to a hidden Cisco defect in 10.x and above…  The first engineer told us that we needed to rebuild an 18 node CUCM cluster from scratch to fix this!  Don’t always believe TAC!

 

Problem Description

 

  • Client complains that auto-registered phones are failing to register
  • Eventually determined that the phone had been previously registered and what subsequently deleted
  • Certain clients say that if they manually configure the phone, it registers
  • Phone logs show a TFTP config error

 

Went on site and noted that there were persistent DB-related syslog alarms being generated.

 

The SDL logs showed the registration failure and a syslog event was logged:

 

00024541.012 |07:28:21.243 |AppInfo |SIPStationD(269) - checkDNsReceived: TotalCount=0
00024541.013 |07:28:21.243 |AppInfo |SIPStationD(269) - sendLineRegisterReq: DN match failed for AUTO-REG
00024541.014 |07:28:21.243 |AppInfo |SIPStationD(269) - sendLineRegisterReq: Phone is autoregistering...sending 200OK followed by a restart request
00024541.015 |07:28:21.243 |AppInfo |SIPStationD(269) - buildServiceControl: Sending callid from mPrimaryLineNum 0: 0UTqUn-9I4FebBfs1swQbo.GQ78HIxyf
00024541.016 |07:28:21.243 |AppInfo |SIPStationD(269) - sendServiceControlNotify: starting servicecontrol NOTIFY timer
00024541.017 |07:28:21.243 |AppInfo |SIPStationD(269) - sendServiceControlNotify: sending service-control NOTIFY
00024541.018 |07:28:21.243 |AppInfo |EndPointTransientConnection - An endpoint attempted to register but did not complete registration Connecting Port:5060 Device name:SEPAAAABBBBCCCC Device IP address:192.168.100.123 Device type:592 Reason Code:7 Protocol:SIP Device MAC address:AAAABBBBCCCC IPAddressAttributes:0 LastSignalReceived:SIPRegisterInd StationState:auto_register App ID:Cisco CallManager Cluster ID:MYCUCM Node ID:mycucmhostame
00024541.019 |07:28:21.243 |AlarmErr |AlarmClass: CallManager, AlarmName: EndPointTransientConnection, AlarmSeverity: Error, AlarmMessage: , AlarmDescription: An endpoint attempted to register but did not complete registration, AlarmParameters: ConnectingPort:5060, DeviceName:SEPAAAABBBBCCCC, IPAddress:192.168.100.123, DeviceType:592, Reason:7, Protocol:SIP, MACAddress:AAAABBBBCCCC, IPAddrAttributes:0, LastSignalReceived:SIPRegisterInd, StationState:auto_register, AppID:Cisco CallManager, ClusterID:MYCUCM, NodeID:mycucmhostame,
00024542.000 |07:28:21.243 |AppInfo |-->RISCMAccess::DeviceTransientConnection(...)
00024543.000 |07:28:21.243 |AppInfo |Device Transient deviceName : SEPAAAABBBBCCCC, IPAddress : 192.168.100.123, IPv6Address : not shown, IPv4Attribute :0, IPv6Attribute :0, Protocol : 2
00024544.000 |07:28:21.243 |AppInfo |DebugMsg deviceName : SEPAAAABBBBCCCC, DeviceType : 592, risClass: 1
00024545.000 |07:28:21.243 |AppInfo |<--RISCMAccess::DeviceTransientConnection(...)
00024541.020 |07:28:21.243 |AppInfo |SIPStationD(269) - Received register from Cisco phone for unconfigured line AUTO-REG; shutting down
00024541.021 |07:28:21.243 |AppInfo |StationTracking::updateMismatchCount: deviceId=SEPAAAABBBBCCCC, mismatchCount=174
00024541.022 |07:28:21.243 |AppInfo |SIPStationD(269) - processVersionStampMismatch: VersionStampMismatch detected, count = 174
00024541.023 |07:28:21.243 |AppInfo |SIPStationD(269) - DevStat-StopClose: LineRegisterReq send failure

 

 

And later we in the same trace see that CUCM sends a NOTIFY for the above:

 

NOTIFY sip:AUTO-REG@192.168.100.123:5060 SIP/2.0
Via: SIP/2.0/UDP 192.168.1.100:5060;branch=z9hG4bK2b1379d1903
From: <sip:192.168.1.100>;tag=789090571
To: <sip:AUTO-REG@192.168.100.123>
Call-ID: bb580e00-6df1b475-27b-247410ac@192.168.1.100
CSeq: 101 NOTIFY
Max-Forwards: 70
Date: Wed, 09 Mar 2016 05:28:21 GMT
User-Agent: Cisco-CUCM11.0
Event: service-control
Subscription-State: active
Contact: <sip:192.168.1.100:5060>
Content-Type: text/plain
Content-Length: 67

action=restart
RegisterCallId={0UTqUn-9I4FebBfs1swQbo.GQ78HIxyf}

 

I traced through the logs and saw that  soon after the NOTIFY was placed on the wire, the following was logged in SDL that highlighted the problem:

 

 

00024556.000 |07:28:21.385 |SdlSig |StationClose |wait |SIPHandler(1,100,80,1) |SIPStationInit(1,100,73,1) |1,100,10,1.591^192.168.100.123^* |[T:N-H:0,N:0,L:0,V:0,Z:0,D:0] CloseStationReason = 0 StationId = SEPAAAABBBBCCCC
00024546.004 |07:28:21.428 |AppInfo |DB: odbc::SQLException(Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UNIQUE INDEX column (Unique Index:assign).)
00024546.005 |07:28:21.429 |AppInfo |DBLException - An error occurred while performing database activities ErrorCode:-239 ExceptionString:Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN App ID:Cisco CallManager Cluster ID:MYCUCM Node ID:mycucmhostame
00024546.006 |07:28:21.429 |AlarmErr |AlarmClass: CallManager, AlarmName: DBLException, AlarmSeverity: Alert, AlarmMessage: , AlarmDescription: An error occurred while performing database activities, AlarmParameters: ErrorCode:-239, ExceptionString:Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN, AppID:Cisco CallManager, ClusterID:MYCUCM, NodeID:mycucmhostame,
00024546.007 |07:28:21.429 |AppInfo |dBProcs - ERROR odbc::SQLException addDevice()
00024546.008 |07:28:21.429 |AppInfo |ProcessDb - ERROR initializing_DbStationAutoRegisterReq - addDevice failed

 

So my initial thought was a bug with orphaned data.  Had to be something along this line as the SDL error was referencing a duplicate.  So , I relunctantly decided to go straight to  TAC for analysis.   I sent this off to TAC and got crickets…  The assigned engineer eventually told me this was an NTP issue which didn’t make sense at all.

Another colleague logged a similar case that we had at another client.  2 re-queues later we got a great engineer who identified  CSCug20588 as the cause of our problem.  It’s a hidden TAC case.

 

The fix?  Well, a RIDICULOUS bug!

 

When auto-registering a phone in 10.x the Auto-Reg config uses a UDT/ULT.  This results in a phone-specific Phone Button Template being created when a phone is inserted into the Informix database, of the form:

 

SEP<MAC-ADDRESS>-SIP-Individual Template

 

The damn PBT was causing the phone not to register!  This obvoiusly remained in the list of PBT’s when the phone was deleted from CUCM.  When it came to AUTO-REG the phone, CUCM clearly does not look to delete/re-add the PBT or simply use the existing one.  Thus the DB insertion error, and the problematic behaviour.

I deleted the PBT and immediately the phone registered.

There’s no fix for this in 10.x and we’re waiting on the BU to confirm an official 11.x SU that will contain the fix for this.  One to watch out for!

 

Tagged , , , ,

CUCM Video Codec Preferences Support – CSCuw53802

Colleague called me for assistance with a video call that wouldn’t set up when dialling in to a VC bridge.

The call flow was from a Polycom endpoint to the new Conductor-managed vTS bridge that I’d installed a week before.  The endpoint could dial into the Rendezvous VC bridge, but calls only had audio.  Some additional testing showed that this wasn’t working for point-to-point calls to Jabber as well, which helped us rule out the bridge completely.

My colleague pulled a Wireshark from Jabber, and the signalling confirmed a codec support issue on the Polycom:

Jabber INVITE with H.264:

Selection_035

However, the response clearly showed that our Polycom unit didn’t like this, and marked the video streams as inactive:

Selection_036

My colleague did some reading up on our somewhat outdated Polycom unit, and we ultimately found that some release keys and a firmware upgrade would be needed.  Customer not too happy, but nothing to do here other than upgrade…

However, my colleague did post a question to me:

Can we not just set some video codec preferences for this call, or at least for the call into the VC bridge?

Well, the answer is… no!

 

As of 11.0 this is not supported.  However, thankfully this does appear to have been considered in the roadmap for CUCM 11.5:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuw53802/

Selection_037

 

This is good news, as the CUCM-based video architecture is now quite mature, having gone through several iterations since it’s inception from CUCM 9.x.

As the market for Skype for Business grows, H.265 begins to create more and more headaches due to patent licensing problems, and Cisco subsequently begins to punt Thor as a potential future video standard, this new feature should hopefully solve some problems down the line.

 

#dontcalltac

Tagged , , , , , , , , , , , , , , ,

ISR G2 PVDM3 Video Conferencing Deprecated

If you are in the process of preparing for the CCIE lab, you will be interested and very happy to see this!  A terrible “feature” finally put out to pasture.  So many caveats and considerations to get this to work 😦

A very good move for Cisco as they continue to migrate their video architecture.

The Feature Deprecation Announcement is below:

http://www.cisco.com/c/en/us/products/collateral/unified-communications/tdm-gateways/bulletin-c25-735945.html

 

In 2009, the Cisco® Integrated Services Routers Generation 2 (ISR G2), along with the Cisco Packet Voice Digital Signal Processor Module (PVDM3), introduced an experimental video conferencing feature with Cisco IOS® Software Release 15.1(4)M. This feature enabled the ISR G2 to provide homogeneous video switching features for selected Cisco IP phone models with resolutions of 720p or lower, such as the Cisco Unified IP Phone 7985G, 9951, and 9971. The platform was limited to video switching and did not support video mixing or overlays.

Video transcoding was also possible for a few channels (fewer than 5) for point-to-point video calls using Cisco Unified Border Element (CUBE).

While the intention was to fully support this feature via the use of a Unified Conferencing Video license, the feature was never fully productized or supported. Cisco provided limited best effort support on an as-needed basis.

This feature will be disabled with Cisco IOS Software Release 15.5(3)M. This issue is addressed with the following defect:

CSCus97643: PVDM3: Cisco IOS Software based video transcoder and conferencing commands purged.

 

The bug itself can be viewed here.

For a full list of IOS-supported features that integrate with CUCM, please see the following:

http://www.cisco.com/c/en/us/products/collateral/routers/3900-series-integrated-services-routers-isr/data-sheet-c78-729824.html

A very useful document in my opinion.

Tagged , , , , , , , , , , , ,

SIP SRST Dialling Fails Due to Header Addressing – CSCur95578

Haven’t been able to post in a while as my current project is keeping me very busy!

As the core multi-cluster deployment is now complete, we’ve moved on to site deployments, which has posed a significant challenge as I am attempting to streamline and automate the user and site configuration for about 3000 sites spread across 9 leaf clusters.

One of challenges was to define a working SRST template to account for some very specific business requirements, including differentiated Class of Restriction in SRST for older SCCP and newer SIP models.  We also opted not to pursue SRST Manager, so a balance was struck between feature support and limiting configuration overhead, as the client’s support  team is quite small for the scope of deployment, and simplicity was the key.  This was all finalized and lab-tested and we were ready to proceed to site deployments and testing SRST on live sites.

However, at the first site test SRST failed!  Problem description was:

  • MGCP fallback was successful
  • SCCP and SIP phones registered successfully
  • SCCP devices were able to make/receive external calls
  • SIP devices could make external calls, but could not receive external calls
  • SIP devices could not make/receive internal calls
  • When making a call from a SIP device, no ring-back was heard, and the remote party’s phone didn’t ring.

Based on the problem description, I was immediately able to determine the following:

  1. Problem was a SIP signalling issue
  2. No ring back – calling party was never receiving 18X
  3. No called party ringing – remote party never received, or wasn’t processing the INVITE from IOS gateway

Very strange, as this was working in the lab!

I looked at my basic voice service voip and voice register global configuration, and could not find anything a-miss:

!
voice service voip
allow-connections sip to sip
allow-connections h323 to h323
allow-connections sip to h323
allow-connections h323 to sip
supplementary-service sip refer
supplementary-service sip moved-temporarily
fax protocol t38 version 0 ls-redundancy 0 hs-redundancy 0 fallback pass-through g711alaw
sip
registrar server
bind all source <loopback-interface>
g729 annexb-all
asymmetric payload full
!

!
voice register global
timeouts interdigit 5
system message Service Interruption
max-dn <#>
max-pool <#>
timezone 29
dialplan-pattern 1 0XXXXXX… extension-length 4 extension-pattern X… demote
!

So, time to look at the call flow signalling…  I pulled debug ccsip messages from the gateway

Sample SIP Trace

005298: Oct 29 10:24:38.192: //-1/xxxxxxxxxxxx/SIP/Msg/ccsipDisplayMsg:
Received:
INVITE sip:0223333733@192.168.1.1;user=phone SIP/2.0
Via: SIP/2.0/UDP 10.1.2.3:5060;branch=z9hG4bK03efbc86
From: “Mr End User” <sip:0223333780@192.168.1.1>;tag=c47295a8075762c24ee36b42-7321fcdb
To: <sip:0223333733@192.168.1.1>
Call-ID: c47295a8-075701fc-0adbec71-182993ef@10.1.2.3
Max-Forwards: 70
Date: Thu, 29 Oct 2015 10:24:37 GMT
CSeq: 101 INVITE
User-Agent: Cisco-CP7841/10.3.1
Contact: <sip:d029fdee-4ea8-b62e-793a-8db0132ff40f@10.1.2.3:5060;transport=udp>
Expires: 180
Accept: application/sdp
Allow: ACK,BYE,CANCEL,INVITE,NOTIFY,OPTIONS,REFER,REGISTER,UPDATE,SUBSCRIBE,INFO
Remote-Party-ID: “Mr End User” <sip:0223333780@192.168.1.1>;party=calling;id-type=subscriber;privacy=off;screen=yes
Supported: replaces,join,sdp-anat,norefersub,resource-priority,X-cisco-srtp-fallback,X-cisco-xsi-8.5.1
Allow-Events: kpml,dialog
Content-Length: 350
Content-Type: application/sdp
Content-Disposition: session;handling=optional

v=0
o=Cisco-SIPUA 19428 0 IN IP4 10.1.2.3
s=SIP Call
t=0 0
m=audio 31580 RTP/AVP 9 0 8 116 18 101
c=IN IP4 10.1.2.3
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:116 iLBC/8000
a=fmtp:116 mode=20
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv

005327: Oct 29 10:24:38.196: //65387/17A79D7C8B2A/SIP/Msg/ccsipDisplayMsg:
Sent:
SIP/2.0 100 Trying
Via: SIP/2.0/UDP 10.1.2.3:5060;branch=z9hG4bK03efbc86
From: “Mr End User” <sip:0223333780@192.168.1.1>;tag=c47295a8075762c24ee36b42-7321fcdb
To: <sip:0223333733@192.168.1.1>
Date: Thu, 29 Oct 2015 10:24:38 GMT
Call-ID: c47295a8-075701fc-0adbec71-182993ef@10.1.2.3
CSeq: 101 INVITE
Allow-Events: kpml, telephone-event
Server: Cisco-SIPGateway/IOS-15.4.3.M1
Content-Length: 0
005361: Oct 29 10:24:38.200: //65388/17A79D7C8B2A/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:0223333733@10.2.3.4:5060 SIP/2.0
Via: SIP/2.0/UDP 10.0.0.1:5060;branch=z9hG4bK4001264
Remote-Party-ID: “Mr End User” <sip:3780@10.0.0.1>;party=calling;screen=yes;privacy=off
From: “Mr End User” <sip:3780@10.0.0.1>;tag=A32FEE28-18F2
To: <sip:0223333733@10.2.3.4>
Date: Thu, 29 Oct 2015 10:24:38 GMT
Call-ID: 17A8D664-7D5E11E5-8B30FF52-8FAD66E0@10.0.0.1
Supported: 100rel,timer,resource-priority,replaces
Min-SE: 1800
Cisco-Guid: 0396860796-2103316965-2334850898-2410505952
User-Agent: Cisco-SIPGateway/IOS-15.4.3.M1
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 101 INVITE
Timestamp: 1446114278
Contact: <sip:3780@10.0.0.1:5060>
Expires: 180
Allow-Events: kpml, telephone-event
Max-Forwards: 69
Content-Type: application/sdp
Content-Disposition: session;handling=required
Content-Length: 324

v=0
o=CiscoSystemsSIP-GW-UserAgent 7919 527 IN IP4 10.0.0.1
s=SIP Call
c=IN IP4 10.0.0.1
t=0 0
m=audio 24612 RTP/AVP 0 8 18 101
c=IN IP4 10.0.0.1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=yes
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20

005363: Oct 29 10:24:38.700: //65388/17A79D7C8B2A/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:0223333733@10.2.3.4:5060 SIP/2.0
Via: SIP/2.0/UDP 10.0.0.1:5060;branch=z9hG4bK4001264
Remote-Party-ID: “Mr End User” <sip:3780@10.0.0.1>;party=calling;screen=yes;privacy=off
From: “Mr End User” <sip:3780@10.0.0.1>;tag=A32FEE28-18F2
To: <sip:0223333733@10.2.3.4>
Date: Thu, 29 Oct 2015 10:24:38 GMT
Call-ID: 17A8D664-7D5E11E5-8B30FF52-8FAD66E0@10.0.0.1
Supported: 100rel,timer,resource-priority,replaces
Min-SE: 1800
Cisco-Guid: 0396860796-2103316965-2334850898-2410505952
User-Agent: Cisco-SIPGateway/IOS-15.4.3.M1
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 101 INVITE
Timestamp: 1446114278
Contact: <sip:3780@10.0.0.1:5060>
Expires: 180
Allow-Events: kpml, telephone-event
Max-Forwards: 69
Content-Type: application/sdp
Content-Disposition: session;handling=required
Content-Length: 324

v=0
o=CiscoSystemsSIP-GW-UserAgent 7919 527 IN IP4 10.0.0.1
s=SIP Call
c=IN IP4 10.0.0.1
t=0 0
m=audio 24612 RTP/AVP 0 8 18 101
c=IN IP4 10.0.0.1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=yes
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20

Suddenly the problem was immediately apparent, but made no sense!  So, what is happening here?

  • Side A phone IP : 10.1.2.3
  • Side B phone IP : 10.2.3.4
  • IOS Loopback IP : 192.168.1.1
  • IOS Ethernet IP : 10.0.0.1

As we can see from the trace:

  1. Despite the Loopback binding, the IOS device was sourcing outbound SIP traffic from the outbound physical interface, instead of the loopback
  2. Outbound INVITE is sent to Side B, but is clearly never processed, as IOS never receives a 1XX Trying/Ringing.  As a result, IOS never sends on a 18X Ringing to Side A, and ring back is never played to the calling party
  3. Multiple INVITEs are re-sent to Side B, which never responds to the call (call setup eventually ended with a  CANCEL from Side A)

Why is this happening!?  Well, the SRST reference in the phone was the Loopback IP, and as a result, the phones registered to the Loopback interface in SRST mode.  As such, they are only going to respond to call setup signalling sourced from this IP – Side B is effectively ignoring the call setup request.

As I’m sure you can guess, this behaviour is not expected, and I immediately thought this was a bug.  What had changed between lab testing and production?  IOS version…  The remote deployment teams had been deploying routers without an upgrade to the Cisco-recommended version that we had tested in the lab.  To validate my conclusion, I looked on cisco.com Bug Search, and got a 1st time hit on the following:

https://tools.cisco.com/bugsearch/bug/CSCur95578

CME-SIP call fails when sip bind is applied under voice service voip

CSCur95578

Description

Symptom:
When a voice gateway falls back to the SRST mode, system message that is configured in the “voice register gloal” section is not displayed on SIP phones

Conditions:
– Voice gateway use the loopback IP address as CME source address
– SIP phones lost the communication to Cisco Unified Call Manager and fall back to SRST mode

Workaround:
Downgrade to an IOS without the fix for CSCup30934 Use a physical IP as CME source address.

We upgraded to the recommended IOS version and re-tested – with immediate positive results.

Conclusion:

This defect outlined a very important point regarding SRST/CME.  In non-trivial deployments with multiple LAN interfaces (or if there are WAN-remote registration devices), always source traffic from a bound interface when traffic is destined to internal devices.  In our case, I could have removed the bind, but the gateway was redundantly connected to 2 core switches with Layer 3 links, which was always going to pose a problem.  Almost without fail, a loopback is the preferred option.

However… some may correctly comment that this may break ITSP call flows if a local SIP Trunking solution is in use – it will, and must be accounted for.  In my base the remote site used ISDN technology, but when a SIP Trunking solution is in use, the ITSP-facing dial peers should be referenced with voice-class sip bind commands.

Great links on this topic here:

 

#dontcalltac

Tagged , , , ,

UCCX – CTI Registration Fails after CUCM Upgrade to 10.5(2) – CSCuq26061

Had a strange one today…

A colleague completed a CUCM upgrade from 10.5 in-version.  No issues were experienced after the minor upgrade.  However, a few days later the client called reporting their contact center and switchboard as down.  This had apparently occurred directly after a power failure in their server room.  Single server BE6K installation.

I jumped onto the gateway, confirming a 404 sent from CUCM to the SIP Gateway.  No routing changes had taken place, so i assumed something was down – immediately thinking CTI.  Their switchboard IVR was fairly complex, so a script had been written on UCCX instead of using Unity Connection AA functionality with Call Handlers.  Reviewing the CTI Ports and Route Points showed me the problem – both “Unknown” – i.e. down since the last reboot.

I next moved to UCCX, and first check CCX Serviceability, noting that Cisco Unified CCX Engine was in Partial Service.  Drilling down to the Sub-System, I could see that Unified CM Telephony Subsystem was marked as OUT OF SERVICE, which made perfect sense due to the non-registration on CUCM.  As all other services were in service, this ruled out any script-related issue as well.  So… time to look at  JTAPI.

After enabling Debugging on the JTAPI logs on CCX, the fault was clear from the logs:

226: Aug 07 13:07:03.201 SAST %JTAPI-JTAPI-7-UNK:[CTIRP_12345]CiscoRegistrationExceptionImpl caught: Device registration failed because of unsupported device IP Addressing capability
227: Aug 07 13:07:08.202 SAST %JTAPI-JTAPI-7-UNK:(P1-JTAPI_1)[MIVR_SS_TEL_PROVIDER-39-0-INIT][CTIRP_12345] getIPAddressingMode= 2
228: Aug 07 13:07:08.202 SAST %JTAPI-JTAPI-7-UNK:(P1-JTAPI_1)[MIVR_SS_TEL_PROVIDER-39-0-INIT][CTIRP_12345]Request: register( myhostname/192.168.1.2, 0, capabilities: “G.711 A-law 64k 30 frame packet size”, “G.711 U-law 64k 30 frame packet size”, algorithmIDs: null, IPv6 address: null)

A quick google (which should have happened sooner!) identified the support forum thread, with an immediate bug match:

https://supportforums.cisco.com/discussion/12405391/cti-route-point-and-cti-ports-not-registering-uccx-105-and-cucm-105-partial

https://tools.cisco.com/bugsearch/bug/CSCuq26061

My only concern was only that I didn’t have an exact version match, as my UCCX was only on version 10.5.1, not 10.6 as quoted in the bug.  However, CUCM did have a similar match.

Workaround was easy – I implemented the CDC as per the documented workaround in CSCuq26061.  In my case, I had to apply this to both the CTI Ports and Route Points.  After a Device Reset, Data Resync, JTAPI Resync (both unnecessary really) and CCX Engine restart, the ports all came up.  I’d recommend BAT here if you face the same.  I tested this on one CTI Port, found it working, then applied to all + CTI RP’s.

Been hitting a few bugs in 10.5 for CCX – hope this helps someone else.

#dontcalltac

Tagged , , , , , , ,
Collaboration Engineer

All things Collaboration - Posts to save for when you need them

Gerry Keleghan's Blog

A Blog about Cisco Unified Communications

ccieme

my personal journey to ccie collaboration

Striving for greatness

Thoughts on emerging tech, open source, and life

Network Experts Blog

“Knowledge comes by eyes always open and working hands.”

SIP Adventures

A unified communications blog by Andrew Prokop

The Cloverhound Blog

Cloverhound Employees Talk Unified Communications and Contact Center

Warcop

Fog navigator. Get out of the clouds. Down to earth solutions. @Warcop

Cisco Collab Engineering Tips

Michael White - CCIE #26626

Darkroomstory

Photography by Manos,

afterthenumber

Thoughts and experiences of a Cisco Collaboration engineer after clearing the CCIE lab...

Longreads

The best longform stories on the web

The Daily Post

The Art and Craft of Blogging

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.