UTF-8 BOM, CUCM BAT and Other Reasons I Hate Windows

Stuck working in a customer environment without my personal laptop, or the ability to install a lot of software.

Hit this bug:



Basically, CUCM doesn’t like BOM headers in text files.  Vanilla Notepad appears to add these by default.  If you’re lucky enough to have a Linux box, this is an easy fix:



On windows, I used Notepad++ to re-encode the files to UTF-8:



BAT Import Fails – Unmapped Exception bad header in block 1 record 15, header magic is not ‘ustar’ or unix-style zeros

If you hit this issue when uploading a BAT file, it’s probably because your bat files are encoded in ANSI, not UTF-8.

Yes, it is a CUCM defect:




For a small number of files, you can simply use vanilla Notepad in Windows, and “Save As”, selecting a new format:



However, when you face this with 50+ BAT files like I did, a useful script goes a long way



Using Multiple SSH Keys for Access Different Git Accounts

At work, I use both my companies internal GitLab and my personal GitHub account.  This requires multiple ssh keys for accessing each respective account.

Under Linux, this requires a ~/.ssh/config file, with a similar setup to the following:


Host github.com
User jonathan
IdentityFile ~/.ssh/id_rsa_github

Host *
User jonathan
IdentityFile ~/.ssh/id_rsa


In the above, I have a specific key for my GitHub account and a default key which I use to service my company’s internal GitLab.  This behaviour overrides the settings in the default ssh_config file.  There are various other parameters configurable as required.

  • Port
  • Protocol
  • LocalForward


The man pages are very helpful in this regard.


Helpful Resources:


Tagged , ,

Failure During Switch-Version: ERROR: Acquiring lock failed

Do you really want to switch between versions ?

Enter (yes/no)? yes
Switching Version and Restarting the Appliance …

Switch version duration can vary depending on the database size
and platform configuration. Please continue to monitor the
switchover process from here.
Waiting ……………………………………

Operation failed

ERROR: Acquiring lock failed

Easily solved with a simple reboot, then retry on the switch-version.

If this does fail, next steps would be:

  • Switch-version using Recovery CD
  • TAC case with Remote Account, and TAC to edit GRUB configuration file



Tagged , , , ,

VMware Tools 10.0 Update Failures Creates Outage – CSCux90747, CSCva09767, CSCuy84075

A friend hit this delightful defect this week:



There are several workarounds, but ultimately there’s a COP file for this – ciscocm.VMwareTools2016c.cop.sgn.  The attached defects cover several initial install and help-me-its-dead scenarios.


It’s a bit of a horror.  Watch out for those VMWare tools updates!



Tagged , , , , , , , , ,

Adding Self-Signed Certificates to Ubuntu Trust Store

I’ve been testing AXL calls using Python, and encountered the following error:


Traceback (most recent call last):
 File "./axl_test.py", line 14, in <module>
 result = client.service.listPhone({'name':'SEP%'},{'name':'','model':''})
 File "/usr/lib/python2.7/dist-packages/suds/client.py", line 566, in __call__
 return client.invoke(args, kwargs)
 File "/usr/lib/python2.7/dist-packages/suds/client.py", line 705, in invoke
 result = self.send(soapenv)
 File "/usr/lib/python2.7/dist-packages/suds/client.py", line 747, in send
 reply = self.options.transport.send(request)
 File "/usr/lib/python2.7/dist-packages/suds/transport/https.py", line 66, in send
 return HttpTransport.send(self, request)
 File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 80, in send
 fp = self.u2open(u2request)
 File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 127, in u2open
 return url.open(u2request, timeout=tm)
 File "/usr/lib/python2.7/urllib2.py", line 429, in open
 response = self._open(req, data)
 File "/usr/lib/python2.7/urllib2.py", line 447, in _open
 '_open', req)
 File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
 result = func(*args)
 File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
 File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
 raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>


This is an easy one, as I’m connecting from my Ubuntu machine to CUCM that’s has a self-signed certificate.  I need to add this to my Ubuntu trust store.


Steps to resolve are:


  1. Install mycertifcate.pem to local machine

  2. cp to /usr/local/share/ca-certificates

  3. Rename certificate to mycertificate.crt

  4. sudo update-ca-certificates


jonathan@mymachine:/usr/local/share/ca-certificates$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs…
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d…

Adding debian:mycertifcate.pem


The certificate will now be concatenated to /etc/ssl/certs/ca-certificates.crt and will be trusted as required.

Tagged , , ,

CSCux67499 – SQL NullPointer Exception causes CUC Subscriber Sync Failures

The later version (11.0 – 11.2) of PCP have a fairly painful caveat when it comes to Unity Connection.  CUC is currently afflicted by a(nother) horrid Informix problem that will cause subscriber sync failures.


The defect is below:


Unfortunately, there’s no workaround available as per the bug description:


User Sync of Unity Connection fails on Prime Collaboration Tool (PCP).
NullPointerException exception is seen in Nice log (PCP).

System or internal error java.lang.NullPointerException
java.sql.SQLException: System or internal error java.lang.NullPointerException
at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:488)
at com.informix.jdbc.IfxSqli.a(IfxSqli.java:9547)
at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2634)
at com.informix.jdbc.IfxSqli.a(IfxSqli.java:1830)
at com.informix.jdbc.IfxSqli.executeStatementQuery(IfxSqli.java:1768)
at com.informix.jdbc.IfxSqli.executeStatementQuery(IfxSqli.java:1699)
at com.informix.jdbc.IfxResultSet.a(IfxResultSet.java:210)
at com.informix.jdbc.IfxStatement.executeQueryImpl(IfxStatement.java:1237)
at com.informix.jdbc.IfxPreparedStatement.executeQuery(IfxPreparedStatement.java:382)
at dfc.ipt.cisco.unity.util.UMPUtil.executeQuery(UMPUtil.java:247)

1) PCP User Sync in executed for Unity Connection
2) Having 5K or 20K End Users record in CUC11.x or 10.x



Please take note of the following, as this does affect planned PCP upgrades:



Nice work from the technical documentation team in listing out the caveat in the upgrade guide!


Before upgrading, you need to install an engineering special in Cisco Unity Connection(10.5.x & 11.0) for successful subscriber synchronization. Contact TAC to obtain the Cisco Unity Connection engineering special for the bug (CSCux67499). If you are using Cisco Unity Connection release below 10.5, then you must upgrade to Cisco Unity Connection version 10.5.x and then apply the engineering special.


A call to TAC to request an ES is going to be required (as of the date of this blog post).  Cross-reference the bug “Fixed In” details and check if this has been included in a later Unity Connection SU.  If in doubt, call TAC for  assistance.



Tagged , , ,

CSCup36072 – CUCM/CUC Patch Upgrade Fails

Hit another horror defect today…  Patch upgrades apparently do not work as of 10.5.x and up!




The defect is described below:



The bug details are below – only workaround is to install with a bootable image!


When doing a fresh install of a Call Manager node, there is an option to use an upgrade patch option.

For CUCM 10.5, this approach cannot be used. Not even by using the bootable image for 10.0 and upgrade file for 10.5

Installing CUCM 10.5 with bootable image of a lower version and upgrade file of CUCM 10.5

None -> Use Bootable image for CUCM 10.5 itself


To resolve, the only “supported” method is to request a bootable image from TAC.


Terrible.  Terrible.  Terrible.

Tagged , , , , ,

CSCug20588 – Auto-Reg Failure – “Could not insert new row – duplicate value in a UN”

Had an issue that’s affected multiple deployments that I’ve been working on.  At one of my clients in particular, this reached an escalation level.  Just sharing this to assist others, as we went through 4 TAC Engineers to get a root cause and eventually this was attributed to a hidden Cisco defect in 10.x and above…  The first engineer told us that we needed to rebuild an 18 node CUCM cluster from scratch to fix this!

Just sharing this to assist others, as we went through 4 TAC Engineers to get a root cause and eventually this was attributed to a hidden Cisco defect in 10.x and above…  The first engineer told us that we needed to rebuild an 18 node CUCM cluster from scratch to fix this!  Don’t always believe TAC!


Problem Description


  • Client complains that auto-registered phones are failing to register
  • Eventually determined that the phone had been previously registered and what subsequently deleted
  • Certain clients say that if they manually configure the phone, it registers
  • Phone logs show a TFTP config error


Went on site and noted that there were persistent DB-related syslog alarms being generated.


The SDL logs showed the registration failure and a syslog event was logged:


00024541.012 |07:28:21.243 |AppInfo |SIPStationD(269) - checkDNsReceived: TotalCount=0
00024541.013 |07:28:21.243 |AppInfo |SIPStationD(269) - sendLineRegisterReq: DN match failed for AUTO-REG
00024541.014 |07:28:21.243 |AppInfo |SIPStationD(269) - sendLineRegisterReq: Phone is autoregistering...sending 200OK followed by a restart request
00024541.015 |07:28:21.243 |AppInfo |SIPStationD(269) - buildServiceControl: Sending callid from mPrimaryLineNum 0: 0UTqUn-9I4FebBfs1swQbo.GQ78HIxyf
00024541.016 |07:28:21.243 |AppInfo |SIPStationD(269) - sendServiceControlNotify: starting servicecontrol NOTIFY timer
00024541.017 |07:28:21.243 |AppInfo |SIPStationD(269) - sendServiceControlNotify: sending service-control NOTIFY
00024541.018 |07:28:21.243 |AppInfo |EndPointTransientConnection - An endpoint attempted to register but did not complete registration Connecting Port:5060 Device name:SEPAAAABBBBCCCC Device IP address: Device type:592 Reason Code:7 Protocol:SIP Device MAC address:AAAABBBBCCCC IPAddressAttributes:0 LastSignalReceived:SIPRegisterInd StationState:auto_register App ID:Cisco CallManager Cluster ID:MYCUCM Node ID:mycucmhostame
00024541.019 |07:28:21.243 |AlarmErr |AlarmClass: CallManager, AlarmName: EndPointTransientConnection, AlarmSeverity: Error, AlarmMessage: , AlarmDescription: An endpoint attempted to register but did not complete registration, AlarmParameters: ConnectingPort:5060, DeviceName:SEPAAAABBBBCCCC, IPAddress:, DeviceType:592, Reason:7, Protocol:SIP, MACAddress:AAAABBBBCCCC, IPAddrAttributes:0, LastSignalReceived:SIPRegisterInd, StationState:auto_register, AppID:Cisco CallManager, ClusterID:MYCUCM, NodeID:mycucmhostame,
00024542.000 |07:28:21.243 |AppInfo |-->RISCMAccess::DeviceTransientConnection(...)
00024543.000 |07:28:21.243 |AppInfo |Device Transient deviceName : SEPAAAABBBBCCCC, IPAddress :, IPv6Address : not shown, IPv4Attribute :0, IPv6Attribute :0, Protocol : 2
00024544.000 |07:28:21.243 |AppInfo |DebugMsg deviceName : SEPAAAABBBBCCCC, DeviceType : 592, risClass: 1
00024545.000 |07:28:21.243 |AppInfo |<--RISCMAccess::DeviceTransientConnection(...)
00024541.020 |07:28:21.243 |AppInfo |SIPStationD(269) - Received register from Cisco phone for unconfigured line AUTO-REG; shutting down
00024541.021 |07:28:21.243 |AppInfo |StationTracking::updateMismatchCount: deviceId=SEPAAAABBBBCCCC, mismatchCount=174
00024541.022 |07:28:21.243 |AppInfo |SIPStationD(269) - processVersionStampMismatch: VersionStampMismatch detected, count = 174
00024541.023 |07:28:21.243 |AppInfo |SIPStationD(269) - DevStat-StopClose: LineRegisterReq send failure



And later we in the same trace see that CUCM sends a NOTIFY for the above:


Via: SIP/2.0/UDP;branch=z9hG4bK2b1379d1903
From: <sip:>;tag=789090571
To: <sip:AUTO-REG@>
Call-ID: bb580e00-6df1b475-27b-247410ac@
CSeq: 101 NOTIFY
Max-Forwards: 70
Date: Wed, 09 Mar 2016 05:28:21 GMT
User-Agent: Cisco-CUCM11.0
Event: service-control
Subscription-State: active
Contact: <sip:>
Content-Type: text/plain
Content-Length: 67



I traced through the logs and saw that  soon after the NOTIFY was placed on the wire, the following was logged in SDL that highlighted the problem:



00024556.000 |07:28:21.385 |SdlSig |StationClose |wait |SIPHandler(1,100,80,1) |SIPStationInit(1,100,73,1) |1,100,10,1.591^^* |[T:N-H:0,N:0,L:0,V:0,Z:0,D:0] CloseStationReason = 0 StationId = SEPAAAABBBBCCCC
00024546.004 |07:28:21.428 |AppInfo |DB: odbc::SQLException(Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UNIQUE INDEX column (Unique Index:assign).)
00024546.005 |07:28:21.429 |AppInfo |DBLException - An error occurred while performing database activities ErrorCode:-239 ExceptionString:Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN App ID:Cisco CallManager Cluster ID:MYCUCM Node ID:mycucmhostame
00024546.006 |07:28:21.429 |AlarmErr |AlarmClass: CallManager, AlarmName: DBLException, AlarmSeverity: Alert, AlarmMessage: , AlarmDescription: An error occurred while performing database activities, AlarmParameters: ErrorCode:-239, ExceptionString:Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN, AppID:Cisco CallManager, ClusterID:MYCUCM, NodeID:mycucmhostame,
00024546.007 |07:28:21.429 |AppInfo |dBProcs - ERROR odbc::SQLException addDevice()
00024546.008 |07:28:21.429 |AppInfo |ProcessDb - ERROR initializing_DbStationAutoRegisterReq - addDevice failed


So my initial thought was a bug with orphaned data.  Had to be something along this line as the SDL error was referencing a duplicate.  So , I relunctantly decided to go straight to  TAC for analysis.   I sent this off to TAC and got crickets…  The assigned engineer eventually told me this was an NTP issue which didn’t make sense at all.

Another colleague logged a similar case that we had at another client.  2 re-queues later we got a great engineer who identified  CSCug20588 as the cause of our problem.  It’s a hidden TAC case.


The fix?  Well, a RIDICULOUS bug!


When auto-registering a phone in 10.x the Auto-Reg config uses a UDT/ULT.  This results in a phone-specific Phone Button Template being created when a phone is inserted into the Informix database, of the form:


SEP<MAC-ADDRESS>-SIP-Individual Template


The damn PBT was causing the phone not to register!  This obvoiusly remained in the list of PBT’s when the phone was deleted from CUCM.  When it came to AUTO-REG the phone, CUCM clearly does not look to delete/re-add the PBT or simply use the existing one.  Thus the DB insertion error, and the problematic behaviour.

I deleted the PBT and immediately the phone registered.

There’s no fix for this in 10.x and we’re waiting on the BU to confirm an official 11.x SU that will contain the fix for this.  One to watch out for!


Tagged , , , ,

Interpreting MGCP Debug Parameters

I manage a number of deployments that still feature ISDN extensively, so debug mgcp packet is something that I’m reading off IOS CLI on a fairly routine basis at the moment.

There are a lot of blog posts that cover reading MGCP debugs pretty well, but one thing that I struggled with for a while was being able to quickly interpret the  debug parameters and what they stood for in each respective MGCP message.  Often these are fairly intuitive, but sometimes a quick point of reference is needed.

I looked around on Cisco forums etc., and really there just isn’t anything that covers this well.  Eventually I resorted to the RFC, and was pleasantly unsurprised that all the parameters were well tabulated and easily reference-able.

I’ve included the list below for reference, but as always it is best to go straight to RFC 3435 to get this information!


3.2.2 Parameter Lines

Parameter lines are composed of a parameter name, which in most cases
   is composed of one or two characters, followed by a colon, optional
   white space(s) and the parameter value.  The parameters that can be
   present in commands are defined in the following table:

   |Parameter name        | Code |  Parameter value                   |
   |BearerInformation     |   B  |  See description (        |
   |CallId                |   C  |  See description (        |
   |Capabilities          |   A  |  See description (        |
   |ConnectionId          |   I  |  See description (        |
   |ConnectionMode        |   M  |  See description (        |
   |ConnectionParameters  |   P  |  See description (        |
   |DetectEvents          |   T  |  See description (        |
   |DigitMap              |   D  |  A text encoding of a digit map.   |
   |EventStates           |   ES |  See description (        |
   |LocalConnectionOptions|   L  |  See description (       |
   |MaxMGCPDatagram       |   MD |  See description (       |
   |NotifiedEntity        |   N  |  An identifier, in RFC 821 format, |
   |                      |      |  composed of an arbitrary string   |
   |                      |      |  and of the domain name of the     |
   |                      |      |  requesting entity, possibly com-  |
   |                      |      |  pleted by a port number, as in:   |
   |                      |      |    Call-agent@ca.example.net:5234  |
   |                      |      |  See also Section         |
   |ObservedEvents        |   O  |  See description (       |
   |PackageList           |   PL |  See description (       |
   |QuarantineHandling    |   Q  |  See description (       |
   |ReasonCode            |   E  |  A string with a 3 digit integer   |
   |                      |      |  optionally followed by a set of   |
   |                      |      |  arbitrary characters (  |
   |RequestedEvents       |   R  |  See description (       |
   |RequestedInfo         |   F  |  See description (       |
   |RequestIdentifier     |   X  |  See description (       |
   |ResponseAck           |   K  |  See description (       |
   |RestartDelay          |   RD |  A number of seconds, encoded as   |
   |                      |      |  a decimal number.                 |
   |RestartMethod         |   RM |  See description (       |
   |SecondConnectionId    |   I2 |  Connection Id.                    |
   |SecondEndpointId      |   Z2 |  Endpoint Id.                      |
   |SignalRequests        |   S  |  See description (       |
   |SpecificEndPointId    |   Z  |  An identifier, in RFC 821 format, |
   |                      |      |  composed of an arbitrary string,  |
   |                      |      |  followed by an "@" followed by    |
   |                      |      |  the domain name of the gateway to |
   |                      |      |  which this endpoint is attached.  |
   |                      |      |  See also Section         |
   |RemoteConnection-     |   RC |  Session Description.              |
   |         Descriptor   |      |                                    |
   |LocalConnection-      |   LC |  Session Description.              |
   |         Descriptor   |      |                                    |

   The parameters are not necessarily present in all commands.  The
   following table provides the association between parameters and
   commands.  The letter M stands for mandatory, O for optional and F
   for forbidden.  Unless otherwise specified, a parameter MUST NOT be
   present more than once.

This table is great as I can simply look up the parameter code in my MGCP packet debug and do a quick interpretation.  I also find this very handy when reading off MGCP Connection ID’s and other call-specific tracing that can be cross-referenced in SDL!


For example, in a simple AUEP (Audit Endpoint), we see:


*Jul 17 11:14:18.332: MGCP Packet received from—>
AUEP 31 S0/SU2/DS1-0/1@RTR1.uc.lab MGCP 0.1
F: X, A, I

What has remote side requested information on?


Firstly, the side has requested information in a comma-separated list:

|RequestedInfo         |   F  |


The list contains the following:

|RequestIdentifier     |   X  |
|Capabilities          |   A  |
|ConnectionId          |   I  |


Simple stuff.



Tagged , , , , , , ,
Collaboration Engineer

All things Technology - Posts to save for when you need them

Gerry Keleghan's Blog

A Blog about Cisco Unified Communications


my personal journey to ccie collaboration

Striving for greatness

Thoughts on DevOps, emerging tech, and open source

Network Experts Blog

“Knowledge comes by eyes always open and working hands.”

SIP Adventures

A unified communications blog by Andrew Prokop

The Cloverhound Blog

Cloverhound Employees Talk Unified Communications and Contact Center


Fog navigator. Get out of the clouds. Down to earth solutions. @Warcop

Cisco Collab Engineering Tips

Michael White - CCIE #26626


Photography by Manos,


Thoughts and experiences of a Cisco Collaboration engineer after clearing the CCIE lab...

The Daily Post

The Art and Craft of Blogging

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.

%d bloggers like this: