Category Archives: cucm

Failure During Switch-Version: ERROR: Acquiring lock failed

Do you really want to switch between versions ?

Enter (yes/no)? yes
Switching Version and Restarting the Appliance …

Switch version duration can vary depending on the database size
and platform configuration. Please continue to monitor the
switchover process from here.
Waiting ……………………………………

Operation failed

ERROR: Acquiring lock failed

Easily solved with a simple reboot, then retry on the switch-version.

If this does fail, next steps would be:

  • Switch-version using Recovery CD
  • TAC case with Remote Account, and TAC to edit GRUB configuration file

 

#dontcalltac

Tagged , , , ,

VMware Tools 10.0 Update Failures Creates Outage – CSCux90747, CSCva09767, CSCuy84075

A friend hit this delightful defect this week:

 

 

There are several workarounds, but ultimately there’s a COP file for this – ciscocm.VMwareTools2016c.cop.sgn.  The attached defects cover several initial install and help-me-its-dead scenarios.

 

It’s a bit of a horror.  Watch out for those VMWare tools updates!

 

#dontcalltac

Tagged , , , , , , , , ,

CSCup36072 – CUCM/CUC Patch Upgrade Fails

Hit another horror defect today…  Patch upgrades apparently do not work as of 10.5.x and up!

 

Selection_041

 

The defect is described below:

 

 

The bug details are below – only workaround is to install with a bootable image!

 

Symptom:
When doing a fresh install of a Call Manager node, there is an option to use an upgrade patch option.

For CUCM 10.5, this approach cannot be used. Not even by using the bootable image for 10.0 and upgrade file for 10.5

Conditions:
Installing CUCM 10.5 with bootable image of a lower version and upgrade file of CUCM 10.5

Workaround:
None -> Use Bootable image for CUCM 10.5 itself

 

To resolve, the only “supported” method is to request a bootable image from TAC.

 

Terrible.  Terrible.  Terrible.

Tagged , , , , ,

CSCug20588 – Auto-Reg Failure – “Could not insert new row – duplicate value in a UN”

Had an issue that’s affected multiple deployments that I’ve been working on.  At one of my clients in particular, this reached an escalation level.  Just sharing this to assist others, as we went through 4 TAC Engineers to get a root cause and eventually this was attributed to a hidden Cisco defect in 10.x and above…  The first engineer told us that we needed to rebuild an 18 node CUCM cluster from scratch to fix this!

Just sharing this to assist others, as we went through 4 TAC Engineers to get a root cause and eventually this was attributed to a hidden Cisco defect in 10.x and above…  The first engineer told us that we needed to rebuild an 18 node CUCM cluster from scratch to fix this!  Don’t always believe TAC!

 

Problem Description

 

  • Client complains that auto-registered phones are failing to register
  • Eventually determined that the phone had been previously registered and what subsequently deleted
  • Certain clients say that if they manually configure the phone, it registers
  • Phone logs show a TFTP config error

 

Went on site and noted that there were persistent DB-related syslog alarms being generated.

 

The SDL logs showed the registration failure and a syslog event was logged:

 

00024541.012 |07:28:21.243 |AppInfo |SIPStationD(269) - checkDNsReceived: TotalCount=0
00024541.013 |07:28:21.243 |AppInfo |SIPStationD(269) - sendLineRegisterReq: DN match failed for AUTO-REG
00024541.014 |07:28:21.243 |AppInfo |SIPStationD(269) - sendLineRegisterReq: Phone is autoregistering...sending 200OK followed by a restart request
00024541.015 |07:28:21.243 |AppInfo |SIPStationD(269) - buildServiceControl: Sending callid from mPrimaryLineNum 0: 0UTqUn-9I4FebBfs1swQbo.GQ78HIxyf
00024541.016 |07:28:21.243 |AppInfo |SIPStationD(269) - sendServiceControlNotify: starting servicecontrol NOTIFY timer
00024541.017 |07:28:21.243 |AppInfo |SIPStationD(269) - sendServiceControlNotify: sending service-control NOTIFY
00024541.018 |07:28:21.243 |AppInfo |EndPointTransientConnection - An endpoint attempted to register but did not complete registration Connecting Port:5060 Device name:SEPAAAABBBBCCCC Device IP address:192.168.100.123 Device type:592 Reason Code:7 Protocol:SIP Device MAC address:AAAABBBBCCCC IPAddressAttributes:0 LastSignalReceived:SIPRegisterInd StationState:auto_register App ID:Cisco CallManager Cluster ID:MYCUCM Node ID:mycucmhostame
00024541.019 |07:28:21.243 |AlarmErr |AlarmClass: CallManager, AlarmName: EndPointTransientConnection, AlarmSeverity: Error, AlarmMessage: , AlarmDescription: An endpoint attempted to register but did not complete registration, AlarmParameters: ConnectingPort:5060, DeviceName:SEPAAAABBBBCCCC, IPAddress:192.168.100.123, DeviceType:592, Reason:7, Protocol:SIP, MACAddress:AAAABBBBCCCC, IPAddrAttributes:0, LastSignalReceived:SIPRegisterInd, StationState:auto_register, AppID:Cisco CallManager, ClusterID:MYCUCM, NodeID:mycucmhostame,
00024542.000 |07:28:21.243 |AppInfo |-->RISCMAccess::DeviceTransientConnection(...)
00024543.000 |07:28:21.243 |AppInfo |Device Transient deviceName : SEPAAAABBBBCCCC, IPAddress : 192.168.100.123, IPv6Address : not shown, IPv4Attribute :0, IPv6Attribute :0, Protocol : 2
00024544.000 |07:28:21.243 |AppInfo |DebugMsg deviceName : SEPAAAABBBBCCCC, DeviceType : 592, risClass: 1
00024545.000 |07:28:21.243 |AppInfo |<--RISCMAccess::DeviceTransientConnection(...)
00024541.020 |07:28:21.243 |AppInfo |SIPStationD(269) - Received register from Cisco phone for unconfigured line AUTO-REG; shutting down
00024541.021 |07:28:21.243 |AppInfo |StationTracking::updateMismatchCount: deviceId=SEPAAAABBBBCCCC, mismatchCount=174
00024541.022 |07:28:21.243 |AppInfo |SIPStationD(269) - processVersionStampMismatch: VersionStampMismatch detected, count = 174
00024541.023 |07:28:21.243 |AppInfo |SIPStationD(269) - DevStat-StopClose: LineRegisterReq send failure

 

 

And later we in the same trace see that CUCM sends a NOTIFY for the above:

 

NOTIFY sip:AUTO-REG@192.168.100.123:5060 SIP/2.0
Via: SIP/2.0/UDP 192.168.1.100:5060;branch=z9hG4bK2b1379d1903
From: <sip:192.168.1.100>;tag=789090571
To: <sip:AUTO-REG@192.168.100.123>
Call-ID: bb580e00-6df1b475-27b-247410ac@192.168.1.100
CSeq: 101 NOTIFY
Max-Forwards: 70
Date: Wed, 09 Mar 2016 05:28:21 GMT
User-Agent: Cisco-CUCM11.0
Event: service-control
Subscription-State: active
Contact: <sip:192.168.1.100:5060>
Content-Type: text/plain
Content-Length: 67

action=restart
RegisterCallId={0UTqUn-9I4FebBfs1swQbo.GQ78HIxyf}

 

I traced through the logs and saw that  soon after the NOTIFY was placed on the wire, the following was logged in SDL that highlighted the problem:

 

 

00024556.000 |07:28:21.385 |SdlSig |StationClose |wait |SIPHandler(1,100,80,1) |SIPStationInit(1,100,73,1) |1,100,10,1.591^192.168.100.123^* |[T:N-H:0,N:0,L:0,V:0,Z:0,D:0] CloseStationReason = 0 StationId = SEPAAAABBBBCCCC
00024546.004 |07:28:21.428 |AppInfo |DB: odbc::SQLException(Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UNIQUE INDEX column (Unique Index:assign).)
00024546.005 |07:28:21.429 |AppInfo |DBLException - An error occurred while performing database activities ErrorCode:-239 ExceptionString:Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN App ID:Cisco CallManager Cluster ID:MYCUCM Node ID:mycucmhostame
00024546.006 |07:28:21.429 |AlarmErr |AlarmClass: CallManager, AlarmName: DBLException, AlarmSeverity: Alert, AlarmMessage: , AlarmDescription: An error occurred while performing database activities, AlarmParameters: ErrorCode:-239, ExceptionString:Error fetching data from datasource: [Informix][Informix ODBC Driver][Informix]Could not insert new row - duplicate value in a UN, AppID:Cisco CallManager, ClusterID:MYCUCM, NodeID:mycucmhostame,
00024546.007 |07:28:21.429 |AppInfo |dBProcs - ERROR odbc::SQLException addDevice()
00024546.008 |07:28:21.429 |AppInfo |ProcessDb - ERROR initializing_DbStationAutoRegisterReq - addDevice failed

 

So my initial thought was a bug with orphaned data.  Had to be something along this line as the SDL error was referencing a duplicate.  So , I relunctantly decided to go straight to  TAC for analysis.   I sent this off to TAC and got crickets…  The assigned engineer eventually told me this was an NTP issue which didn’t make sense at all.

Another colleague logged a similar case that we had at another client.  2 re-queues later we got a great engineer who identified  CSCug20588 as the cause of our problem.  It’s a hidden TAC case.

 

The fix?  Well, a RIDICULOUS bug!

 

When auto-registering a phone in 10.x the Auto-Reg config uses a UDT/ULT.  This results in a phone-specific Phone Button Template being created when a phone is inserted into the Informix database, of the form:

 

SEP<MAC-ADDRESS>-SIP-Individual Template

 

The damn PBT was causing the phone not to register!  This obvoiusly remained in the list of PBT’s when the phone was deleted from CUCM.  When it came to AUTO-REG the phone, CUCM clearly does not look to delete/re-add the PBT or simply use the existing one.  Thus the DB insertion error, and the problematic behaviour.

I deleted the PBT and immediately the phone registered.

There’s no fix for this in 10.x and we’re waiting on the BU to confirm an official 11.x SU that will contain the fix for this.  One to watch out for!

 

Tagged , , , ,

Interpreting MGCP Debug Parameters

I manage a number of deployments that still feature ISDN extensively, so debug mgcp packet is something that I’m reading off IOS CLI on a fairly routine basis at the moment.

There are a lot of blog posts that cover reading MGCP debugs pretty well, but one thing that I struggled with for a while was being able to quickly interpret the  debug parameters and what they stood for in each respective MGCP message.  Often these are fairly intuitive, but sometimes a quick point of reference is needed.

I looked around on Cisco forums etc., and really there just isn’t anything that covers this well.  Eventually I resorted to the RFC, and was pleasantly unsurprised that all the parameters were well tabulated and easily reference-able.

I’ve included the list below for reference, but as always it is best to go straight to RFC 3435 to get this information!

 


3.2.2 Parameter Lines

Parameter lines are composed of a parameter name, which in most cases
   is composed of one or two characters, followed by a colon, optional
   white space(s) and the parameter value.  The parameters that can be
   present in commands are defined in the following table:


    ------------------------------------------------------------------
   |Parameter name        | Code |  Parameter value                   |
   |----------------------|------|------------------------------------|
   |BearerInformation     |   B  |  See description (3.2.2.1).        |
   |CallId                |   C  |  See description (3.2.2.2).        |
   |Capabilities          |   A  |  See description (3.2.2.3).        |
   |ConnectionId          |   I  |  See description (3.2.2.5).        |
   |ConnectionMode        |   M  |  See description (3.2.2.6).        |
   |ConnectionParameters  |   P  |  See description (3.2.2.7).        |
   |DetectEvents          |   T  |  See description (3.2.2.8).        |
   |DigitMap              |   D  |  A text encoding of a digit map.   |
   |EventStates           |   ES |  See description (3.2.2.9).        |
   |LocalConnectionOptions|   L  |  See description (3.2.2.10).       |
   |MaxMGCPDatagram       |   MD |  See description (3.2.2.11).       |
   |NotifiedEntity        |   N  |  An identifier, in RFC 821 format, |
   |                      |      |  composed of an arbitrary string   |
   |                      |      |  and of the domain name of the     |
   |                      |      |  requesting entity, possibly com-  |
   |                      |      |  pleted by a port number, as in:   |
   |                      |      |    Call-agent@ca.example.net:5234  |
   |                      |      |  See also Section 3.2.1.3.         |
   |ObservedEvents        |   O  |  See description (3.2.2.12).       |
   |PackageList           |   PL |  See description (3.2.2.13).       |
   |QuarantineHandling    |   Q  |  See description (3.2.2.14).       |
   |ReasonCode            |   E  |  A string with a 3 digit integer   |
   |                      |      |  optionally followed by a set of   |
   |                      |      |  arbitrary characters (3.2.2.15).  |
   |RequestedEvents       |   R  |  See description (3.2.2.16).       |
   |RequestedInfo         |   F  |  See description (3.2.2.17).       |
   |RequestIdentifier     |   X  |  See description (3.2.2.18).       |
   |ResponseAck           |   K  |  See description (3.2.2.19).       |
   |RestartDelay          |   RD |  A number of seconds, encoded as   |
   |                      |      |  a decimal number.                 |
   |RestartMethod         |   RM |  See description (3.2.2.20).       |
   |SecondConnectionId    |   I2 |  Connection Id.                    |
   |SecondEndpointId      |   Z2 |  Endpoint Id.                      |
   |SignalRequests        |   S  |  See description (3.2.2.21).       |
   |SpecificEndPointId    |   Z  |  An identifier, in RFC 821 format, |
   |                      |      |  composed of an arbitrary string,  |
   |                      |      |  followed by an "@" followed by    |
   |                      |      |  the domain name of the gateway to |
   |                      |      |  which this endpoint is attached.  |
   |                      |      |  See also Section 3.2.1.3.         |
   |RemoteConnection-     |   RC |  Session Description.              |
   |         Descriptor   |      |                                    |
   |LocalConnection-      |   LC |  Session Description.              |
   |         Descriptor   |      |                                    |
    ------------------------------------------------------------------

   The parameters are not necessarily present in all commands.  The
   following table provides the association between parameters and
   commands.  The letter M stands for mandatory, O for optional and F
   for forbidden.  Unless otherwise specified, a parameter MUST NOT be
   present more than once.

This table is great as I can simply look up the parameter code in my MGCP packet debug and do a quick interpretation.  I also find this very handy when reading off MGCP Connection ID’s and other call-specific tracing that can be cross-referenced in SDL!


 

For example, in a simple AUEP (Audit Endpoint), we see:

 

*Jul 17 11:14:18.332: MGCP Packet received from 192.168.123.213:2427—>
AUEP 31 S0/SU2/DS1-0/1@RTR1.uc.lab MGCP 0.1
F: X, A, I
<—

What has remote side requested information on?

 

Firstly, the side has requested information in a comma-separated list:

|RequestedInfo         |   F  |

 

The list contains the following:

|RequestIdentifier     |   X  |
|Capabilities          |   A  |
|ConnectionId          |   I  |

 

Simple stuff.

 

#dontcalltac

Tagged , , , , , , ,

CUCM Video Codec Preferences Support – CSCuw53802

Colleague called me for assistance with a video call that wouldn’t set up when dialling in to a VC bridge.

The call flow was from a Polycom endpoint to the new Conductor-managed vTS bridge that I’d installed a week before.  The endpoint could dial into the Rendezvous VC bridge, but calls only had audio.  Some additional testing showed that this wasn’t working for point-to-point calls to Jabber as well, which helped us rule out the bridge completely.

My colleague pulled a Wireshark from Jabber, and the signalling confirmed a codec support issue on the Polycom:

Jabber INVITE with H.264:

Selection_035

However, the response clearly showed that our Polycom unit didn’t like this, and marked the video streams as inactive:

Selection_036

My colleague did some reading up on our somewhat outdated Polycom unit, and we ultimately found that some release keys and a firmware upgrade would be needed.  Customer not too happy, but nothing to do here other than upgrade…

However, my colleague did post a question to me:

Can we not just set some video codec preferences for this call, or at least for the call into the VC bridge?

Well, the answer is… no!

 

As of 11.0 this is not supported.  However, thankfully this does appear to have been considered in the roadmap for CUCM 11.5:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuw53802/

Selection_037

 

This is good news, as the CUCM-based video architecture is now quite mature, having gone through several iterations since it’s inception from CUCM 9.x.

As the market for Skype for Business grows, H.265 begins to create more and more headaches due to patent licensing problems, and Cisco subsequently begins to punt Thor as a potential future video standard, this new feature should hopefully solve some problems down the line.

 

#dontcalltac

Tagged , , , , , , , , , , , , , , ,

Accessing the jabber-config.xml Jabber Configuration File

Often when working on Cisco Jabber installations, you will be required to edit an existing jabber-config.xml file to change client parameters or support new functionality, or simply as part of an audit or system assessment want to have a look at what customization is being used for Jabber already.

There are several ways to access the jabber-config.xml file from your CUCM/TFTP server or Jabber client PC.

When troubleshooting, it is often helpful to know the methods to view from both the TFTP server and the Jabber client PC.  Discrepancies can be a good starting point for troubleshooting.

Method 1 – CUOS

From CUOS, run the following command:

admin:file view tftp jabber-config.xml

 

Method 2 – Browser

From your browser, enter the following:

http://X.X.X.X:6970/jabber-config.xml

where X.X.X.X is the CUCM (i.e. TFTP Server) IP Address

 

Method 3 – TFTP Client

Use a TFTP Client such as Windows TFTP client or TFTPD32 and GET the jabber-config.xml file.

 

Method 4 – Windows Client PC

Fetch this from the following path:

%USERPROFILE%\AppData\Roaming\Cisco\Unified Communications\Jabber\CSF\Config

You may need to enable showing of hidden files on the Windows desktop to reach this.

 

If you know of more, please let me know!

#dontcalltac

Tagged , , , , , ,

VoIP Bandwidth Calculations

If you’re preparing for a your CCIE Written or Lab exams, or (far more importantly) if you need to perform bandwidth sizing for a VoIP design or implementation, it’s critical to be able to accurately calculate the amount of bandwidth that your precious voice streams are going to need as they traverses your network.

Vik Mahli makes an excellent overview of what this means in terms of a QoS configuration here, but skips the calculation part altogether in order to focus on the LAN QoS config that he wished to cover in his post.  Naturally, this created some confusion which was carried over into the comments section!

I encourage you to read the comments, specifically Mark Holloway’s length post on this topic.  Mark makes an excellent reference here to the QoS SRND, and is well worth a read!  Please see this link and refer to where Mark and Vik use a 93bkps base for G.711 calls from.

Building on some of the discussions there, I’d like to provide some additional detail on how these calculations can be performed in this post.

Finally, please take note that this does not account for signalling bandwidth requirements.  This is a different discussion – ping me and I’ll blog about this too! 🙂

 

Performing VoIP Bandwidth Calculations

 

Calculation Assumptions

Firstly, we will have to make some protocol-related assumptions for sizing purposes:

  • IP – 20 bytes
  • User Datagram Protocol (UDP) – 8 bytes
  • Real-Time Transport Protocol (RTP) – 12 bytes
  • Compressed Real-Time Protocol (cRTP) reduces the IP/UDP/RTP headers to 2 or 4 bytes
  • Multilink Point-to-Point Protocol (MP) – 6 bytes
  • Frame Relay Forum (FRF).12 Layer 2 (L2)– 6 bytes
  • End-of-frame flag on MP and Frame Relay frames – 1 byte
  • Ethernet L2 headers– 18 bytes

 

Codec-Specific Information

Secondly, we need to know some values that are codec-specific:

  • Voice Payload Size
  • Codec Bitrate

Some values worth keeping in mind here are:

  • G711
    • Bit Rate – 64 kbps
    • Default Payload size – 160 bytes
  • G729
    • Bit Rate – 8 kbps
    • Default Payload size – 20 bytes
  • iLBC 20ms
    • Bit Rate – 15.2 kbps
    • Default Payload size – 38 bytes
  • iLBC 30ms
    • Bit Rate – 13.33 kbps
    • Default Payload size – 50 bytes

 

There is a detailed discussion of these values and their explanations here.  I won’t go into more detail in this post.  Please refer to the above document for clarity.

NOTE:

For differing Millesecond Packet Sizes (e.g. G.729 20ms/30ms), the codec bitrate is always constant.  Therefore the payload size can always be scaled as a ratio using the formula of:

codec bit rate = codec sample size / codec sample interval

More on this later in a worked example…

 

Bandwidth Calculation Formulas

From the TAC Online Help, we’re presented with a fairly simple calculation:

Total Packet Size = (Media Access Header - Layer 2) + (IP/UDP/RTP Header) + (voice payload)
Voice Packets Per Second (PPS) = codec bit rate / voice payload size
Bandwidth = [ (voice packet size) + 1 byte (frame flag) ] * PPS

Finally, it is best practice to allow for a 5.0% overhead in all calculations.

 

Some Worked Examples

This seems simple enough, so let’s put it into practice in some worked examples!

 

Example 1 : 3 Concurrent G.711 Calls over Ethernet

Total Packet Size = [ (18) + (40) + (160) ] = 218 bytes 

Packets per Second = [ 64 kbps * 1000 bits per kilobit ] / [ 160 bytes (default voice payload) * 8 bits per byte ]  = 50 pps

Bandwidth = [ ( 218 bytes ) + (0 bytes ) ] * 50 pps * 8 bites/byte = 87200 bits/sec = 87.2 kbps

5.0% Overhead87.2 kbps * 1.05 = 91.560 kbps

Concurrent Calls = 3 * 91.560 kbps = 274.68 kbps

 

Example 2 : 5 Concurrent G.729 Calls over PPP with Compressed RTP

Total Packet Size = [ (6) + (2) + (20) ] = 28 bytes 

Packets per Second = [ 8 kbps * 1000 bits per kilobit ] / [ 20 bytes (default voice payload) * 8 bits per byte ]  = 50 pps

Bandwidth = [ ( 28 bytes ) + (1 byte) ] * 50 pps * 8 bites/byte = 87200 bits/sec = 11.6 kbps

5.0% Overhead = 11.6 kbps * 1.05 = 12.18 kbps

Concurrent Calls = 5 * 12.18 kbps = 60.9 kbps

 

Example 3 : 10 Concurrent iLBC Calls over Ethernet using a 20ms bitrate

Total Packet Size = [ (18) + (40) + (38) ] = 96 bytes

Packets per Second =[ 15.2 kbps * 1000 bits per kilobit ]  / [ 38 bytes (default voice payload) * 8 bits per byte ]  = 50 pps

Bandwidth = [ ( 96 bytes ) + (0 bytes ) ] * 50 pps * 8 bites/byte = 38400 bits/sec = 38.4 kbps

5.0% Overhead = 38.4 kbps * 1.05 = 40.32 kbps

Concurrent Calls = 10 * 40.32 kbps = 403.20 kbps

The TAC Tool does not cover iLBC calculations, but we can see that there’s nothing special going on here.  We just needed to know the bitrate values for the codec and the calculations are effectively the same.

 

Example 4 : 5 Concurrent iLBC Calls over Ethernet using a 30ms bitrate

Total Packet Size = [ (18) + (40) + (50) ] = 108 bytes

Packets per Second =[ 13.3 kbps * 1000 bits per kilobit ]  / [ 50 bytes (default voice payload) * 8 bits per byte ]  = 33.3 pps

Bandwidth = [ ( 108 bytes ) + (0 bytes ) ] * 33.3 pps * 8 bites/byte = 28800 bits/sec = 28.8 kbps

5.0% Overhead = 28.8 kbps * 1.05 = 30.24 kbps

Concurrent Calls = 5 * 30.24 kbps = 151.20 kbps

 

Example 5 : 2 Concurrent G.729 Calls over Ethernet using a 40ms bitrate

OK, let’s end it off with a challenge 🙂

As I said above, the bitrate is always a constant for any codec:

codec bit rate = codec sample size / codec sample interval

Therefore, we will need to calculate our new Payload Sample Size for 40ms instead of 20ms:

Codec Sample Size = 8 kbps * 40ms  / (8bits/byte) = 40 bytes

See?  Not so hard!  If you’re following, you can actually see that you could have just used a ratio of:

New Payload Size = Old Payload Size * (New Sampling Interval / Old Sampling Interval)

 

OK, I digress.  Now, carrying on as before:

Total Packet Size = [ (18) + (40) + (40) ] = 98 bytes

Packets per Second =[ 8 kbps * 1000 bits per kilobit ]  / [ 40 bytes (default voice payload) * 8 bits per byte ]  = 25 pps

Bandwidth = [ ( 98 bytes ) + (0 bytes ) ] * 25 pps * 8 bites/byte = 19600 bits/sec = 19.60 kbps

5.0% Overhead =19.60 kbps * 1.05 = 20.58 kbps

Concurrent Calls = 2 * 40.32 kbps = 41.16 kbps

As you can see, the payload size may have doubled, but the packet throughput has halved!

 

Example 6 : 8 Concurrent G.722 Calls over Ethernet using a 10ms bitrate

Building on the ideas presented in the previous example:

New Payload Size = 160 kbps * (10/20) = 80 kbps

Total Packet Size = [ (18) + (40) + (80) ] = 138 bytes

Packets per Second =[ 64 kbps * 1000 bits per kilobit ]  / [ 80 bytes (default voice payload) * 8 bits per byte ]  = 100 pps

Bandwidth = [ ( 138 bytes ) + (0 bytes ) ] * 100 pps * 8 bites/byte = 110400 bits/sec = 110,4 kbps

5.0% Overhead =110.40 kbp * 1.05 = 115.92  kbps

Concurrent Calls = 8 *115.92 kbps = 927,36 kbps

OK, at this point not much more to cover! 🙂

 

A Final Note on Sampling Rates…

I highlighted the PPS value in Example 5 in red to make a final point on this topic.

As you can see, the payload size may have doubled, but the packet throughput has halved.

 

Where does all of this come  into play?  Well, there are two competing considerations here:

  1. Longer Durations (i.e. a lower PPS value) reduces Network Overheads
  2. Increased Packet Sizes decreases the resiliency to Network Errors (such as packet loss)

 

Considerations such as physical hardware capabilities, link stability, available bandwidth, PBX feature sets (and don’t forget that of your Peering Partners!)  etc. can all be considered when deciding on a preferred sampling rate for your VoIP network – a lot for the VoIP designer to think about when making decisions on what codecs (and respective capabilities) to support!

 

Online References

I have managed to find a number of excellent documents to help you.

 

For a detailed breakdown of how to perform these calculations:

 

A TAC tool, and simply the best out there with a nice GUI:

 

Configuring Bandwidth-based CAC for CUBE:

 

For quick and dirty calculations that don’t take into account network overheads:

 

Additional links to various sites that offer similar tools:

 

Hope this helps all you CCIE aspirants and VoIP network designers!

 

#dontcalltac

Tagged , , , , ,

Cisco Support Community Collaboration Videos

Thanks to Java for sharing this – can’t believe I’ve never come across this!

A really, really awesome resource for a number of Collab-related support topics.  Some excellent videos on topics such as:

  • CUOS certificates
  • Self-Provisioning
  • Conference Now
  • PCD
  • TelePresence
  • VCS
  • Service Discovry and UDS
  • Various phone model factory reset procedures

 

 

Enjoy! 🙂

Tagged , , , , , , , , , ,

CUOS Certificate Expiration Issues and Regeneration

I arrived on site at a remote client picking up a deployment that was done some 18 months ago for additional scope of work.  However, when I arrived to begin, I encountered some strange behaviour that I struggled to attribute to anything obvious.

What I saw was:

 

  • High number of AXL connections on CUCM:

admin:utils diagnose test

Log file: platform/log/diag3.log

Starting diagnostic test(s)
===========================
test – disk_space : Passed (available: 1796 MB, used: 12360 MB)
skip – disk_files : This module must be run directly and off hours
test – service_manager : Passed
test – tomcat : Passed
test – tomcat_deadlocks : Passed
test – tomcat_keystore : Passed
test – tomcat_connectors : Passed
test – tomcat_threads : Passed
test – tomcat_memory : Passed
test – tomcat_sessions : FailedThe following web applications have an unusually large number of active sessions: axl. Please collect all of the Tomcat logs for root cause analysis: file get activelog tomcat/logs/*
skip – tomcat_heapdump : This module must be run directly and off hours
test – validate_network : Passed
test – raid : Passed
test – system_info : Passed (Collected system information in diagnostic log)
test – ntp_reachability : Passed
test – ntp_clock_drift : Passed
test – ntp_stratum : Passed
skip – sdl_fragmentation : This module must be run directly and off hours
skip – sdi_fragmentation : This module must be run directly and off hours

 

  • Backups Failing

 

  • Jabber services listed as” UNKNOWN” from the GUI, even though listed as “STARTED” from Serviceability and from CLI:

 

Selection_006

 

  • Immediate Switch-Version failed after an upgrade due to SELinux failing to upgrade correctly, causing differing behaviour on different servers:
    – Java errors on login to server and CLI scripts unable to run due to permissions issues
    – A Cisco DB service refusing to start!
    – Kicked out of ssh session directly after authentication – SELinux issue

 

 

The SELinux issue actually prevented a switch-back as this could not be initiated from CLI.  To resolve this I had to complete a manual partition switch using the CUCM/CUC Recovery CD as well as a File System Check and Repair!

 

This was all very, very strange!

 

Once back to a “stable” platform after the partition swap and file-system check, I again attempted an upgrade, which this time failed:

04/20/2016 19:46:49 upgrade_install.sh|Started auditd…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Started setroubleshoot…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Changed selinux mode to enforcing|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Cleaning up rpm_archive…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Removing /common/rpm-archive/10.5.2.12901-1|<LVL::Info>
04/20/2016 19:46:50 upgrade_install.sh|File:/usr/local/bin/base_scripts/upgrade_install.sh:604, Function: main(), Upgrade Failed — (1)|<LVL::Error>
04/20/2016 19:46:50 upgrade_install.sh|Parse argument status=upgrade.stage.error|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|_set_upgrade_status_attribute: status set to upgrade.stage.error|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|is_upgrade_lock_available: Upgrade lock is not available.|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|is_upgrade_in_progress: Already locked by this process (pid: 32606).|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 32606)|<LVL::Debug>

 

This time, I had selected an immediate switch-version instead of using a 2-stage process after a successful upgrade.  Again, I could confirm a successful upgrade:

 

04/20/2016 19:39:01 post_upgrade|Post Upgrade RTMTFinish|<LVL::Notice>
04/20/2016 19:39:01 post_upgrade|========================= Upgrade complete. Awaiting switch to version. =========================|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|Post-upgrade processing complete|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|Application install on inactive partition complete|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|L2 upgrade… Run selinux_on_inactive_partition|<LVL::Info>

 

So now, I concentrated on the post-upgrade switch-version logging, and picked up on this:

 

04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /etc/opt/cisco/elm/client/.security/trust_certs/Cisco_Root_CA_M1.pem->system_u:object_r:cisco_etc_t:s0 failed:’Operation not permitted’|<LVL::Debug>
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /etc/opt/cisco/elm/client/.security/trust_certs/Cisco_Root_CA_M1.der->system_u:object_r:cisco_etc_t:s0 failed:’Operation not permitted’|<LVL::Debug>
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /home/sftpuser/sftp_connect.sh->specialuser_u:object_r:user_home_t:s0 failed:’Operation not permitted’|<LVL::Debug>

 

It was then that I became very confused.  These strange issues together all pointed to **certificate expiration**.  However, the system had been installed less than 18 months ago, so this didn’t make sense!  I checked the certificates and confirmed – all expired.  Then it dawned on me.  When we had installed the system, we had staged the network and voice at the same time.  We’d pointed NTP to the firewall – which must have, prior to go-live, been set as a master by our firewall team … with the wrong date-time! 😦

So, root cause found, but now to resolve?!

 

Certificates are a tricky business in CUCM since the implementation of Security by Default.  I don’t intend to cover this topic here (it’s absolutely huge), but I will say this – changing certificates on CUCM can run the risk of getting you fired.

If you don’t know 100% what you are doing, you could cause an outage that will require serious skill to repair, an emergency TAC engagement,  manual phone intervention or a combination of the 3.

 

So, I now needed to complete Certificate Regeneration on CUCM, CUC and IM&P.

Some notes here:

  • My cluster was not in Mixed Mode – my life just got a whole lot easier!
  • The client didn’t have a Private CA, so we were forced to use Self-Signed.  A major consideration if you have signed certs.
  • I was working on a BE6K – with no redundancy – so for me…  ITL files are going to be an issue.  I must to do a “Pre- 8.0 Rollback” and blank all the ITL files before I start.
  • Multi-App integrations are probably going to require manual certificate download/import between clusters for IM&P

 

Following the above process, I carefully:

  1. Blanked all ITL files, with necessary TVS and TFTP service restarts as required
  2. Stopped TFTP Services
  3. Regenerated all applicable certificates as per the above guide
  4. Manually deleted “-trust” certificates as per the guide – I preferred the GUI to the CLI for this – watch out for related bugs in the above guide.
  5. Restarted all applicable services
  6. Removed Pre-8.0 Rollback Enterprise Parameter ITL blanking
  7. Again, restarted TVS and TFTP services
  8. Imported regenerated certs for intra- and inter-cluster communications

 

An Important Note:

When restart TFTP, always wait 5-10mins for the TFTP files to rebuild on the server.  More haste, Less-Job-Come-Monday.

 

Rebooted IM&P, and immediately saw that all the affected IM&P services were listed correctly:

Selection_010.png

 

Helpful links on SBD (Security by Default) and CUOS Certificates in General:

 

 

Check out the ITLRecovery enhancements in 10.x and above!

 

#dontcalltac

Tagged , , , , , ,
Collaboration Engineer

All things Collaboration - Posts to save for when you need them

Gerry Keleghan's Blog

A Blog about Cisco Unified Communications

ccieme

my personal journey to ccie collaboration

Striving for greatness

Thoughts on emerging tech, open source, and life

Network Experts Blog

“Knowledge comes by eyes always open and working hands.”

SIP Adventures

A unified communications blog by Andrew Prokop

The Cloverhound Blog

Cloverhound Employees Talk Unified Communications and Contact Center

Warcop

Fog navigator. Get out of the clouds. Down to earth solutions. @Warcop

Cisco Collab Engineering Tips

Michael White - CCIE #26626

Darkroomstory

Photography by Manos,

afterthenumber

Thoughts and experiences of a Cisco Collaboration engineer after clearing the CCIE lab...

Longreads

The best longform stories on the web

The Daily Post

The Art and Craft of Blogging

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.