Monthly Archives: April 2016

Cisco Support Community Collaboration Videos

Thanks to Java for sharing this – can’t believe I’ve never come across this!

A really, really awesome resource for a number of Collab-related support topics.  Some excellent videos on topics such as:

  • CUOS certificates
  • Self-Provisioning
  • Conference Now
  • PCD
  • TelePresence
  • VCS
  • Service Discovry and UDS
  • Various phone model factory reset procedures

 

 

Enjoy! ūüôā

Tagged , , , , , , , , , ,

CUOS Certificate Expiration Issues and Regeneration

I arrived on site at a remote client picking up a deployment that was done some 18 months ago for additional scope of work.  However, when I arrived to begin, I encountered some strange behaviour that I struggled to attribute to anything obvious.

What I saw was:

 

  • High number of AXL connections on CUCM:

admin:utils diagnose test

Log file: platform/log/diag3.log

Starting diagnostic test(s)
===========================
test – disk_space : Passed (available: 1796 MB, used: 12360 MB)
skip – disk_files : This module must be run directly and off hours
test – service_manager : Passed
test – tomcat : Passed
test – tomcat_deadlocks : Passed
test – tomcat_keystore : Passed
test – tomcat_connectors : Passed
test – tomcat_threads : Passed
test – tomcat_memory : Passed
test – tomcat_sessions : FailedThe following web applications have an unusually large number of active sessions: axl. Please collect all of the Tomcat logs for root cause analysis: file get activelog tomcat/logs/*
skip – tomcat_heapdump : This module must be run directly and off hours
test – validate_network : Passed
test – raid : Passed
test – system_info : Passed (Collected system information in diagnostic log)
test – ntp_reachability : Passed
test – ntp_clock_drift : Passed
test – ntp_stratum : Passed
skip – sdl_fragmentation : This module must be run directly and off hours
skip – sdi_fragmentation : This module must be run directly and off hours

 

  • Backups Failing

 

  • Jabber services listed as” UNKNOWN” from the GUI, even though listed as “STARTED” from Serviceability and from CLI:

 

Selection_006

 

  • Immediate Switch-Version failed after an upgrade due to¬†SELinux failing to upgrade correctly, causing differing behaviour on different servers:
    – Java errors on login to server and CLI scripts unable to run due to permissions issues
    – A Cisco DB service refusing to start!
    – Kicked out of ssh session directly after authentication – SELinux issue

 

 

The SELinux issue actually prevented a switch-back as this could not be initiated from CLI.  To resolve this I had to complete a manual partition switch using the CUCM/CUC Recovery CD as well as a File System Check and Repair!

 

This was all very, very strange!

 

Once back to a “stable” platform after the partition swap and file-system check, I again attempted an upgrade, which this time failed:

04/20/2016 19:46:49 upgrade_install.sh|Started auditd…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Started setroubleshoot…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Changed selinux mode to enforcing|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Cleaning up rpm_archive…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Removing /common/rpm-archive/10.5.2.12901-1|<LVL::Info>
04/20/2016 19:46:50 upgrade_install.sh|File:/usr/local/bin/base_scripts/upgrade_install.sh:604, Function: main(), Upgrade Failed — (1)|<LVL::Error>
04/20/2016 19:46:50 upgrade_install.sh|Parse argument status=upgrade.stage.error|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|_set_upgrade_status_attribute: status set to upgrade.stage.error|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|is_upgrade_lock_available: Upgrade lock is not available.|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|is_upgrade_in_progress: Already locked by this process (pid: 32606).|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 32606)|<LVL::Debug>

 

This time, I had selected an immediate switch-version instead of using a 2-stage process after a successful upgrade.  Again, I could confirm a successful upgrade:

 

04/20/2016 19:39:01 post_upgrade|Post Upgrade RTMTFinish|<LVL::Notice>
04/20/2016 19:39:01 post_upgrade|========================= Upgrade complete. Awaiting switch to version. =========================|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|Post-upgrade processing complete|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|Application install on inactive partition complete|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|L2 upgrade… Run selinux_on_inactive_partition|<LVL::Info>

 

So now, I concentrated on the post-upgrade switch-version logging, and picked up on this:

 

04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /etc/opt/cisco/elm/client/.security/trust_certs/Cisco_Root_CA_M1.pem->system_u:object_r:cisco_etc_t:s0 failed:’Operation not permitted’|<LVL::Debug>
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /etc/opt/cisco/elm/client/.security/trust_certs/Cisco_Root_CA_M1.der->system_u:object_r:cisco_etc_t:s0 failed:’Operation not permitted’|<LVL::Debug>
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /home/sftpuser/sftp_connect.sh->specialuser_u:object_r:user_home_t:s0 failed:’Operation not permitted’|<LVL::Debug>

 

It was then that I became very confused. ¬†These strange issues together all pointed to **certificate expiration**. ¬†However, the system had been installed less than 18 months ago,¬†so this didn’t make sense! ¬†I checked the certificates and confirmed – all expired. ¬†Then it dawned on me. ¬†When we had installed the system, we had staged the network and voice at the same time. ¬†We’d pointed NTP to the firewall – which must have, prior to go-live, been set as a master by our¬†firewall team … with the wrong date-time! ūüė¶

So, root cause found, but now to resolve?!

 

Certificates are a tricky business in CUCM since the implementation of Security by Default. ¬†I don’t intend to cover this topic here (it’s absolutely huge), but I will say this – changing certificates on CUCM can run the risk of getting you fired.

If you don’t know 100% what you are doing, you could¬†cause an outage that will require¬†serious skill to repair, an emergency TAC¬†engagement, ¬†manual phone intervention or a combination of the 3.

 

So, I now needed to complete Certificate Regeneration on CUCM, CUC and IM&P.

Some notes here:

  • My cluster was not in Mixed Mode – my life just got a whole lot easier!
  • The client didn’t have a Private CA, so we were forced to use Self-Signed. ¬†A major consideration if you have signed certs.
  • I¬†was¬†working on a BE6K – with no redundancy – so for me… ¬†ITL files are going to be an issue. ¬†I must¬†to do a “Pre- 8.0 Rollback” and blank all the ITL files before I start.
  • Multi-App integrations are¬†probably going to require manual certificate download/import between clusters for IM&P

 

Following the above process, I carefully:

  1. Blanked all ITL files, with necessary TVS and TFTP service restarts as required
  2. Stopped TFTP Services
  3. Regenerated all applicable certificates as per the above guide
  4. Manually deleted “-trust” certificates as per the guide – I preferred the GUI to the CLI for this – watch out for related bugs in the above guide.
  5. Restarted all applicable services
  6. Removed Pre-8.0 Rollback Enterprise Parameter ITL blanking
  7. Again, restarted TVS and TFTP services
  8. Imported regenerated certs for intra- and inter-cluster communications

 

An Important Note:

When restart TFTP, always wait 5-10mins for the TFTP files to rebuild on the server.  More haste, Less-Job-Come-Monday.

 

Rebooted IM&P, and immediately saw that all the affected IM&P services were listed correctly:

Selection_010.png

 

Helpful links on SBD (Security by Default) and CUOS Certificates in General:

 

 

Check out the ITLRecovery enhancements in 10.x and above!

 

#dontcalltac

Tagged , , , , , ,

Gathering CUOS Installation Information Prior to Upgrade

It may sound obvious, but prior to any upgrade Рprepare for the eventuality of a system rebuild.  I hit an upgrade today that required both a Recovery CD partition swap and file system check on the original Active Partition to recover the system, after SELinux failures occurred after a switch-version.

For a rebuild, collect the following from CLI prior to upgrade:

  1. show network eth0 

  2. show status

  3. show version active

  4. utils ntp config

  5. show web-security

 

This will collect IP/Hostname, DNS, NTP/Timezone, firmware and certificate details(last one is critical!).

This is all negated if you have Answer Files set up to begin with and/or are using PCD.

 

HTH

Tagged , , ,

Active Unassigned DN’s – Updating with SQL

Update to a previous post

Disclaimer :

Provided as is Рyou break your box all on your own buddy!  Lab this first.

 The offending DNs can be queried using:

select n.dnorpattern from numplan n
left outer join devicenumplanmap m on m.fkdevice = n.pkid
where m.fkdevice is null
and n.tkpatternusage = ‘2’
and n.iscallable = ‘t’

  • m.fkdevice is null assures¬†the Route Plan element is unassigned
  • tkpatternusage = ‘2’¬†matches DNs only
  • iscallable=’t’ defines the “Active” checkbox on the DN page – and our offending behaviour.

Some help came from here.

Construct your own UPDATE¬†SQL statement to fix it! ¬†As assistance – start¬†here, it’s not too difficult.

#dontcalltac

Tagged , , , , , ,

Prime Collaboration Provisioning and the Unassigned DN

Something to watch out for!

When using Prime Collaboration for auto-provisioning of ¬†users/lines for Self-Provisioning in CUCM, the DNs will be added as “Active” by default.

Active Check Box

The Active check box, which only displays for unassigned directory numbers, determines whether the directory number gets loaded and used by Cisco Unified Communications Manager. By checking the check box, the directory number gets loaded and used by Cisco Unified Communications Manager. For example, the directory number belonged to an employee who left the company. The directory number had certain settings that were configured, such as call forwarding to voice-messaging system. By leaving the directory number active, a call that is intended for the directory number will get forwarded. This eliminates the need to reconfigure another employee to have the same call-forwarding options. If the check box is not checked, the directory number will not get loaded by Cisco Unified Communications Manager, which results in settings that are configured for that DN to not be used (for example, call forward destinations), and callers will not get their call forwarded properly.

Please see the CUCM System Guide.  My links for 8.6, but still 100% relevant.

The impact is that when doing site migrations, sites already deployed onto the new cluster will fail to dial users that, although pre-provisioned in CUCM, have not yet Self-Provisioned their endpoints. ¬†Nice way to create an outage, sending all calls a VM box that hasn’t even been set up!

I haven’t found a Prime-driven solution for this. ¬†Sadly, we’re back to BAT tools to get the working with Export/Import… ¬†Or SQL.

Tagged , , , ,

WebEx in Ubuntu 14.04+ for x64 systems

As per Cisco’s WebEx System Requirements specification, WebEx is only supported in x86 for Linux, and for Ubuntu specifically in 12.x and 14.x with Gnome.

However, most¬†modern machines use the x64 architecture, so this does create a problem. ¬†I had some fun sorting out this one ( I need it for work, so it’s pretty important I guess ūüôā ). ¬†The steps to resolve are fairly well documented, and in summary are as follows:

 

  1. WebEx requires Java.  The WebEx system requirements guide says Java 6, but it works just fine (eventually!) in Java 8.
  2. Regarding browsers, Chrome just doesn’t work – walk away. ¬†Need Firefox here!
  3. Java may require certain security exceptions in certain cases to get this working
  4. Even after initiating a Webex, in-meeting features such as audio, desktop share etc. are not going to work. ¬†At the time of meeting initiation, WebEx will prompt the user to accept the download of¬†a number of shared libraries (.so files) ¬†to the user’s /home directory, that are stored under /home/myuser/.webex/. ¬†These will contain package dependencies that will prevent most features from working, and should be addressed using this process.

 

There is however still a caveat¬†listed here that needs to be taken into consideration. ¬†It’s¬†important to¬†install¬†the correct icedtea plugin to remove any conflicts:

 

sudo apt-get -y remove icedtea-7-plugin:i386 icedtea-netx:i386
sudo apt-get install openjdk-7-jre:i386 libxmu6:i386 icedtea-7-plugin firefox
sudo update-alternatives --auto mozilla-javaplugin.so

 

It definitely doesn’t look too pretty in Ubuntu, but hey, for my purposes (support) I’m happy it’s working!

Tagged , , , , ,

Migrating to Ubuntu – Practical Guide for Voice/Network Engineers

I’ve recently migrated from Windows to Ubuntu. ¬†I was issued a new Dell laptop at the office after suffering through an HP device for… too long!

I decided to take the opportunity to migrate to Ubuntu. ¬†It’s been a long time coming. ¬†I wasn’t new to the OS, but I’m not a guru either. ūüôā

 

INSTALLATION AND SETUP

The key to success here is a good plan before even creating your LiveCD/USB and installing the OS!

CONVERTING FROM WINDOWS

Some planning is involved to make this a success.  These guides helped me:

 

DISK PARTITIONING

Ensure that you have a good plan on your disk partitioning, especially for /, /home, /boot, swap:

DUAL BOOT WITH WINDOWS

Very good guide – boot loader consideration, partitioning are key here:

OS INSTALLATION

The installation itself is straightforward.

 

GRUB2 BOOTLOADER

Becomes quite important, especially for non-trivial and dual boot.

 

BASIC UBUNTU DESKTOP SETUP

CLI COMMANDS

Just the basics for day 0 if you need it:

TEXT EDITOR – VI

I’m quite a fan. ¬†For GUI-based editors, I like¬†Geany as well (quite a lot like Notepad++¬†in Windows – its pretty much an IDE).

Noteinstall vim immediately! vi has issues with certain keys in 14.04.

FILE RENAMING – REMOVING WHITESPACE FROM MIGRATED WIN DOCS

Sounds silly, but this is a really key issue to resolve.

Migrate files, then update them.  Thankfully, this is what BASH was made for, and a one-liner find does this perfectly!

 

FILE SHARING AND SAMBA

Key topic in mixed office environments.

 

PRINTING

Setting up network printers for an office environment.

 

MAIL

I chose to use Thunderbird as my mail client. ¬†Exchange generally is¬†a problem, but there are some solutions… Took me a while to get this working, but wasn’t too emotional.

MIGRATING PST MAIL ARCHIVES

This process is key.  The realpst package is awesome!

INTEGRATING TO EXCHANGE FROM THUNDERBIRD

Use Exquilla if you can.

This is not a free tool!

 

INTEGRATING TO OWA FROM THUNDERBIRD

I ended up using this as our exchange team couldn’t get me the EWS integration information I needed for Exquilla and auto-discovery didn’t auth with my account.

This is an excellent solution.  Watch out for package dependencies needed (JDK will be required) and untrusted sources.

CALENDAR INTEGRATION

I used DavMail for this again with DavCal.

I also integrated Gmail with DavCal.

 

THUNDERBIRD CUSTOMIZATION

Thunderbird may require some tweaking to have your client act more like Outlook РI needed this to comply to office mail standards with signatures and reply/forwarding behaviour.

Consider installing the SmartTemplate4 add-on:

Creating customized HTML mail signatures:

 

PRINTING

Easy to set up.  A good read as well

 

VIRTUALIZATION

I decided to go for VirtualBox.  Just some pointers that for things that I needed to get working with a Ubuntu Host/Win7 Guest setup.

I had some performance issues (v. high host CPU) related to:

SHARING USB

Works with vmware-tools if you using vmware, but some considerations if you use VB:

SHARING FOLDERS

Not too difficult to get working:

However, you do need to have Guest Tools loaded:

LEAVE IN WINDOWS

Some things are not better and for non-trivial tasks I didn’t even look at open source alternatives:

  • Visio
  • Office

 

PROGRAMS / SOFTWARE IN LINUX

Just the basics – it’s a great blog! ūüôā

NETWORK ENGINEERING TOOLS

Just a selection of Voice/Network Engineering tools that I found useful to immediately use in Ubuntu. ¬†This is what Linux is made for, and why I migrated I guess ūüôā

VOIP/SIP TOOLS

Just to get started, the usual suspects:

  • RTMT
  • TranslatorX
  • audacity¬†– IVR-related stuff
  • Winimage for Answer Files? – wrote a simple BASH script for that!
  • SIP Workbench in Windows – will look for a decent alternative and update…

NETWORKING TOOLS

Just some to get started with:

  • geany¬† – I start with this because every good engineer needs a good text editor. ¬†For GUI-based this is my favourite
  • nmap – n00bs like me can ease into the CLI by starting¬†with¬†zenmap
  • gns3 – much better in linux!
  • wireshark + tshark
  • pac – excellent PERL-based replacement for SecureCRT!

A lot of scripting in Python and BASH will replace awkward Windows programs…

VPNs

As a consulting engineer, I have a number of client VPNs to connect to Рcovering PPTP, Cisco VPN and AnyConnect/JunOs/FortiClient.

Here’s a general Ubuntu overview:

PPTP

This is a stock standard for smaller clients.  There are occasional errors, but most are easily solved by tailing /var/log/syslog.

VPNC

I’d had tremendous issues with this, usually related to poor encryption on the server-side configuration that Linux doesn’t accept.

I wrote this wrapper script that deals with the challenges relating to 1DES weak encryption-related issues if you want to use the terminal to initiate the connection:

#!/bin/sh
sudo vpnc –local-port 0 –enable-1des /etc/vpnc/myvpncconffile.conf

I named it vpnc-connect-wrapper, made it executable and placed in it /home/myuser/bin. ¬†It’s now available to initiate weaker connections. ¬†I saw some solutions re-naming the default binary, which I didn’t like.

Hard-coded with the .conf filename for now… ¬†Will add shell options later. ¬†It’s fairly trivial to do.

The Network Manager Gnome GUI also has added a weak encryption Advanced Setting option to deal with this as well

UPDATE:

I still can’t get this working for multiple VPN connections. ¬†I’ve resorted to doing this in my Windows VM. ¬†I see many online posts with the same logs, and no solution in 14.04 or 14.10. ¬†Not sure about later versions, but its been broken since before 12.04.

SSL VPN:

Got this working as well!

CHECKPOINT SNX

Used to be available as a Linux command line tool, but sadly is now only available through a browser connection – a number of dependencies to get this working, including installing Java.

 

 

OS ADMINISTRATION

LINUX KERNEL

Quite advanced.  Keep for reference for later:

 

PACKAGE MANAGEMENT

Just an overview for Linux newbies…

LINUX SYSTEM DEBUGGING

This really is an excellent tutorial on this topic

 

HARDWARE TROUBLESHOOTING

A basic overview of common troubleshooting tools and tips:

 

BASIC SYS-ADMIN HACKS

Some sys-admins tasks to get you going.  Some were useful early on.

BACKUPS + SYSTEM IMAGING

Key topics when you are new to an OS and you wanna back out a f&^k up.

 

USEFUL LINUX FORUMS

I got a lot of value out of these:

Hope this helps a few folks!

Tagged , , , , , , , ,
Collaboration Engineer

All things Collaboration - Posts to save for when you need them

Gerry Keleghan's Blog

A Blog about Cisco Unified Communications

ccieme

my personal journey to ccie

Striving for greatness

Thoughts on emerging tech, open source, and life

Network Experts Blog

‚ÄúKnowledge comes by eyes always open and working hands.‚ÄĚ

SIP Adventures

A unified communications blog by Andrew Prokop

The Cloverhound Blog

Cloverhound Employees Talk Unified Communications and Contact Center

Warcop

Fog navigator. Get out of the clouds. Down to earth solutions. @Warcop

Cisco Collab Engineering Tips

Michael White - CCIE #26626

Darkroomstory

Photography by Manos,

afterthenumber

Thoughts and experiences of a Cisco Collaboration engineer after clearing the CCIE lab...

Longreads

The best longform stories on the web

The Daily Post

The Art and Craft of Blogging

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.