Thanks to Java for sharing this – can’t believe I’ve never come across this!
A really, really awesome resource for a number of Collab-related support topics. Some excellent videos on topics such as:
Enjoy! 🙂
I arrived on site at a remote client picking up a deployment that was done some 18 months ago for additional scope of work. However, when I arrived to begin, I encountered some strange behaviour that I struggled to attribute to anything obvious.
What I saw was:
admin:utils diagnose test
Log file: platform/log/diag3.log
Starting diagnostic test(s)
===========================
test – disk_space : Passed (available: 1796 MB, used: 12360 MB)
skip – disk_files : This module must be run directly and off hours
test – service_manager : Passed
test – tomcat : Passed
test – tomcat_deadlocks : Passed
test – tomcat_keystore : Passed
test – tomcat_connectors : Passed
test – tomcat_threads : Passed
test – tomcat_memory : Passed
test – tomcat_sessions : Failed – The following web applications have an unusually large number of active sessions: axl. Please collect all of the Tomcat logs for root cause analysis: file get activelog tomcat/logs/*
skip – tomcat_heapdump : This module must be run directly and off hours
test – validate_network : Passed
test – raid : Passed
test – system_info : Passed (Collected system information in diagnostic log)
test – ntp_reachability : Passed
test – ntp_clock_drift : Passed
test – ntp_stratum : Passed
skip – sdl_fragmentation : This module must be run directly and off hours
skip – sdi_fragmentation : This module must be run directly and off hours
The SELinux issue actually prevented a switch-back as this could not be initiated from CLI. To resolve this I had to complete a manual partition switch using the CUCM/CUC Recovery CD as well as a File System Check and Repair!
This was all very, very strange!
Once back to a “stable” platform after the partition swap and file-system check, I again attempted an upgrade, which this time failed:
04/20/2016 19:46:49 upgrade_install.sh|Started auditd…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Started setroubleshoot…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Changed selinux mode to enforcing|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Cleaning up rpm_archive…|<LVL::Info>
04/20/2016 19:46:49 upgrade_install.sh|Removing /common/rpm-archive/10.5.2.12901-1|<LVL::Info>
04/20/2016 19:46:50 upgrade_install.sh|File:/usr/local/bin/base_scripts/upgrade_install.sh:604, Function: main(), Upgrade Failed — (1)|<LVL::Error>
04/20/2016 19:46:50 upgrade_install.sh|Parse argument status=upgrade.stage.error|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|_set_upgrade_status_attribute: status set to upgrade.stage.error|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|is_upgrade_lock_available: Upgrade lock is not available.|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|is_upgrade_in_progress: Already locked by this process (pid: 32606).|<LVL::Debug>
04/20/2016 19:46:50 upgrade_install.sh|release_upgrade_lock: Releasing lock (pid: 32606)|<LVL::Debug>
This time, I had selected an immediate switch-version instead of using a 2-stage process after a successful upgrade. Again, I could confirm a successful upgrade:
04/20/2016 19:39:01 post_upgrade|Post Upgrade RTMTFinish|<LVL::Notice>
04/20/2016 19:39:01 post_upgrade|========================= Upgrade complete. Awaiting switch to version. =========================|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|Post-upgrade processing complete|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|Application install on inactive partition complete|<LVL::Info>
04/20/2016 19:39:01 upgrade_manager.sh|L2 upgrade… Run selinux_on_inactive_partition|<LVL::Info>
So now, I concentrated on the post-upgrade switch-version logging, and picked up on this:
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /etc/opt/cisco/elm/client/.security/trust_certs/Cisco_Root_CA_M1.pem->system_u:object_r:cisco_etc_t:s0 failed:’Operation not permitted’|<LVL::Debug>
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /etc/opt/cisco/elm/client/.security/trust_certs/Cisco_Root_CA_M1.der->system_u:object_r:cisco_etc_t:s0 failed:’Operation not permitted’|<LVL::Debug>
04/20/2016 19:46:38 upgrade_manager.sh|(CAPTURE) /sbin/restorecon.orig set context /home/sftpuser/sftp_connect.sh->specialuser_u:object_r:user_home_t:s0 failed:’Operation not permitted’|<LVL::Debug>
It was then that I became very confused. These strange issues together all pointed to **certificate expiration**. However, the system had been installed less than 18 months ago, so this didn’t make sense! I checked the certificates and confirmed – all expired. Then it dawned on me. When we had installed the system, we had staged the network and voice at the same time. We’d pointed NTP to the firewall – which must have, prior to go-live, been set as a master by our firewall team … with the wrong date-time! 😦
So, root cause found, but now to resolve?!
Certificates are a tricky business in CUCM since the implementation of Security by Default. I don’t intend to cover this topic here (it’s absolutely huge), but I will say this – changing certificates on CUCM can run the risk of getting you fired.
If you don’t know 100% what you are doing, you could cause an outage that will require serious skill to repair, an emergency TAC engagement, manual phone intervention or a combination of the 3.
So, I now needed to complete Certificate Regeneration on CUCM, CUC and IM&P.
Some notes here:
Following the above process, I carefully:
An Important Note:
When restart TFTP, always wait 5-10mins for the TFTP files to rebuild on the server. More haste, Less-Job-Come-Monday.
Rebooted IM&P, and immediately saw that all the affected IM&P services were listed correctly:
Helpful links on SBD (Security by Default) and CUOS Certificates in General:
Check out the ITLRecovery enhancements in 10.x and above!
#dontcalltac
It may sound obvious, but prior to any upgrade – prepare for the eventuality of a system rebuild. I hit an upgrade today that required both a Recovery CD partition swap and file system check on the original Active Partition to recover the system, after SELinux failures occurred after a switch-version.
For a rebuild, collect the following from CLI prior to upgrade:
show network eth0
show status
show version active
utils ntp config
show web-security
This will collect IP/Hostname, DNS, NTP/Timezone, firmware and certificate details(last one is critical!).
This is all negated if you have Answer Files set up to begin with and/or are using PCD.
HTH
Update to a previous post…
Disclaimer :
Provided as is – you break your box all on your own buddy! Lab this first.
The offending DNs can be queried using:
select n.dnorpattern from numplan n
left outer join devicenumplanmap m on m.fkdevice = n.pkid
where m.fkdevice is null
and n.tkpatternusage = ‘2’
and n.iscallable = ‘t’
Some help came from here.
Construct your own UPDATE SQL statement to fix it! As assistance – start here, it’s not too difficult.
#dontcalltac
Something to watch out for!
When using Prime Collaboration for auto-provisioning of users/lines for Self-Provisioning in CUCM, the DNs will be added as “Active” by default.
Active Check Box
The Active check box, which only displays for unassigned directory numbers, determines whether the directory number gets loaded and used by Cisco Unified Communications Manager. By checking the check box, the directory number gets loaded and used by Cisco Unified Communications Manager. For example, the directory number belonged to an employee who left the company. The directory number had certain settings that were configured, such as call forwarding to voice-messaging system. By leaving the directory number active, a call that is intended for the directory number will get forwarded. This eliminates the need to reconfigure another employee to have the same call-forwarding options. If the check box is not checked, the directory number will not get loaded by Cisco Unified Communications Manager, which results in settings that are configured for that DN to not be used (for example, call forward destinations), and callers will not get their call forwarded properly.
Please see the CUCM System Guide. My links for 8.6, but still 100% relevant.
The impact is that when doing site migrations, sites already deployed onto the new cluster will fail to dial users that, although pre-provisioned in CUCM, have not yet Self-Provisioned their endpoints. Nice way to create an outage, sending all calls a VM box that hasn’t even been set up!
I haven’t found a Prime-driven solution for this. Sadly, we’re back to BAT tools to get the working with Export/Import… Or SQL.
As per Cisco’s WebEx System Requirements specification, WebEx is only supported in x86 for Linux, and for Ubuntu specifically in 12.x and 14.x with Gnome.
However, most modern machines use the x64 architecture, so this does create a problem. I had some fun sorting out this one ( I need it for work, so it’s pretty important I guess 🙂 ). The steps to resolve are fairly well documented, and in summary are as follows:
There is however still a caveat listed here that needs to be taken into consideration. It’s important to install the correct icedtea plugin to remove any conflicts:
sudo apt-get -y remove icedtea-7-plugin:i386 icedtea-netx:i386 sudo apt-get install openjdk-7-jre:i386 libxmu6:i386 icedtea-7-plugin firefox sudo update-alternatives --auto mozilla-javaplugin.so
It definitely doesn’t look too pretty in Ubuntu, but hey, for my purposes (support) I’m happy it’s working!
I’ve recently migrated from Windows to Ubuntu. I was issued a new Dell laptop at the office after suffering through an HP device for… too long!
I decided to take the opportunity to migrate to Ubuntu. It’s been a long time coming. I wasn’t new to the OS, but I’m not a guru either. 🙂
The key to success here is a good plan before even creating your LiveCD/USB and installing the OS!
Some planning is involved to make this a success. These guides helped me:
http://www.makeuseof.com/tag/migrating-from-windows-7-to-ubuntu/
http://www.cnet.com/how-to/how-to-migrate-from-windows-to-ubuntu/
Ensure that you have a good plan on your disk partitioning, especially for /, /home, /boot, swap:
http://www.howtogeek.com/howto/35676/how-to-choose-a-partition-scheme-for-your-linux-pc/
http://www.howtogeek.com/114503/how-to-resize-your-ubuntu-partitions/
Very good guide – boot loader consideration, partitioning are key here:
The installation itself is straightforward.
http://www.dedoimedo.com/computers/ubuntu-14-04-install-guide.html
http://www.makeuseof.com/tag/migrating-from-windows-7-to-ubuntu/
Becomes quite important, especially for non-trivial and dual boot.
Just the basics for day 0 if you need it:
I’m quite a fan. For GUI-based editors, I like Geany as well (quite a lot like Notepad++ in Windows – its pretty much an IDE).
Note – install vim immediately! vi has issues with certain keys in 14.04.
Sounds silly, but this is a really key issue to resolve.
Migrate files, then update them. Thankfully, this is what BASH was made for, and a one-liner find does this perfectly!
Key topic in mixed office environments.
http://www.howtogeek.com/howto/29167/3-easy-ways-to-connect-to-windows-shared-folders-from-linux/
http://unix.stackexchange.com/questions/141335/how-to-access-windows-file-server-from-linux
http://www.howtogeek.com/176471/how-to-share-files-between-windows-and-linux/
Setting up network printers for an office environment.
http://smallbusiness.chron.com/add-network-printer-linux-57531.html
https://www.linux.com/learn/tutorials/774476-how-to-manage-printers-in-linux
I chose to use Thunderbird as my mail client. Exchange generally is a problem, but there are some solutions… Took me a while to get this working, but wasn’t too emotional.
This process is key. The realpst package is awesome!
https://www.exratione.com/2013/11/importing-email-from-outlook-on-windows-to-thunderbird-on-ubuntu/
http://manpages.ubuntu.com/manpages/raring/man1/readpst.1.html
Use Exquilla if you can.
This is not a free tool!
I ended up using this as our exchange team couldn’t get me the EWS integration information I needed for Exquilla and auto-discovery didn’t auth with my account.
This is an excellent solution. Watch out for package dependencies needed (JDK will be required) and untrusted sources.
http://yllus.com/2013/09/28/using-mozilla-thunderbird-access-outlook-web-access-owa-e-mail-account/
http://www.webupd8.org/2015/05/on-demand-system-tray-for-ubuntu.html
I used DavMail for this again with DavCal.
I also integrated Gmail with DavCal.
Thunderbird may require some tweaking to have your client act more like Outlook – I needed this to comply to office mail standards with signatures and reply/forwarding behaviour.
Consider installing the SmartTemplate4 add-on:
Creating customized HTML mail signatures:
Easy to set up. A good read as well
I decided to go for VirtualBox. Just some pointers that for things that I needed to get working with a Ubuntu Host/Win7 Guest setup.
I had some performance issues (v. high host CPU) related to:
Works with vmware-tools if you using vmware, but some considerations if you use VB:
Not too difficult to get working:
http://askubuntu.com/questions/270394/access-shared-folders-on-an-ubuntu-host-and-a-windows-xp-guest
However, you do need to have Guest Tools loaded:
Some things are not better and for non-trivial tasks I didn’t even look at open source alternatives:
Just the basics – it’s a great blog! 🙂
Just a selection of Voice/Network Engineering tools that I found useful to immediately use in Ubuntu. This is what Linux is made for, and why I migrated I guess 🙂
Just to get started, the usual suspects:
Just some to get started with:
A lot of scripting in Python and BASH will replace awkward Windows programs…
As a consulting engineer, I have a number of client VPNs to connect to – covering PPTP, Cisco VPN and AnyConnect/JunOs/FortiClient.
Here’s a general Ubuntu overview:
This is a stock standard for smaller clients. There are occasional errors, but most are easily solved by tailing /var/log/syslog.
http://www.cyberciti.biz/tips/howto-configure-ubuntu-fedora-linux-pptp-client.html
Your distro should have a pre-installed GUI for this – Ubuntu 14.04 did.
Watch out for this issue (GRE tunnelling kernel-related error) that requires a modprobe!
I’d had tremendous issues with this, usually related to poor encryption on the server-side configuration that Linux doesn’t accept.
http://askubuntu.com/questions/488435/cisco-vpn-configuration-steps
http://ubuntuhandbook.org/index.php/2013/08/install-cisco-vpn-client-ubuntu-13-04-13-10/
I wrote this wrapper script that deals with the challenges relating to 1DES weak encryption-related issues if you want to use the terminal to initiate the connection:
#!/bin/sh
sudo vpnc –local-port 0 –enable-1des /etc/vpnc/myvpncconffile.conf
I named it vpnc-connect-wrapper, made it executable and placed in it /home/myuser/bin. It’s now available to initiate weaker connections. I saw some solutions re-naming the default binary, which I didn’t like.
Hard-coded with the .conf filename for now… Will add shell options later. It’s fairly trivial to do.
The Network Manager Gnome GUI also has added a weak encryption Advanced Setting option to deal with this as well
UPDATE:
I still can’t get this working for multiple VPN connections. I’ve resorted to doing this in my Windows VM. I see many online posts with the same logs, and no solution in 14.04 or 14.10. Not sure about later versions, but its been broken since before 12.04.
Got this working as well!
http://askubuntu.com/questions/134715/fortinet-ssl-vpn-client-and-ubuntu-12-04
Very easy to set up with this GUI client. Only different between the Linux GUI and the Windows GUI is that window defaults to port 443, whereas Linux defaults to 10443.
Used to be available as a Linux command line tool, but sadly is now only available through a browser connection – a number of dependencies to get this working, including installing Java.
Quite advanced. Keep for reference for later:
Just an overview for Linux newbies…
This really is an excellent tutorial on this topic
A basic overview of common troubleshooting tools and tips:
Some sys-admins tasks to get you going. Some were useful early on.
Key topics when you are new to an OS and you wanna back out a f&^k up.
I got a lot of value out of these:
Hope this helps a few folks!
All things Collaboration - Posts to save for when you need them
A Blog about Cisco Unified Communications
my personal journey to ccie
Thoughts on emerging tech, open source, and life
“Knowledge comes by eyes always open and working hands.”
A unified communications blog by Andrew Prokop
Cloverhound Employees Talk Unified Communications and Contact Center
Fog navigator. Get out of the clouds. Down to earth solutions. @Warcop
Michael White - CCIE #26626
Photography by Manos,
Home
Thoughts and experiences of a Cisco Collaboration engineer after clearing the CCIE lab...
The best longform stories on the web
The Art and Craft of Blogging
The latest news on WordPress.com and the WordPress community.
