Taking advantage of Cisco’s zero day protection, Cisco FirePOWER checks and downloads the latest signature files from the cloud throughout the day. Once the Cisco FirePOWER system has been configured and tuned up, it can run mostly autonomously without human intervention. Until one day you discovered either the Management Center or some of your sensors are throwing out health alerts for high disk space utilization. High disk space utilization can cause software update to fail. The IPS may fail to function as it rises to critical level. In this session, we’ll walk through the common causes and ways to resolve Cisco FirePOWER high disk space utilization issues on both the Management Center and the IPS sensors. They can be 7000 and 8000 series physical appliances or virtual machines.
Here, I will demonstrate the troubleshooting steps on a Management Center first and followed by the sensors. And the things can be done to improve disk utilization and system performance.
Cisco FirePOWER High Disk Space Utilization on Management Center (formally Defense Center)
When you received disk utilization health warning concerning the Management Center, you should verify its disk usage per directory using CLI.
Verify disk utilization per directory
Use a user account with admin rights. SSH to the Management Center and su to root using the same password.
login as: jwang
Using keyboard-interactive authentication.
Password:
Last login: Mon Mar 28 16:40:29 2016 from 192.168.31.77
Copyright 2004-2014, Cisco and/or its affiliates. All rights reserved. Sourcefire a registered trademark of Sourcefire, Inc. All other trademarks are property of their respective owners.
Sourcefire Linux OS v5.4.0 (build 126)
Sourcefire Virtual Defense Center 64bit v5.4.1.6 (build 40)
jwang@DC:~$
jwang@DC:~$ sudo su -
Password:
root@DC:/# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/root ext3 3.1G 1.3G 1.8G 42% /
devtmpfs devtmpfs 4.2G 58k 4.2G 1% /dev
/dev/sda1 ext2 104M 49M 50M 50% /boot
/dev/sda7 ext2 257G 96G 149G 40% /var
none tmpfs 4.2G 8.2k 4.2G 1% /dev/shm
root@DC:/#Find any directory is over 85%. The system generates warning when any directory is 85% utilized and critical when it reaches 90%. You’ll be focusing on cleaning up and pruning files in those directories.
Common issue 1: Local Backup Files
If you configured scheduled backup jobs to run, it’ll likely use up your disk space because the Management Center does not have a file rotation mechanism in place. Backups which have been copied to another device can be safely deleted. Or, simply prune old backup files by only keeping the recent ones.
Navigate to System > Tools > Backup/Restore, check any old backup files and click the Delete button.
To prevent backup files from filling up disk space, it is recommended to configure a remote backup storage. Check out How to Backup and Restore FirePOWER Management Center.
Common Issue 2: Software Updates
Patches for old software versions can be deleted, whether you have applied them already or decided not to use them.
Navigate to System > Updates, and click the Delete button to the right of any old patches you would like to delete.
Above are the most common issues causing the Management Center to run out of disk space. Next we are looking in to the facts that cause the sensor to fill up disk space.
Cisco FirePOWER High Disk Space Utilization on FirePOWER Sensors
Verify disk utilization per directory
Navigate to Devices > Device Management and locate the sensor’s IP addresses.
Use a user account with admin rights. SSH to the sensor.
login as: jwang
Using keyboard-interactive authentication.
Password:
Last login: Wed Jun 15 16:59:17 2016 from jwang.corp.com
Copyright 2004-2014, Cisco and/or its affiliates. All rights reserved. Sourcefire is a registered trademark of Sourcefire, Inc. All other trademarks are property of their respective owners.
Sourcefire Linux OS v5.4.0 (build 126)
Sourcefire 3D7110 v5.4.0.6 (build 35)Issue the “show disk” command to display per directory disk utilization.
Use “show disk-manager” command to display per service disk utilization report.
Observe the outputs from the commands. Find any directory that is over 85%, as well as any services that is using substantial disks and getting close to its maximum allowed limit.
Common Issue 1: Software Updates
Log into the GUI of the sensor and delete old updates in the same manner. Note you are accessing the sensor directly via an Internet browser (instead of the Management Center). They look very similar. If the sensor is remote, make sure you can reach it over VPN. Or use a jump host that is located on the same network as the remote sensor.
Navigate to System > Updates, and click the Delete button to the right of any old patches you would like to delete.
Common issue 2: Local Backup Files
Local backups will not get pruned by the pruning process. They must be deleted by the user.
There is feature to configure remote backups. It is highly recommended.
Please go to: System > Tools > Backup/Restore
Then click Help > Online
This will reveal a very well laid out explanation of your backup options. You can do the same for any tab in the Sensor or DC GUI to obtain information on a selected area. For step by step instruction, check out –
Cisco FirePOWER – How to Backup and Restore
Common issue 3: A Few Other Places to Check
Commands below that you could use to check the needed locations and remove the necessary files to free up disk space. This will need to be done with a restart to disk manager which will recalculate the disk quota.
First you need to enter the “expert” mode to be able to run the advanced commands:
> expert
jwang@Sourcefire:~$
jwang@Sourcefire:~$ls -lh /var/common/
Look for any cores copy them via scp to the management center and then pull them off the management center in case you need to provide them to TAC. Then you may proceed with a removal of these.
/var/sf/detection_engines/
Once here you can go into each of the UUID for the detection engines and then look for instance numbers. This will look like the below example. Go into each instance and remove the backup/conn-unified
20971496 /var/sf/detection_engines/4399a26e-713e-11e3-ba8a-a46ba9fa1326/instance-3/backup/conn-unified.log.1463844936
20971496 /var/sf/detection_engines/4399a26e-713e-11e3-ba8a-a46ba9fa1326/instance-3/backup/conn-unified.log.1465854991/var/tmp/
Here you want to look for any “Apply” files and remove these.
After doing all of this please proceed to restart disk manager with the below commands.
1)pmtool status | grep ‘diskmanager’
Verify diskmanager is running and its current pid.
2)pmtool restartbyid diskmanager
3)pmtool status | grep ‘diskmanager’
Verify diskmanager is running and its pid has changed.
After done all above, you should re-apply Access Control Policies to each sensor for them to update their health status. Until you re-apply, the health status may remain in warning state. Just keep it in mind.
Check Cisco’s documentation on Troubleshoot Excessive Disk Utilization on Sourcefire Appliances
