Quick Fix: Increase Root Partition Size in VCSA 6.7

Full disclosure, I’m stealing Anthony Spiteri’s ‘Quick Fix’ used in the title of this post.  It’s somewhat related to a post Anthony published yesterday so I hope he doesn’t mind.  Over this past week both Anthony and I upgraded our vCenters from 6.7 to 6.7 U1.  While our problems were slightly different they both had a common issue.  Our root partitions on our Tiny install of VCSA 6.7 had either run out of space or near out of space causing upgrade issues to U1.

While Anthony was adamant in finding the root cause of the issue to free up space my solution was much simpler --just increase the space on the root volume.  I’ve seen this a number of times with Tiny installs of VCSA so I didn’t want to bother troubleshooting a Home Lab problem through the night.

TL:DR

See bottom of post for the full description of my issue.

Issue:

The VCSA root partition is at 100% or near 100% with less than 10% free space available.

root@vc01 [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs 4.9G 792K 4.9G 1% /dev/shm
tmpfs 4.9G 696K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 11G 11G 0 100% /
…
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/log_vg-log 9.8G 2.3G 7.0G 25% /storage/log
root@vc01 [ ~ ]#

Resolution:

Increase the size of /dev/sda3

Edit the settings of the VCSA VM and located Hard Disk 1 (This disk represents device SDA).  Increase the size of the disk from 12 GB (for a Tiny appliance deployment).  In the example below I have increase it to  20 GB.

Click OK and reboot the appliance.

After the reboot SSH to the VCSA and enter the shell.

There’s a few different commands you can run at this point (see my full issue below) but the easiest command to run is below

/usr/lib/applmgmt/support/scripts/autogrow.sh

As its name implies this will auto grow the root partition with the available space just allocated.

You should now see something like below.

root@vc01 [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs 4.9G 792K 4.9G 1% /dev/shm
tmpfs 4.9G 696K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 19G 9.1G 8.5G 52% /
…
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/log_vg-log 9.8G 2.3G 7.0G 25% /storage/log
root@vc01 [ ~ ]#

The root partition should now have sufficient space for normal operation or upgrades.

 

 

Full Issue:

During an attempt to update my VCSA from version 6.7 to 6.7 U1 I encountered the following error in the VAMI.

Update installation failed. vCenter is non-operational

Attempting to log onto the appliance may also cause intermediate issues.

Error in method invocation Field messages missing from Structure com.vmware.vapi.std.errors.service_unavailable

Checking the space usage on my appliance showed that I only had 5% free space which appeared to not be sufficient for an appliance upgrade.

root@vc01 [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs 4.9G 792K 4.9G 1% /dev/shm
tmpfs 4.9G 696K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 11G 9.1G 900M 95% /
…
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/log_vg-log 9.8G 2.3G 7.0G 25% /storage/log
root@vc01 [ ~ ]#

After a little bit of troubleshooting and attempting to locate the correct vCenter Hard Disk that /dev/sda3 represented.  I identified that Hard Disk 1 was the correct vCenter disk.  /dev/sda3 has multiple partitions and the actual size of /dev/sda is 12GB for a Tiny VCSA install.

To correct the issue I increase the size of this disk to 20 GB, rebooted the appliance, and ran the below command.

/usr/lib/applmgmt/support/scripts/lvm_cfg.sh storage lvm autogrow

This command auto grew the root partition with the available space on the disk that I had just allocated.  I later found out that autogrow.sh is a much simpler command to run and that lvm_cfg.sh wasn’t available anymore after upgrading to VCSA 6.7 U1.