When you start the service manually using following command it gives error.
Syntax
service-control --start vmware-vpxd
Error
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False
2019-02-08T22:44:52.954Z Service vpxd state STOPPED
Error executing start on service vpxd. Details {
"resolution": null,
"detail": [
{
"args": [
"vpxd"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"vpxd"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vpxd'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Error in VPXD log
2019-02-08T22:44:53.608Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Starting VMware VirtualCenter 6.5.0 build-8024368 2019-02-08T22:44:53.608Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Log directory: /var/log/vmware/vpxd. 2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=Main] Account name: root 2019-02-08T22:44:53.609Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxUtil] [LoadMachineInstanceUuid] Local instance UUID: 6f0cbee2-56b1-43d3-b13a-348208627e07 2019-02-08T22:44:53.614Z error vpxd[7FB5996C4800] [Originator@6876 sub=Main] Init failed. Exception: N7Vmacore15SystemExceptionE(No space left on device) --> [context]zKq7AVECAAAAADBxegAOdnB4ZAAAeF4rbGlidm1hY29yZS5zbwAAEBcbAMppGAC8HisAtJgaAH6mGgG4w6d2cHhkAAEBxqcBCsinAeEoVQG9+1QBap9TAuAFAmxpYmMuc28uNgABdZdT[/context] 2019-02-08T22:44:53.615Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down 2019-02-08T22:44:53.615Z info vpxd[7FB5996C4800] [Originator@6876 sub=SupportMgr] Wrote uptime information 2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] Setting VDB delay statements queue size to 11000 transactions for 11 GB RAM dedicated to vpxd. 2019-02-08T22:44:53.616Z info vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc 2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=CryptUtil] [static bool Vpx::Common::CryptUtil::UnmungePasswordToBuffer(VpxDecrypter*, const string&, char*, size_t)] invalid decrypter 2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from 00000:0 2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType]: Database error: ODBC error: (00000) - 2019-02-08T22:44:53.617Z error vpxd[7FB5996C4800] [Originator@6876 sub=Default] Error getting configuration info from the database 2019-02-08T22:44:53.617Z warning vpxd[7FB5996C4800] [Originator@6876 sub=Main] Database not initialized. Nothing to unlock 2019-02-08T22:44:53.617Z info vpxd[7FB5996C4800] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now
This issue may caused because of Inodes are full on vCenter appliance. More details on Inodes
To see current storage usage of iNodes, run following command.
root@vCenterServer [ / ]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on devtmpfs 4115632 531 4115101 1% /dev tmpfs 4117336 4 4117332 1% /dev/shm tmpfs 4117336 690 4116646 1% /run tmpfs 4117336 16 4117320 1% /sys/fs/cgroup /dev/sda3 712704 712704 0 100% / tmpfs 4117336 77 4117259 1% /tmp /dev/sda1 32768 305 32463 1% /boot /dev/mapper/log_vg-log 1638400 14332 1624068 1% /storage/log /dev/mapper/dblog_vg-dblog 1638400 22 1638378 1% /storage/dblog /dev/mapper/db_vg-db 1638400 2854 1635546 1% /storage/db /dev/mapper/seat_vg-seat 13107200 4430 13102770 1% /storage/seat /dev/mapper/netdump_vg-netdump 655360 11 655349 1% /storage/netdump /dev/mapper/autodeploy_vg-autodeploy 1638400 13 1638387 1% /storage/autodeploy /dev/mapper/updatemgr_vg-updatemgr 6553600 273 6553327 1% /storage/updatemgr /dev/mapper/imagebuilder_vg-imagebuilder 1638400 14 1638386 1% /storage/imagebuilder /dev/mapper/core_vg-core 6553600 15 6553585 1% /storage/core
This command will give you iNode counts
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
sample output
875 /usr/bin
1417 /opt/vmware/lib/python2.7/test
1584 /usr/lib/vmware-vsphere-ui/server/work/deployer/s/global/39/0/h5ngc.war/resources/libs/angular-i18n
5362 /usr/share/man/man3
607550 /var/spool/mqueue
Go to path where you see most of the inodes, then get file count.
[/var/spool/mqueue ]# ls -ltrh | wc 607551 5467952 35237911
Delete inodes, take snapshot of vCenter VM for safer side.
find /var/spool/mqueue/ -type f -print0 | xargs -0 rm -f
Verify space for inodes again using command ‘df -i
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on devtmpfs 4115632 531 4115101 1% /dev tmpfs 4117336 4 4117332 1% /dev/shm tmpfs 4117336 690 4116646 1% /run tmpfs 4117336 16 4117320 1% /sys/fs/cgroup /dev/sda3 712704 105154 607550 15% / tmpfs 4117336 77 4117259 1% /tmp /dev/sda1 32768 305 32463 1% /boot /dev/mapper/log_vg-log 1638400 14332 1624068 1% /storage/log /dev/mapper/dblog_vg-dblog 1638400 22 1638378 1% /storage/dblog /dev/mapper/db_vg-db 1638400 2854 1635546 1% /storage/db /dev/mapper/seat_vg-seat 13107200 4430 13102770 1% /storage/seat /dev/mapper/netdump_vg-netdump 655360 11 655349 1% /storage/netdump /dev/mapper/autodeploy_vg-autodeploy 1638400 13 1638387 1% /storage/autodeploy /dev/mapper/updatemgr_vg-updatemgr 6553600 273 6553327 1% /storage/updatemgr /dev/mapper/imagebuilder_vg-imagebuilder 1638400 14 1638386 1% /storage/imagebuilder /dev/mapper/core_vg-core 6553600 15 6553585 1% /storage/core
