Ubuntu 20.04 Template Problem

[rastopato]

problem

Deployment of VMs from Ubuntu 20.04.06 LTS stopped working after upgrading ACS to 4.21 and VMware to 8.0.3. Every VM deployed from this template is stuck at boot with corrupted ext4 filesystem errors.

I tried every possible combination of preparing new ubuntu template, disk adapters, bios/uefi, etc, etc, nothing worked.
Ubuntu 22.04 and 24.04 has no problem.

I tried it on environment where are previously deployed Vms from Ubuntu 20.04 template without issues. Now, after upgrade, it is the same error.
I got this error in all my production and testing environments.

Finally i tried it on environment on ACS 4.20.1 and VMware 8.0.0 and there is no problem with Ubuntu 20.04.

management-server.log.zip

versions

ACS: 4.21
VMware vCenter 8.0.3
Vmware ESXi 8.0.3
Ubuntu 20.04.06 LTS
Primary storage is VMFS datastores on SAN storage

The steps to reproduce the bug

1. Create Template with Ubuntu 20.04.06 LTS
2. Deploy new VM from it
3. It should be stack at the boot with filesystem corruption error.
4. If you not prepare any /etc/cloud/cloud.cfg.d/ files, it maybe start, but you cant even write username/password because of corrupted filesytem

What to do about it?

Please resolve it, or suggest workaround.

[weizhouapache]

You may refer to #7572 and #9625

[rastopato]

i'm not sure, what you are refering to. esx 8U3 is supported from version 4.20.1

[weizhouapache]

i'm not sure, what you are refering to. esx 8U3 is supported from version 4.20.1

if you go through the issue #7572, you might know that we face a similar issue with cloudstack systemvm template since VMware 8.0.1. The root cause is still unknown but we have addressed the issue by #9625. To be clear

• add fsck.mode=force fsck.repair=yes to boot args
• add depmod -a in init.sh

The same issue might impact user templates too. This is why I asked you to have a look.

[rastopato]

I assumed if these issues are closed and compatibility matrix is ok with ESX 8.0.3, it will be ok. Anyway, all other templates are ok with 8.0.3, except Ubuntu 20.04. I added fsck.mode, fsck.repair and depmod -a. I'm now little step closer, but it is still not working:

[weizhouapache]

I assumed if these issues are closed and compatibility matrix is ok with ESX 8.0.3, it will be ok. Anyway, all other templates are ok with 8.0.3, except Ubuntu 20.04. I added fsck.mode, fsck.repair and depmod -a. I'm now little step closer, but it is still not working:

You may need to run "depmod -a" when create the template.
CloudStack uses packer, it is one of the last steps in the process, see #11128

[rastopato]

I did, of course. Now I tried to change the way i ran it, and i ran it at the end, and it looks promising now. I will let you tomorrow.

[rastopato]

Well, no luck. i run depmod -a, set fsck in the grub. Fsck is running during booting process, fix a lot of things, but VM is obviously still corrupted:

[sureshanaparti]

@rastopato We fixed similar file system errors with the below options mentioned by @weizhouapache already, in this PR: #9625. Maybe, it's hitting different issue now. Can you try to create a VM in vCenter/ESXi directly using this template and check.

• add fsck.mode=force fsck.repair=yes to boot args
• add depmod -a in init.sh

[rastopato]

Hi guys, i've done a little bit more troubleshooting based on your comment and I found out some issues. When i prepared Template VM i export it from VMware using vmware ovftool.

When I import this ova to the target vmware environment, it is working with no problems.

When I import this ova to the ACS as template it is saved on NFS storage in separate directory with some files in it. There is whole .ova file with different name (3ff4f989-6751-3607-9fd1-63c1429da88d.ova) and most probably the same .ova but unpacked to vmdk, ovf, nvram and mf file.

When I try deploy OVF template directly on VMware from these unpacked 4 files, i got error: "Failed to deploy OVF package. Cause: A specified parameter was not correct: The checksum(s) from the provided manifest file do not match the content of file(s): ubuntu2004.ovf."

When I copy whole .ova file from NFS storage (for example 3ff4f989-6751-3607-9fd1-63c1429da88d.ova), unpack it manually and Deploy it as OVF directly on vmware, it is working

Then i tried to delete 4 files from directory on NFS and unpack there that .ova file, delete all templates from vCenter and also from primary storage. Then i tried deploy VM again from ACS, but still no success - corrupted filesystem.

I just realize that all Templates which were working on previous versions are now broken. I was able to repair some of them by simply regenerate and import new template (Ubuntu 22.04, 24.04, RHEL9.x), some of them are broken with corrupted filesystems also after regenerating it (Ubuntu 20.04, Rocky Linux 9.x, CentOS Stream 9)

[rastopato]

It is definitely corrupting filesystem when VMware Templates are creating. During the process of creating instance, multiple VMware templates are created by deploying .ovf template from NFS storage to Primary NTFS storage. This templates are already corrupted, when i try to deploy Vm from it directly in VMware.

[RosiKyu]

@rastopato, I have tested this and was not able to reproduce the issue.

Environment

Component	Version
CloudStack	4.21.0.0
VMware ESXi	8.0.3.0
Primary Storage	VMFS
Management Server	Oracle Linux 8

Templates Tested

Template Name	Deploy-as-is	Status
Ubuntu 20.04	false	Ready
Ubuntu 20.04 deploy-as-is	true	Ready
Ubuntu 20.04 from lab deploy as is	true	Ready

(localcloud) 🐱 > list templates templatefilter=all filter=name,deployasis,hypervisor,format,isready
{
  "count": 5,
  "template": [
    {
      "deployasis": false,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "Ubuntu 20.04"
    },
    {
      "deployasis": true,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "Ubuntu 20.04 deploy-as-is"
    },
    {
      "deployasis": true,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "Ubuntu 20.04 from lab deploy as is"
    },
    {
      "deployasis": false,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "CentOS 5.3(64-bit) no GUI (vSphere)"
    },
    {
      "deployasis": false,
      "format": "OVA",
      "hypervisor": "VMware",
      "isready": true,
      "name": "SystemVM Template (vSphere)"
    }
  ]
}

Storage

(localcloud) 🐱 >  list storagepools
{
  "count": 2,
  "storagepool": [
    {
      "clusterid": "42a694eb-1ed5-479e-bfca-04389400e770",
      "clustername": "p1-c1",
      "created": "2026-02-05T10:35:55+0000",
      "details": {},
      "disksizeallocated": 70852298752,
      "disksizetotal": 160792838144,
      "disksizeused": 15406727168,
      "hasannotations": false,
      "hypervisor": "VMware",
      "id": "d173e0a1-40db-3053-8caf-f20f437151cf",
      "ipaddress": "VMFS datastore: /Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
      "istagarule": false,
      "managed": false,
      "name": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
      "overprovisionfactor": "2.0",
      "path": "/Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
      "podid": "8ed4917e-5f74-44f6-b395-891580cf7c47",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "type": "VMFS",
      "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
      "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
    },
    {
      "clusterid": "42a694eb-1ed5-479e-bfca-04389400e770",
      "clustername": "p1-c1",
      "created": "2026-02-05T10:35:57+0000",
      "details": {},
      "disksizeallocated": 31960596480,
      "disksizetotal": 160792838144,
      "disksizeused": 6465519616,
      "hasannotations": false,
      "hypervisor": "VMware",
      "id": "a72b7e2f-47aa-3c24-9d23-2d2468ab16f0",
      "ipaddress": "VMFS datastore: /Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
      "istagarule": false,
      "managed": false,
      "name": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
      "overprovisionfactor": "2.0",
      "path": "/Trillian/ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri2",
      "podid": "8ed4917e-5f74-44f6-b395-891580cf7c47",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "CLUSTER",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "type": "VMFS",
      "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
      "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
    }
  ]
}

VMs Deployed

VM Name	State	Template
VM-07ef5514...	Running	Ubuntu 20.04 (deployasis=false)
test-deploy-as-is	Running	Ubuntu 20.04 deploy-as-is (deployasis=true)
test-lab-dai	Running	Ubuntu 20.04 from lab (deployasis=true)

list virtualmachines filter=name,state,templatename

(localcloud) 🐱 > list virtualmachines filter=name,state,templatename
{
  "count": 3,
  "virtualmachine": [
    {
      "name": "VM-07ef5514-382c-4014-b1b9-6041f60ea63a",
      "state": "Running",
      "templatename": "Ubuntu 20.04"
    },
    {
      "name": "test-deploy-as-is",
      "state": "Running",
      "templatename": "Ubuntu 20.04 deploy-as-is"
    },
    {
      "name": "test-lab-dai",
      "state": "Running",
      "templatename": "Ubuntu 20.04 from lab deploy as is"
    }
  ]
}

All three VMs boot successfully and keyboard input works at console login prompt.

Checksum Verification on Secondary Storage

Verified that extracted OVA files on secondary storage match the manifest:

cat ubuntu-focal-20.04-cloudimg.mf
SHA256(ubuntu-focal-20.04-cloudimg.vmdk)= 192b5c869b99a0db91f6567fe29f89d9b98984a3b16220ef06ab7320de779411
SHA256(ubuntu-focal-20.04-cloudimg.ovf)= 603242f28820af7fe2afb621990ce130b6c6a1cb10ed481b9254cadb83fd4547

sha256sum ubuntu-focal-20.04-cloudimg.ovf ubuntu-focal-20.04-cloudimg.vmdk
603242f28820af7fe2afb621990ce130b6c6a1cb10ed481b9254cadb83fd4547  ubuntu-focal-20.04-cloudimg.ovf ✓
192b5c869b99a0db91f6567fe29f89d9b98984a3b16220ef06ab7320de779411  ubuntu-focal-20.04-cloudimg.vmdk ✓

Checksums match — no corruption detected during OVA extraction.

The OVA extraction and VM deployment work correctly.

To help us investigate further, could you please provide:

1. Your problematic Ubuntu 20.04 OVA template (or a download link) so we can test with the exact same template
2. Template registration settings - was "Read Instance settings from OVA" enabled?

If you can share the OVA that consistently fails, we can compare it against templates that work and try identifying what triggers the corruption.

[RosiKyu]

I tried deploying your template and got "Operating System not found" - it won't boot at all.

Can you confirm:

1. Does your template use BIOS or UEFI firmware?
2. What disk controller?

In my test, the deployment had vmdk errors during first power-on attempts, and the VM eventually powered on but can't find the OS.

I have also tested with the official Ubuntu 20.04 cloud image: https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.ova

First deployment on ESXi 8.0.3 - boots cleanly, NO EXT4 errors. Can you test this same image on your ESXi 8.0.3 cluster?

[rastopato]

i'm using UEFI Secure Boot and pvscsi root disk controller. I'm creating template with VMversion 20.
Image from ubuntu is working for me as well.

[RosiKyu]

@rastopato, I have re-tested with your template:

Test Environment

• CloudStack: 4.21
• ESXi: 8.0.3 build-24022510
• Primary Storage: VMFS on iSCSI (LIO-ORG target, VMW_SATP_ALUA with SCSI reservations)

Template: Ubuntu2004.ova

• SHA-512: 4552487d45487477f9d8b97a67c4fba2d745bd20f733c9162a7fb24876abc612ae725800cda04300ce48a137dce994e3ce16b13f0668cfda27122c9ba36e13fc

(localcloud) 🐱 > list templates id=831221a1-5ab5-4591-ae8e-f06e9f05666e templatefilter=all
{
  "count": 1,
  "template": [
    {
      "account": "admin",
      "arch": "x86_64",
      "bits": 0,
      "checksum": "4552487d45487477f9d8b97a67c4fba2d745bd20f733c9162a7fb24876abc612ae725800cda04300ce48a137dce994e3ce16b13f0668cfda27122c9ba36e13fc",
      "childtemplates": [],
      "created": "2026-02-09T19:26:35+0000",
      "crossZones": false,
      "deployasis": false,
      "details": {
        "rootDiskController": "pvscsi"
      },
      "directdownload": false,
      "displaytext": "Reporter Ubuntu 20.04 template",
      "domain": "ROOT",
      "domainid": "47f1fcd3-027d-11f1-848e-1e00ef0003c4",
      "domainpath": "/",
      "downloaddetails": [
        {
          "datastore": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-sec1",
          "datastoreId": "118b6b6e-c668-4054-a81e-20bebd9daafa",
          "datastoreRole": "Image",
          "downloadPercent": "100",
          "downloadState": "DOWNLOADED"
        },
        {
          "datastore": "ref-trl-10863-v-Mol8-rositsa-kyuchukova-esxi-pri1",
          "datastoreId": "d173e0a1-40db-3053-8caf-f20f437151cf",
          "datastoreRole": "Primary",
          "downloadPercent": "100",
          "downloadState": "DOWNLOADED"
        }
      ],
      "forcks": false,
      "format": "OVA",
      "hasannotations": false,
      "hypervisor": "VMware",
      "id": "831221a1-5ab5-4591-ae8e-f06e9f05666e",
      "isdynamicallyscalable": false,
      "isextractable": false,
      "isfeatured": false,
      "ispublic": false,
      "isready": true,
      "name": "Ubuntu-20.04-reporter-template",
      "ostypeid": "78242165-027d-11f1-848e-1e00ef0003c4",
      "ostypename": "Ubuntu 20.04 LTS",
      "passwordenabled": false,
      "physicalsize": 1221115392,
      "requireshvm": true,
      "size": 21474836480,
      "sshkeyenabled": false,
      "status": "Download Complete",
      "tags": [],
      "templatetype": "USER",
      "url": "http://10.0.35.49:8888/ubuntu2004.ova",
      "zoneid": "ece6ad13-731c-4dc9-a052-102c801e87e9",
      "zonename": "ref-trl-10863-v-Mol8-rositsa-kyuchukova"
    }
  ]
}

VM Configuration (matching your setup)

• Boot type: UEFI Secure Boot
• Root disk controller: pvscsi
• OS: Ubuntu 20.04.6 LTS

Results

Filesystem: CLEAN - No corruption

• No EXT4 errors in dmesg
• VM boots and runs normally

My test environment also uses iSCSI-backed VMFS with SCSI reservations (ALUA) - architecturally similar to SAN storage. Both environments run Cloudstack 4.21 and ESXi 8.0.3

Despite similar env configuration, I cannot reproduce the corruption. I did observe sporadic VMDK locking during some deployments, but these resolved automatically - likely due to the retry mechanism in PR #10586 (fixed in 4.20.1) which retries PowerOn operations when lock issues occur.

This points to something specific in your environment:

• ESXi build/patch level - What is your exact build number?
• SAN vendor - Different vendors may have different interactions with ESXi 8.0.3 ?
• Storage drivers/VIBs - Any third-party VIBs installed?

This appears to be a VMware/SAN vendor-specific issue rather than a Cloudstack defect.

[rastopato]

Well, iSCSI is definitely not comparable.

ESXi build doesn't matter, i tried old ones, as well as last one 25067014.

We have enterprise setup: HPE servers, Cisco SAN and FC storage from HPE and/or DELL. This is reliable and trusted setup for many years with no issues with some custom vibs, etc. I have same results on different storage devices.

You have sporadic problem with accessing files during VM PowerOn.
I have problem with accessing file during template creation EVERY TIME i'm deploying new template. This is not during power on. This is when you deploying first instance from new ACS Template, ACS is calling VMware and creating multiple VMware templates (c935dc91d4013091a603258617947536). During this process there is multiple errors with accessing vmdk file, because is locked.
That's why I think there is a problem with corrupted filesystem.

This should be fixed here: #12335
I wish i could test it.

When i deploy instance from the same template on cluster with 8.0.0, there is no such error and no filesystem corruption.

[weizhouapache]

Well, iSCSI is definitely not comparable.

ESXi build doesn't matter, i tried old ones, as well as last one 25067014.

We have enterprise setup: HPE servers, Cisco SAN and FC storage from HPE and/or DELL. This is reliable and trusted setup for many years with no issues with some custom vibs, etc. I have same results on different storage devices.

You have sporadic problem with accessing files during VM PowerOn. I have problem with accessing file during template creation EVERY TIME i'm deploying new template. This is not during power on. This is when you deploying first instance from new ACS Template, ACS is calling VMware and creating multiple VMware templates (c935dc91d4013091a603258617947536). During this process there is multiple errors with accessing vmdk file, because is locked. That's why I think there is a problem with corrupted filesystem.

This should be fixed here: #12335 I wish i could test it.

When i deploy instance from the same template on cluster with 8.0.0, there is no such error and no filesystem corruption.

@rastopato
all these issues (image is corrupted, file is locked, power on failed, etc) appeared in VMware 8.0.1
we have put many efforts on the root cause diagnosis, but could not find .
for the 2nd and 3rd issues, we added the retry mechanism which seems to reduce the impact.
for the 1st issue, the root cause is still unknown. we have fixed the systemvm templates by "depmod -a". However, for existing user templates, we have no clue.