Azure cloud-init: hanging at modules-final stage

Question

I'm experiencing some issues with Azure cloud-init on Ubuntu 20.04 and am looking for some guidance on the next steps for troubleshooting.

At the moment, I'm just using a test cloud-init file, testing installing my required packages. This is the cloud-init file. The custom apt configuration is in there because of waagent was locking the process (ref. Azure cloud-init: Failed to install packages):

#cloud-config

apt:
  conf: |
    Acquire::Retries "60";
    DPkg::Lock::Timeout "60";

package_update: true

# Install packages required for NetBox
packages:
  - python3
  - python3-pip
  - python3-venv
  - python3-dev
  - build-essential
  - libxml2-dev
  - libxslt1-dev
  - libffi-dev
  - libpq-dev
  - libssl-dev
  - zlib1g-dev
  - nginx

When I create the Azure VM with this configuration I get no errors, but in checking the job status 'modules-final' is always in the 'running' state and never seems to complete.

azureuser@cloud-init-test:~$ cat /var/lib/cloud/data/status.json 
{
 "v1": {
  "datasource": "DataSourceAzure [seed=/dev/sr0]",
  "init": {
   "errors": [],
   "finished": 1671618221.4859056,
   "start": 1671618217.8323092
  },
  "init-local": {
   "errors": [],
   "finished": 1671618215.7414432,
   "start": 1671618214.2585633
  },
  "modules-config": {
   "errors": [],
   "finished": 1671618332.374849,
   "start": 1671618275.386914
  },
  "modules-final": {
   "errors": [],
   "finished": null,
   "start": 1671618332.9963179
  },
  "modules-init": {
   "errors": [],
   "finished": null,
   "start": null
  },
  "stage": "modules-final"
 }
}

If I re-run the process manually (ref. How to re-run cloud-init without reboot), then I get the following:

azureuser@cloud-init-test:~$ sudo cloud-init modules --mode=final
Cloud-init v. 22.4.2-0ubuntu0~20.04.2 running 'modules:final' at Wed, 21 Dec 2022 11:42:19 +0000. Up 665.68 seconds.
Ign:1 http://security.ubuntu.com/ubuntu focal-security InRelease
Ign:2 http://archive.ubuntu.com/ubuntu focal InRelease
...

(I had to chop the list of URLs to avoid having the question flag as spam :)). This list just scrolls seemingly without timeout. I'm not familiar with what this means, but it seems like the URLs are where packages are held so I get the feeling that it's trying to run the package installs but failing and retrying?

Does anyone have any insight into how I can proceed to troubleshoot?

Thanks!

score 2 · Accepted Answer · answered Dec 26 '22 at 13:29

I tried to reproduce the same in my environment and got below results:

Try to ping the URLs to check whether you are able to do so like below enter image description here Try to Sudo ping 8.8.8.8 inside your distro (Ubuntu). If you cannot reach network, search for network related issues in wsl.

If network is reachable, it's either DNS problem or host being blocked by your network.

To check DNS, try Sudo ping archive.ubuntu.com.

Step 1: sudo  systemctl restart systemd-resolved.service  
Step 2: sudo cloud-init modules --mode=final

Execute the commands below if above steps doesn't give you required output

sudo  systemctl unmask systemd-resolved.service  
sudo  systemctl enable systemd-resolved.service  
sudo  systemctl start systemd-resolved.service

Thanks for taking the time to respond. It was DNS (it's always DNS). I was getting caught out because there was a script that runs post cloud-init that was adding in the DNS resolver config, so when I was testing manually after the package installs had failed, everything looked fine. I ended up moving the addition of the DNS resolvers into the bootcmd portion of the cloud-init file and everything works. Thanks again. — AndyB, Jan 03 '23 at 10:21

Azure cloud-init: hanging at modules-final stage

1 Answers1