12

I took over a project that is running on Ansible for server provisioning and management. I'm fairly new to Ansible but thanks to the good documentation I'm getting my head around it. Still I'm having an error which has the following output:

failed: [build] (item=[u'software-properties-common', u'python-pycurl', u'openssh-server', u'ufw', u'unattended-upgrades', u'vim', u'curl', u'git', u'ntp']) => {"failed": true, "item": ["software-properties-common", "python-pycurl", "openssh-server", "ufw", "unattended-upgrades", "vim", "curl", "git", "ntp"], "msg": "Failed to lock apt for exclusive operation"}

The playbook is run with sudo: yes so I don't understand why I'm getting this error (which looks like a permission error). Any idea how to trace this down?

- name: "Install very important packages"
  apt: pkg={{ item }} update_cache=yes state=present
  with_items:
    - software-properties-common # for apt repository management
    - python-pycurl # for apt repository management (Ansible support)
    - openssh-server
    - ufw
    - unattended-upgrades
    - vim
    - curl
    - git
    - ntp

playbook:

- hosts: build.url.com
  sudo: yes
  roles:
    - { role: postgresql, tags: postgresql }
    - { role: ruby, tags: ruby }
    - { role: build, tags: build }
supersize
  • 13,764
  • 18
  • 74
  • 133
  • can you also show us part of the playbook that's causing this issue along with the ansible command that you're executing? – Abhishek Balaji R Jul 23 '17 at 19:53
  • @AbhishekBalajiR edited – supersize Jul 23 '17 at 19:58
  • 1
    can you add `become: true` in the Install task? if that doesn't work, it might be worthwhile to check if you are able to manually install packages, otherwise try restarting the server. There are some suggestions here: [link](https://github.com/ansible/ansible/issues/4355) – Abhishek Balaji R Jul 23 '17 at 20:10
  • This is a very common situation when provisioning Ubuntu (and likely some other distributions). You try to run Ansible while automatic updates are running in background (which is what happens right after setting up a new machine). As APT uses semaphore, Ansible gets kicked out. Search AskUbuntu or Unix.SE for suggestions. The playbook is ok, you can verify it by running it later. – techraf Jul 24 '17 at 00:23

3 Answers3

18

I just had the same issue on a new VM. I tried many approaches, including retrying the apt commands, but in the end the only way to do this was by removing unattended upgrades.

I'm using raw commands here, since at this point the VM doesn't have Python installed, so I need to install that first, but I need a reliable apt for that.

Since it is a VM and I was testing the playbook by resetting it to a Snapshot, the system date was off, which forced me to use the date -s command in order to not have problems with the SSL certificate during apt commands. This date -s triggered an unattended upgrade.

So this snippet of a playbook is basically the part relevant to disabling unattended upgrades in a new system. They are the first commands I'm issuing on a new system.

- name: Disable timers for unattended upgrade, so that none will be triggered by the `date -s` call.
  raw: systemctl disable --now {{item}}
  with_items:
    - 'apt-daily.timer'
    - 'apt-daily-upgrade.timer'

- name: Reload systemctl daemon to apply the new changes
  raw: systemctl daemon-reload

# Syncing time is only relevant for testing, because of the VM's outdated date.
#- name: Sync time
#  raw: date -s "{{ lookup('pipe', 'date') }}"

- name: Wait for any possibly running unattended upgrade to finish
  raw: systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true

- name: Purge unattended upgrades
  raw: apt-get -y purge unattended-upgrades    

- name: Update apt cache
  raw: apt-get -y update

- name: If needed, install Python
  raw: test -e /usr/bin/python || apt-get -y install python

Anything else would cause apt commands to randomly fail because of locking issues caused by unattended upgrades.

Ikar Pohorský
  • 4,617
  • 6
  • 39
  • 56
Daniel F
  • 13,684
  • 11
  • 87
  • 116
  • Hi, I've tried the above code, but Ansible just tells me that systemctl, or systemd-run, or apt-get command cannot be found. – Mark R Jan 19 '23 at 11:35
11

This is a very common situation when provisioning Ubuntu (and likely some other distributions). You try to run Ansible while automatic updates are running in background (which is what happens right after setting up a new machine). As APT uses semaphore, Ansible gets kicked out.

The playbook is ok and the easiest way to verify is to run it later (after automatic update process finishes).

For a permanent resolution, you might want to:

  • use an OS image with automatic updates disabled
  • add an explicit loop in the Ansible playbook to repeat the failed task until it succeeds
techraf
  • 64,883
  • 27
  • 193
  • 198
  • This was it! :) – supersize Jul 24 '17 at 09:04
  • 3
    There is also a trick to wait for init-script to finish: https://serverfault.com/questions/855872/how-to-wait-for-user-data-script-to-run-when-starting-ec2-instances-with-ansible/855898#855898 – Konstantin Suvorov Jul 24 '17 at 10:13
  • @KonstantinSuvorov Are you sure it's a part of the boot process? APT automatic updates run cyclically in background. Even on a system which is running without restarts. – techraf Jul 24 '17 at 12:18
  • In general, apt updates can surely occur anytime. But when we talk about server provisioning, apt lock is very often held by cloud-init, this is what my remark about. – Konstantin Suvorov Jul 24 '17 at 12:22
  • Interesting... So if I reverted a machine from a snapshot it will update before `boot-finished` was created? I can't wait to test... :) – techraf Jul 24 '17 at 12:34
0

I also got this error, This is what fixed my issue.

name: update apt package
  become: yes
  apt:
    name: '*'
    state: latest
    update_cache: true
  register: apt_get_status    <<<<<<<<<<<<<<
  until: apt_get_status is success  <<<<<<<<
  retries: 60
  delay: 10
manish roy
  • 1
  • 1
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 18 '23 at 12:05