That's what I come up with:
- name: Get directory listing
  find:
    path: "{{ directory }}" 
    file_type: any
    hidden: yes
  register: directory_content_result
- name: Remove directory content
  file:
    path: "{{ item.path }}" 
    state: absent
  with_items: "{{ directory_content_result.files }}" 
  loop_control:
    label: "{{ item.path }}" 
First, we're getting directory listing with find, setting 
- file_typeto- any, so we wouldn't miss nested directories and links
- hiddento- yes, so we don't skip hidden files
- also, do not set recursetoyes, since it is not only unnecessary, but may increase execution time.
Then, we go through that list with file module. It's output is a bit verbose, so loop_control.label will help us with limiting output (found this advice here).
But I found previous solution to be somewhat slow, since it iterates through the content, so I went with:
- name: Get directory stats
  stat:
    path: "{{ directory }}"
  register: directory_stat
- name: Delete directory
  file:
    path: "{{ directory }}"
    state: absent
- name: Create directory
  file:
    path: "{{ directory }}"
    state: directory
    owner: "{{ directory_stat.stat.pw_name }}"
    group: "{{ directory_stat.stat.gr_name }}"
    mode: "{{ directory_stat.stat.mode }}"
- get directory properties with the stat
- delete directory
- recreate directory with the same properties.
That was enough for me, but you can add attributes as well, if you want.