gsutil / gcloud storage file listing sorted date descending?

Question

Is there no way to get a file listing out from a Google Cloud Storage bucket that is sorted by date descending? This is very frustrating. I need to check the status of files that are uploaded and the bucket has thousands of objects.

gsutil ls does not have the standard linux -t option.

Google cloud console also lists it but does not offer sorting options.

score 79 · Answer 1 · edited Apr 29 '20 at 12:16

79

I use this as a workaround:

gsutil ls -l gs://[bucket-name]/ | sort -k 2

This outputs full listing including date as the second field, sort -k 2 then sorts by this field.

edited Apr 29 '20 at 12:16

Gerard

2,832
3
27
39

answered Aug 06 '18 at 14:07

torno

1,162
1
8
14

1

This is best answer to me. – Nam G VU Jun 02 '19 at 23:47
1

Short explanation of why this works: The second field in the ls output is a timestamp, `-k 2` sorts by this field. – Gerard Feb 15 '20 at 14:47
2

One downside to be aware of for large buckets: it has to download the entire list of items before the sorting can be applied. – xaviert Jul 23 '20 at 13:19
This does alphabetical sorting, so if you want to sort by size, this does not work. – Soumitri Pattnaik May 27 '21 at 11:48
This sorts by oldest first how do I reverse it? – Nathan McKaskle Oct 22 '21 at 14:52
1

@NathanMcKaskle I tried `sort -r -k 2` but the issue is that the "TOTAL: X objects..." line will show up first. Ended up doing `sort -k 2 | tail -n 2 | head -n 1`. – Jason Fry Dec 13 '21 at 17:49
Is there a way to delete older files from the list. Say retaining only latest 5 at any given moment – 7guyo Feb 08 '22 at 08:08
@7guyo you should be able to just use `head -5` or `tail -5` depending on the order of your sort. e.g. `sort -k 2 | tail -5` – fantabolous Aug 13 '22 at 07:15

Mike Schwartz · Answer 2 · 2018-03-02T14:01:50.967

6

The only ordering supported by GCS is lexicographic.

As a workaround, if it's possible for you to name your objects with a datestamp, that would give you a way to list objects by date.

edited Mar 02 '18 at 14:01

answered Mar 01 '18 at 14:46

Mike Schwartz

11,511
1
33
36

1

I'm guessing this is by design? To allow infinitely long directories / buckets? – Dhiraj Gupta Mar 02 '18 at 05:54
1

I've been looking for official documentation of this feature (lexicographic object list ordering) and have been unable to find it. Are there any links that you know of? – Mar 22 '18 at 19:01
The ordering must be deterministic since they allow paging of results, it's just not necessarily alphabetic (or controllable?). I'd love to see some official documentation on this as well as I haven't yet been able to find it. – Sebastien Martin Jun 28 '18 at 19:08
3

I too was looking for official documentation about this behavior. I found this reference: https://cloud.google.com/storage/docs/listing-objects "This page shows you how to list the objects stored in your Cloud Storage buckets, which are ordered in the list lexicographically by name". However, it's not on the official API page: https://cloud.google.com/storage/docs/json_api/v1/objects/list – bboe Feb 19 '20 at 18:55

gsutil / gcloud storage file listing sorted date descending?

2 Answers2

Linked