20

I am trying to automate the process of syncing my web assets with Google Cloud Storage. I basically need to copy everything in my development directory up to the cloud. However, I need to ignore the .git directory and some other irrelevant files.

I can't just do a gsutil cp -R . <dest> because that takes absolutely everything, including .git. I tried find . | fgrep git | gsutil cp -I <dest> but that flattens all directories and puts them in root!

Is there a way I can solve this with gsutil or do I have to do a loop in script which uploads all directories (except .git) with -R and then uploads individual files in current directory?

Augustin
  • 2,444
  • 23
  • 24
Shahbaz
  • 10,395
  • 21
  • 54
  • 83
  • Possible duplicate of [rsync will not exclude hidden files in gsutil 4.15](http://stackoverflow.com/questions/33702631/rsync-will-not-exclude-hidden-files-in-gsutil-4-15) – mimming Dec 30 '15 at 00:29

3 Answers3

42

You could use a command like:

gsutil rsync -x '\.git.*' dev_dir gs://your-bucket

See Google Storage - rsync - Synchronize content of two buckets/directories

Dan Dar3
  • 1,199
  • 13
  • 23
Mike Schwartz
  • 11,511
  • 1
  • 33
  • 36
  • 3
    This is an appropriate answer. Not sure why it got downvoted. – sirugh Jan 18 '16 at 16:43
  • This has the downside of not being able to use `-z js,css,json,html,svg` to put these files in GCS compressed with gzip. This makes the storage cost slightly smaller, but the key value of it is that **without this the files will be served from the GCS bucket without compression**. – Greg Dubicki May 19 '23 at 10:09
8

You have two options:

A) Remove the git files after they are uploaded:

gsutil rm gs://bucket/\*.git\*

B) Use find to exclude git files:

find . -not -path '*/.git' -type f -printf '%P\n' | xargs -I '{}' gsutil cp '{}' gs://bucket/'{}'

Source: https://groups.google.com/forum/#!topic/gsutil-discuss/zoHhkTPhiNc

It would've been much easier if gsutil implemented rsync, this would've been easier with their --exclude flag.

Aziz Saleh
  • 2,687
  • 1
  • 17
  • 27
0

Your web assets are text files - JS, CSS, etc. - which you want to serve compressed, don't you?

Then you need to be aware that GCS requires you to upload such files compressed, to serve them compressed!

Therefore you want to use the -z parameter of gsutil to compress such files.

Expanding on Aziz Saleh's answer then, you probably want to do this:

path '*/.git' -type f -printf '%P\n' | xargs -I '{}' gsutil cp -z js,css,json,html,htm,xml '{}' gs://bucket/'{}'

(You cannot use -z parameter with the Mike Schwartz's gsutil rsync solution.)

Read more about gsutil cp here and about "Transcoding of gzip-compressed files" here.

Greg Dubicki
  • 5,983
  • 3
  • 55
  • 68