Deal with reserved characters when syncing volumes with different formatting

Question

I have a NAS and I want to copy some directories on it to an exfat formatted external HDD.

Problem is paths / filenames may contain reserved characters for exfat (see https://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations), (but they are UTF-8 encoded) for example:

/home/fricadelle/mount_point/my_mp3s/Jake "the snake" O'Brian/03 - is it music ?.mp3

Here " and ? are a problem. cp -r this folder or filename directly to my HDD results in cannot create regular file and No such file or directory errors.

I have not found yet a satisfactory solution. This How to bulk-rename files with invalid encoding or bulk-replace invalid encoded characters? suggests to bulk rename everything. I don't like that because the files play ok on my ubuntu desktop or mac, or my network audio player. I want to keep those characters in my NAS. I think I would also have to copy it first on my desktop, then bulk rename, then copy it over to my external HDD.

Ideally I would like to pass some option to some rsync-like that would map all forbidden characters to, say, space, on the fly. So that the file names / folder names are altered only on my external HDD.

JamesIsIn · Answer 1 · 2020-01-08T12:18:40.363

I had to solve this problem for my music player sync script because it's 2020 and FAT (and NTFS) still can't handle question marks or colons.

PMPSync.sh

I chose to replace any instance of a ? or a : with __ (double underscores) as this was likely to never produce a problem. You could use other similar solutions and you could expand your list of replaceable characters to fit your needs.

function func_PMP_naughtyFilesystem() { 
    # if the PMP uses FAT or NTFS then sub certain characters with __ 
    printf '%s\n' "Some file systems are unable to handle certain characters.  " 
    printf '%s\n' "FAT and NTFS file systems in particular are bad about this.  " 
    printf '%s\n' "If you would like we can change : and ? into __ to avoid sync failures.  " 
    read -rep "Is your PMP formatted to use one of these lesser file systems?  (Y/n)  " -n1 naughtyFilesystem 
    printf '\n' 
} 

function func_CreateSourceAndDestination() { 
    # 
    for (( i = 0 ; i < ${#files_syncSource[@]} ; i++ )) ; do 
        files_syncDestination[${i}]="${files_syncSource[${i}]#${directory_MusicLibraryRoot_source}}" 
        # printf '%s\n' "${files_syncDestination[${i}]}" # useful in debugging 
        if [ "${naughtyFilesystem}" != "n" ] ; then 
            files_syncDestination[${i}]="${files_syncDestination[${i}]//\:/__}" 
            files_syncDestination[${i}]="${files_syncDestination[${i}]//\?/__}" 
        fi 
        file_destinationPath="$( dirname -- "${directory_PMPRoot_destination}${files_syncDestination[${i}]}" )" 
        if [ ! -d "${file_destinationPath}" ] ; then 
            mkdir -p "${file_destinationPath}" 
        fi 
        # try process substitution 
        # rsync --files-from=<( printf "%s\n" "${files[@]}" ) source destination 
        # rsync -rltDvPm --files-from=<( printf "%s\n" "${files_syncSource[${i}]}" ) "${files_syncDestination[${i}]}" 
        # the above needs to lose the interation and destination should just be PMProot I guess 
        rsync -rltDvPmz "${files_syncSource[${i}]}" "${directory_PMPRoot_destination}${files_syncDestination[${i}]}" 
    done 
}

First I ask the user if they want to make the substitutions and then unless they answer no I make those substitutions just prior to making each sync.

Let me know if you have any questions.

Deal with reserved characters when syncing volumes with different formatting

1 Answers1