Getting all FTP directory/file listings recursively in one call

Question

I'm creating a backup program that backs data up to FTP. To archive effectively, I need to do several file attribute comparisons. Right now, I am using FluentFTP and calling FtpClient.FileExists FtpClient.GetFileSize, and FtpClient.GetModifiedTime per file. This is obviously not very optimal.

If I could download the FTP directory's entire tree structure in a single call along with file attributes, it would improve back-up speed substantially. The alternative is to build a local index, but then I have to make sure it's being updated properly and also account for the possibility of it getting corrupted.

Is there any way to do this other than roll my own solution?

Answer 1

FTP offers these commands to retrieve an information about remote files:

• NLST to retrieve names of files in certain directory – Supported by all servers.
• SIZE and MDTM to retrieve size and modification file of certain file – Supported by virtually all servers.
• LIST to retrieve directory listing, including file attributes – Supported by all servers, but the listing is not standardized and thus not really machine-readable. Though most servers will support *nix style listing (like *nix ls command). Though each will have its quirks.
• MLSD to retrieve directory listing, including file attributes in machine readable format – This command is least supported. It's particularly not supported by *nix vsftpd and Windows IIS.

So you do not have to use FtpClient.GetFileSize and FtpClient.GetModifiedTime per file. Use FtpClient.GetListing per directory. Internally it uses MLSD, if supported by the server. Otherwise it falls back to LIST and tries to parse the listing.

---

Some FTP servers (like ProFTPD) do support a non-standard proprietary -R switch to the LIST command that will make them return listing across all subfolders. FluentFTP supports that too (FtpListOption.Recursive). Though note that FluentFTP uses -R with LIST only, while it prefers using MLSD, if the server supports it. So to make sure LIST -R is used, you need to use both FtpListOption.Recursive and FtpListOption.ForceList.

If your server does not support the -R switch, you have to implement recursion yourself. Or use an FTP client that has API for it.

For example with my WinSCP .NET assembly, you can use Session.EnumerateRemoteFiles:

IEnumerable<RemoteFileInfo> allFiles =
    session.EnumerateRemoteFiles("/", null, EnumerationOptions.AllDirectories);