What's the procedure to re-download locally deleted files using SFTP Inbound

Question

As per this doc couldn't find the right process to re-downloading a locally removed file from remote SFTP.

The requirement is, to delete local file which already been fetched from remote SFTP and use sftp-inbound-adapter (DSL configuration) to re-fetch that same file when required. In this implementation, MetadataStore haven't been persisted into any external system like PropertiesPersistingMetadataStore or Redis Metadata Store. So as per doc, MetadataStore stored in In-Memory.

Couldn't find any way to remove meta data of that remote file from MetadataStore to re-fetch the locally deleted file using file_name. And don't have any clue, how should this removeRemoteFileMetadata() callback needs to be implemented (according to this doc).

Configuration class contain followings:

    @Bean
    public IntegrationFlow fileFlow() {
        SftpInboundChannelAdapterSpec spec = Sftp.inboundAdapter(sftpConfig.getSftpSessionFactory())
                .preserveTimestamp(true)
                .patternFilter(Constants.FILE_NAME_CONVENTION)
                .remoteDirectory(sftpConfig.getSourceLocation())
                .autoCreateLocalDirectory(true)
                .deleteRemoteFiles(false)
                .localDirectory(new File(sftpConfig.getDestinationLocation()));

        return IntegrationFlows
                .from(spec, e -> e.id("sftpInboundAdapter").autoStartup(false)
                        .poller(Pollers.fixedDelay(5000).get()))
                .channel(MessageChannels.direct().get())
                .handle(message -> {
                    log.info("Fetching File : " + message.getHeaders().get("file_name").toString());
                })
                .get();
    }

Towfiqul Islam · Answer 1 · 2020-05-29T13:50:04.437

I tried to solve this and I used Tanvir Hossain's reference code. I coded like this.

@Bean
public IntegrationFlow fileFlow() {
    SftpInboundChannelAdapterSpec spec = Sftp
            .inboundAdapter(sftpConfig.getSftpSessionFactory())
            .preserveTimestamp(true)
            .filter(sftpFileListFilter())
            .localFilter(systemFileListFilter())
            .remoteDirectory(sftpConfig.getSourceLocation())
            .autoCreateLocalDirectory(true)
            .deleteRemoteFiles(false)
            .localDirectory(new File(sftpConfig.getDestinationLocation()));

    return IntegrationFlows
            .from(spec, e -> e.id("sftpInboundAdapter").autoStartup(false)
                    .poller(Pollers.fixedDelay(5000).get()))
            .channel(MessageChannels.direct().get())
            .handle(message -> {
                log.info("Fetching File : " 
                        + message.getHeaders().get("file_name").toString());
            })
            .get();
}


private FileSystemPersistentAcceptOnceFileListFilter systemFileListFilter() {

    return new FileSystemPersistentAcceptOnceFileListFilter(store(), prefix);
}


private ChainFileListFilter<ChannelSftp.LsEntry> sftpFileListFilter() {

    ChainFileListFilter<ChannelSftp.LsEntry> chainFileListFilter = 
                                                new ChainFileListFilter<>();
    chainFileListFilter.addFilters(
            new SftpPersistentAcceptOnceFileListFilter(store(), prefix),
            new SftpSimplePatternFileListFilter(sftpConfig.getFileFilterValue())
    );
    return chainFileListFilter;
}

@Bean
public SimpleMetadataStore store() {
    return new SimpleMetadataStore();
}

and my Controller for removing metadata is like below :

public class Controller { 

    private final SimpleMetadataStore simpleMetadataStore;

    public Controller(SimpleMetadataStore simpleMetadataStore) {

           this.simpleMetadataStore = simpleMetadataStore;

    }

    @GetMapping("/test/remove-metadata/{type}/{fileName}")
    @ResponseBody
    public String removeFileMetadata(
             @PathVariable("fileName") String fileName,
             @PathVariable("type") String type
    ) {
        String prefix = definedPrefix;
        String filePath = "";
        if(type.equals("local")){
            filePath = "/local/storage/path/" + fileName;
        }else if(type.equals("remote")){
            filePath = fileName
        }
        String key = prefix + filePath;

        simpleMetadataStore.remove(key);
        return key;
    }

}

I am getting my desired file. It is re-fetching file for me.

Thanks, i was adding `SftpPersistentAcceptOnceFileListFilter` filter and `SftpSimplePatternFileListFilter` incorrectly. this helped a lot. — Tanvir Hossain, May 29 '20 at 13:03
But in your controller, you used a type to choose absolute path or just file name. Which one worked for you? — Tanvir Hossain, May 29 '20 at 13:10
I added these for testing purpose. But the **remote** type worked for me. — Towfiqul Islam, May 29 '20 at 13:16

Gary Russell · Accepted Answer · 2020-05-28T20:32:07.097

2

Use a ChainFileListFilter, with a SftpSimplePatternFileListFilter and a SftpPersistentAcceptOnceFileListFilter.

Use a SimpleMetadataStore to store the state in memory (or some other MetadataStore).

new SftpPersistentAcceptOnceFileListFilter(store, "somePrefix");

Then, store.remove(key) where key is somePrefix + fileName.

Use a similar filter in the localFilter with FileSystemPersistentAcceptOnceFileListFilter.

edited May 28 '20 at 20:32

answered May 28 '20 at 20:01

Gary Russell

166,535
14
146
179

1

do i need to create a `SimpleMetadataStore` bean like this ? `@Bean public SimpleMetadataStore store() { return new SimpleMetadataStore(); }` – Tanvir Hossain May 28 '20 at 20:54
1

should i use absolute path of the _**remote file**_ or the _**local file**_ or just the _**file name**_ to construct the `key`? – Tanvir Hossain May 28 '20 at 21:21
1

For the sftp filter, it's the remote file name; for the file system filter it's the absolute path. See the `fileName()` method in each filter. It's called from `buildKey()` in the abstract class. – Gary Russell May 28 '20 at 21:27

What's the procedure to re-download locally deleted files using SFTP Inbound

2 Answers2