CURL fails with exit code 18 for SFTP with just few bytes remaining for file sizes less than120KB , but not when I add strace to sshd

Question

There is weird issue with curl in Amazon Linux 2 server.

It fails with exit code 18 with just about ~1MB bytes to read. So to troubleshoot the issue, I have setup local SFTP server.

so my curl command is like this: curl -u testuser:12345 -o /tmp/testfile sftp://127.0.0.1/home/testuser/testfile

Also the issue doesn't appear if the file size is less than ~120KB.

None of the below changes helped:

• Not using -o
• Quoting the url
• using --insecure
• using localhost instead of 127.0.0.1
• using -f before the url

debug output using -vvv doesn't tell me anything,

trace output using --trace doesn't tell me anything,

tcpdump data is encrypted and I can't tell what's going on.

I wish I can share all outputs. But I am working on a remote desktop with no way to transfer any files. So I can't share anything.

So, what's going on?

And yes, if I use sftp session and get command and that works.

curl output:

{ [30000 bytes data]
* transfer closed with 277200 bytes remaining to read
 35  420k   35  150k    0     0  1195k      0 --:--:-- --:--:-- --:--:-- 1201k
 35  420k   35  150k    0     0  1179k      0 --:--:-- --:--:-- --:--:-- 1179k
* Closing connection
curl: (18) transfer closed with 277200 bytes remaining to read

sftp-server debug3 log:

Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 0: realpath
Apr 03 21:31:30 my-host sftp-server[3159]: realpath "."
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 0: sent names count 1
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 1: open flags 1
Apr 03 21:31:30 my-host sftp-server[3159]: open "/home/testuser/testfile2" flags READ mode 0100644
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 1: sent handle handle 0
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 2: stat
Apr 03 21:31:30 my-host sftp-server[3159]: stat name "/home/testuser/testfile2"
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 2: sent attrib have 0xf
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 3: read "/home/testuser/testfile2" (handle 0) off 0 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 3: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 4: read "/home/testuser/testfile2" (handle 0) off 30000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 4: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 5: read "/home/testuser/testfile2" (handle 0) off 60000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 5: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 6: read "/home/testuser/testfile2" (handle 0) off 90000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 6: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 7: read "/home/testuser/testfile2" (handle 0) off 120000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 7: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 8: read "/home/testuser/testfile2" (handle 0) off 150000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 8: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 9: read "/home/testuser/testfile2" (handle 0) off 180000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 9: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 10: read "/home/testuser/testfile2" (handle 0) off 210000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 10: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 11: read "/home/testuser/testfile2" (handle 0) off 240000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 11: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 12: read "/home/testuser/testfile2" (handle 0) off 270000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 12: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 13: read "/home/testuser/testfile2" (handle 0) off 300000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 13: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 14: read "/home/testuser/testfile2" (handle 0) off 330000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 14: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 15: read "/home/testuser/testfile2" (handle 0) off 360000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 15: sent data len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 16: read "/home/testuser/testfile2" (handle 0) off 390000 len 19600
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 16: sent data len 19600
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 17: read "/home/testuser/testfile2" (handle 0) off 409600 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 17: sent data len 21400
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 18: read "/home/testuser/testfile2" (handle 0) off 431000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 18: sent status 1
Apr 03 21:31:30 my-host sftp-server[3159]: sent status End of file
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 19: read "/home/testuser/testfile2" (handle 0) off 461000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 19: sent status 1
Apr 03 21:31:30 my-host sftp-server[3159]: sent status End of file
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 20: read "/home/testuser/testfile2" (handle 0) off 491000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 20: sent status 1
Apr 03 21:31:30 my-host sftp-server[3159]: sent status End of file
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 21: read "/home/testuser/testfile2" (handle 0) off 521000 len 30000
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 21: sent status 1
Apr 03 21:31:30 my-host sftp-server[3159]: sent status End of file
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: request 22: read "/home/testuser/testfile2" (handle 0) off 551000 len 12400
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 22: sent status 1
Apr 03 21:31:30 my-host sftp-server[3159]: sent status End of file
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 23: close handle 0
Apr 03 21:31:30 my-host sftp-server[3159]: close "/home/testuser/testfile2" bytes read 431000 written 0
Apr 03 21:31:30 my-host sftp-server[3159]: debug3: request 23: sent status 0
Apr 03 21:31:30 my-host sftp-server[3159]: sent status Success
Apr 03 21:31:30 my-host sftp-server[3159]: debug1: read eof
Apr 03 21:31:30 my-host sftp-server[3159]: session closed for local user testuser from [127.0.0.1]
Apr 03 21:31:30 my-host sshd[3157]: Received disconnect from 127.0.0.1 port 47854:11: Shutdown
Apr 03 21:31:30 my-host sshd[3157]: Disconnected from 127.0.0.1 port 47854
Apr 03 21:31:30 my-host sshd[3154]: pam_unix(sshd:session): session closed for user testuser
{ [16384 bytes data]
100  420k  100  420k    0     0  4020k      0 --:--:-- --:--:-- --:--:-- 4047k
100  420k  100  420k    0     0  3953k      0 --:--:-- --:--:-- --:--:-- 3953k

Connection #0 to host 127.0.0.1 left intact

Answer 1

I assume you have used 127.0.0.1 as a mask for the real address, or you have some forwarding going on.

Have you tried:

curl -u testuser:12345 -o /tmp/testfile scp://127.0.0.1/home/testuser/testfile

scp is designed to work on a single channel unlike sftp, and also scp does not require a confirmation of each pack before the next is sent. You may find scp will be able to transverse the firewall you are trying to bypass.