I have about 17k files in a directory. When I run ls directory, I have to wait for about 15-20 seconds before the results are displayed. On the other hand, when I run ls directory | wc -l or ls directory | grep .xyz, the results are displayed immediately.
Why does this happen and is there a way to fix this?
I'm going to guess that you're using Linux.
If your ls command is aliased such that it shows files and folders in colour, then it needs to find out each item's permissions (a stat() call) and whether it has any "file capabilities" set (a getxattr() call) in order to choose the right colour. Depending on file system, these calls can be fairly slow if the required metadata hasn't been cached in RAM yet. [Extended attributes often live in the data area, so each getxattr results in HDD seeks.]
On the other hand, ls | when redirected to a pipe automatically disables colouring, so it no longer needs to do any extra checks – just a straightforward readdir() loop which returns the file name and type, and the kernel likely even implements read-ahead for that.
nonsense
Use strace or perf trace to check which system calls, if any, are taking a long time.
ls sorts output by default. Try ls -f to disable sorting and colorizing output.
Two things:
ls first and ls | wc -l later, its possible the former will read from your HDD and the latter will read cached data. If so, ls initially "stalls" and prints nothing for few seconds. Another ls will start printing almost immediately, as long as the cached data is still there. If you started with ls | wc -l in the first place, it would have to wait for HDD to supply data.stty speed will show you some value, but I think it doesn't matter for a virtual terminal. Still, displaying characters and scrolling takes time (see this question). Passing the same data through a pipe is faster.