You can test how much does time takes by wrapping it into another time:
time time true
Note: I'm adding true which is the simple command which 'do nothing'.
true - do nothing, successfully - man true
On mine, it shows:
real:0m0.000s user:0m0.000s sys:0m0.000s
Even with 20 occurrences:
$ time time time time time time time time time time time time time time time time time time time time true true
real:0m0.000s user:0m0.000s sys:0m0.000s
Here are 1000:
$ eval $(for i in $(seq 1 1000); do printf "time "; done) true
real:0m0.000s user:0m0.000s sys:0m0.000s
man time gives some clues how the time is calculated:
Most information shown by time is derived from the wait3(2) system call. The numbers are only as good as those returned by wait3(2). On systems that do not have a wait3(2) call that returns status information, the times(2) system call is used instead. However, it provides much less information than wait3(2), so on those systems time reports the majority of the resources as zero.
For further technical details, check: man 2 wait3 or man wait4.
To debug time on Linux, you can use strace, e.g.
$ strace -f time true
To count time of syscalls (-c) and summarise the time differences (-s), run:
$ strace -fwc time true
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
30.91 0.007723 69 112 write
27.19 0.006794 6794 1 wait4
20.44 0.005107 340 15 13 execve
7.43 0.001855 1855 1 clone
2.94 0.000734 122 6 6 access
...
Here is how to debug time 10 times:
$ strace -fwc $(echo $(for i in $(seq 1 10); do printf "time "; done)) true | head
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
87.35 1.057704 105770 10 wait4
7.33 0.088807 79 1120 write
2.25 0.027275 222 123 112 execve
1.37 0.016571 1657 10 clone
0.32 0.003913 71 55 mmap
0.32 0.003844 116 33 33 access
...
And 1000 times:
$ strace -fwc $(echo $(for i in $(seq 1 100); do printf "time "; done)) true | head
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
98.66 112.115311 1121153 100 wait4
0.77 0.874091 78 11200 write
0.24 0.271382 226 1203 1102 execve
0.16 0.179477 1795 100 clone
0.03 0.037229 123 303 303 access
0.03 0.036702 73 505 mmap
...
Obviously running under the debugger, the time is going to be slower when running the actual command, but it can give the general idea what kind of syscalls are in use and how much time they can use compared to others.