When you return from a function, the type of the return value is determined from the function prototype, not what it's written as-is. If the type doesn't match, it'll be implicitly converted to the correct type. So this example function:
double foo(void) { return 1; }
Actually returns double(1), or equivalently, 1.0. It does not return an int because 1 is int. It does, however, convert your int to a double value to match the function's return value type as declared.
So coming to your question, the innermost function called is double f(int), and the second function called is int f(double), and the outermost function called is double f(int). This matched the output you see: aba.
f( f( f(1) ) );
↑  ↑  ↑
|  |  calls double f(int)
|  calls int f(double)
calls double f(int)