First, the inline specification on a function is just a hint.  The compiler can (and often does) completely ignore the presence or absence of an inline qualifier.  With that said, a compiler can inline a recursive function, much as it can unroll an infinite loop.  It simply has to place a limit on the level to which it will "unroll" the function.
An optimizing compiler might turn this code:
inline int factorial(int n)
{
    if (n <= 1)
    {
        return 1;
    }
    else
    {
        return n * factorial(n - 1);
    }
}
int f(int x)
{
    return factorial(x);
}
into this code:
int factorial(int n)
{
    if (n <= 1)
    {
        return 1;
    }
    else
    {
        return n * factorial(n - 1);
    }
}
int f(int x)
{
    if (x <= 1)
    {
        return 1;
    }
    else
    {
        int x2 = x - 1;
        if (x2 <= 1)
        {
            return x * 1;
        }
        else
        {
            int x3 = x2 - 1;
            if (x3 <= 1)
            {
                return x * x2 * 1;
            }
            else
            {
                return x * x2 * x3 * factorial(x3 - 1);
            }
        }
    }
}
In this case, we've basically inlined the function 3 times.  Some compilers do perform this optimization.  I recall MSVC++ having a setting to tune the level of inlining that would be performed on recursive functions (up to 20, I believe).