I see that you have copied the code from an existing answer. Sadly, that answer contains errors!
Before using the FYL2X instruction, read the description in the manual:
This instruction calculates (ST(1)*log2(ST(0))), stores the result in register ST(1), and pops the FPU register stack. The source operand in ST(0) must be a non-zero positive number.
The FYL2X instruction is designed with a built-in multiplication to optimize the calculation of logarithms with an arbitrary positive base (b).
logbx = (log2b)-1 * log2x
The error stems from not noticing that the multiplier is a reciprocal. We can easily remove it because (log2b)-1 = logb2 and then we substitute 10 for b. We therefore need as multiplier log102 which the FPU provides through its FLDLG2 instruction.
num     REAL8 3.324
res     REAL8 ?
        ...
        FLDLG2           ; ST0 = log10(2)
        FLD     num      ; ST1 = log10(2), ST0 = num
        FYL2X            ; ST0 = log10(num)
        FSTP    res
To calculate the natural logarithm (aka 'ln'), the only change would be to use FLDLN2 (instead of FLDLG2) to obtain the multiplier loge2.
To calculate the binary logarithm, the only change would be to use FLD1 (instead of FLDLG2) to obtain the trivial multiplier 1.
An algorithm that converts a floating point number into its textual representation is not all that simple, but maybe next fixed-point trick is all that you need:
- multiply the floating point number by 1000 (for a 3-digit precision)
- store the product to memory as integer
- convert (*) the integer into text (repeated divisions by 10)
- put a decimal point character between the last 3 digits and all the digits in front of them
num     REAL8 3.324
quad    dq ?
kilo    dw 1000
        ...
        FLDLG2           ; ST0 = log10(2)
        FLD     num      ; ST1 = log10(2), ST0 = num
        FYL2X            ; ST0 = log10(num)
        FIMUL   kilo     ; ST0 = 1000 * log10(num)
        FISTP   quad
        ...
(*) Displaying numbers with DOS explains how converting an integer to text works. Look through the 'with DOS' mention and focus on the how and the why of the conversion itself.