OpenGL: Rotate lookAt camera axis in a compute shader

Question

I'm making a ray marching demo and I have a problem rotating the camera axis. I'm using quaternions to rotate the mouse and casts rays to that direction just fine. An example of the problem: If I turn 180° on yaw and press W it goes backwards instead of forward. I tried many things like translating and didn't work out for me.

I'm wondering if this is a wrong way to create a moving camera for a ray casting algorithm.
The lookAt function:

    vec4 rotor = quatRotor(vec3(0., -1., 0.), -iMouse.x);
    rotor      = quatMult(rotor, quatRotor(vec3(1., 0., 0.), iMouse.y));

    vec3 f     = normalize(center - eye);
    vec3 r     = normalize(cross(vec3(0.0, 1.0, 0.0), f));
    vec3 u     = normalize(cross(f, r));

    float fov    = radians(45);
    vec4 dir     = vec4(normalize(uv.x * r + uv.y * u + f * fov), 1.0);

    mat4 m;
    m[ 0 ][ 0 ] = r.x;
    m[ 0 ][ 1 ] = r.y;
    m[ 0 ][ 2 ] = r.z;
    m[ 1 ][ 0 ] = u.x;
    m[ 1 ][ 1 ] = u.y;
    m[ 1 ][ 2 ] = u.z;
    m[ 2 ][ 0 ] = -f.x;
    m[ 2 ][ 1 ] = -f.y;
    m[ 2 ][ 2 ] = -f.z;
    m[ 3 ][ 0 ] = center.x;
    m[ 3 ][ 1 ] = center.y;
    m[ 3 ][ 2 ] = center.z;
    m[ 3 ][ 3 ] = 1.0;

    dir     = m * dir;
    dir.xyz = quatRotate(dir.xyz, rotor);

    return dir;   

The eye variable holds the WASD key values which are uniforms. Their initialized values are vec3(0, 0, -1) for xyz respectively.
The center is vec3(0,0,0) and the uv variable is the current pixel I'm drawing.