The syntax you are using is not valid in C. You can get pretty close to this syntax in context of assignment operator by using a compound literal 
tfrac multf(tfrac a, tfrac b)
{
  tfrac res;
  res = (tfrac) { a.num * b.num, a.den * b.den };
  return res;
}
Note the (tfrac) part before the {}.
But in your case you can simply use initialization instead of assignment
tfrac multf(tfrac a, tfrac b)
{
  tfrac res = { a.num * b.num, a.den * b.den };
  return res;
}
However, returning to compound literals again, if you prefer, you can use a compound literal to turn the whole thing into an one-liner
tfrac multf(tfrac a, tfrac b)
{
  return (tfrac) { a.num * b.num, a.den * b.den };
}