Chaining the || operator with multiple values like (str[i]=='u'||'o'||'i'||'e'||'a') or (str[i]==('u'||'o'||'i'||'e'||'a')) is not used to check if a value is one of a set of values.
The || operator is the logical OR operator.  It treats both of its operands as boolean values and evaluates to either 0 or 1 depending on the operands.  The use of this operator is detailed in section 6.5.14 of the C standard:
2 Each of the operands shall have scalar type.
3 The || operator shall yield 1 if either of its operands compare unequal to 0; otherwise, it yields 0.  The result has type int.
4 Unlike the  bitwise | operator, the || operator  guarantees  left-to-right  evaluation;  if  the second operand is evaluated, there is a sequence point between the evaluations of the first and  second  operands.  If  the  first  operand  compares  unequal  to  0,  the  second  operand  is not evaluated.
Since C doesn't have a true boolean type, any integer value (which includes character constants) can be an operand to ||.  So any non-zero value is considered true and zero is considered false.  Also, note from paragraph 4 above that this operator has "short-circut" evaluation, meaning that the right side won't be evaluated if the result of the operator is known just by looking at the left side.
Now lets apply this to your expressions.  First:
if (str[i]=='u'||'o'||'i'||'e'||'a')
Because we're dealing with multiple operators here, we need to apply the operator precedence rules detailed here.  Since the equality comparison operator == has higher precedence than the logical OR operator ||, this parses as follows:
if ((str[i]=='u')||'o'||'i'||'e'||'a')
So first we evaluate str[i]=='u'.  This will be either 0 or 1 depending on whether str[i] is 'u' or not.  Then we evaluate the first ||, so we have either 1||'o' or 0||'o'.
In the first case the left operand is 1 so as per paragraph 4 above the right side is not evaluated, which includes the other || operators so the final result is 1, i.e. true which is the desired result.  In the second case 0 is false so then we look at the right side which is 'o'.  This is a character constant whose value is value used to encode the character 'o'.  If your system uses ASCII (which it most likely does) this value is 111.  Because this is a non-zero value the whole expression 0||'o' evaluates to 1 i.e. true.  Again because of the short-circuit behavior of || the next || operator isn't evaluated since the left side is true.  This means the above expression is always true.
Now moving to your second expression:
if (str[i]==('u'||'o'||'i'||'e'||'a'))
The first thing that is evaluated is 'u'||'o'.  The 'u' character has an ASCII code of 117 which is non-zero, so the first || results in 1 and the right side, which includes the remaining || operators are not evaluated.  So now you have str[i] == 1.  Unless str contains non-printable characters, you'll never find a character whose encoding is 1, so this expression will always evaluates to 0, i.e. false.
C doesn't have a built in operator that checks if a value is a member of a set, which means you either need to check str[i] each character explicitly:
if ((str[i]=='u') || (str[i]=='o') || (str[i]=='i') || (str[i]=='e') || (str[i]=='a'))
Or you can create an array of characters to check and loop through them:
char vowels[5] = "aeiou";   // an array of char, but NOT a string
int found = 0;
for (int j = 0; j < sizeof(vowels); j++) {
    if (str[i] == vowels[j]) {
        found = 1;
        break;
    }
}
if (found) {
    ...
Or you can use the strchr to loop through the values for you:
if (strchr("aeiou", str[i]))
Or use a switch with fallthrough cases:
switch(str[i]) {
case 'a':
case 'e':
case 'i':
case 'o':
case 'u':
    // do something
    break;
default:
    // do something else
}