[* is a regular command, similar to grep, find, or cat. You should be able to find it in /bin. Since it's a separate program, the shell will perform its normal set of expansions before handing [ its arguments.
As has been mentioned, since you're using * in your tests, you're getting glob expansions. Note that even if you use quotes, such as 'hel*', this probably won't work as you hope, because [ does not support patterns. In the case of h*o working, that is likely due to the presence of a file named hello in your current directory, and no other files matching that pattern. If it does work without a hello file, you may have an odd implementation, and your script is likely to fail on other systems.
Depending on your needs, there are a couple of options. Bash, Zsh, and some other shells have the [[ builtin. Since it is a builtin, it can give its arguments special treatment, including avoiding glob expansion. Additionally, it can do pattern matching. Try
var1=hello
if [[ "$var1" = hel* ]]; then
echo success
fi
Also, note the lack of quotes around the pattern. Without quotes, hel* is treated as a pattern by [[, with quotes (single or double), "hel*" is treated literally.
If you need wider compatibility, such as for shells without [[, you can use grep:
var1=hello
if echo "$var1" | grep -qe 'hel.*' ; then
echo success
fi
No [ or [[ necessary here, but the quotes around 'hel.*' are.
*Some shells actually do have [ builtin, but this is for efficiency purposes. It should still behave identically to the separate executable, including having its arguments subjected to the shell's normal "mangling."