I'll start by answering your last question:
Also, how many fingers do computers have?
Computers only have one finger... However, that doesn't tell the whole story, since computers also usually have a lot of hands (like 32 or 64 in most computers ;).
By definition, anything digital "is a data technology that uses discrete (discontinuous) values." We can represent said discrete values in real-world electronics by using transistors as digital switches. Transistors, when used in switching mode, can either let current flow or not. Thus, we can represent the output of a transistor as either ground (no connection, 0V), or the supply voltage (or some percentage of) - the equivalents of false and true from boolean logic.
To use these discrete states to represent numeric values, we see that each transistor can represent two discrete states - or a number from 0 to 1. This is part of the base 2 number system, which follows the same steps as our base 10 system (and base 3, 4, 5, base 16, and so on and so fourth). If we represent the number 52 in these various number systems, where each digit can range from 0 to N-1, where N is the base of the number system, we have:
Base 2: 1 1 0 1 0 0 110100_2
2^5 2^4 2^3 2^2 2^1 2^0
Base 4: 3 1 0 310_4
4^2 4^1 4^0
Base 10: 5 2
10^1 10^0 52_10
Base 16: 3 4 34_16
16^1 16^0
Now, in the case of binary numbers, you can see that we can represent exponentially larger numbers (like every other number system), by adding more digits - or in the digital computer case, by adding more discrete transistors in parallel with the system. This is why a 32-bit (unsigned) integer can store any number from 0 to (2^32) - 1.
Again, since we can only represent two discrete states electronically, the only way for us to represent more is to extend these states with more "on-off" states, or by adding more base-2 numbers in parallel. This is why everything in the computer world is based off of powers of two - this is the only way we can represent values in a computer.
It should be noted that this is incredibly different then an analog computer, which literally can have an infinite number of state values. This introduces precision errors into values, which are unavoidable - also why digital computers are preferred (less information entropy, better ways of storing, compressing, encrypting, and no information degradation).
It should also be noted that we use base 2 numbers because our transistors can only have two states. If we were capable of making three-state (I'm not talking about tri-state logic here, which simply uses a high-impedance state) or four-state transistor, then we could surely create computers using a different number system.
However, binary is the "tried and true", and there is no real advantage to using a different number system aside from the fact that you might need to use less discrete components to store as many values. For example, when we turned 52 into the base-4, base-10, and base-16 equivalents, you can see that as the base of the number system increases, naturally the number of individual digits required to hold that number decreases.
However, transistors are cheap, small, and tiny - so we have no problems representing very large values (which is also why we've made the switch to 64-bit computing - we can do more operations on larger numbers in less time).