We only need to test 1 in 625 candidate numbers.
Either solution A:
upper_limit = 10**4
lower_limit = int(10**3.5) + 1
rem = lower_limit % 625
if rem > 0:
    lower_limit = lower_limit - rem + 625
for n in xrange(lower_limit, upper_limit, 625):
    if n % 16 in [1, 15]:
        print {1: n, 15: n+1}[n%16]
        break
or Solution B:
print (1 * (-39) * 16 + 0 * 1 * 625) % 10000
Read on for explanation.
Starting from the brute-force list-comprehension that tests all candidates:
from math import ceil
[n for n in xrange(ceil(10**3.5), 10000) if (n*n) % 10000 == n]
(The ceil rounds the square root of 10**7 up to nearest integer).
(Notice that 10000 is the first number whose square has 9 digits, and the loop will not test that number.)
... or if you'd like to early-terminate as soon as you find a solution:
for n in xrange(ceil(10**3.5), 10000):
    if (n*n) % 10000 == n:
        print n
        break
But consider: we are looking for numbers n**2 - n = n*(n-1) which are divisible by 10**4 = 2**4 * 5**4.
- either 
n or n-1 is odd; so the other one would have to be divisible by the full 2**4 = 16.  Similarly, you can't have both n and (n-1) be divisible by 5; 
- so we need either 
n or (n-1) to be divisible by 5**4 = 625. 
- if either one (
n or (n-1)) is divisible by both 625 and 16, then that number is divisible by 10000.  No such number has four digits, so it must be that either n or (n-1) is divisible by 625, and the other one is divisible by 16. 
- We can thus restrict our search space to looking only at multiples of 
625 which have four digits; we just have to be careful to remember that the multiple of 625 might be either n or (n-1); the other one has to be divisible by 16. 
So:
upper_limit = 10**4
lower_limit = ceil(10**3.5)
rem = lower_limit % 625
if rem > 0:
    lower_limit = lower_limit - rem + 625
for n in xrange(lower_limit, upper_limit, 625):
    if (n-1) % 16 == 0:
        print n
        break
    if (n+1) % 16 == 0:
        print n+1
        break
Or if you test n instead of (n-1), and combine both condition branches into n % 16 in [1, 15], and for compactness, you could print {1: n, 15: n+1}[n%16].
This is Solution A. (Also, you can certainly replace n%16 with n & 0xf if you prefer.)
But wait! All of this can in fact be done using the...
We want to find n such that either:
n = 0 (mod 625) and n - 1 = 0 (mod 16),
or:
n - 1 = 0 (mod 625) and n = 0 (mod 16).
So in each case we have two equations, with coprime moduli, solved for the same number n:
n = 0 (mod 625) and n = 1 (mod 16), 
or else
n = 1 (mod 625) and n = 0 (mod 16).
Now (in both cases) we would use the Extended Euclidean Algorithm to find m1 and m2 such that 16*m1 + 625*m2 = 1.  It turns out that -39*16 + 1*625 = 1, which leads to the Solution B above, from the second case.  (Note: the first case would yield instead 625, whose square does end in 0625, but doesn't count as a solution.)
For completeness, here is an implementation for the Extended Euclidean Algorithm.  The second and third outputs are the m1 and m2; in our case 1 and -39 in some order.
def extended_gcd(a, b):
    last_remainder, remainder = abs(a), abs(b)
    x, last_x, y, last_y = 0, 1, 1, 0
    while remainder:
        last_remainder, (quotient, remainder) = remainder, divmod(last_remainder, remainder)
        x, last_x = last_x - quotient*x, x
        y, last_y = last_y - quotient*y, y
    return last_remainder, last_x * ((-1)**(a < 0)), last_y * ((-1)**(b < 0))