-allow forks
-allow only private forks
-no forks
These three options are only available for private repos.
- Allow forks means it both support public forks (the fork version can be seen publicly) and private forks (the fork version can only seen by the person who forked it). 
- Allow only private forks means it allow private-only forks. 
- No forks means you repo don't allow any fork. 
And fundamentally, to fork a project (take the source from someone's repository at certain point in time, and apply your own diverging changes to it), you would clone the remote repository to create a copy of it, then do your own work in your local repository and commit changes.
For differences between git and mercurial type of repositories, you can check out this post: What is the Difference Between Mercurial and Git?