4

I want to download a forum where I only have access via my username and my password.

I tried the following:

C:\wget.exe wget -k -m -E -p -np -R viewtopic.php*p=*,memberlist.php*,faq.php*,posting.php*,search.php*,ucp.php*,viewonline.php*,*sid*,*view=print*,*start=0* -o log.txt http://www.myforum1234.com/forum/categories/discussions

So this is the command I enter in my cmd. Because when I click on the wget.exe a black window appears, but very fast disappears. But I think this is also a correct way (I use Windows XP)?

My problem is that the results show that wget could not download the forum, because it could not login. So the login page is shown (downloaded), but nothing more. I was logged in, when I exercised the command.

I am not a wget professional, so I am not sure if my command is correct? I copied this from another post. A simple C:\wget.exe wget http://www.theforumurl.com did not work.

EDIT:

I now also tried

C:\wget.exe wget -k -m -E -p -np -R *start=0* -o log.txt http://www.myforum.com/forum/categories/discussions

But the same problem here.

2nd EDIT concerning the link in the first comment:

I now tried

C:\wget.exe wget -k -m -E -p -np -R *start=0* -o log.txt http://www.myforum.com/forum/categories/discussions --post-data="username&password=1234"

But again, same problem!

When I hoover over the login button I can see the following URL:

http://www.myforum.com/user/popupLogin

Do I have to use this one?

3rd EDIT:

I also tried to add the username:password@ before the www., so like this:

C:\wget.exe wget -k -m -E -p -np -R  *start=0* -o log.txt http://user:passw@www.myforum.com/forum/categories/

The result is the same, I can see that the login did not work.

4th EDIT:

I also tried according to this thread:

C:\wget.exe wget --save-cookies cookies.txt --post-data 'user=usern&password=passw' http://www.myforum.com/user/popupLogin

C:\wget.exe wget --load-cookies cookies.txt -p http://www.myforum.com/forum/categories/

But again, same problem!!

5th EDIT:

I think I now isolated the source code of the login-button:

<div class="forumSignup">
          <a href="http://www.myforum.com/user/popupLogin" class="Button SignInPopup">Login</a> </div>

6th EDIT:

I also tried it with HTTrack, but the problem is the same: The login does not work. Another problem seems to be that the forum itslef uses the URL www.mywebsite.com/forum, but the login is required for the www.mywebsite.com. So when I use e.g. something like username:pass@www.mywebsite.com the mywebsite is captured, but not the forum. When I use username:pass@www.mywebsite.com/forum the login does not work and nothing is captured.

Community
  • 1
Stat Tistician
  • 53

4 Answers4

2

First of all, you would do C:\wget.exe -k -m …, you don't repeat wget name.

Since login into the forum seems complicated (it can get complex even for simple sites), the best solution is probably to log in with your browser and then give the cookies* to wget (either put in a file and use --load-cookies or pass them directly with --header "Cookie: name=value").

* The way of extracting them vary a bit depending on your browser.

Ángel
  • 1,393
0

It's difficult to mirror a site with login using wget. You need expert knowledge to use wget. Currently you pass user name and password, cookies, and needed switchs.
Additional things to do.
1. Avoid mirroring until everything is o.k., As recursively downloading page force the webserver to add your IP to blacklist. (try to save single page)
2. Fake wget as a browser as Most web forums hate download managers. see this answer for more info.

Best solution

The best and easiest way to mirror this kind of site is to use ** scrap book **. It's a firefox plugin. All you need to do is launch firefox, login to site, right click -> save page as, filter by domain. see this answer to efficiently mirror a site.

Community
  • 1
totti
  • 870
0

Wget interprets <pass>@serveraddress as port.

To specify a username and password, use the --user and --password switches :

wget --user username --password passw http://...
harrymc
  • 498,455
0

If you have access to cookie data on a browser (firefox has its own cookie browser under options->privacy, but there are plugins to ease this task), perform a manual login to the forum, search for all the cookies for that domain and store them in the cookies.txt file, it would probably work with your previous command:

C:\wget.exe wget --load-cookies cookies.txt -p http://www.myforum.com/forum/categories/

Some logins pages are way too complex to try to perform the task in a single command line.

Remember to include ALL the cookies for the whole domain (search for "myforum.com", not just "www.myforum.com")

NuTTyX
  • 2,716