FlashGot - a beginners guide to easy downloading with Firefox

FlashGot is the best Firefox download manager integration around. That is their words and they may very well be true. FlashGot ( http://flashgot.net/ ) is a tool to integrate the website with a download manager of your choice.

In regard to download managers then we are in business with tools like DownThemAll ( http://www.downthemall.net/ ), which is free and the commercial tool ReGet ( http://www.reget.com/en/ ). I will demonstrate both here in this micro tutorial. We will also have a quick look into some of the configuration possibilities of FlashGot.

Quote: "Spend less time downloading and more time on organizing". 

The image screenshot below shows the Preferences for FlashGot. From the red marked circles you can see that I have a possibility to choose the default download manager as either DTA (Down Them All), ReGet or the build in browser download capability. I am most familiar with ReGet as it gives me the most information and control, but the choice matters little as both tools manage to download properly.

FlashGot supports a wide range of download managers whereas the most popular probably is DownThemAll, FlashGet, GetRight, jDownloader and ReGet.

FlashGot basically has two modes, either you choose to download 1 link or you choose to download the entire page, and everything that is linked to from the page, one level deep. Let's see how things looks using DownThemAll and choosing 'FlashGot All'.

If we switch the default download manager to ReGet then we get a slightly different picture although they are quite similar. With DTA some files are checked as which to download. With ReGet all files are picked as to be downloaded. Both tools require you to hand pick which, because in this case you may want only the default size video and the HD largest quality video.

With both DTA and ReGet the next step is to click OK and let the downloading begin. If you right click a link and choose 'FlashGot Link' which is the single file download mode then you get thrown directly into the single file download mode of the download manager you have set the preference too. Again it is the OK button and your download starts.

The force of download managers like DTA and ReGet are their ability to stop and resume huge downloads with multiple threads. Downloading a 2.18Gb file may take a little while and the best way is to use a tool like DTA or ReGet. Give it a try.

Offline Explorer - site ripping for beginners

Here is a small tutorial with pictures for you to better understand the steps in creating a useable Offline Explorer project which downloads what you want and nothing more.

First off, you will have to get the tool. Which you buy/get from http://www.metaproducts.com. The cheapest version is about 50 USD and the most expensive is about 500 USD. The price for the product may seem quite high, but believe me, even the most expensive version is worth the cost if you excell in siteripping. It will save you an emmense amount of time. The current version is 5.6, my own version is an old one I bought a long time ago and will be using v4.8 for demonstration use.

The target site for this demonstration is http://www.betanews.com. Betanews is a site which holds some downloadable files (programs), which mainly are the newest versions of various software. We will have a wack at the WWW & Internet section of tools. Have a look below for a peek of the site.

I'd suggest you visit the site and pan the links, so that you have some feeling of what we're about to download. Click the circled item and see what happends.

Make note of the URL in the address bar of the browser and start Offline Explorer and create a New Project. Type the URL and it will look something like the picture below.


One of the things you should make not of is what you tell OE to download. Leave it as it is for now and take not that you are downloading from the site and 1 level down in the sites hierarchy.

Click OK, and start the project. Quickly switch the top tab's to Queue. There you will be able to see what OE intends to download. It will look something like the picture below.


You can pause the downloading by pressing F9, pressing F9 again will resume the download. For more information on how to make cool use of this feature then look for HAM download here on this site.

Scroll down in the list and notice the links with the /download/ path. These are the ones of interest, the rest is "junk" and without value to us. This can be compared to make note of the links for the .jpg and the .avi / .wmv files on a site (IF that is what you want - hehe).


So we have to filter the /download/ links. Go back into your project and find 'URL Filters -> Directory' and switch to 'View included directories keyword'. Type '/download/' without the ' and add it to the list of keywords are included. Also go back up in the project and find 'File Filters -> Archive'. Add the two items 'rar' and 'exe' to the list of what to download and uncheck everything else under the File Filters. Be sure that the File Filters -> Archive is set to 'Load using URL filters settings'. Take a close look at the screenshots below. Delete the files downloaded and start the project again.


One thing you will notice is that you don't actually get the files (.exe and .rar) files because you still have a setting to download only 1 level deep. So increase the level to 2. Look below.


Delete your downloaded files and start the project again. You will see that you lack the files because they use another link structure than '/download/'. The keyword you need to include is '/sendfile/'. Add this to the directories keywords like shown below.


This time you will be able to see the list below, and that is what you will download. Let the project finish and explore the files you downloaded. If you do not know where the files are located then click the 'Options' button of OE and look under 'File Location'.


What you hopefully have learned is the use of OEEE's Filter settings and how to limit what you download. The list of keywords can greatly improve the quality of you download.

Going down on Lexi

Downloading a website.

One solo site named 'All over Lexi' uses php as the main cgi based url method. Now sometimed downloading cgi sites can give you alot of unwanted stuff.

The site presents 4 pages with picture sets and 1 page with some videos. As there are way more pictures than videos, the 1st gets precedense. Browse around and look at some images to see wether the webmaster is on drugs or clean. This site a clean webmaster, which is good.

Looking at a picture and then rightclicking it to see the url will give you something like:

http://www.site.com/members/pictures/hallway_lg/lexi-lg-154.jpg

Check more pictures to see wether this way of layout sticks. With this site it does. All images are in the formats:

http://www.site.com/members/pictures/pinkoncouch_lg/lexi-lg-001.jpg
http://www.site.com/members/pictures/hallway_lg/lexi-lg-154.jpg

So only the set names and numbering differs. First we have to get the setnames. So go back to page 1 and save it to your hdd, do the same with page 2, 3 and 4. With those nice saved you have a go at it with GREP. A nice regular expression like "(?<=href=\").*?_lg" does the job nicely and renders the result below.

click to enlarge

Mark it and copy it to Notepad, UltraEdit, EditPro or whatever editor you use. Preferrable an editor which handles column mode. Next we need to find out how many pictures to download from each set. Luckily the drug free webmaster wrote this on the pages. So we check the html and find out that some newline characters will obstruct our intentions. Remove all tabs, newlines etc from the 4 htm files and save them again. You can do that by running

tr -d "\n\r\t" < p1.htm >p1.html
tr -d "\n\r\t" < p2.htm >p2.html
tr -d "\n\r\t" < p3.htm >p3.html
tr -d "\n\r\t" < p4.htm >p4.html

The files with the full extension are now newline free. With those 4 new files we run a GREP to extract the number of images to download:

grep -Poh "[0-9]* pictures" p*.html

This will give you a new list and mark the important stuff and paste it into your editor COLUMN WISE.

click to enlarge

So you have to write a few things in column mode in your editor to complete your list. And once done search replace ' ' (space) with nothing. Which will concatanate the text to a proper list of url's with macros. (see the article MACRO downloading).

click to enlarge

Copy your download list to a new project in OEEE and insert your username and password so that each url will begin like http://yourusername:yourpassword@www.

No need for cookies or referers. Hit Download and let it run. Download the vidoes manually, there are only 5. Handle candids, wallpapers and webcams as you see fit.

HAM downloads

HAM downloading is a lazy way to obtain your download links, but it is rather effective. The basis of a HAM download is to grab the urls the site produces while in pause. Let's take Ann Angel as an example. Visit her site and login, go to the pictures page (if that is what you are on the look for) and rightclick anywhere and choose "Download with ReGet". I assume you have installed ReGet Deluxe and configured it to expert mode.

Once ReGet pops up you can switch tab and go to the HTTP properties tab. See it below.

Click to enlarge

Since the site is a FORM site, then we cannot use direct username and password in OEEE. We have to obtain the cookie which hold the information needed to login to the site. Mark the line and copy the cookie to your clipboard.

Now start OEEE and create a new project. Paste the cookie from your clipboard into the project. Make it look something like below.

Click to enlarge

Now change the maximum simultanious downloads in OEEE into 1. Do that by changing the Options in OEEE, see below.

Click to enlarge

With that done you hit the Suspend button so that all downloads are set to pause. After that click the Queue tab in OEEE. Now you hit Download button in OEEE (everything is in pause).

Click to enlarge

Here comes the nice thing hit F9 twice. This will turn off Suspend for a brief second (allowing the link to be downloade) and then put OEEE into Suspend mode right after that. This will make OEEE resolve the 1st download into all subsequent links in level 1. Now all you have to do is to mark all the lines you wish to download anc choose Copy URL.

Click to enlarge

You can now do paste the list back into OEEE (hit the Stop button first) or paste the url's into Notepad for some GREP processing or whatever you wish for, example deduct url filters.

MACRO downloading

Using macros when downloading is a powerful thing. Offline Explorer Enterprise Edition can help you quite a deal. If you have 10 pictures on a http server like this:

http://www.server.com/pictures/image01.jpg
http://www.server.com/pictures/image02.jpg
http://www.server.com/pictures/image03.jpg
http://www.server.com/pictures/image04.jpg
http://www.server.com/pictures/image05.jpg
http://www.server.com/pictures/image06.jpg
http://www.server.com/pictures/image07.jpg
http://www.server.com/pictures/image08.jpg
http://www.server.com/pictures/image09.jpg
http://www.server.com/pictures/image10.jpg

Then you can download all 10 directly by using a numbering macro in OEEE. Write this in your download box.

http://www.server.com/pictures/image{:01..10}.jpg

That will do the job and generate 10 urls when you hit the run/play button in OEEE. Now you can spiff up the numbering macro a bit by using a enumerater step. Let's say that the pictures are numbered 01,03,05,07,09,11 instead of a direct squence. Then you can create your macro like this:

http://www.server.com/pictures/image{:01..11|2}.jpg

That will generate the urls:

http://www.server.com/pictures/image01.jpg
http://www.server.com/pictures/image03.jpg
http://www.server.com/pictures/image05.jpg
http://www.server.com/pictures/image07.jpg
http://www.server.com/pictures/image09.jpg
http://www.server.com/pictures/image11.jpg

Now this is all handy. The big advantage is being able to use multiple macros in the same command. Very often you need set names to your download list. If the webmaster of the site of your choice is drug free (which is almost none) then the sets will be numbered 001, 002 and so forth. But this is almost NEVER the case, although Ann-Angel's webmaster does that - thank you for that.

The easiest way of obtaining set names is to visit whatever gallery page that holds the episodes/sets you wish to download. Let's try with an example. Reality Kings has a site called Mikes In Brazil. Visit the url hxxp://www.mikesinbrazil.com and save the intro page to your hdd. It should be the page which is is the commercial page which lists a bunch of episodes.

Each of the episodes has a episode name. This is hidden a little. But if you let your mouse hover over the Watch Trailer button then you can spot the episode name as the first part of the wmv file. Example sol.wmv. The episode name is: sol.

With your saved file run this command on the command line:

grep -Po "(?<=trailer=)[a-z0-9]*" mikesinbrazil.html >set_names_mib.txt

If you do not redirect the output to the text file set_names_mib.txt then your result could look very much like the picture below.

Click to enlarge

With a file of episode names we can enhance our OEEE macro download a bit. We can now construct it to use the set names and a numbering macro.

http://www.server.com/{:file=c:\set_names_mib.txt}/pictures/image{:001..321}.jpg

The macro download will pr0duce 321*8 = 2568 direct urls for downloading images. The urls will look something like:

http://www.server.com/sol/pictures/image001.jpg
http://www.server.com/sol/pictures/image002.jpg
...
http://www.server.com/sol/pictures/image321.jpg
http://www.server.com/natashia2/pictures/image001.jpg
http://www.server.com/natashia2/pictures/image002.jpg
... and so forth

I hope this will help you in your downloading.

Downloading

This summary is not available. Please click here to view the post.