Beginners guide to site ripping (4) - Organizing

ORGANIZE IT. This is an art form. Close to a religion. There are many ways of organizing a site rip into something manageble.

You should organize a siterip because of several reasons. Here is a few that hopefully will make you consider your layout.

1. Your layout should be easily browseable
2. Your layout should enable anyone looking at the site to find the exact set fast
   
3. Your layout should have an order or sequence, dated or numbered
4. Your layout should include all possible information about every set
5. You should never rename files, keep the site names. It makes it easier for others to find specific files
6. Always go for the higest quality of pictures and the best quality of videos, delete the rest
   
7. Avoid organizing by model, organize by chronological representation
8. Keep as flat a structure as possible. It is easier to browse and is CD/DVD/BD burn friendly
9. Identify videos clearly by putting them in a specific folder or marking the foldernames clearly. Not everybody likes it all mixed up
10. Do not keep stuff in your layout which has not been directly downloaded from the site. Do not put home created stuff into your layout
11. Make sure your rip/layout is complete to the date. Do not leave stuff out of the layout without clearly signaling what has been left out
12. If your layout is media spefic (CD/DVD/BD) then keep the same layout on all medias.
13. Your layout should be extensible
14. If you make csv's after you have organized your siterip, then create a comment on the 1st. line which tells the date of the siterip, and possible notes to what is NOT in the csv. If you favour extensibility then make notes to anyone who may extend your work
15. If a site is messy then include set numbers/codes in the foldernames so that anyone other than you can find his or her ways around the site and your layout. It should be possible to easily match your layout to the site
16. Avoid having foldernames longer than 64 characters long due to media burnability
17. Avoid national characters in foldernames as well as special characters. Replace & with 'and', replace , with - and so forth. Keep it simple
    

This article is not about using your Offline Browser to filter away all the unwanted stuff, this is something you eventually will learn. And in the case you are not using an Offline Browser for the job then it is probably ReGetm DownThemAll or GetRight which helps you.

Manual
The simplest way of organizing is the manual way. You quietly write the directory names yourself, either by copy/paste from the site to your Windows Explorer, Total Command or DirectoryOpus. And build everything up by hand. Unzipping manually and everything.

If you are maintaining a site on a regular basis then this is the easiest way to go. It is seldom beneficial to go tool crazy when you just have to download and add a set or two. It would recommend that everybody try stuff manually. And a site of up til 100 sets can easily be done manual.



File tool
There seems to be 3 tools of preference. They are: Windows Explorer, Total Commander and DirectoryOpus.

I have personally never been a fan of Total Commander ( http://www.ghisler.com/ ), but the tools is helpful for zip files among others and used by a skilled person it can be a deadly weapon.

DirectoryOpus ( http://www.gpsoft.com.au/ )can do everything for you. You can script things and unzip etc. Highly configureable and nice to work with.

Windows Explorer still a pretty handy tool in the hands of a skilled person. The unzip capabilities are too poor. But it is there and if you master the tool then it can be pretty effective.



The editor
If you do not go 100% manual on stuff then chances are you have made lists in a editor. Notepad is always there but not quite good for the job of making either download or organizer or folder renamer lists. You need something a bit more powerful.

Excel is good for lists, you can easily sort stuff and export it to different formats or just simply copy/paste from it.

UltraEdit ( http://www.ultraedit.com/ )is a allround editor. Pretty good at everything. Column mode supported, which is a big plus. Easy macro recording and playback. My favourite for editing. Column mode is extremely useful for making both download lists and also directory name lists. Writing directly column mode can also be very beneficial when creating dated folder names.

EditPad Pro ( http://www.editpadpro.com/ ) same the same capabilities as UltraEdit, and a very good regular expression capable search/replace tool. It is more a decision about religion in regard to what editor you use. But I'd advice you to go for either UltraEdit or EdiPad Pro, or maybe both if your wallet is thick enough. There are other alternatives to editors and some are also free of charge. You may consider SlickEdit http://www.slickedit.com/, Notepad++ http://notepad-plus.sourceforge.net/uk/site.htm or even EditPlus http://www.editplus.com/.



Helper files (text and BAT)
It can be quite useful to make BAT files for helping you with making directories, unzipping or moving files around. There is not far from creating a text file in your editor with all the foldernames into making a batch file. Take a look at the two pictures.

Click to enlarge (text file)

Click to enlarge (batch file)

The text file version (to top most) just gives you the directory names. The batch (BAT) file version put a MKDIR (MaKe DIRectory) command infront of the foldername and if you save the BAT file to the directory and the doubleclick it then it creates all the folders for you, and you have less manual work.

You can make more advanced BAT files, but it will require you to learn the commands which can be used in the BAT file. All command line tools can be used. So stuff like CD (Change Directory), REN (REName), DEL (DELete) and MOVE (MOVE) are available to your disposal. Both WinRAR and PKZip / UNZip also have command line versions of their tools so you can unzip alot of files with one BAT file.

Take a look at the unzip.bat file below which enters every directory and unzip's all the zip files in the directory and once finished it deletes all the zip files in the directory and finally goes back (cd ..) to the original directory.

Click to enlarge

There are tools for creating batch files (BAT). One of the tools is called DO which is an old tool from 2002. It is commandline based and you can download it here. Be sure to read the do.txt file for help.


Common layouts
The common standard towards layout is a) Browsable, b) Easy to find specific sets and c) Maintainable. If sets are dated, then use dates. Keep the dates in ISO standard which means year-month-day. If dates are not used the use numbers in sequence. Below you have 6 pictures which you can click to see common layouts. Click each picture to enlarge.

Beginners guide to site ripping (3) - Tools

In the article "Beginners guide to site ripping (2) - Check & CSV" we spoke of differnet tools to make a csv.

There are quite a few tools which come into question in regard to csvs. The best are listed below.

• The!Checker v4.20 (2006) http://mitglied.lycos.de/vhorch/piccheck.htm
• Hunter v2.1 (1999) http://broquard-ebooks.com/hunter/Default.htm
• PhotoServe Professional Verify v1.01.39 (2008)
  
• PServeCheck Ultimate v4.5.1.45 (2008) http://w4nd0rn.blogspot.com
  

• ScanSort v1.81 (1999) http://www.geocities.com/SouthBeach/Pier/3193/scansort.html

• fastCSV v0.09b (2005) http://wdispatcher.tripod.com/

• CSV Workshop v1.0.0.6 (2003) http://home.comcast.net/~brazilwares/

The first 4 tools are all collection managers. Which means that they can make a csv and they can sort files according to csvs (ecsv's). They can keep all your csvs in views for later and manage your collections of csv'd material.

The 5th tool, ScanSort, is not a collection manager, it is a tool for sorting contents according to the layout of a ecsv file. ScanSort is a command line tool and thus has no graphical interface.

The 6th tool, fastCSV is a tool for making csvs, it cannot sort files according to an ecsv. This tool is also a commandline tool and has no graphical user interface.

The 7th tool, CSV Workshop, is a tool for viewing csvs and finding problems among ecsvs. This includes internal and external dupes and other nasty things.

All 7 tools are designed for Windows, but there should be a possibility to port PSPV to other platforms as this is written in Perl mainly.

Most of the tools are not maintained any longer and some of them suffer from age, as example CSV Workshop which cannot correctly display the proper internal size of a ecsv file if it holds data above 4Gb.

I digged up some old tutorials for you to read.

Download: Muleskinners guide to Hunter
Download: PigVomit's guide to Hunter
Download: PWA's guide to The!Cheker
Download: PigVomit's guide to The!Checker
Download: ScanSort How To
Download: CSV Workshop screenshots and guide

Note!
PSPV (PhotoServe Professional Verify) is similar to PSCU (PServeCheck Ultimate) and both tools are very new tools and they both gets maintained on a constant basis. Both tools are by far way more advanced (not complex!) and faster than any of the other tools listed.

Both tools are rather intuitive and can peform a bunch of tasks. Some of the features in these tools are as follows.

• Automatic download of updated csvs from repository sites.
• Disk space management which automatically moves contents around when needed.
• Hunting/sorting from raw, zip and rar files.
• Integrated uploads of updated csvs.
• Integrated and automatic leech of collections.
• Mass Miss/Wrong and Needs csv fill possibilities.
• Automatic mass collection setup.
• Internal dupe and unwanted file detection.
• Cache for ultra fast sorting/hunting.
• Complete filter views for collection groups.
• View filters for: ongoing, pre-final, final, need to check, ready to burn, burned.
• Direct integration to Nero and ImgBurn.
• HTML reports and backup status.
• Empty directory cleanup.
• CSV and report cleanup.
• Leech priority.

World premiere - PSCU

Click to enlarge

More screenshots here, here and here.

MG is proud to offer you the first public release of PServeCheck Ultimate Edition v4.5.1.45 right here. As with PSPV (the Perl relative to PSCU) this tools is a community tool which has been restricted for a select few for about 6 years. Read the file 'readme.txt' before you run the program and bare in mind that PSCU and PSPV are not meant to be standalone collection managers, as with Hunter or The!Checker. What you get here today is only 1/4th of the full package.

A big salute and thanks to The2nd for quickly assembling a 'standalone' version of PSCU for release here on MG, alongside being the author of PSCU.

Also thanks to the staff at VG for the triggerpack which follows the release. Shouts to Austria for the programming and to Canada for the servers and service at VG.

PSCU quick install:
Just a quick note. There are 2 groups among the trigges which are extendable. The groups are: 'Custom' and 'My Personal Collections' where you can add your own custom stuff. When PSCU refers to the mIRC path then point it the proper directory of the unzip. Edit and check the file remote.ini before you launch the executable the first time.

We will put an article about usage of PSCU.

At the end of this article PSPV is still community tools and thus not open for everybody to use.

Beginners guide to site ripping (2) - Check & CSV

Let's assume you have managed to download a site, any site for that matter. But at this point you will have ended up with a folder structure of some kind. And you will have some folders with plain .jpg's and maybe even some with .zip files. You also have a folder with some videos, .wmv or .avi probably or maybe some .mov files.

Before you do ANYTHING!, to the files...reorganize, unzip, delete, move, rename.....ANYTHING!. Then this is the expert advice.

Make a csv of the raw rip - ALWAYS!

Why? - you may ask. Well there is 1 simpel answer to that. If you fuck up and accidently delete something or copy/move stuff to the wrong places then the raw site-rip csv will organize the entire thing back to the original places and you can start over. You can even spot/find the file you accidently deleted.

"Make a csv?" WTF is that ? you may ask yourself. A csv is a simpel file which can be used for organizing and checking files. You need tools to create csvs and you also need tools for organizing/sorting stuff according to a csv. We will eventually get to the tools in an article. But to skip the jazz, then go search for tools like PicCheck / The!Checker, Hunter, ScanSort, PSP Verify, PServeCheck, fastCSV etc. The csv files are acutally ecsv (Extended Comma Seperated Values) file and it has a specific layout like this: (excerpt only)

20080527-video-full.wmv,105742805,3925D04D,\20080527 - Vicky - Pantyhose\,
001.jpg,3065686,B75F6804,\20080529 - Natasha - Wall Bars\,
002.jpg,2998642,5B2B26A0,\20080529 - Natasha - Wall Bars\,
...

There are 5 parts of a ecsv file. And the order of the 5 parts are of importance.

1. Filename
2. Size counted in bytes
3. CRC32 value in hexadecimal format
4. Complete path for the file
5. Optional comment

The tools listed earlier will help you make such a file. The ecsv files have a naming convention. There are two often used conventions, one is the WSC convention and the other is the ET/VG convention.

WSC looks like this: MVCD2_Mystique_Magazine_Vault_CD2_5744.csv

ET look like this: Watch4Beauty-DVD12(Pre-Final)_1903.csv

WSC put the so called trigger part as the first part of the ecsv filename, in the example it is MVCD2. The next part is the descriptive name (Mystique Magazine Vault). The 3rd part tells you information about media size of the ecsv OR the year of the ecsv. The last part shows how many files the ecsv covers. Finally the extension: csv.

ET uses a different approach, there is no trigger part, instead there is a descriptive name followed by a - and then a media descripter (DVD12). This is always the case and there is a counter together with the media description. Immediately following the media descripter is a ecsv state identifyer. There are 6 different states (normally).

1. Ongoing, means the whole (...) is omitted. Which means that the csv still changes and probably does not contain sufficient files to be a full CD or DVD.
2. (Pre-Final). In this case the ecsv has reached the proper number of files so that a full CD or DVD or whatever can be filled if one were to burn the contents the ecsv coversto a media.
3. (Pre-Final-1). The ecsv is still not 100% closed and the maintainer changed something. Maybe added a missing file, maybe renamed a folder. A change compared to the Pre-Final ecsv.
4. (Pre-Final-2). The 2nd change from PF (Pre-Final).
5. (Pre-Final-3). The 3rd change from PF.
6. (Pre-Final-4). The 4th and hopefully last change from the PF ecsv.
7. (Final). The ecsv is now closed and no changes will be made to it any further. You could burn the contents the ecsv covers to a media and feel save that it does not change.
8. (Final-1). The ecsv maker made a minor fuck-up and the (Final) should be discharded.
9. (Final-2). More errors and the F1 should be discharded the F2 should be used.
   
10. (Final-3). More errors and the F2 should be discharded the F3 should be used.
11. (Final-4). More errors and the F3 should be discharded the F4 should be used.
12. (Re-Burn). The ecsv maker fucked up good. And there was an error. So stuff was re-organized / cleaned and people should re-burn the media.
13. (Re-Burn-1) -> (Re-Burn-4). At this point someone will have shot the ecsv maker, for continiously making errors and having to re-do them over and over again.

Most often you will see stuff going from 'ongoing' into (Pre-Final) maybe even (Pre-Final-1) and then to (Final).

I will not speak further about the ecsvs and how to use them. Just make a ecsv of your downloads so that you can use a tool organize the contents back to the original places.

Before you begin to clean up your downloads and put stuff into proper folders then you need to check the contents to see if it actually is any good.

Picture checking
You can use Hunter, jCheck or jpgfix to check the jpeg pictures.

Video checking
As far as I know there does not exist any tools to verify wether a video (avi, wmv or mov) is good. But there are three good indicators. Play it with Fast Forward and watch that it is playable. Secondly, randomly click the seeker bar in your video player to find out wether it is seekable. And lastly, always check the last few seconds to see wether the video ends like it should and not premature.

Hunter needs some understnading and can be quite complex to uses. JpgFix is a commandline tools may be not suit everyone. jCheck is a simple Windows dialog with one purpose. So it should be straight forward using that one.

Beginners guide to site ripping (1)

When you are faced with the need to download a site from the web then there are several options.

You can always go ahead and spend some days or weeks using FireFox or Internet Explorer and click every picture and video and choose "Save To" or "Save As". Just writing the path's for the jpg's and the videos may take up alot of your time. And chances are you may miss something. So what to do?

Get a download manager!

But which one ? I do not know many download managers, but some of the most known ones out there are:

• Down Them All http://www.downthemall.net/
  
• Get Right http://www.getright.com/
  
• ReGet http://www.reget.com/
  

They can help you when you click around and they will queue up your downloads. And you can stop at one day and continue on another.

You still have to visit every page and right clicking and choose "Download With ...". But you can do it much faster and you do not have to wait for downloads to complete before you begin a new download.

To circumvent having to click every link or at least every page you can get other tools. In general these tools are called Offline Browsers. They can automatically click every link on a website and altomaticially download every file from the website. So basically it is like if you took a copy of the website to your own harddrive. Some of the more common tools in the Offline Browser category are:

Get an Offline Browser!

• HTTP Weazel http://www.metaexception.com/en/products/HTW/
  
• HTTrack http://www.httrack.com/
  
• Offline Explorer http://www.metaproducts.com/
• Black Widow http://softbytelabs.com/us/bw/index.html

Personally I started off with the Weazel and had a short fling to the Track before I meet the one and only OE. But that is just personal experience. The Weazel is easy and so is the Widow. The learning curve for the Track is a little higher and OE can be too.

Offline Browsers start by visiting the page you point them to. Then they automatically click every link on that page which leads them to all pages at "level 2". On level 2 they download everything to your harddrive and then they move on to click all links on level 2 so that they reach level 3 and so on.

These tools can work during the night, so you can leave your computer on and let it download a website while you sleep. The tools can also work as if there were 2,3 or 8 or more people downloading the site. So you are not limited to just 1 "virtual" person clicking. If you play around then avoid settings above 8, it is not nice for the website...and if the same person is hammering on a website with 8 differen "people" all wanting to download then chances are the webmaster may get pissed off. Set the limit to 2-3 and maybe even 4.

Downloading the Internet, please wait....

Now, this feature of automatically copying an entire website is simply great. BUT!, what if the website links to Google? Then you begin visiting every link Google has and all the sites it points to ? YEP!, you will begin downloading the entire internet.....And I bet you that you will run dry from memory and harddrive space before the download is finished.

Download only 1 website...

To avoid downloading the internet you have to limit your Offline Browser to only download from the website you want. You can most often do this by only downloading from "the starting server" or only download from "the starting domain".

Perhaps you do not want small thumbnails and other small graphic pieces which are displayed on every page. Maybe you do not even want to save the .htm / .html files. If this is the case then tick off stuff as .gif, .htm etc. This will tell your Offline Browser to NOT to save these file types to your harddrive.

I suggest you play around with an Offline Browser and a good site could be this site. Make a new project and download everything from the site only and avoid downloadin anything which does not belong to this website.

Browse the contents which was downloade and play around with settings which may allow you to both download this website and the graphics too.

Getting the proper tease

One of the issues with the ONTE site is the hassel with downloading the proper higest quality version of the picture sets. Some sets come in plain size, others in both plain and large sizes and some, luckily most these days, comes in ultra high quality too.

This article will give you one way to filter out all the unwanted versions of the zipped picture sets.

In order for this to work you have to go to every month you wish to download.

Then you should save every month on the site to a htm file, no need to save the graphics along, just keep it to html only. Secondly you have to remove all newlines, carrige returns and tabs from the htm files..

You can accomplish that by running a TR command on the commandline in the OS of your choice.

The TR should look like the one below.

tr -d "\n\r\t" < onteXYZ.htm > onteXYZ.html

Do that on every page you saved. Once they are all saved to their .html counterpart, then run the command below.

grep -Po "http://[\w.]+/members/(.....)?zips/[-\w]+\.zip(?=..[\w ]+.?\([\d]+.[\d]+ MB\)./div../div.)" onte*.html

That will extract all the picture zip files in the higest quality available. No need to filter stuff manually. And no need to clean up after the download.

What the grep actually does is to look for any http:// link which links to a zip file and have some trailings which consists of two ending div's.

Have a look below to better understand the result.

Click to enlarge

Tease and please for show

The csv for the Only Tease site has had a rest of updates since January 2008. The previous maintainer is MIA and that put a stop to a very good csv set.

BUT!, times change and I can happily annouce that new updates are on the way. A source has sprung and this means that new csvs for the ONTE site is arriving at VG and mirrors.

So that you can all enjoy some nice sorting of your downloads.

Here's some cover teasers.

TXT2WJR gets graphical

New version, now with GUI, bugfixes and new feature.

Program purpose:

Convert a text file with url's and an optional text file with directory names into a proper ReGet Deluxe download queue. Ready for downloading.

New features:

• Appends to an existing ReGet queue if one is choosen
  
• Set cookie (optional)
• Set referer (optional)
• Set username (optional)
• Set password (optional)
• Handles target directories as both absolut and relative or plain.
  

You can have a quick look at the GUI for TXT2WJR v0.04. This tiny GUI is made with Microsoft C# language and some stuff called WPF which is hidden inside .NET framework v3.5. It's all some new Microsoft Windows Siesta (sorry Vista) jazz. Anyway, it is my first attempt ever on writing a C# program, at the same time it is my first time ever with .NET and even my first time with WPF (Windows Presentation Foundation). So bare with me if it looks and acks like something the cat dragged in.

Click to enlarge

This package will be available as a ZIP download hosted on MediaFire sometime pretty darn soon. (When I bother to package it, hehe).

Beginners RegEx - part 4 - repeats and boundries

This is a addition to the former article about metacharacters. And a little new feature which allows us to tell how many occurances we want to allow of a centain character.

We have seen that * means zero or more, and that + mean one or more of the preceeding characters.

In this case we will use dates as the taget for our example. Consider some dates like the following.

30 April, 2008
14 May 2008
30. April 2008
1 - May - 2008 Description
2nd May 2008
Date: 12TH - 4 - 2008
02-05-2008

All valid dates to my eye, so we want to make a regular expression which assures us that we get such dates if we look for them. Let's outline some of the rules for dates. I will skip the real validation test regarding invalid dates like the 65th February. But merely consider the format.

• Day comes first
• Month comes second
• Year comes last
• Day is 1 or 2 digits long
• Month is letters or a number which 1 or 2 digits long
• Year is 4 digits long
• We have some kind of seperators between day, month and year

Repeats - minimum and maximum
Those are the basic rules for the date formats we wish to accept and match. The first thing we need to assure is that the digit versions of day, month and year do not exceed their natural limitations of 1-2 digits, 1-2 digits and 4 digits. This can be done by setting minimum and maximum.

[0-9]{1,2}

The character class [0-9] must be repeated atleast 1 and maximum 2 times in sequence. So every number between 0 and 99 is good. {minimum,maximum} of the preceeding character or character class. There is a flavour of the {minimum,maximum} specifier. It is used in a case like this:

[0-9]{4}

Repeats - exact count
Using just a {exact count} can be useful when we want to make assure that the year in our dates is exactly 4 digits.

One other thing which could become useful for us is to find the boundry of the dates we wish to match. What we want is to check from the boundry of the date to the ending boundry of the date. Let's consider a simpel version which could match one of our dates.

[0-9]{1,2}-[0-9]{1,2}-[0-9]{4}

Boundries
That one could match 02-05-2008 with success. But what if the date wasn't actually a date, but part of a calculation like 2002-05-20089124, then we would still get a match on some of it, hence 2002-05-20089124. Which is not something we want. For the purpose regular expressions has a metacharacter called \b (boundry). What \b actually does is to distinguish between what is considered a word ( \w = [a-zA-Z0-9_] ) and anything else. So if we re-write the previous regular expression into:

\b[0-9]{1,2}-[0-9]{1,2}-[0-9]{4}\b

Then we get the desired effect. Hence we are saying that we have to have some kind of boundry first then some digits and - (hyphen's) and finally a boundry again. This well allow us to not confuse 2002-05-20089124 with a date.



Multiple choices
There is also the issue about the month being spelled instead of written with digits. This means that we will have to adjust out month part a bit to allow either digits (as now [0-9]{1,2}) or letters ( [a-zA-Z]+ ). The subexpression [a-zA-Z]+ says letters a-z and A-Z and we have to have a minimum of 1 letter and maximum unlimited; but only letters. So we adjust our regex alittle again. This time we use the logical OR construction ( thispart | orthispart ). The expression will look like this:

\b[0-9]{1,2}-([a-zA-Z]+|[0-9]{1,2})-[0-9]{4}\b

Metacharacters for digits
Now all the 0-9 becomes a bit confusing with all the 1,2 numbers in there, so we re-write it using the \d (digit) metacharacter.

\b\d{1,2}-([a-zA-Z]+|\d{1,2}+)-\d{4}\b

Seperators between day, month and year
We're getting there. But we still lack a few issues. Namely the seperating characters. We have to accept , comma, . dot, - hyphen, space, tab etc. as seperating character. If you read the previous article you can find that \s is space, tab, newline etc., so a character class like [\s,.-] will cover our possible seperator. And since we need to have atleast 1 seperator between our day, month and year, we tweak the character class to [\s,.-]+ and we modify it to look like below.

\b\d{1,2}[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

Extensions to day
It's is almost complete we can now have any number of seperators between our day, month and year. So all we need is to allow days like: 1st, 2nd, 3rd,4th etc. We basically need to allowed some text extensions to the day member of our expression. We could do this by a simple subexpression like (st|nd|rd|th). The trick is that the dates may contain the extensions or they may not, so we have to make the subexpression optional. So we add a ? to the subexpression and get (st|nd|rd|th)?. Our expression looks like this:

\b\d{1,2}(st|nd|rd|th)?[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

So what do we need to assure ?, basically the only things left is that the dates MAY have some kind of seperation between the day and the extension. Such as 1 st or 2 nd. So lets inject our seperator subexpression [\s,.-] in between and let us make it optional as to zero or more times. We do that with a *, so we get:

\b\d{1,2}[\s,.-]*(st|nd|rd|th)?[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

Case in-sensitive
What else ?. What if the day was written like 1 St ? There is some case sensitive issues. We could extend our subexpression (st|nd|rd|th)? into (st|nd|rd|th|St|Nd|Rd|Th)? or we could make the whole thing in-case sensitive. That is done by (?i), so now the expression looks like:

\b\d{1,2}[\s,.-]*(?i)(st|nd|rd|th)?[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

And I believe that's is as good as it gets today. Take time to elaborate the regular expression. There is a few nitty gritty things one can do to make it really tight.

Beginners RegEx - part 3 - metacharacters

In "Beginners RegEx - part 1" we saw some character classes (the [a-z] style) to march any character between a and z. We also saw that you can mix character classes to include more characters by using styles like [a-z0-9] to match every letter from a to z and every digit from 0 to 9. And if we wanted allow the uppercase versions of the letters we'd have to write something like [a-zA-Z0-9].

Today we'll look at some metacharacters \char, which can make life easier when using character classes [ characters ].

Here is a short list of some character class metacharacters

\t = Tabulator
\n = Newline
\r = Carrige return
\s = Any white space character, ie. space, tab, newline, formfeed etc.
\S = Any character which is NOT a whitespace
\w = Word, which is basically [a-zA-Z0-9_]
\W = Any character which is NOT defined as being a word
\d = Digit, which is basically [0-9]
\D = Any character which is NOT defined as being a digit

So now we can construct some tighter regular expressions which eventually we be easier to read, once you grow acustomed to it.

So if we have a situation in a html file which looks like this:


Then we can construct a regular expression which matches the url part of the html by writing the following regex.

http://[\w\s/.-]+

This will give us a result like:

We see a new feature of regular expressions in the regex above. One we haven't discussed yet. It is the + (plus), which tells us to match atleast one or more occurances of the preceeding character class. Opposite to the * which means match zero or more occurances of the preceeding character or character class.

Let's take the regex apart and explain what is going on.

Click to enlarge

One of the things you may wonder about, hopefully, is that inside the character class we wrote . (dot) to allow marching of the dots between the url name parts w4nd0rn.blogspot.com and the final . just before the type (.html).

Inside a character class [ ] the . (dot) does not mean any character what so ever. In a character class it means just what it is, ie. . (dot).

Beginners RegEx - part 2 - negation and zero-width positive lookbehind

We're going to concentrate on negating ranges and a regex feature called zero-width positive lookbehind. Dom't let the words scare you. It is harmless.

Negation is used to tell the regex that it should match everything which is NOT in a range []. Zero-width positive lookbehind is used to get the regex to match characters which will be used to find the stuff we are looking for, but to avoid getting the stuff as the result from our regex. The latter sounds really cryptic, but is quite useful.

Zero-width positive lookbehind. Have a look at the regex below

(?<=.H1.)[^<]*

The first part of the regex is the zero-width positive lookbehind (?<=text). The second is a range [characters] and the last is * which matches zero or more of the preceeding range. Let's take it from the top.

(?<=text) tells that we are looking for text, but if we find a match then we do not want the text "printed" as part of the result. In the case we are looking for .H1. which, if we were searching a html file, would match <H1> (header size 1). Remember that . (dot) matches any character so . (dot) before H1 matches both the < before and the > after the H1. The only real restriction when using (?<=text) is that the text has be of fixed length. So it is not possible to put ? and * inside a (?<=text) because these could change the length of the zero width look behind sub expression.

[^<]* is a little tricky to understand. First of, it is a range we specify. So anything inside the [ and ] is what we are looking for. The * after the range tells us that whatever we find that matches the range can be repeated zero or more times. Inside the range we use 1 metacharacter, the ^.

^ as the very first character in a range means "anything but". So if we had [^a] means that we want to match anything but the letter a. With the [^<] we look for anything but the character <, and as we have a * succeding the range [^<]* we look for zero or more occurances of characters which are NOT <. Practically we get a match on a html string like this:

<H1>here is some header text</H1>

The first part of the full regex finds <H1>, but omits it in the result, the second part fines the text after the first < and matches everything till it reaches a <. So the result is that we get the text inside the html tags <H1> and </H1>.

You can negate more than 1 character when using ranges. A regex like [^<=@] tells us that we want anything besides the characters < and = and finally @.

Beginners RegEx

Here's a short article which could be called "RegEx for dummies" or beginners. The most important thing about all of this is to read the primer (below).

PRIMER: Regular Expressions (RegEx) is used to find text which matches certain criteria. RegEx does NOT find words or sentences. It finds single characters. So a regular expression like "gin" does not find the word "gin" it finds the letter "g" followed by the letter "i" followed by the letter "n". Make a not of this because it will become much easier this way.

So here is a few examples followed by explanation of some of the meta characters in RegEx.

RegEx: "gin" matches Beginners.

RegEx: "gin?" matches Begixxers. Because the n is optional (? = preceeding character is optional)

RegEx: "(ht|f)tp://" matches both http:// and ftp:// (| = an logical or between the enclosed character sequences)

RegEx: "[1-2][0-9]" matches any number sequence where the 1st digit is 1 or 2 and the following digit is between 0 and 9. So numbers between 10-29. But also 10000 or 294.51 or 14degrees.

RegEx: "/[a-z0-9]/" marches any sequence where a / is followed by letters from a-z or digits from 0 to 9 and then followed by a /

RegEx: "s.x" matches any sequence where a s is followed by ANY character and then followed by a x. So sex, six, sux, sax are all good matches.

RegEx "http:.*\.zip" matches anything followed by after the letters http: 0 or more times and then finally .zip. The * means reperition 0 or more times . means any character \. means the literal dot itself.


So here is a tiny explanation:

? = The preceeding character is optional. Can be there or not
(x|y) = A sub group where either x or y is required. One could make them both optional by (x|y)?, see ?
[a-z0-9/] = A range for 1 character which can be any letter between a-z, 0-9 or a /.
. = Means any character what so ever.
* = Like ? it tells that the preceeding character can be repeated 0 or an unlimited time.
\. = The literal dot itself. It is "escaped" by the \ to tell that we mean . and not the .(any char)

That's all. Download EditPad Pro and load/write some text, use the search (Ctrl+F) and write your first regular expressions to try searching.


















In later articles we shall construct more "game on" kind of regular expressions. And explore tools like GREP which has a slightly different and less featured regex than EditPad Pro.

Making the URL list

If you've bother to watch the video tutorial on how to easily transform an url list file into a ReGet queue with all nessary information, then you must face the job of creating an URL list file.

Now this can be done manually by copying the nessary information from the html files in your browser....OR....you may lean against regular expressions and GREP.

NOOOOOOOO!, I can hear you scream...too complex!

Yeeees!, I shout and tell you to watch the video tutorial on how to produce a URL list file by using regex. I will make some articles on the regular expression language. But for now.... watch the videoclip and decide for yourself if it is worth using.



Download videoclip here: urllist.avi.zip

Now, you may say "I could have done this with ReGet and IE integration or FireFox and FlashGot." and you're right. But when the contents spans 25 pages or hundreds then using ReGet integration or FlashGot becomes a hazzel. That is why I introduce this way of doing things as it is easier if you have to produce URL lists for several pages by this method. Using only 1 html file is to keep the complexity to a minimum.

Practical TXT2WJR

Here you have a video tutorial with a practical example of how to use the tool. The video takes advantage of the newest version of TXT2WJR, which is v0.03. You can download it here.

Be advised that it is better if you download the video (XViD format) and view it as the image may seem blurred here on the blog.




Download video file for better view: http://www.mediafire.com/?zjzwsymflwe