Beginners RegEx - part 4 - repeats and boundries

This is a addition to the former article about metacharacters. And a little new feature which allows us to tell how many occurances we want to allow of a centain character.

We have seen that * means zero or more, and that + mean one or more of the preceeding characters.

In this case we will use dates as the taget for our example. Consider some dates like the following.

30 April, 2008
14 May 2008
30. April 2008
1 - May - 2008 Description
2nd May 2008
Date: 12TH - 4 - 2008
02-05-2008

All valid dates to my eye, so we want to make a regular expression which assures us that we get such dates if we look for them. Let's outline some of the rules for dates. I will skip the real validation test regarding invalid dates like the 65th February. But merely consider the format.

• Day comes first
• Month comes second
• Year comes last
• Day is 1 or 2 digits long
• Month is letters or a number which 1 or 2 digits long
• Year is 4 digits long
• We have some kind of seperators between day, month and year

Repeats - minimum and maximum
Those are the basic rules for the date formats we wish to accept and match. The first thing we need to assure is that the digit versions of day, month and year do not exceed their natural limitations of 1-2 digits, 1-2 digits and 4 digits. This can be done by setting minimum and maximum.

[0-9]{1,2}

The character class [0-9] must be repeated atleast 1 and maximum 2 times in sequence. So every number between 0 and 99 is good. {minimum,maximum} of the preceeding character or character class. There is a flavour of the {minimum,maximum} specifier. It is used in a case like this:

[0-9]{4}

Repeats - exact count
Using just a {exact count} can be useful when we want to make assure that the year in our dates is exactly 4 digits.

One other thing which could become useful for us is to find the boundry of the dates we wish to match. What we want is to check from the boundry of the date to the ending boundry of the date. Let's consider a simpel version which could match one of our dates.

[0-9]{1,2}-[0-9]{1,2}-[0-9]{4}

Boundries
That one could match 02-05-2008 with success. But what if the date wasn't actually a date, but part of a calculation like 2002-05-20089124, then we would still get a match on some of it, hence 2002-05-20089124. Which is not something we want. For the purpose regular expressions has a metacharacter called \b (boundry). What \b actually does is to distinguish between what is considered a word ( \w = [a-zA-Z0-9_] ) and anything else. So if we re-write the previous regular expression into:

\b[0-9]{1,2}-[0-9]{1,2}-[0-9]{4}\b

Then we get the desired effect. Hence we are saying that we have to have some kind of boundry first then some digits and - (hyphen's) and finally a boundry again. This well allow us to not confuse 2002-05-20089124 with a date.



Multiple choices
There is also the issue about the month being spelled instead of written with digits. This means that we will have to adjust out month part a bit to allow either digits (as now [0-9]{1,2}) or letters ( [a-zA-Z]+ ). The subexpression [a-zA-Z]+ says letters a-z and A-Z and we have to have a minimum of 1 letter and maximum unlimited; but only letters. So we adjust our regex alittle again. This time we use the logical OR construction ( thispart | orthispart ). The expression will look like this:

\b[0-9]{1,2}-([a-zA-Z]+|[0-9]{1,2})-[0-9]{4}\b

Metacharacters for digits
Now all the 0-9 becomes a bit confusing with all the 1,2 numbers in there, so we re-write it using the \d (digit) metacharacter.

\b\d{1,2}-([a-zA-Z]+|\d{1,2}+)-\d{4}\b

Seperators between day, month and year
We're getting there. But we still lack a few issues. Namely the seperating characters. We have to accept , comma, . dot, - hyphen, space, tab etc. as seperating character. If you read the previous article you can find that \s is space, tab, newline etc., so a character class like [\s,.-] will cover our possible seperator. And since we need to have atleast 1 seperator between our day, month and year, we tweak the character class to [\s,.-]+ and we modify it to look like below.

\b\d{1,2}[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

Extensions to day
It's is almost complete we can now have any number of seperators between our day, month and year. So all we need is to allow days like: 1st, 2nd, 3rd,4th etc. We basically need to allowed some text extensions to the day member of our expression. We could do this by a simple subexpression like (st|nd|rd|th). The trick is that the dates may contain the extensions or they may not, so we have to make the subexpression optional. So we add a ? to the subexpression and get (st|nd|rd|th)?. Our expression looks like this:

\b\d{1,2}(st|nd|rd|th)?[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

So what do we need to assure ?, basically the only things left is that the dates MAY have some kind of seperation between the day and the extension. Such as 1 st or 2 nd. So lets inject our seperator subexpression [\s,.-] in between and let us make it optional as to zero or more times. We do that with a *, so we get:

\b\d{1,2}[\s,.-]*(st|nd|rd|th)?[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

Case in-sensitive
What else ?. What if the day was written like 1 St ? There is some case sensitive issues. We could extend our subexpression (st|nd|rd|th)? into (st|nd|rd|th|St|Nd|Rd|Th)? or we could make the whole thing in-case sensitive. That is done by (?i), so now the expression looks like:

\b\d{1,2}[\s,.-]*(?i)(st|nd|rd|th)?[\s,.-]+([a-zA-Z]+|\d{1,2})[\s,.-]+\d{4}\b

And I believe that's is as good as it gets today. Take time to elaborate the regular expression. There is a few nitty gritty things one can do to make it really tight.

Beginners RegEx - part 3 - metacharacters

In "Beginners RegEx - part 1" we saw some character classes (the [a-z] style) to march any character between a and z. We also saw that you can mix character classes to include more characters by using styles like [a-z0-9] to match every letter from a to z and every digit from 0 to 9. And if we wanted allow the uppercase versions of the letters we'd have to write something like [a-zA-Z0-9].

Today we'll look at some metacharacters \char, which can make life easier when using character classes [ characters ].

Here is a short list of some character class metacharacters

\t = Tabulator
\n = Newline
\r = Carrige return
\s = Any white space character, ie. space, tab, newline, formfeed etc.
\S = Any character which is NOT a whitespace
\w = Word, which is basically [a-zA-Z0-9_]
\W = Any character which is NOT defined as being a word
\d = Digit, which is basically [0-9]
\D = Any character which is NOT defined as being a digit

So now we can construct some tighter regular expressions which eventually we be easier to read, once you grow acustomed to it.

So if we have a situation in a html file which looks like this:


Then we can construct a regular expression which matches the url part of the html by writing the following regex.

http://[\w\s/.-]+

This will give us a result like:

We see a new feature of regular expressions in the regex above. One we haven't discussed yet. It is the + (plus), which tells us to match atleast one or more occurances of the preceeding character class. Opposite to the * which means match zero or more occurances of the preceeding character or character class.

Let's take the regex apart and explain what is going on.

Click to enlarge

One of the things you may wonder about, hopefully, is that inside the character class we wrote . (dot) to allow marching of the dots between the url name parts w4nd0rn.blogspot.com and the final . just before the type (.html).

Inside a character class [ ] the . (dot) does not mean any character what so ever. In a character class it means just what it is, ie. . (dot).

Beginners RegEx - part 2 - negation and zero-width positive lookbehind

We're going to concentrate on negating ranges and a regex feature called zero-width positive lookbehind. Dom't let the words scare you. It is harmless.

Negation is used to tell the regex that it should match everything which is NOT in a range []. Zero-width positive lookbehind is used to get the regex to match characters which will be used to find the stuff we are looking for, but to avoid getting the stuff as the result from our regex. The latter sounds really cryptic, but is quite useful.

Zero-width positive lookbehind. Have a look at the regex below

(?<=.H1.)[^<]*

The first part of the regex is the zero-width positive lookbehind (?<=text). The second is a range [characters] and the last is * which matches zero or more of the preceeding range. Let's take it from the top.

(?<=text) tells that we are looking for text, but if we find a match then we do not want the text "printed" as part of the result. In the case we are looking for .H1. which, if we were searching a html file, would match <H1> (header size 1). Remember that . (dot) matches any character so . (dot) before H1 matches both the < before and the > after the H1. The only real restriction when using (?<=text) is that the text has be of fixed length. So it is not possible to put ? and * inside a (?<=text) because these could change the length of the zero width look behind sub expression.

[^<]* is a little tricky to understand. First of, it is a range we specify. So anything inside the [ and ] is what we are looking for. The * after the range tells us that whatever we find that matches the range can be repeated zero or more times. Inside the range we use 1 metacharacter, the ^.

^ as the very first character in a range means "anything but". So if we had [^a] means that we want to match anything but the letter a. With the [^<] we look for anything but the character <, and as we have a * succeding the range [^<]* we look for zero or more occurances of characters which are NOT <. Practically we get a match on a html string like this:

<H1>here is some header text</H1>

The first part of the full regex finds <H1>, but omits it in the result, the second part fines the text after the first < and matches everything till it reaches a <. So the result is that we get the text inside the html tags <H1> and </H1>.

You can negate more than 1 character when using ranges. A regex like [^<=@] tells us that we want anything besides the characters < and = and finally @.

Beginners RegEx

Here's a short article which could be called "RegEx for dummies" or beginners. The most important thing about all of this is to read the primer (below).

PRIMER: Regular Expressions (RegEx) is used to find text which matches certain criteria. RegEx does NOT find words or sentences. It finds single characters. So a regular expression like "gin" does not find the word "gin" it finds the letter "g" followed by the letter "i" followed by the letter "n". Make a not of this because it will become much easier this way.

So here is a few examples followed by explanation of some of the meta characters in RegEx.

RegEx: "gin" matches Beginners.

RegEx: "gin?" matches Begixxers. Because the n is optional (? = preceeding character is optional)

RegEx: "(ht|f)tp://" matches both http:// and ftp:// (| = an logical or between the enclosed character sequences)

RegEx: "[1-2][0-9]" matches any number sequence where the 1st digit is 1 or 2 and the following digit is between 0 and 9. So numbers between 10-29. But also 10000 or 294.51 or 14degrees.

RegEx: "/[a-z0-9]/" marches any sequence where a / is followed by letters from a-z or digits from 0 to 9 and then followed by a /

RegEx: "s.x" matches any sequence where a s is followed by ANY character and then followed by a x. So sex, six, sux, sax are all good matches.

RegEx "http:.*\.zip" matches anything followed by after the letters http: 0 or more times and then finally .zip. The * means reperition 0 or more times . means any character \. means the literal dot itself.


So here is a tiny explanation:

? = The preceeding character is optional. Can be there or not
(x|y) = A sub group where either x or y is required. One could make them both optional by (x|y)?, see ?
[a-z0-9/] = A range for 1 character which can be any letter between a-z, 0-9 or a /.
. = Means any character what so ever.
* = Like ? it tells that the preceeding character can be repeated 0 or an unlimited time.
\. = The literal dot itself. It is "escaped" by the \ to tell that we mean . and not the .(any char)

That's all. Download EditPad Pro and load/write some text, use the search (Ctrl+F) and write your first regular expressions to try searching.


















In later articles we shall construct more "game on" kind of regular expressions. And explore tools like GREP which has a slightly different and less featured regex than EditPad Pro.

How to find the invisible

When you are using regular expressions to grab text from files, especially from HTML files you often run into a case where the easiest way to find the specif information you are looking for is to use the html tags as qualifiers.

As example:

<STRONG>Here is the text I want to grab</STRONG>

In a case like above the only good qualifier is the HTML tags.

But you do not want a list like

<STRONG>Here is the first occurance</STRONG>
<STRONG>This could be the second occurance</STRONG>

Instead you want only the text inside the specific HTML tags. To accomplish that you should use a advanced regular expression feature called: Zero-width positive lookbehind.

With zero-width positive lookbehind you can use one part as qualifier and another to grab. Let's make a RegEx which handles the above.

(?<=<STRONG>)[a-zA-Z ]*

The magic part of the RegEx is the zero-width positive lookbehind, which specifies that it should look for the occurance <STRONG>. It is done by (?<=plain text). You can only have plain text inside the zero-width positive lookbehind part.

The second part of the RegEx is the matching of any number of characters between a-z and also A-Z plus ' ' (space). As the character < is NOT a part of the matching text then the RegEx will not match the </STRONG>.

I hope this is useful to you.