Shane Duffy

.NET Regular Expressions

Posted on: August 14, 2007

What the Heck is a Regular Expression Anyway?

I’m sure you are familiar with the use of “wildcard” characters for pattern matching. For example, if you want to find all the Microsoft Word files in a Windows directory, you search for “*.doc“, knowing that the asterisk is interpreted as a wildcard that can match any sequence of characters. Regular expressions are just an elaborate extension of this capability.

In writing programs or web pages that manipulate text, it is frequently necessary to locate strings that match complex patterns. Regular expressions were invented to describe such patterns. Thus, a regular expression is just a shorthand code for a pattern. For example, the pattern “\w+” is a concise way to say “match any non-null strings of alphanumeric characters”. The .NET framework provides a powerful class library that makes it easy to include regular expressions in your applications. With this library, you can readily search and replace text, decode complex headers, parse languages, or validate text.

Some Simple Examples

Searching for Elvis

Suppose you spend all your free time scanning documents looking for evidence that Elvis is still alive. You could search with the following regular expression:

1. elvis Find elvis

This is a perfectly valid regular expression that searches for an exact sequence of characters. In .NET, you can easily set options to ignore the case of characters, so this expression will match “Elvis”, “ELVIS”, or “eLvIs”. Unfortunately, it will also match the last five letters of the word “pelvis”. We can improve the expression as follows:

2. \belvis\b Find elvis as a whole word

Now things are getting a little more interesting. The “\b” is a special code that means, “match the position at the beginning or end of any word”. This expression will only match complete words spelled “elvis” with any combination of lower case or capital letters.

Suppose you want to find all lines in which the word “elvis” is followed by the word “alive.” The period or dot “.” is a special code that matches any character other than a newline. The asterisk “*” means repeat the previous term as many times as necessary to guarantee a match. Thus, “.*” means “match any number of characters other than newline”. It is now a simple matter to build an expression that means “search for the word ‘elvis’ followed on the same line by the word ‘alive’.”

3. \belvis\b.*\balive\b Find text with “elvis” followed by “alive”

With just a few special characters we are beginning to build powerful regular expressions and they are already becoming hard for we humans to read.

Let’s try another example.

Determining the Validity of Phone Numbers

Suppose your web page collects a customer’s seven-digit phone number and you want to verify that the phone number is in the correct format, “xxx-xxxx”, where each “x” is a digit. The following expression will search through text looking for such a string:

4. \b\d\d\d-\d\d\d\d Find seven-digit phone number

Each “\d” means “match any single digit”. The “-” has no special meaning and is interpreted literally, matching a hyphen. To avoid the annoying repetition, we can use a shorthand notation that means the same thing:

5. \b\d{3}-\d{4} Find seven-digit phone number a better way

The “{3}” following the “\d” means “repeat the preceding character three times”.

Basics of .NET Regular Expressions

Let’s explore some of the basics of regular expressions in .NET.

Special Characters

You should get to know a few characters with special meaning. You already met “\b“, “.“, “*“, and “\d“. To match any whitespace characters, like spaces, tabs, and newlines, use “\s” . Similarly, “\w” matches any alphanumeric character.

Let’s try a few more examples:

6. \ba\w*\b Find words that start with the letter a

This works by searching for the beginning of a word (\b), then the letter “a”, then any number of repetitions of alphanumeric characters (\w*), then the end of a word (\b).

7. \d+ Find repeated strings of digits

Here, the “+” is similar to “*”, except it requires at least one repetition.

8. \b\w{6}\b Find six letter words

Try these and start experimenting by inventing your own expressions. Here is a table of some of the characters with special meaning:

.

Match any character except newline

\w

Match any alphanumeric character

\s

Match any whitespace character

\d

Match any digit

\b

Match the beginning or end of a word

^

Match the beginning of the string

$

Match the end of the string

Table 1. Commonly used special characters for regular expressions

In the beginning

The special characters “^” and “$” are used when looking for something that must start at the beginning of the text and/or end at the end of the text. This is especially useful for validating input in which the entire text must match a pattern. For example, to validate a seven-digit phone number, you might use:

9. ^\d{3}-\d{4}$ Validate a seven-digit phone number

This is the same as example (5), but forced to fill the whole text string, with nothing else before or after the matched text. By setting the “Multiline” option in .NET, “^” and “$” change their meaning to match the beginning and end of a single line of text, rather than the entire text string.

Escaped characters

A problem occurs if you actually want to match one of the special characters, like “^” or “$“. Use the backslash to remove the special meaning. Thus, “\^“, “\.“, and “\\“, match the literal characters “^“, “.“, and “\“, respectively.

Repetitions

You’ve seen that “{3}” and “*” can be used to indicate repetition of a single character. Later, you’ll see how the same syntax can be used to repeat entire subexpressions. There are several other ways to specify a repetition, as shown in this table:

*

Repeat any number of times

+

Repeat one or more times

?

Repeat zero or one time

{n}

Repeat n times

{n,m}

Repeat at least n, but no more than m times

{n,}

Repeat at least n times

Table 2. Commonly used quantifiers

Let’s try a few more examples:

10. \b\w{5,6}\b Find all five and six letter words

11. \b\d{3}\s\d{3}-\d{4} Find ten digit phone numbers

12. \d{3}-\d{2}-\d{4} Social security number

13. ^\w* The first word in the line or in the text

Try the last example with and without setting the “Multiline” option, which changes the meaning of “^“.

Character Classes

It is simple to find alphanumerics, digits, and whitespace, but what if we want to find anything from some other set of characters? This is easily done by listing the desired characters within square brackets. Thus, “[aeiou]” matches any vowel and “[.?!]” matches the punctuation at the end of a sentence. In this example, notice that the “.” And “?” lose their special meanings within square brackets and are interpreted literally. We can also specify a range of characters, so “[a-z0-9]” means, “match any lowercase letter of the alphabet, or any digit”.

Let’s try a more complicated expression that searches for telephone numbers.

14. \(?\d{3}[) ]\s?\d{3}[- ]\d{4} A ten digit phone number

This expression will find phone numbers in several formats, like “(800) 325-3535” or “650 555 1212”. The “\(?” searches for zero or one left parentheses, “[) ]” searches for a right parenthesis or a space. The “\s?” searches for zero or one whitespace characters. Unfortunately, it will also find cases like “650) 555-1212” in which the parenthesis is not balanced. Below, you’ll see how to use alternatives to eliminate this problem.

Negation

Sometimes we need to search for a character that is NOT a member of an easily defined class of characters. The following table shows how this can be specified.

\W

Match any character that is NOT alphanumeric

\S

Match any character that is NOT whitespace

\D

Match any character that is NOT a digit

\B

Match a position that is NOT the beginning or end of a word

[^x]

Match any character that is NOT x

[^aeiou]

Match any character that is NOT one of the characters aeiou

Table 3. How to specify what you don’t want

15. \S+ All strings that do not contain whitespace characters

Later, we’ll see how to use “lookahead” and “lookbehind” to search for the absence of more complex patterns.

Alternatives

To select between several alternatives, allowing a match if either one is satisfied, use the pipe “|” symbol to separate the alternatives. For example, Zip Codes come in two flavors, one with 5 digits, the other with 9 digits and a hyphen. We can find either with this expression:

16. \b\d{5}-\d{4}\b|\b\d{5}\b Five and nine digit Zip Codes

When using alternatives, the order is important since the matching algorithm will attempt to match the leftmost alternative first. If the order is reversed in this example, the expression will only find the 5 digit Zip Codes and fail to find the 9 digit ones. We can use alternatives to improve the expression for ten digit phone numbers, allowing the area code to appear either delimited by whitespace or parenthesis:

17. (\(\d{3}\)|\d{3})\s?\d{3}[- ]\d{4} Ten digit phone numbers, a better way

Grouping

Parentheses may be used to delimit a subexpression to allow repetition or other special treatment. For example:

18. (\d{1,3}\.){3}\d{1,3} A simple IP address finder

The first part of the expression searches for a one to three digit number followed by a literal period “\.“. This is enclosed in parentheses and repeated three times using the “{3}” quantifier, followed by the same expression without the trailing period.

Unfortunately, this example allows IP addresses with arbitrary one, two, or three digit numbers separated by periods even though a valid IP address cannot have numbers larger than 255. It would be nice to arithmetically compare a captured number N to enforce N<256, but this is not possible with regular expressions alone. The next example tests various alternatives based on the starting digits to guarantee the limited range of numbers by pattern matching. This shows that an expression can become cumbersome even when looking for a pattern that is simple to describe.

19. ((2[0-4]\d|25[0-5]|[01]?\d\d?)\.){3}(2[0-4]\d|25[0-5]|[01]?\d\d?) IP finder

When subexpressions are grouped with parentheses, the text that matches the subexpression is available for further processing in a computer program or within the regular expression itself. By default, groups are numbered sequentially as encountered in reading from left to right, starting with 1.

A “backreference” is used to search for a recurrence of previously matched text that has been captured by a group. For example, “\1” means, “match the text that was captured by group 1”. Here is an example:

20. \b(\w+)\b\s*\1\b Find repeated words

This works by capturing a string of at least one alphanumeric character within group 1 “(\w+)“, but only if it begins and ends a word. It then looks for any amount of whitespace “\s*” followed by a repetition of the captured text “\1” ending at the end of a word.

It is possible to override the automatic numbering of groups by specifying an explicit name or number. In the above example, instead of writing the group as “(\w+)“, we can write it as “(?<Word>\w+)” to name this capture group “Word“. A backreference to this group is written “\k<Word>“. Try this example:

21. \b(?<Word>\w+)\b\s*\k<Word>\b Capture repeated word in a named group

Using parentheses, there are many special purpose syntax elements available. Some of the most common are summarized in this table:

Captures

(exp)

Match exp and capture it in an automatically numbered group

(?<name>exp)

Match exp and capture it in a group named name

(?:exp)

Match exp, but do not capture it

Lookarounds

(?=exp)

Match any position preceding a suffix exp

(?<=exp)

Match any position following a prefix exp

(?!exp)

Match any position after which the suffix exp is not found

(?<!exp)

Match any position before which the prefix exp is not found

Comment

(?#comment)

Comment

Table 4. Commonly used Group Constructs

We’ve already talked about the first two. The third “(?:exp)” does not alter the matching behavior, it just doesn’t capture it in a named or numbered group like the first two.

Positive Lookaround

The next four are so-called lookahead or lookbehind assertions. They look for things that go before or after the current match without including them in the match. It is important to understand that these expressions match a position like “^” or “\b” and never match any text. For this reason, they are known as “zero-width assertions”. They are best illustrated by example:

(?=exp)” is the “zero-width positive lookahead assertion”. It matches a position in the text that precedes a given suffix, but doesn’t include the suffix in the match:

22. \b\w+(?=ing\b) The beginning of words ending with “ing”

(?<=exp)” is the “zero-width positive lookbehind assertion”. It matches the position following a prefix, but doesn’t include the prefix in the match:

23. (?<=\bre)\w+\b The end of words starting with “re”

Here is an example that could be used repeatedly to insert commas into numbers in groups of three digits:

24. (?<=\d)\d{3}\b Three digits at the end of a word, preceded by a digit

Here is an example that looks for both a prefix and a suffix:

25. (?<=\s)\w+(?=\s) Alphanumeric strings bounded by whitespace

Negative Lookaround

Earlier, I showed how to search for a character that is not a specific character or the member of a character class. What if we simply want to verify that a character is not present, but don’t want to match anything? For example, what if we are searching for words in which the letter “q” is not followed by the letter “u”? We could try:

26. \b\w*q[^u]\w*\b Words with “q” followed by NOT “u”

Run the example and you will see that it fails when “q” is the last letter of a word, as in “Iraq”. This is because “[^q]” always matches a character. If “q” is the last character of the word, it will match the whitespace character that follows, so in the example the expression ends up matching two whole words. Negative lookaround solves this problem because it matches a position and does not consume any text. As with positive lookaround, it can also be used to match the position of an arbitrarily complex subexpression, rather than just a single character. We can now do a better job:

27. \b\w*q(?!u)\w*\b Search for words with “q” not followed by “u”

We used the “zero-width negative lookahead assertion”, “(?!exp)”, which succeeds only if the suffix “exp” is not present. Here is another example:

28. \d{3}(?!\d) Three digits not followed by another digit

Similarly, we can use “(?<!exp)", the “zero-width negative lookbehind assertion”, to search for a position in the text at which the prefix “exp” is not present:

29. (?<![a-z ])\w{7} Strings of 7 alphanumerics not preceded by a letter or space

Here is one more example using lookaround:

30. (?<=<(\w+)>).*(?=<\/\1>) Text between HTML tags

This searches for an HTML tag using lookbehind and the corresponding closing tag using lookahead, thus capturing the intervening text but excluding both tags.

Comments please

Another use of parentheses is to include comments using the “(?#comment)” syntax. A better method is to set the “Ignore Pattern Whitespace” option, which allows whitespace to be inserted in the expression and then ignored when the expression is used. With this option set, anything following a number sign “#” at the end of each line of text is ignored. For example, we can format the preceding example like this:

31. Text between HTML tags, with comments

(?<= # Search for a prefix, but exclude it
<(\w+)> # Match a tag of alphanumerics within angle brackets
) # End the prefix

.* # Match any text

(?= # Search for a suffix, but exclude it
<\/\1> # Match the previously captured tag preceded by "/"
) # End the suffix

Greedy and Lazy

When a regular expression has a quantifier that can accept a range of repetitions (like “.*“), the normal behavior is to match as many characters as possible. Consider the following regular expression:

32. a.*b The longest string starting with a and ending with b

If this is used to search the string “aabab”, it will match the entire string “aabab”. This is called “greedy” matching. Sometimes, we prefer “lazy” matching in which a match using the minimum number of repetitions is found. All the quantifiers in Table 2 can be turned into “lazy” quantifiers by adding a question mark “?“. Thus “*?” means “match any number of repetitions, but use the smallest number of repetitions that still leads to a successful match”. Now let’s try the lazy version of example (32):

33. a.*?b The shortest string starting with a and ending with b

If we apply this to the same string “aabab” it will first match “aab” and then “ab”.

*?

Repeat any number of times, but as few as possible

+?

Repeat one or more times, but as few as possible

??

Repeat zero or one time, but as few as possible

{n,m}?

Repeat at least n, but no more than m times, but as few as possible

{n,}?

Repeat at least n times, but as few as possible

Table 5. Lazy quantifiers

What did we leave out?

I’ve described a rich set of elements with which to begin building regular expressions; but I left out a few things that are summarized in the following table. Many of these are illustrated with additional examples in the project file. The example number is shown in the left-hand column of this table.

#

Syntax

Description

\a

Bell character

\b

Normally a word boundary, but within a character class it means backspace

\t

Tab

34

\r

Carriage return

\v

Vertical tab

\f

Form feed

35

\n

New line

\e

Escape

36

\nnn

Character whose ASCII octal code is nnn

37

\xnn

Character whose hexadecimal code is nn

38

\unnnn

Character whose Unicode is nnnn

39

\cN

Control N character, for example carriage return (Ctrl-M) is \cM

40

\A

Beginning of a string (like ^ but does not depend on the multiline option)

41

\Z

End of string or before \n at end of string (ignores multiline)

\z

End of string (ignores multiline)

42

\G

Beginning of the current search

43

\p{name}

Any character from the Unicode class named name, for example \p{IsGreek}

(?>exp)

Greedy subexpression, also known as a non-backtracking subexpression. This is matched only once and then does not participate in backtracking.

44

(?<x>-<y>exp) or (?-<y>exp)

Balancing group. This is complicated but powerful. It allows named capture groups to be manipulated on a push down/pop up stack and can be used, for example, to search for matching parentheses, which is otherwise not possible with regular expressions. See the example in the project file.

45

(?im-nsx:exp)

Change the regular expression options for the subexpression exp

46

(?im-nsx)

Change the regular expression options for the rest of the enclosing group

(?(exp)yes|no)

The subexpression exp is treated as a zero-width positive lookahead. If it matches at this point, the subexpression yes becomes the next match, otherwise no is used.

(?(exp)yes)

Same as above but with an empty no expression

(?(name)yes|no)

This is the same syntax as the preceding case. If name is a valid group name, the yes expression is matched if the named group had a successful match, otherwise the no expression is matched.

47

(?(name)yes)

Same as above but with an empty no expression

Table 6. Everything we left out. The left-hand column shows the number of an example in the project file that illustrates this construct.

Article based off the work of Jim Hollenhorst @ http://www.codeproject.com/KB/dotnet/regextutorial.aspx Follow up pieces will be created to this regex piece.

Advertisements

6 Responses to ".NET Regular Expressions"

What a thorough article on regular expressions! I learned a few tricks I hadn’t seen before.

very interesting, but I don’t agree with you
Idetrorce

The article was great!

But, i’m confused by ldetrorce.

What kind of a person would say they don’t agree with a list of facts? If something is clearly wrong (i.e. doesn’t function as the article states) then point it out. But, you can’t “disagree”.

Or, if there is some small matter of opinion and you have a useful opposing viewpoint, then post your thoughts instead of an empty blanket statement.

Great article…totally demystifies RegEx.
Not sure what Idetrorce disagrees with.

Thanks for the article!

I have an application in wich i am usin regular expression for text box so it should not accept text like
it is done but problem is it is not accepting sign in anywhere in the text box
e.g x<y
then it gives error
i am using reg expression as " ^[^]+\w*$”

Sounds like a great article thank you Shane so much for writing it. It takes a lot of time and energy to put something like that together.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Subscribe to this blog now!

Top Clicks

  • None
August 2007
M T W T F S S
« Jul   Sep »
 12345
6789101112
13141516171819
20212223242526
2728293031  
%d bloggers like this: