Regular Expressions
Contents
13.3. Regular Expressions¶
Regular expressions (or regex for short) are special patterns that we use
to extract parts of strings.
Think about the format of a Social Security Number (SSN) like 134-42-2012
.
To describe this format, we
might say that SSNs consist of three digits, then a dash, two digits, another
dash, then four digits. Regexes let us capture this pattern in code. For
instance, a simple regex for SSNs could look like:
...-..-....
Although we haven’t yet introduced the syntax and special characters associated with regular expressions, or even how to “parse” them, you can probably make sense of the pattern above. It follows closely the written description of a SSN that we just gave. The syntax of regular expressions is fortunately quite simple to learn; we introduce nearly all of the syntax in this section alone.
As we introduce the concepts, we tackle some of the examples described in an
earlier section, and show how to carry out the tasks with regular expressions.
Almost all programming languages have a library to match patterns using regular
expressions, making regular expressions useful regardless of the specific
programming language. We use some of the common methods available in the Python
built-in re
module to accomplish the tasks from the examples. These methods
are summarized in a table at the end of this section, where the basic usage and
return value are briefly described. Since we only cover a few of the most
commonly used methods, you may find it useful to consult the official
documentation on the re
module as well.
Regular expressions are based on searching a string one character (aka literal) at a time for a pattern. We call this notion concatenation of literals.
13.3.1. Concatenation of Literals¶
Concatenation is best explained with a basic example. Suppose we are looking
for the pattern cat
in the string Scatter!
. Here’s how a computer matches
literal patterns like cat
:
Begin with the first character in the string (
S
).Check whether it matches the first character in the pattern (
c
).If there isn’t a match, move onto the next character of the string.
If there is a match, check the rest of the pattern (
a
, thent
).If the entire pattern matches, report that the pattern was found.
Figure 13.1 contains a diagram of
the idea behind this search through the string
one character at a time. The pattern “cat” is found within the word
Scatter!
in
positions 1-3 in the string. Once you get the hang of this process, you can
move on to the richer set of patterns; they all follow from this basic
paradigm.
Fig. 13.1 To match literal patterns, the computer moves along the string and checks whether the entire pattern is matched.¶
Note
In the example above we observe that regular expressions can match patterns that appear anywhere in the input string. In Python, this behavior differs depending on the method used to match the regex—some methods only return a match if the regex appears at the start of the string; some methods return a match anywhere in the string.
Character Classes
We can make patterns more flexible by using a character class
(also known as a character set), which
lets us specify a collection of equivalent characters to match. This allows us
to create more relaxed matches. To create a character class, wrap the set of
desired characters in brackets [ ]
.
For example, the pattern [0123456789]
means “match any literal
within the brackets”—in this case, any single digit.
Then, the following regular expression matches three digits.
[0123456789][0123456789][0123456789]
This is such a commonly used character class that
there is a shorthand notation for the range of digits, [0-9]
. Character
classes allow us to create a regex for SSNs:
[0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]
Two other ranges that are commonly used in character classes are [a-z]
for
lowercase and [A-Z]
for uppercase letters. We can combine ranges with other
equivalent characters and use partial ranges. For example [a-cX-Z27]
is
equivalent to the character class, [abcXYZ27]
.
Let’s return to our original pattern cat
and modify it to include two character classes:
c[oa][td]
This pattern matches cat
, but it also matches cot
, cad
, and cod
:
Regex: c[oa][td]
Text: The cat eats cod, cads, and cots, but not coats.
Matches: ↑↑↑ ↑↑↑ ↑↑↑ ↑↑↑
Negated Character Classes
A negated character class matches any character
except those between the square brackets. To create a negated character
class, place the caret symbol as the first character after the left sqaure
bracket. For example, [^0-9]
matches any character except a digit.
Wildcard Character
When we really don’t care what the literal is, we can specify this with the
period character .
. This matches any character except a newline.
Escaping Meta Characters
We have now seen several special characters, called meta characters: [
and
]
denote a character class ^
switches to a negated character class, .
represents any character, and -
denotes a range. But, sometimes we might want
to create a pattern that matches one of these literals. When this happens, we
must escape it with a backslash. For example, we can match the literal left
bracket character using the regex \[
.
Regex: \[
Text: Today is [2022/01/01]
Matches: ↑
Next, we will show how quantifiers can help create a more compact and clear regular expression for SSNs.
13.3.2. Quantifiers¶
To create a regex to match SSNs, we wrote:
[0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]
This matches 3 digits, a dash, 2 more digits, a dash, and 4 more digits.
Quantifiers allow us to match multiple consecutive appearances of a literal. We
specify the number of repetitions by placing the number in curly braces { }
.
# We add the `r` character before the quotes to make a raw string, which makes
# regexes easier to write in Python
ssn_re = r'[0-9]{3}-[0-9]{2}-[0-9]{4}'
# Python's builtin re module has methods for matching regexes
import re
re.findall(ssn_re, 'My SSN is 382-34-3840.')
['382-34-3840']
# The pattern shouldn't match phone numbers
re.findall(ssn_re, 'My phone is 382-123-3842.')
[]
A quantifier always modifies the character or character class to its immediate left. The following table shows the complete syntax for quantifiers.
Quantifier |
Meaning |
---|---|
{m, n} |
Match the preceding character m to n times. |
{m} |
Match the preceding character exactly m times. |
{m,} |
Match the preceding character at least m times. |
{,n} |
Match the preceding character at most n times. |
Shorthand Quantifiers
Some commonly used quantifiers have a shorthand:
Symbol |
Quantifier |
Meaning |
---|---|---|
|
{0,} |
Match the preceding character 0 or more times |
|
{1,} |
Match the preceding character 1 or more times |
|
{0,1} |
Match the preceding charcter 0 or 1 times |
We use the *
character instead of {0,}
in the following examples.
Quantifiers are greedy
Quantifiers will return the longest match possible. This sometimes results in surprising behavior. Since a SSN starts and ends with a digit, we might think the following shorter regex will be a simpler approach for finding SSNs. Can you figure out what went wrong in the matching?
ssn_re = r'[0-9].+[0-9]'
re.findall(ssn_re, 'My SSN is 382-34-3842 and hers is 382-34-3333.')
['382-34-3842 and hers is 382-34-3333']
In many cases, using a more specific character class prevents these false “over” matches:
ssn_re = r'[0-9\-]+[0-9]'
re.findall(ssn_re, 'My SSN is 382-34-3842 and hers is 382-34-3333.')
['382-34-3842', '382-34-3333']
Literal concatenation and quantifiers are two of the core concepts in regular expressions. Next, we’ll introduce two more core concepts: alternation and grouping.
13.3.3. Alternation and Grouping to Create Features¶
Character classes let us match multiple options for a single literal.
We can use alternation to match multiple options for a group of literals.
For example, in the food safety example in
Chapter 9 we marked violations related to
body parts by seeing if the violation had the substring
hand
, nail
, hair
, or glove
.
We can use the |
character in a regex to specify this alteration:
body_re = r"hand|nail|hair|glove"
re.findall(body_re, "unclean hands or improper use of gloves")
['hand', 'glove']
re.findall(body_re, "Unsanitary employee garments hair or nails")
['hair', 'nail']
log_entry
'169.237.46.168 - - [26/Jan/2004:10:47:58 -0800]"GET /stat141/Winter04 HTTP/1.1" 301 328 "http://anson.ucdavis.edu/courses""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)"'
Grouping using parentheses
Every set of parentheses specifies a regex group, which allows us to extract parts of a pattern. For example, we can use groups to extract the day, month, year, and time from the web server log entry.
# This pattern matches the entire timestamp
time_re = r"\[[0-9]{2}/[a-zA-z]{3}/[0-9]{4}:[0-9:\- ]*\]"
re.findall(time_re, log_entry)
['[26/Jan/2004:10:47:58 -0800]']
# Same regex, but we use parens to make regex groups...
time_re = r"\[([0-9]{2})/([a-zA-z]{3})/([0-9]{4}):([0-9:\- ]*)\]"
# ...which tells findall() to split up the match into its groups
re.findall(time_re, log_entry)
[('26', 'Jan', '2004', '10:47:58 -0800')]
As we can see, re.findall
returns a list of tuples containing the individual
components of the date and time of the web log.
We have introduced a lot of terminology, so in the next section we’ll bring it all together into a set of tables for easy reference.
13.3.4. Reference Tables¶
We conclude this section with a few tables that summarize order of
operation, meta characters, and shorthands for character classes.
We also provide tables summarizing the handful of methods in the re
Python library that we have used in this section.
The four basic operations for regular expressions, concatenation, quantifying, alternation, and grouping have an order of precedence, which we make explicit in the table below.
Operation |
Order |
Example |
Matches |
---|---|---|---|
concatenation |
3 |
|
|
alternation |
4 |
|
|
quantifying |
2 |
|
|
grouping |
1 |
c(at)? |
|
The following table provides a list of the meta characters introduced in this section, plus a few more. The column labeled “Doesn’t Match” gives examples of strings that the example regexes don’t match.
Char |
Description |
Example |
Matches |
Doesn’t Match |
---|---|---|---|---|
. |
Any character except \n |
|
abc |
ab |
[ ] |
Any character inside brackets |
|
car |
jar |
[^ ] |
Any character not inside brackets |
|
car |
bar |
* |
≥ 0 or more of previous symbol, shorthand for {0,} |
|
bbark |
dark |
+ |
≥ 1 or more of previous symbol,shorthand for {1,} |
|
bbpark |
dark |
? |
0 or 1 of previous symbol, shorthand for {0,1} |
|
she |
the |
{n} |
Exactly n of previous symbol |
|
hellooo |
hello |
| |
Pattern before or after bar |
|
we |
e |
\ |
Escape next character |
|
[hi] |
hi |
^ |
Beginning of line |
|
ark two |
dark |
$ |
End of line |
|
noahs ark |
noahs arks |
Additionally, we provide a table of shorthands for some commonly used character
sets. These shorthands don’t need [ ]
.
Description |
Bracket Form |
Shorthand |
---|---|---|
Alphanumeric character |
|
|
Not an alphanumeric character |
|
|
Digit |
|
|
Not a digit |
|
|
Whitespace |
|
|
Not whitespace |
|
|
We used the following methods in re
in this section. The names of the methods
are indicative of the functionality they perform: search or
match a pattern in a
string; find all cases of a pattern in a string; substitute all occurrences
of a pattern with a substring, and split a string into pieces at the pattern.
Each, requires a pattern and string to be specified, and some have
extra arguments.
The table below provides the format of the method usage and a
description of the return value.
Method |
Return value |
---|---|
|
truthy match object if the pattern is found anywhere in the string, otherwise |
|
truthy match object if the pattern is found at the beginning of the string, otherwise |
|
list of all matches of |
|
string where all occurrences of |
|
list of the pieces of |
Regex and pandas
As seen in the previous section, pandas
Series objects have a .str
property
that supports string manipulation using Python string methods. Conveniently,
the .str
property also supports some functions from the re
module. The
table below shows the analogous functionality from the above table of the re
methods. Each requires a pattern.
See the pandas
docs for a complete list of
string methods.
Method |
Return value |
---|---|
|
Series of booleans indicating whether the |
|
list of all matches of |
|
Series with all matching occurrences of |
|
Series of lists of strings around given |
Regular expressions are a powerful tool, but are somewhat notorious for being difficult to read and debug. We close with some advice for regexes.
Develop your regular expression on simple test strings to see what the pattern matches.
If a pattern matches nothing, try weakening it by dropping part of the pattern. Then tighten it incrementally to see how the matching evolves. (Online regex checking tools can be very helpful here).
Use raw strings whenever possible for cleaner patterns, especially when a pattern includes a backslash.
When you have lots of long strings, consider using compiled patterns because they can be faster to match (see
compile()
in there
library).
While powerful, regular expressions are terrible at certain types of problems. Don’t use them to:
Parse hierarchical structures such as JSON or HTML; use a parser instead.
Search for complex properties, like palindromes and balanced parentheses.
Validating a complex feature, such as a valid email address.
In the next section, we carry out an example text analysis. We’ll clean the data using regular expressions and string manipulation, convert the text into quantitative data, and analyze the text via these derived quantities.