I was asked recently to check if someone had been accessing a web service outside the office and at first I thought that would be quite simple.

I would just go and check the access logs on the server and list any IP addresses that did not match the office IP address.

To do this I would use grep.

What is grep?

Grep stands for global regular expression print and is a command-line tool for searching files that match a pattern.

I love the ease at which you can use grep to search through log files on servers with ease.

A simple example of grep can be found below where we search for a domain name in an access log:

$ grep "ad-nav.co.uk" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...

This would then list all lines that contain “ad-nav.co.uk” in the access.log file.

Although this is a very basic use of grep I needed to do more with it.

I needed to check access to a specific site where the request was not from the IP address of the office or my home address.

I needed to build a query or pattern with the AND, OR and NOT operators.

Using AND, OR and NOT operators in grep

Simply put grep has no AND, OR and NOT operators. However, you can still achieve what you need to.

OR operator

OR can be achieved many ways in grep.

You can use a pipe like you would in regex:

$ grep "ad-nav.co.uk\|adamstacey.co.uk" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
adamstacey.co.uk:80 196.0.0.1 - [10/May/2019:14:18:32 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

You can use the grep option -E for extended regex.

$ grep -E "ad-nav.co.uk|adamstacey.co.uk" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
adamstacey.co.uk:80 196.0.0.1 - [10/May/2019:14:18:32 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

There is also a shortcut command that works like the option -E called egrep that does exactly the same thing without having to specify the option.

$ egrep "ad-nav.co.uk|adamstacey.co.uk" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
adamstacey.co.uk:80 196.0.0.1 - [10/May/2019:14:18:32 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

AND operator

To achieve AND in grep, we need to use the -E option and some regex magic.

$ grep -E "ad-nav.co.uk.*10/May/2019" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

It is important to note that the above searches for the patterns in order, so will only match the first pattern followed by the second, so if we had the date before the web address that would not get matched.

To search for the web address and date in any order on a line you can do this using the pipe for multiple patterns in a regex.

$ grep -E "ad-nav.co.uk.*10/May/2019|10/May/2019.*ad-nav.co.uk" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
[10/May/2019:14:18:32 +0100] - ad-nav.co.uk:80 196.0.0.1...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

Another way to achieve AND and in my opinion the easiest way, especially if you are not familiar with regex is to chain the grep commands.

$ grep "ad-nav.co.uk" access.log | grep "10/May/2019"
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

What we are effectively saying is do my first grep and then in the results of that grep do another grep and show me the end results.

NOT operator

The NOT operator in grep is achieved by using the inverted option -v.

$ grep -v "adamstacey.co.uk" access.log
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:13:15:41 +0100]...
ad-nav.co.uk:80 196.0.0.1 - [10/May/2019:14:19:24 +0100]...

References

Photo credit

unsplash-logoRoel Dierckens

Adam Stacey

I am Adam Stacey, the guy behind AdNav! I setup AdNav as a way to write up any technical challenges, how I overcame them, opinions on tech and much rambling. I try and cut through any technical jargon to make it friendly and easy to understand.

View all posts

Add comment

Your email address will not be published.

Adam Stacey

I am Adam Stacey, the guy behind AdNav! I setup AdNav as a way to write up any technical challenges, how I overcame them, opinions on tech and much rambling. I try and cut through any technical jargon to make it friendly and easy to understand.