How do I use grep -Po regex options to match across multiple lines without being greedy?

Question

I would like to capture only successful nmap scan results and exclude results that did not return useful information. I've listed my desired grep output below that I want.

I tried using (?s) to enable DOTALL to make . include line breaks so that I can match/capture across multiple lines, but the problem is that it appears to disable the use of \n which I want to use as part of my pattern.

I'm trying to use a lookahead but I know the .* is greedy and I think it's matching the longest string which is basically the entire file. I want it to use the shortest string instead.

How can I dynamically capture successful nmap scan results in the following text file using Grep's -Po regex options?

desired output:

Nmap scan report for 10.11.1.72                       
Host is up (0.028s latency).                          
                                                      
PORT    STATE SERVICE                                 
111/tcp open  rpcbind                                 
| nfs-ls: Volume /home                                
|   access: Read Lookup NoModify NoExtend NoDelete NoExecute                                                
| PERMISSION  UID   GID   SIZE  TIME                 FILENAME                                               
| drwxr-xr-x  0     0     4096  2015-09-17T13:21:59  .
| drwxr-xr-x  0     0     4096  2015-01-07T10:56:34  ..                                                     
| drwxr-xr-x  1013  1013  4096  2015-09-17T13:21:47  jenny                                                  
| drwxr-xr-x  1012  1012  4096  2015-09-17T13:21:40  joe45                                                  
| drwxr-xr-x  1011  1011  4096  2015-09-17T13:21:52  john                                                   
| drwxr-xr-x  1014  1014  4096  2019-10-27T23:48:51  marcus                                                 
| drwxr-x---  0     1010  4096  2015-01-08T16:01:31  ryuu                                                   
|_                                                    
| nfs-showmount:                                      
|_  /home 10.11.0.0/255.255.0.0                       
| nfs-statfs:                                         
|   Filesystem  1K-blocks  Used       Available  Use%  Maxfilesize  Maxlink                                 
|_  /home       7223800.0  2059608.0  4797244.0  31%   8.0T         32000

Here is my current command that I'm starting with:

grep -Poz '(?s)\d+\.\d+\.\d+\.\d+.*Nmap' test2

test2 file:

### SCAN RESULTS ###

Nmap scan report for 10.11.1.39
Host is up (0.041s latency).

PORT    STATE    SERVICE
111/tcp filtered rpcbind

Nmap scan report for 10.11.1.44
Host is up (0.043s latency).

PORT    STATE  SERVICE
111/tcp closed rpcbind

Nmap scan report for 10.11.1.50
Host is up (0.043s latency).

PORT    STATE    SERVICE
111/tcp filtered rpcbind

Nmap scan report for 10.11.1.71
Host is up (0.040s latency).

PORT    STATE  SERVICE
111/tcp closed rpcbind

Nmap scan report for 10.11.1.72
Host is up (0.040s latency).

PORT    STATE SERVICE
111/tcp open  rpcbind
| nfs-ls: Volume /home
|   access: Read Lookup NoModify NoExtend NoDelete NoExecute
| PERMISSION  UID   GID   SIZE  TIME                 FILENAME
| drwxr-xr-x  0     0     4096  2015-09-17T13:21:59  .
| drwxr-xr-x  0     0     4096  2015-01-07T10:56:34  ..
| drwxr-xr-x  1013  1013  4096  2015-09-17T13:21:47  jenny
| drwxr-xr-x  1012  1012  4096  2015-09-17T13:21:40  joe45
| drwxr-xr-x  1011  1011  4096  2015-09-17T13:21:52  john
| drwxr-xr-x  1014  1014  4096  2019-10-27T23:48:51  marcus
| drwxr-x---  0     1010  4096  2015-01-08T16:01:31  ryuu
|_
| nfs-showmount: 
|_  /home 10.11.0.0/255.255.0.0
| nfs-statfs: 
|   Filesystem  1K-blocks  Used       Available  Use%  Maxfilesize  Maxlink
|_  /home       7223800.0  2068516.0  4788336.0  31%   8.0T         32000

Nmap scan report for 10.11.1.73
Host is up (0.041s latency).

PORT    STATE    SERVICE
111/tcp filtered rpcbind

Nmap scan report for 10.11.1.75
Host is up (0.041s latency).

PORT    STATE    SERVICE
111/tcp filtered rpcbind

Nmap scan report for 10.11.1.79
Host is up (0.041s latency).

PORT    STATE    SERVICE
111/tcp filtered rpcbind

Nmap scan report for 10.11.1.101
Host is up (0.041s latency).

PORT    STATE  SERVICE
111/tcp closed rpcbind

Thanks @Barmar I tried using cat test2 | grep -Poz '(?s)\d+\.\d+\.\d+\.\d+.*?Nmap' as suggested but still produces same result :( — spookymonkey, Aug 18 '21 at 21:45
I tried it and it seemed to work correctly. The output didn't include `scan report for` between each match. — Barmar, Aug 18 '21 at 21:48
If you don't want `Nmap` included at the end of each result, use a lookahead: `.*?(?=Nmap)` — Barmar, Aug 18 '21 at 21:49

score 0 · Answer 1 · answered Aug 18 '21 at 21:50

0

Use a non-greedy quantifier followed by a lookahead.

grep -Poz '(?s)\d+\.\d+\.\d+\.\d+.*?(?=Nmap)' test2

answered Aug 18 '21 at 21:50

Barmar

741,623
53
500
612

1

Sorry @Barmar, this didn't work for me. It just outputs the entire file except it starts at the first ip address. I just want to grab the successful results such as 10.11.1.72 in the file. I just figured out how to filter it after playing around with it and I did use a lookahead. Thanks for the suggestion for the lookahead! – spookymonkey Aug 19 '21 at 05:37
Can you edit the question and show the desired output? – Barmar Aug 19 '21 at 14:02
I updated the question to be more clear and I also included desired output. sorry for the confusion! – spookymonkey Aug 19 '21 at 15:32

spookymonkey · Answer 2 · 2021-08-19T15:17:26.290

Finally figured out how to do this, probably not the prettiest way of doing it but it works...

command:

grep -Poz 'Nmap scan report.+\nHost is up.+\n\nPORT.+\n\d+.+\n\|(.|\n)+?(?=\n\n)' test2

output:

Nmap scan report for 10.11.1.72                                                                                   
Host is up (0.041s latency).                                                                                      
                                                                                                                  
PORT    STATE SERVICE                                                                                             
111/tcp open  rpcbind                                                                                             
| nfs-ls: Volume /home                                                                                            
|   access: Read Lookup NoModify NoExtend NoDelete NoExecute                                                      
| PERMISSION  UID   GID   SIZE  TIME                 FILENAME                                                     
| drwxr-xr-x  0     0     4096  2015-09-17T13:21:59  .                                                            
| drwxr-xr-x  0     0     4096  2015-01-07T10:56:34  ..                                                           
| drwxr-xr-x  1013  1013  4096  2015-09-17T13:21:47  jenny                                                        
| drwxr-xr-x  1012  1012  4096  2015-09-17T13:21:40  joe45                                                        
| drwxr-xr-x  1011  1011  4096  2015-09-17T13:21:52  john                                                         
| drwxr-xr-x  1014  1014  4096  2019-10-27T23:48:51  marcus                                                       
| drwxr-x---  0     1010  4096  2015-01-08T16:01:31  ryuu                                                         
|_                                                                                                                
| nfs-showmount:                                                                                                  
|_  /home 10.11.0.0/255.255.0.0                                                                                   
| nfs-statfs:                                                                                                     
|   Filesystem  1K-blocks  Used       Available  Use%  Maxfilesize  Maxlink                                       
|_  /home       7223800.0  2059600.0  4797252.0  31%   8.0T         32000

notes:

had to specify unique filter for first 5 lines to exclude unsuccessful scans
it's important to use -z as this allows for matching \n
it was necessary to use OR expression (.|\n)*? to match all text across multiple lines
used a lookahead (?=\n\n) to specify end of match
make sure to use +? to make modifier non-greedy so that it matches the shortest string instead of longest string

You'll want to get rid of the [useless use of `cat`](https://stackoverflow.com/questions/11710552/useless-use-of-cat) — tripleee, Aug 19 '21 at 08:50
There's nothing in the question that suggests that you only want to match `Host is up` reports. — Barmar, Aug 19 '21 at 14:04

How do I use grep -Po regex options to match across multiple lines without being greedy?

2 Answers2