-3

So I have to following bash code:

for i in `cat list1.txt`; do
  cat list2.txt |awk '/$i/{flag=1;next}/Flag2/{flag=0}flag'
done

Of course that the $i doesn't work because it has to be properly passed from bash to AWK, problem is: I tried multiple things, with -v and etc, but it didn't work. Thoughts?

Marcus
  • 3
  • 4
  • 1
    Please [edit] your question to show us a sample of your two input files and the corresponding desired output. It looks like you should be reading both files using awk, rather than attempting to use a shell loop. – Tom Fenech Jan 15 '19 at 17:48
  • It's unclear what you're trying to accomplish. – TrebledJ Jan 15 '19 at 17:48
  • Single quotes make `'$i'` a literal dollar sign followed by an i. – Paul Hodges Jan 15 '19 at 17:50
  • 1
    [edit] your question to include concise, testable sample input and expected output so we can help you come up with the right solution. There's an excellent chance you don't need the shell loop at all but we can't say for sure without seeing what it is you're trying to do (as opposed to how you're trying to do it). – Ed Morton Jan 15 '19 at 18:07

1 Answers1

4

First, c.f. this page to explain why not to use 

for i in `cat list1.txt`

...ever.

Second, this for why not to use 

cat list2.txt | awk ...

Sorry to harp. Now...try

while read -r val || [[ -n "$val" ]] 
do awk "/$val/ { flag=1; next } /Flag2/ { flag=0 } flag" list2.txt
done < list1.txt

awk in double-quotes...not ideal. Or, as Charles suggests, use -v (always listen to Charles & Ed...)

while read -r val || [[ -n "$val" ]] 
do awk -v i="$val" '
     $0 ~ i  { flag=1; next } 
     /Flag2/ { flag=0; }
     flag
   ' list2.txt
done < list1.txt

Still waiting for file samples. Please give us a peek at the format of these files so I can actually run a valid test.

Note the || [[ -n "$val" ]] is only needed if there's a chance the last record won't have a newline.

Paul Hodges
  • 13,382
  • 1
  • 17
  • 36
  • Should probably suggest `awk -v` for passing shell variables to awk safely. I assume the OP wants to treat `$val` as a regex, which will look more like `awk -v val="$val" '$0 ~ val { ... }'` (not tested, YMMV, but I'm 98% sure we *do* have Q&A entries with duplicative, tested answers to essentially that effect). – Charles Duffy Jan 15 '19 at 17:52
  • The "useless use of cat" cargo cult is a very strange thing. It honestly has little to no merit, but its followers must think it's incredibly urgent because they bring it up every chance they can. – Swiss Jan 15 '19 at 17:56
  • It's a low-consequence bad habit that's pretty easy to avoid, thus encouraging actual thought be put into design and structure. Is it horrible? Not usually. Do I write better code because I try to avoid it? Generally, yes. – Paul Hodges Jan 15 '19 at 17:57
  • 1
    Using cat to open a filehandle at the start of a pipeline chain while building that pipeline piece by piece is much nicer in practice than having to move the filename around constantly while you're adding pieces. Not using cat saves you a single PID from a pool of tens of thousands of PIDs. – Swiss Jan 15 '19 at 17:59
  • Sometimes it's worthwhile. If you thought it through and made an informed decision, the argument has served its purpose. – Paul Hodges Jan 15 '19 at 18:04
  • 1
    `/**i**/` will not work. You want something like `$0 ~ i` but idk what those `*`s are for. – Ed Morton Jan 15 '19 at 18:05
  • maybe back-quoted? meh. editing it out till better minds wade in. Still learning `awk`. – Paul Hodges Jan 15 '19 at 18:07
  • Again, hoping OP will chime in with file formats and more info, lol – Paul Hodges Jan 15 '19 at 18:07
  • Paul - `i` is an awk variable. `/i/` is a repexp for the character `i` and in the code `/i/` is shorthand for `$0 ~ /i/`. What you want instead if `$0 ~ i` which is searching for a regexp match on the contents of the variable `i` rather than looking for the character `i`. I have no idea why you're putting any `*`s in there at all. What you proibably really want though is `index($0,i)` for a string rather than regexp comparison but that will do a partial match. idk... until the OP tells us more about what they're trying to do there's just too many possibilities to suggest a solution. – Ed Morton Jan 15 '19 at 18:11
  • I typed wrongly the **$i** because of the Bold option from Stackoverflow. Therefore the ** is indeed, not needed. The While thing worked for me (from PH). My struggle is to pass, safely, the var from Bash to AWK, and using the -v option didn't work out for me, although I've read quite few examples from Google. I'm aware of the "cat" thing, I was just pasting from my shell, and should have used the proper educated way. – Marcus Jan 15 '19 at 18:51
  • wrt `My struggle is to pass, safely, the var from Bash to AWK, and using the -v option didn't work out for me` - if you tell us in what way it `didn't work out` for you then we can help but if you decide to give up and just double quote the whole script then you're opening yourself up to a LOT of problems. Best advice from someone with 25+ years of shell and awk experience - don't do it... – Ed Morton Jan 15 '19 at 19:06
  • Now I get it. OP typed `/**$i**/` because he wanted it emphasized, but in code it kept it as a literal. Still wouldn't have worked because `/i/` is looking for the letter i instead of seeing a variable. Trying another edit - input always welcome. – Paul Hodges Jan 15 '19 at 20:35
  • 3
    @Swiss, using `cat` means that your input comes in on a FIFO, not a seekable file handle. For tools like `sort`, `tail`, `wc -c` or the like that can be a **huge** difference -- think about reading through all of a 5GB file when with a real handle you could just jump straight to the end, or reading it front-to-back to sort instead of being able to split into processes/threads that handle subsets of the file in parallel and merge their results together. ` – Charles Duffy Jan 15 '19 at 20:48
  • 2
    @PaulHodges wrt your comment `always listen to Charles & Ed` - if you stop by Chicago tomorrow you can literally listen to both of us live and in person as we work together :-). – Ed Morton Jan 15 '19 at 21:52
  • @Charles Duffy File seek makes sense if the command supports a file argument and opens the file handle itself, but does that apply for IO redirection? How would something like `sort` know that `FD0` is now seekable and not just a FIFO? – Swiss Jan 15 '19 at 23:35
  • 1
    @Swiss, the easiest approach is to *try*, which is exactly what `tail` &c. do -- they try to do the efficient thing, and fall back to just reading from the beginning if they get an error. If you pass in a FIFO, you guarantee that seek and tell will fail, prompting fallback behavior from anything that might otherwise benefit from them. See the `-ESPIPE` error documented in `man 2 lseek`. – Charles Duffy Jan 15 '19 at 23:36
  • @Ed, a four-hour round trip from St Louis would make for a very long lunch, but I might just do that one day to buy you guys a pie and a beer if you know a really good deep-dish place. ;) – Paul Hodges Jan 16 '19 at 14:28
  • @EdMorton I will be honest with you, although I can have my regex sentences in place - I know, what I can call "sufficient", to process what I need to. Of course, as you may have noticed, I don't Excel in AWK. I've read among of bunch of SO QAs, similar as this one: https://stackoverflow.com/questions/19075671/how-do-i-use-shell-variables-in-an-awk-script But I was unable to find a way to pass along the variable, I've tried like these i.e.: awk -v var=$i '/var/{flag=1;next}/Server/{flag=0}flag' file1.txt awk -v awkvar="$i" '{gsub /awkvar/{flag=1;next}/Server/{flag=0}flag}' file1.txt – Marcus Jan 16 '19 at 18:01
  • 1
    I suspect you're passing it fine but seeking it wrongly - rather than `/var/`, use `$0 ~ var` as suggested above. – Paul Hodges Jan 16 '19 at 19:27
  • @Marcus your code has exactly the problem I described in [my earlier comment to Paul](https://stackoverflow.com/questions/54204172/for-i-in-cat-file-read-variable-from-awk-regex/54204326?noredirect=1#comment95235937_54204326) – Ed Morton Jan 16 '19 at 23:54