0

I wrote shell script in AIX 7.1 and it's not executed in a proper order.

The shell script is

receive 2 parameter $param and $filename

listoffiles='ls ${param}/*.txt'
awk 'FNR-1' ${listoffiles} >> ${param}/${filename}
mv ${param}/*.txt ${param}/archive

My objective is to combine lines in ${listoffiles} to one file, excluding the header of each file. Afterwards, I'd like to move the ${listoffiles} files, including resulting ${param}/${filename} to a folder (let's say it's "archive" folder). ${filename} should refer to parameter and will give file unique name per execution call and will always have ".txt" extension.

The problem is: if there are 3 (or more) script execution in the same time, the result will be:

  1. One execution will result in a proper order

  2. Other execution will result in mv ${param}/*.txt ${param}/archive to be executed first before awk 'FNR-1' ${listoffiles} >> ${param}/${filename}

What am I doing wrong here? Or is there any way to guarantee for script to strict with it's execution step? (I've tried adding && or ; but the result stays the same)

Kioels
  • 51
  • 1
  • 11
  • 3
    Shell scripts require exquisite precision in the use of spaces. The assignment must have no spaces on either side of the equals sign. This means that the `awk` script is given no files to process. Learn to debug with `bash -x` (and/or `sh -x`). Even when you fix that, the shell will not expand or glob `$param/*.txt` when it comes from expanding `${listoffiles}`. – Jonathan Leffler Dec 12 '14 at 02:36
  • @JonathanLeffler, ...it won't do the expansion, to be sure, but why wouldn't it do the glob (if the quoting style was corrected so that the expansion could be performed)? Which is not at all to say that I'm defending that code; storing glob expressions in scalar variables is of course an abomination. – Charles Duffy Dec 12 '14 at 02:41
  • @CharlesDuffy: Oh — you're right. If comments were editable for longer, I'd fix it. If the `$param` is handled outside of `$listoffiles` (e.g. `listoffiles='*.txt'; echo "$param"/$listoffiles`), then the globbing will occur. The globbing will occur if you have a directory called `$param` (as opposed to the expansion of `$param`). Basically, you have to think hard about the way the shell is going to handle the [expansions](http://www.gnu.org/software/bash/manual/bash.html#Shell-Expansions). – Jonathan Leffler Dec 12 '14 at 02:46
  • @Kioels, that's not good practice either. See http://mywiki.wooledge.org/ParsingLs – Charles Duffy Dec 12 '14 at 03:07
  • The symptom you describe is _exactly_ what one would expect if `param` were the same between more than one explanation, and utterly inexplicable if it is not. (It's not that any given instance is running commands out-of-order, but rather that instance A is renaming files while instance B is creating them). – Charles Duffy Dec 12 '14 at 03:54
  • If you use a tool such as `sysdig` to monitor process invocations, I _promise_ you that you'll see something along these lines occur. – Charles Duffy Dec 12 '14 at 03:54

2 Answers2

2

Don't hardcode a name such as combine.txt; instead, use mktemp to create a unique temporary file for your instance:

tempfile=$(mktemp "$param/combine.XXXXXX")
awk 'FNR-1' "$param"/*.txt >"$tempfile"
mv "$tempfile" "$param/archive"

Using unique filenames will allow concurrent instances of your script to operate without interacting with each other.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • sorry, i've edited the question. There should be another parameter giving a filename. The filename will always have ".txt" extension. – Kioels Dec 12 '14 at 03:07
  • If `param` is a directory, and you have two concurrent processes modifying that directory, how could a solution even *possibly* allow concurrent operation? Your `*.txt` isn't going to be isolated to content created or edited by the current shell as a matter of course. – Charles Duffy Dec 12 '14 at 03:10
  • If what you want _is_ possible, you need to explain it better, so I can understand how or why it's possible. – Charles Duffy Dec 12 '14 at 03:11
  • $param should be unique per execution. Let's say first process $param refer to '/dat/firstfolder' and second process $param refer to '/dat/secondfolder', the problem is why does the second process won't do the script in sequence. – Kioels Dec 12 '14 at 03:16
  • Ahh. With that clarification, the answer is simple: The script as written *does* run in order, always. Thus, your question is based on a misdiagnosis; the actual issue is elsewhere. Have you followed the advice Jonathan gave you and reproduced the issue with `bash -x`? – Charles Duffy Dec 12 '14 at 03:37
0

Since you collect the list of files first, and then do stuff to it, by the time it gets around to doing stuff to it, the file may have been moved by another process.

You may want to try locking the files - see How to prevent a script from running simultaneously? - specify a filename that relates to the file you are about to process, and give it a short timeout. it will fail if another process has done the same. You can skip this one, because another thread is processing it.

Also make sure (as @Charles Duffy correctly points out) that each process writes to a different file.

Community
  • 1
  • 1
AMADANON Inc.
  • 5,753
  • 21
  • 31
  • in my case, the execution of the script will always need a unique $param per execution. there should be no problem regarding locking files since every process is unique – Kioels Dec 12 '14 at 03:10