run terminal command for all files in a folder

Question

I have a models folder which contains a bunch of sql files models/mart/table1.sql, models/mart/table2.sql, models/mart/table3.sql

I run this command manually on the terminal:

dbt run-operation generate_model_yaml --args '{"model_name": "table1"}'

However, instead of running it individually for each table, I want to include it in the Bitbucket pipeline. How can I modify the command such that it runs in a a loop? It should extract the tablename (filename) for all files from a specified folder (models/mart) and then run the command accordingly by replacing the model_name by the filename each time?

pipelines:
  custom: 
    dbt-run-default:
          - step:
              name: 'Compile'
              image: fishtownanalytics/dbt:1.0.0
              script:
                - cd dbt_4flow
                - dbt deps --profiles-dir ./profiles

score 0 · Answer 1 · answered Mar 16 '22 at 11:36

0

Try this Shellcheck-clean code:

#! /bin/bash -p

sql_dir=$1

shopt -s nullglob
for sql_path in "$sql_dir"/*.sql; do
    sql_name=${sql_path##*/}
    table_name=${sql_name%.sql}
    dbt run-operation generate_model_yaml --args '{"model_name": "'"${table_name}"'"}'
done

It's a program that takes the path to a directory containing '.sql' files as a command line argument.
The shopt -s nullglob prevents the "$sql_dir"/*.sql in the code expanding to garbage if there are no '.sql' files in the directory.
See Removing part of a string (BashFAQ/100 (How do I do string manipulation in bash?)) for an explanation of ${sql_path##*/} and ${sql_name%.sql}.

answered Mar 16 '22 at 11:36

pjh

6,388
2
16
17

how can I run it as a single command from my terminal? When I try, I get this: ```zsh: parse error near `do'``` – x89 Mar 16 '22 at 14:11
@x89, it's Bash code so it can't be run from a Zsh terminal. The question is tagged 'bash'. The easiest way to run it as a single command from your terminal is to put it in a file, make the file executable, and put it in a directory on your `PATH`. See [Add a bash script to path](https://stackoverflow.com/q/20054538/4154375). – pjh Mar 16 '22 at 14:23
`zsh` can be configured to understand Bash extensions, too; but obviously, if you were asking for a Bash solution, that's what you are going to get. – tripleee Mar 18 '22 at 06:27

run terminal command for all files in a folder

1 Answers1