Argo: How to pip install libraries in yaml with a 'Script' template?

Question

Context - Basic PG docker is running with described credentials, want to perform an Argo workflow which creates a table, adds some data etc separately.

While defining argo yaml for aforementioned requirements, getting a 'no library found psycopg' (PFB)

Where can one do pip install of needed libraries? I understand i can create a docker container with this script and a libraries installation CMD before. Is there no way to install libraries to perform simple python scripts using 'script' template purely?

Reference: https://github.com/argoproj/argo-workflows/blob/master/examples/scripts-python.yaml

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: scripts-python
spec:
  entrypoint: python-script-example
  templates:
  - name: python-script-example
    steps:
    - - name: creating-emp-tbl
        template: create-emp-tbl
    - - name: print
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "{{steps.creating-emp-tbl.outputs.result}}"

  - name: create-emp-tbl
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        from psycopg2 import connect
        conn = connect(
            database="postgres",
            user="postgres",
            host="localhost",
            port=5432,
            password="mysecretpassword",
        )

        cursor = conn.cursor()

        try:
            cursor.execute(
                "CREATE TABLE EMPLOYEES (id serial PRIMARY KEY, age integer, team varchar);"
            )
            print("created")
        except:
            print("couldn't create table")

        conn.commit()
        cursor.close()
        conn.close()
  - name: print-message
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo result was: {{inputs.parameters.message}}"]

I feel like the only way to do it is building your own image — maxisam, Sep 22 '22 at 18:49

score 0 · Answer 1 · answered Feb 13 '23 at 11:24

One way you might achieve this is by programmatically installing dependencies at the start of your script, as discussed in Installing python module within code

As an example of a Script Workflow that installs a dependency, e.g., ulid:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: workflow-draft-
spec:
  entrypoint: draft-script
  templates:
 
    - name: draft-script
      steps:
      - - name: try-out-pip-install
          template: draft

    - name: draft
      script:
        image: python:alpine3.16
        command: ["python"]
        source: |
          import pip
          pip.main(['install', 'ulid'])
      
          import ulid
          print("ulid", ulid.ulid())

Argo: How to pip install libraries in yaml with a 'Script' template?

1 Answers1