How to use Google Data Prep API using Python

Question

Google Just launched the new API. Link is here.

I want to know what is the host in this case as they are using example.com and using the port 3005.

I am also following this article. But this does not provide example code.

Answer 1

If you open the Dataprep Console and navigate to Settings > Access Tokens you can click on Generate New Token. This will bring up the token, that you can copy to clipboard, but also the instructions on which base endpoint to use:

---

Then, clicking on a particular Recipe (see image below) will modify the browser URL to be in the form of:

https://clouddataprep.com/flows/<FLOW_ID>?recipe=<RECIPE_ID>&tab=recipe

We'll keep RECIPE_ID> so that our request body (dataprep-request.json) is something like this:

{
  "wrangledDataset": {
    "id": <RECIPE_ID>
  }
}

Then, we can call JobGroups Create:

curl https://api.clouddataprep.com/v4/jobGroups \
  -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @dataprep-request.json

The response will be similar to this output (I used python3 -m json.tool to pretty print the JSON):

{
    "sessionId": "<SESSION_ID>",
    "reason": "JobStarted",
    "jobGraph": {
        "vertices": [
            4479390,
            4479391
        ],
        "edges": [
            {
                "source": 4479390,
                "target": 4479391
            }
        ]
    },
    "id": <JOB_GROUP_ID>,
    "jobs": {
        "data": [
            {
                "id": 4479390
            },
            {
                "id": 4479391
            }
        ]
    }
}

Now, with the retrieved <JOB_GROUP_ID> we can use the JobGroups Get endpoint:

curl https://api.clouddataprep.com/v4/jobGroups/<JOB_GROUP_ID> \
  -H "Authorization: Bearer $TOKEN"

Response:

{
    "id": <JOB_GROUP_ID>,
    "name": null,
    "description": null,
    "ranfrom": "ui",
    "ranfor": "recipe",
    "status": "InProgress",
    "profilingEnabled": true,
    "runParameterReferenceDate": "2019-12-08T21:49:33.000Z",
    "createdAt": "2019-12-08T21:49:35.000Z",
    "updatedAt": "2019-12-08T21:49:36.000Z",
    "workspace": {
        "id": REDACTED
    },
    "creator": {
        "id": REDACTED
    },
    "updater": {
        "id": REDACTED
    },
    "snapshot": {
        "id": 4226057
    },
    "wrangledDataset": {
        "id": <RECIPE_ID>
    },
    "flowRun": null
}