csv upload function works, but error if I want to fill it with content

Question

I wrote a function that creates a report for me and uploads it to S3. However, I have problems filling the CSV file with content. Here you can see the code:

import boto3
import re
import csv

def lambda_handler(event,context):
    client = boto3.client('ce')
    response = client.get_cost_and_usage(
        TimePeriod={
            'Start': "2019-05-15",
            'End':  "2019-07-05"
        },
        Granularity='MONTHLY',
        Metrics=['BlendedCost'],
        GroupBy=[
            {
                'Type': 'TAG',
                'Key': 'Project'
            },
        ]
    )

    csv_testerinho = csv.writer(open("/tmp/csv_testerinho.csv", "w+"))
    csv_testerinho.writerow(["Account Name", "Month", "Cost"])

    #writing rows in csv
    for detail in response:
       csv_testerinho.writerow([response['Start'],
                                response['End'],
                                response['BlendedCost']
                                ])

    client = boto3.client('s3')
    client.upload_file('/tmp/csv_testerinho.csv', 'bucket_name','final_testerinho.csv')

When I execute the code I get the following error:

Response:
{
  "errorMessage": "'Start'",
  "errorType": "KeyError",
  "stackTrace":
}

What would I have to do to fill the CSV with the information I get through the API?

response is the client.get_cost_and_usage with the filters of the Information I want — mango5k, Aug 12 '19 at 15:30
Sure, but what is the client.get_cost_and_usage with the filters of the Information you want? — Stop harming Monica, Aug 12 '19 at 15:40
This command gives the information over API from AWS Cost Explorer. — mango5k, Aug 12 '19 at 15:42

jarmod · Accepted Answer · 2019-08-12T20:01:08.340

0

You should retrieve the start and end times using the following where n is an index into the ResultsByTime list:

response['ResultsByTime'][n]['TimePeriod']['Start']
response['ResultsByTime'][n]['TimePeriod'['End']

Or you could write:

for result in response['ResultsByTime']:
    start = result['TimePeriod']['Start']
    end = result['TimePeriod']['End']

So, applying this to your code:

with open("/tmp/csv_testerinho.csv", "w+") as fd:
    csv_testerinho = csv.writer(fd)
    csv_testerinho.writerow(["Start", "End", "Cost"])

    for result in response['ResultsByTime']:
        start = result['TimePeriod']['Start']
        end = result['TimePeriod']['End']
        total_cost = 0.0
        for group in result['Groups']:
            cost = group['Metrics']['BlendedCost']['Amount']
            total_cost += float(cost)
        csv_testerinho.writerow([
            start,
            end,
            total_cost
        ])

You will need to double-check how I have retrieved and aggregated the blended cost because it's not completely trivial to work out how to do this. If you print out the response dict object you get back, you will see what it contains.

BTW you seem to be writing column headers Account Name, Month and Cost but writing rows containing start time, end time, and cost which looks like a problem.

See the get_cost_and_usage reference for more details of the response.

edited Aug 12 '19 at 20:01

answered Aug 12 '19 at 16:05

jarmod

71,565
16
115
122

I don't know if I misunderstand you, but where's the write function? If I were to execute the code with your loop, the CSV would still be empty after all. – mango5k Aug 12 '19 at 17:36
Your code is incorrectly trying to retrieve values from `response`. For example you have written `response['Start']` which is what is causing the `KeyError` (indicating that there is no `Start` key in the response). If you read the usage reference you will see that `response` is a dict, it contains a list named `ResultsByTime`, and each element of that list contains a `TimePeriod` dict which itself, finally, contains `Start` and `End`. I have shown you how to correctly access start/end time values in the response. You can then modify your code as needed. – jarmod Aug 12 '19 at 18:13
Okay, I think I know what you mean now. But shouldn't the code at least write the columns "Account Name", "Month" and "Cost" in the CSV without the values? Even that doesn't happen (line 22) – mango5k Aug 12 '19 at 18:36
Your code fails and exits before it gets to the point that it uploads the CSV to S3. – jarmod Aug 12 '19 at 18:37
But then I'm surprised that an empty CSV is uploaded to S3 when I comment out the part that concerns the information. With the line of code ```csv_testerinho.writerow(["Account Name", "Month", "Cost"])``` at least the name should be there?! – mango5k Aug 12 '19 at 18:42
You have not closed the CSV file and its contents have not been flushed to disk, so the file on disk at the time of upload is empty. There are two ways to resolve this. See https://stackoverflow.com/questions/3347775/csv-writer-not-closing-file – jarmod Aug 12 '19 at 18:51
Nice, I've been able to fix that, thank you very much. I think I lack a lot of general knowledge. Now I still have to manage to transfer the information and I am happy :D – mango5k Aug 12 '19 at 19:19
If the answer provided is helpful and solves the problem, please accept it as the answer so others know it's been solved. – jarmod Aug 12 '19 at 19:24
When I use your solution in my code, I get a syntax error: Syntax error in module 'lambda_function': expected an indented block related to the following line: ```start = result['TimePeriod']['Start']```. But yes, if I can get it to work, I will gladly accept your answer. – mango5k Aug 12 '19 at 19:34
Okay, I didn't say anything, the code's perfect, it's on me. But now he tells me the following error ```"errorMessage": "string indices must be integers"``` does that mean, I still have to convert strings into integers somehow? – mango5k Aug 12 '19 at 19:43
Have updated code in my response. The blended cost appears to be in `Metrics` rather than `Totals` and is grouped (hence the aggregation). – jarmod Aug 12 '19 at 20:02
Super many thanks for your time and effort! But I still don't understand the problem with the Integer and how you changed it, can you explain it for understanding? – mango5k Aug 12 '19 at 20:09
The problem causing "TypeError: string indices must be integers" was because I had accidentally mis-typed the for loop second time around as `for result in response` when it should have been `for result in response['ResultsByTime']` The former returns the keys of the response dict, for example "GroupDefinitions". The next line of code was evaluating `result['TimePeriod']` which, in that case, was invalid because `result` was a string and you can only index a string using integers (for example: result[0] is "G", result[1] is "r"). – jarmod Aug 12 '19 at 20:17

csv upload function works, but error if I want to fill it with content

1 Answers1