How to fix memory leak in my application?

Question

In my GAE app I add rows to Google Spreadsheet.

taskqueue.add(url='/tabletask?u=%s' % (user_id),
retry_options=taskqueue.TaskRetryOptions(task_retry_limit=0),
                            method='GET')

class TableTaskHandler(webapp2.RequestHandler):
    def get(self):
        user_id = self.request.get('u')
        if user_id:
            try:
                tables.add_row(
                    user_id
                )
            except Exception, error_message:
                pass



def get_google_api_service(scope='https://www.googleapis.com/auth/spreadsheets', api='sheets', version='v4'):
    ''' Login to Google API with service account and get the service
    '''
    service = None
    try:
        credentials = AppAssertionCredentials(scope=scope)
        http = credentials.authorize(httplib2.Http(memcache))
        service = build(api, version, http=http)
    except Exception, error_message:
        logging.exception('Failed to get Google API service, exception happened - %s' % error_message)
    return service

def add_row(user_id, user_name, project_id, question, answer, ss_id=SPREADSHEET_ID):
    service = get_google_api_service()
    if service:
        values = [
            [
                user_id, user_name, project_id, question, answer # 'test1', 'test2'
            ],
            # Additional rows ...
        ]
        body = {
            'values': values
        }
        # https://developers.google.com/sheets/api/guides/values#appending_values
        response = service.spreadsheets().values().append(
            spreadsheetId=ss_id,
            range='A1:E1000',
            valueInputOption='RAW',
            body=body).execute()

I add many tasks with different row values.

In result I get critical errors 'Exceeded soft private limit of 128 Mb with 158 Mb' after servicing 5 requests in total.

What could be wrong here?

There are several posts here about memory issues arising from task queues. It is fairly complicated so I suggest reviewing those posts and then updating your question if needed. Here is one example: https://stackoverflow.com/questions/33036334/memory-leak-in-google-ndb-library — new name, Sep 22 '17 at 13:02
@JeffO'Neill, that question is about NDB and not task queues. And I've implemented task queues only when I faced with memory leakage (had it before task queues usage). — LA_, Sep 22 '17 at 16:14
I'm running into a similar memory leak with service.spreadsheets().values() either get or append if the range is larger than 100. Have you managed to fix this? @LA_ — Borko Kovacev, Oct 09 '17 at 21:06
@BorkoKovacev, no, I haven't fixed it - I moved to Google Fusion Tables — LA_, Oct 12 '17 at 07:30
@LA_ have you tuck with fusion tables? I am wondering because I am having the exact same issue you had. — BrainPermafrost, Jan 07 '20 at 19:20

score 1 · Answer 1 · answered Oct 04 '17 at 11:52

At first glance there’s nothing special in your code that might lead to a memory leak. I don’t think anybody can locate it unless he’s very deeply familiar with the 3rd party libraries used and their existing bugs. So I’d approach the problem as follows:

First lets find out where exactly memory is leaking and whether it’s leaking at all.

Refer to tracemalloc, memory_profiler, heapy or whatever else you’re familiar with. Most profilers available are listed here Which Python memory profiler is recommended?

Expected outcome: you clearly know where exactly the memory is leaking, up to a code line / python expression
If the problem is in a 3rd party code, try to dig deeper into its code and figure out what’s up there
Depending on p.2 outcome

a. Post another SO question like ‘why this python code excerpt leads to a memory leak’ - ideally it should be a standalone code snippet that leads to a weird behavior free of any third party libraries and reproducible locally. Environment specification - at least python version, is appreciated

b. If the problem is in a 3rd party library and you’ve located the problem, open a bug report on github/anywhere the target project is hosted

c. If the problem is clearly in a 3rd party library and you’re unable to find the cause, open a ticket describing the case with the profiler's report attached

score 0 · Answer 2 · edited Oct 02 '17 at 10:39

0

It seems to be that you are running instance class B1 or F1, which has a memory limit of 128 MB.

A possible solution would be to use a higher instance class. But please keep in mind that choosing a different instance class will have impact on your pricing and quotas.

edited Oct 02 '17 at 10:39

Palec

12,743
8
69
138

answered Sep 22 '17 at 09:20

Michael Meyer

2,179
3
24
33

Thanks, Michael, this is exactly what I did, but I have memory leakage even with B4 instance. – LA_ Sep 22 '17 at 16:15

How to fix memory leak in my application?

2 Answers2

Linked