0

Problem you have encountered: I have deployed a spring boot application (backend) on Google App Engine. For past few days, I am getting below mentioned error intermittently: Error: Process terminated because the request deadline was exceeded. (Error code 123)

Description:

I have deployed a spring boot application (backend) on Google App Engine. For past few days, I am getting below mentioned error intermittently: Error: Process terminated because the request deadline was exceeded. (Error code 123)

The application uses CloudSQL (MySQL) database. I have setup below mentioned auto-scale properties as well:

       <sessions-enabled>true</sessions-enabled>
    <warmup-requests-enabled>true</warmup-requests-enabled>
    <instance-class>F2</instance-class>
    <automatic-scaling>
        <target-cpu-utilization>0.65</target-cpu-utilization>
        <min-instances>10</min-instances>
        <max-instances>20</max-instances>
        <min-idle-instances>5</min-idle-instances>
        <max-idle-instances>6</max-idle-instances>
        <min-pending-latency>30ms</min-pending-latency>
        <max-pending-latency>500ms</max-pending-latency>
        <max-concurrent-requests>10</max-concurrent-requests>
    </automatic-scaling>
    <inbound-services>
          <service>warmup</service>
    </inbound-services>

What you expected to happen: The application hosted on GAE shouldn't fail with this intermittent error

Steps to reproduce: No steps available, its intermittent.

Other information (workarounds you have tried, documentation consulted, etc): I have tried multiple different combinations for the configurations of autoscaling, etc.

Maddy
  • 1
  • 1
    This issue as mentioned is intermittent, and causes slowness in the API response (as it is a backend application) and ultimately fails with 500 Http error code with the error message mentioned in the details. around "latency": "100.205186s" – Maddy Jan 07 '21 at 10:19
  • Hello, does your application makes requests calls to an external services ? I found a similar question on [SO](https://stackoverflow.com/questions/30913255/how-to-solve-process-terminated-because-the-request-deadline-was-exceeded-err) where it was suggested 2 possible improvements: use warmup requests and call external services in a Task Queue. – marian.vladoi Jan 08 '21 at 10:44
  • hello, - yes, I have handled the warmup requests - could you please confirm what sort of external services you are referring to? – Maddy Jan 11 '21 at 05:24
  • Hi, as per the previous comment from the SO post: https://stackoverflow.com/questions/30913255/how-to-solve-process-terminated-because-the-request-deadline-was-exceeded-err. It refers to the two approaches warmup requests and external services. External service in a TaskQueue:https://cloud.google.com/appengine/docs/standard/python/taskqueue – Saigeetha Sundar Jan 18 '21 at 16:04

1 Answers1

0

I just hat the exact same issue in a GAE Standard Java 8 Application - after A LOT of trial and error I found the issue was related to Cloud SQL - one symptom was that autoscaled instances restarted (probably crashed) frequently as if there was something stuck. The "Error: Process terminated because the request deadline was exceeded. (Error code 123)" also produced no further logs.

The solution (in our case) was (because that was the only thing we changed right before the error appeared all the time) that we frequently used a Cloud SQL Query in our application that went like this:

case when column=1 then 1 else -1 end

column could be NULL in some (rare) cases - no problem with our normal SQL client we tested this with, but for some weird reason, Cloud SQL and JDBC has a problem with this causing these instance issue

changing to

case when coalesce(column,0)=1 then 1 else -1 end

so that the there would never be a comparison against NULL in the case statement solved the problem

GeraldDC
  • 11
  • 1