502 Server Error sometime on Google Compute Engine

Question

I set up a server on Google Compute Engine with Apache server on Ubuntu 16.04.4 LTS. It's protected with IAP.

It was fine all along for about 6 months but now some of the users encounter 502 Server Error.

I already checked the following links

• Some 502 errors in GCP HTTP Load Balancing [Changed the Apache KeepAliveTimeout to 620]
• 502 response coming from errors in Google Cloud LoadBalancer [Removed ajax requests]

But the problem is still there.

Here is the error message from one of the log.

{
     httpRequest: {…}  
     insertId:  "170sg34g5fmld90"  
     jsonPayload: {
      @type:  "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"   
      statusDetails:  "failed_to_pick_backend"   
     }
     logName:  "projects/sggc-web01/logs/requests"  
     receiveTimestamp:  "2018-03-14T07:21:55.807802906Z"  
     resource: {…}  
     severity:  "WARNING"  
     spanId:  "44a49bf1b3893412"  
     timestamp:  "2018-03-14T07:21:53.048717425Z"  
     trace:  "projects/sggc-web01/traces/f35119d8571f20df670b0d53ab6b3210"  
    }

Please help me to trace and fix the issue. Thank you!

Answer 1

The error is not being caused by the server but the load balancer.

For the error we can see in the statusDetails "failed_to_pick_backend" it is being caused because all the instances were unhealthy (or still are) when it tries to establish the connection.

This can be because:

1 - The CPU usage of the instances were too high and they weren't able to answer the health check request from the load balancer showing as unhealthy to it.

2 - The health checks aren't being allowed in the firewall (I doubt this can be the reason if it worked before)