0

I'm going to write a quick little SF service to report endpoints from a service to a load balancer. That part is easy and well understood. FabricClient. Find Services. Discover endpoints. Do stuff with load balancer API.

But I'd like to be able to deal with a graceful drain and shutdown situation. Basically, catch and block the shutdown of a SF service until after my app has had a chance to drain connections to it from the pool.

There's no API I can find to accomplish this. But I kinda bet one of the services would let me do this. Resource manager. Cluster manager. Whatever.

Is anybody familiar with how to pull this off?

Jerome Haltom
  • 1,670
  • 2
  • 17
  • 23

1 Answers1

0

From what I know this isn't possible in a way you've described.

Service Fabric service can be shutdown by multiple reasons: re-balancing, errors, outage, upgrade etc. Depending on the type of service (stateful or stateless) they have slightly different shutdown routine (see more) but in general if the service replica is shutdown gracefully then OnCloseAsync method is invoked. Inside this method replica can perform a safe cleanup. There is also a second case - when replica is forcibly terminated. Then OnAbort method is called and there are no clear statements in documentation about guarantees you have inside OnAbort method.

Going back to your case I can suggest the following pattern:

  1. When replica is going to shutdown inside OnCloseAsync or OnAbort it calls lbservice and reports that it is going to shutdown.
  2. The lbservice the reconfigure load balancer to exclude this replica from request processing.
  3. replica completes all already processing requests and shutdown.

Please note that you would need to implement startup mechanism too i.e. when replica is started then it reports to lbservice that it is active now.

In a mean time I like to notice that Service Fabric already implements this mechanics. Here is an example of how API Management can be used with Service Fabric and here is an example of how Reverse Proxy can be used to access Service Fabric services from the outside.

EDIT 2018-10-08

In order to abstract receive notifications about services endpoints changes in general you can try to use FabricClient.ServiceManagementClient.ServiceNotificationFilterMatched Event.

There is a similar situation solved in this question.

Oleg Karasik
  • 959
  • 6
  • 17
  • Yeah. I need to be able to do it from outside the specific service though. Some are guest executables. I'm running on prem, not Azure. I'm trying to duplicate whatever it is that's going on in Azure that drains the App GW or LB properly. – Jerome Haltom Sep 20 '18 at 14:57
  • @wasabi correct me if I am wrong but are you saying that this service would act as GW or LB? Or this service is going to communicate with external LB or GW? – Oleg Karasik Sep 20 '18 at 18:01
  • It's going to communicate with an external LB or GW. I want to make sure that LB has been drained before the service actually begins to stop. – Jerome Haltom Oct 06 '18 at 16:28
  • @wasabi I have updated the answer with more information. – Oleg Karasik Oct 08 '18 at 07:43
  • Thanks! This looks like part of my problem. And a useful one at that. I think stalling the shutdown is still not answered, though, right? – Jerome Haltom Oct 13 '18 at 22:08