How to make Ignite act non-blocking with TcpDiscoveryVmIpFinder like it does with TcpDiscoveryMulticastIpFinder?

Question

Background:

• I have a web app, spring-based.
• locally on a dev machine I have 2 tomcat instances that runs same app - that way I test how web farm nodes communicates to each other
• I use Jelastic cloud for app deployment
• On Jelastic It's not running as a web-farm, but rolling update mechanism is used (while AppV1 is running at NodeA and handling user requsts I start AppV2@NodeB, warm it up and redirect user requests to it. Goal is to let NodeB copy all sessions from NodeA)

Intention

• Current release version is using 3rd server based on NodeJS as a shortcut to MessageBus between nodes. But recently I spotted Ignite and thought it would be great to decrease number of platforms (nodejs) and get just everything in Java.
• So I replaced NodeJS-based messaging with Ignite messaging. Ignite is initialized using Spring XML config and org.apache.ignite.IgniteSpringBean
• When running my app locally with TcpDiscoveryMulticastIpFinder it works perfectly. NodeA starts even if there is no NodeB started. When I start NodeB it smoothly joins cluster and nodes are connected to each other and communicating perfectly. Most important thing here is that I can start and stop nodes any time and I have non-blocking error-free operation using Ignite messaging.

Issue

• But on Jelastic I don't have multicast, so I have to explicitly define list of IP addresses (use TcpDiscoveryVmIpFinder), which is ok - I have kind of static host names for each node. BUT now when NodeA starts it blocks until it connects to NodeB. And if NodeB is not there whole app crash (failes to deploy).

Question is How to make it work in scenario

1. TcpDiscoveryVmIpFinder is used
2. NodeA starts (while NodeB is not started)
3. Expectation: NodeA operates any amount of time correctly (of course I can't send messages to remotes, because there are no remotes connected - that is ok)
4. At any time NodeB starts
5. Expectation: NodeA and NodeB find each other and communicate to each other (basically for sessions replication and other messages)
6. NodeA stopped
7. Expectation: NodeB continues to operate normally serving user requests
8. Now switch: NodeA becomes NodeB and vice versa; repeat from step 3

Answer 1

Most likely you missed one important thing related to TcpDiscoveryVmIpFinder. Both nodes A and B have to contain their own IP addresses in the IP finder's addresses list. This is highlighted in this readme section [1]. Otherwise you'll get into the situation like you have when node A has to wait while node B joins the cluster.

Set this IP finder for both the nodes and everything should work as required.

                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <value>hodeA_ip_address_or_host_name:47500..47509</value>
                            <value>hodeB_ip_address_or_host_name:47500..47509</value>
                        </list>
                    </property>
                </bean>

[1] https://apacheignite.readme.io/docs/cluster-config#static-ip-based-discovery

Answer 2

I found workaround (not a solution). If you know right solution - please let me know by adding answer below.

I subclassed TcpDiscoverySpi and override resolvedAddresses() in order to test candidates whether they are reachable or not.

That solution is not perfect because now if NodeA didn't find NodeB it will never retry to find NodeB. However that is not a showstopper since once NodeB starts it reaches NodeA and they start talking to each other. So effectively Rolling Updates mechanism works for me.

Method isCandidateRespond is quick-n-dirty solution and most likely will vary case-by-case.

Here is a code snippet:

public class TcpDiscoverySpiPrecheckingImpl extends TcpDiscoverySpi {
    private Logger log = Logger.getLogger(getClass());

    @Override
    protected Collection<InetSocketAddress> resolvedAddresses() throws IgniteSpiException {
        Collection<InetSocketAddress> candidates = super.resolvedAddresses();
        Collection<InetSocketAddress> approved = new ArrayList<>();
        for (InetSocketAddress candidate : candidates) {
            if (isCandidateRespond(candidate)) {
                approved.add(candidate);
            }
        }
        return approved;
    }

    private boolean isCandidateRespond(InetSocketAddress candidate) {
        if (log.isTraceEnabled()) {
            log.trace("Checking if remote node responds: " + candidate);
        }
        URL url = null;
        try {
            url = new URL("http://" + candidate.getHostName() + "/");
            try (InputStream stream = url.openStream()) {
                return true;
            }
        } catch (Throwable t) {
            log.info("Candidate remote node didn't respond: " + url + ". Reason: " + t.getMessage());
            return false;
        }
    }
}