6

I'm using Cassandra 3.10 with 6 Nodes cluster having replication Factor 2. [please consider the IP's have been masked and not original].

Datacenter: DC1
===========================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  100.55.10.124  212.33 MiB  256          65.9%             8328123a-678c-4027-a136-06b3626ab017  rack1
UN  100.55.10.125  216.92 MiB  256          67.1%             af76123c-6880-43d6-bff8-b139731242bd  rack1
UN  100.55.10.126  216.94 MiB  256          67.0%             f541cdc1-aa2f-4a30-3612-5d4291421ec3  rack1
Datacenter: DC2
===========================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  100.55.10.130  198.15 MiB  256          60.7%             332eb914-bf14-4d0d-9c38-47fc4b0952f1  rack2
UN  100.55.10.131  229.48 MiB  256          70.8%             cafc702f-83d4-479a-b117-ada75274e0ba  rack2
UN  100.55.10.132  223.14 MiB  256          68.5%             8a9c03ae-ed5a-4cb9-9540-0ca97a04640e  rack2

cluster was working fine, but these days i see below "INFO" in the system logs 

INFO  [Native-Transport-Requests-3] 2017-04-19 03:21:19,684 Message.java:617 - Unexpected exception during request; channel = [id: 0x7acf3f0c, L:/100.55.10.124:9042 ! R:/30.75.50.2:39263]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.39.Final.jar:4.0.39.Final]

I was trying to analyze the issue, but couldn't make it yet.

i've looked at following JIRA issues around this, but wasn't able to find anything helpful.
i've also referred to few stackoverflow links couldnt find anything conclusive.

i am using CQL v3.4.4. Is there anyone, who is familiar with above scenario?

Community
  • 1
  • 1
segFaulter
  • 180
  • 9
  • An old post, but solved same issue, this is because of a client application on IP 30.75.50.2 is closing/kiling connection with Cassandra node. This will cause the same INFO message. – Anil Kapoor Feb 08 '19 at 05:09

1 Answers1

1

As this request is showing "Native-Transport-Requests" in that case would request to check the output of "nodetool tpstasts" command.

If you see a high blocked count on "Native-Transport-Requests" in that case you plan to increase max_queued_native_transport_requests.

CQL Native Protocol is the way the Cassandra driver communicates with the server. This includes all reads, writes, schema changes, etc. There are a limited number of threads available to process incoming requests.

When all threads are in use, some requests wait in a queue (pending). If the queue fills up, some requests are silently rejected (blocked) and the server never replies.

https://support.datastax.com/hc/en-us/articles/360031470531-High-blocked-NTR-count-during-increased-workload-on-Cassandra-node

Yogesh Pingle
  • 51
  • 3