4

I'm writing a kibana plugin and a logstash pipeline. For my tests, I just wrote a logstash input like that:

input {

   beats {
        port => 9600
        ssl => false
        ssl_verify_mode => "none"
    }

}

But when I try to open a connection with node (code above):

invoke = (parameters, id, port, host) => {
        var fs = require('fs');

        console.log(`Sending message in beats, host= ${host}, port= ${port}, message= ${parameters.message}`);

        var connectionOptions = {
            host: host,
            port: port
        };

        var client = lumberjack.client(connectionOptions, {rejectUnauthorized: false, maxQueueSize: 500});

        client.writeDataFrame({"line": id + " " + parameters.message});
    }

logstash gives to me "invalid version of beats protocol: 22" and "invalid version of beats protocol: 3":

Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 22
        at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.11.jar:?]
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.11.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        ... 9 more
[2020-08-11T07:49:47,954][INFO ][org.logstash.beats.BeatsHandler] [local: 172.22.0.40:9600, remote: 172.22.0.1:33766] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
[2020-08-11T07:49:47,955][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 3
        at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.11.jar:?]
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.11.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
        ... 11 more
tshepang
  • 12,111
  • 21
  • 91
  • 136
Pikappa
  • 181
  • 1
  • 2
  • 11
  • 2
    The `beats` input uses a proprietary protocol that is used by the [elastic beats](https://www.elastic.co/beats/) or a [custom beat](https://www.elastic.co/guide/en/beats/devguide/current/new-beat.html), which is written in Go, I don't think it will work in any other use case. Why are you sending your data using node.js to a beats input? I think you should use the TCP input. – leandrojmp Aug 11 '20 at 14:07
  • Because my plugin (developed in node.js) has to simulate the behavior of the standard file beat – Pikappa Aug 11 '20 at 14:17
  • 1
    Oh, I see. This error normally appears when there is a mismatch in the TLS configuration between the sender beats and logstash, for example logstash is expecting no TLS communication, but the sender is using TLS. But you would need to check in the lib documentation how to enable or disable the TLS. For the code you shared it seems to be disabled. – leandrojmp Aug 11 '20 at 14:37

3 Answers3

0
  1. Instead of use beats input you could try to use tcp input.

Example:

input {
  tcp {
    port => "9600"
    codec => "json"
  }
}
  1. If you are using beats input and you want to use Logstash to perform additional processing on the data collected by Filebeat, you need to configure Filebeat to use Logstash.

To do this, you edit the Filebeat configuration file to disable the Elasticsearch output by commenting it out and enable the Logstash output by uncommenting the Logstash section:

output.logstash:
     hosts: ["127.0.0.1:5044"]

You can read more on https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html

Wallace Jackson
  • 194
  • 3
0

For understanding more about this error we will need to see the filbeat.yml. Input plugin that you have used is a valid input plugin but in filbert.yml you might have not had output.logstash value or probably made some other mistakes. Can you please check if you have sent the output to Elasticsearch or Logstash ?

Please ensure you have this line of code in your filebeat.yml

output.logstash:
  hosts: ["127.0.0.1:5044"]
Incepter
  • 2,711
  • 15
  • 33
sidharth vijayakumar
  • 1,190
  • 5
  • 29
0

There could be many cases of that in my case issue was related to my filebeat.yml i was having below error on my logstash server

    nd it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:253) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
        at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.2.6.jar:?]
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.2.6.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
        ... 11 more

When i check my filebeat so it says connection refused

root@ip-10-0-8-193:~# filebeat test output
elasticsearch: http://10.0.13.37:5044...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.0.13.37
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... ERROR Get "http://10.0.13.37:5044": read tcp 10.0.8.193:34940->10.0.13.37:5044: read: connection reset by peer

when i closely check my logs so i found one misconfiguration from errors

[2022-10-12T11:08:06,107][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.14.30:9200"]}

above error means i didn't configure outputs correctly i closed elasticsearch line but miss output line

Solution In my case
I went back to my filebeat.yml and make the required changes

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #  hosts: ["localhost:9200"]

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.0.13.37:5044"]

make sure we properly commented elasticsearch output

Mansur Ul Hasan
  • 2,898
  • 27
  • 24