4

I'm having problems trying to connect to Elasticsearch (ES) on an EC2 instance from my local linux box via the EC2 instance public ip i.e. curl [PUBLIC_IP]:9200

I followed the steps in this guide: https://github.com/miztiik/elk-stack/tree/master/ElasticSearch.

My ES version is 6.8.9

Here's what's working and what's not:

  • On ES EC2 instance: curl localhost:9200 works
  • On another instance with same VPC: curl [PUBLIC_IP]:9200 works
  • On my local linux box: curl [PUBLIC_IP]:9200 doesn't work, however telnet [PUBLIC_IP] 9200 works i.e. it connects and gives me the escape character '^]'

My /etc/elasticsearch/elasticsearch.yml config has the following:

http.enabled: true
http.port: 9200
network.host: 0.0.0.0
http.cors.allow-origin: "*"
http.cors.enabled: true

There is only one (new) security group attached to the EC2 instance, which has the following inbound rules:

enter image description here

I also confirmed that the EC2 instance is in a public subnet i.e. connected to an internet gateway.

Thanks for any help.

Update

I also installed Apache httpd on the instance and rechecked everything. Here is the current state of things:

  • I can ping, telnet and connect to the web server (:80) from the outside.
  • I cannot connect to Elasticsearch (:9200) or Kibana (:5601) from the outside. All these I can however do within the VPC from another instance.
zenith7
  • 151
  • 1
  • 3
  • 8

2 Answers2

2

This sounds firewall related. Check on the ECE2 security group and either modify the default Sec group or create new one and associate it with your instance. For a test, modify your inbound as for your port as: 

0.0.0.0/0 IPv4

And set network host as follows

network.host: _ec2 # if using the plugin

Otherwise 

network.host: "{elastic_ip}”

If your ece2 instance doesn’t have public dns, you will have to edit your/etc/hosts file and add the IP address of your instance

network.bind_host

This specifies which network interface(s) a node should bind to in order to listen for incoming requests. A node can bind to multiple interfaces, e.g. two network cards, or a site-local address and a local address. Defaults to network.host.

network.publish_host The publish host is the single interface that the node advertises to other nodes in the cluster, so that those nodes can connect to it. Currently an Elasticsearch node may be bound to multiple

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

https://discuss.elastic.co/t/elasticsearch-only-accessible-from-localhost/65782/3

https://www.elastic.co/blog/running-elasticsearch-on-aws

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#describing-security-group

How do I enable remote access/request in Elasticsearch 2.0?

salah-1
  • 1,299
  • 11
  • 15
  • Haven't I already done this? If you click the "inbound rules" hyperlink in my post, it shows my inbound rules associated with the security group. I have "Custom TCP TCP 9200 0.0.0.0/0". I don't have high enough reputation to paste that image inline to my post. – zenith7 May 22 '20 at 03:22
  • Huh, thought they were little different. Are u sure this security group is associated with the instance? Is it the default one ? – salah-1 May 22 '20 at 03:24
  • I associated a new security group to my ES EC2 instance when i created that instance. There is only one security group associated to the instance, and those are the inbound rules I have set. Where do I set "network.host: _ec2" ? In my elasticsearch.yml file? If I do so elasticsearch can't restart. I get "java.net.UnknownHostException: _ec2: Name or service not known" in my ES log file. – zenith7 May 22 '20 at 04:53
  • Correct , in the elasticsearch.yml file – salah-1 May 22 '20 at 05:01
  • Ok, I installed the discovery-ec2 plugin. I also had to set network.host to _ec2 _ (i.e. underscore on either side), not _ec2. Restarted EC2, but still unable to connect via curl [PUBLIC_IP]:9200 on linux box. – zenith7 May 22 '20 at 05:48
  • Wait , is your instance reachable publicly? See updated answer. If this is the problem, didn’t need the plugin :) – salah-1 May 22 '20 at 05:54
  • Yes my instance has a Public IP/DNS. Should I uninstall the plugin and just set network.host back to 0.0.0.0 ? I will be where I started again :) – zenith7 May 22 '20 at 06:06
  • It’s ok, leave plugin till we find what’s going on. Look at elastic logs and firewall logs after you try your connections from ur box. Also, can you ping the instance and get some response – salah-1 May 22 '20 at 10:29
  • I restarted from scratch with a new EC2 instance. The only difference is I installed ES 6.x (previous was 5.x) as I found out I can restore my 5.x snapshots using 6.x, but not with 7.x. I didn't install discovery-ec2 plugin this time. I also associated an Elastic IP to the instance. I can ping the public ip from my linux box, however only after i added 'ICMP - IPv4' to inbound rules of the security group. Still the same issue - cannot connect to [public_ip]:9200 from linux box. I'll check elastic and firewall logs more thoroughly. Thanks for all your help so far. – zenith7 May 24 '20 at 05:40
  • If you can ping or reach the box on port :80 from your machine then, the problem is related to elastic search config – salah-1 May 24 '20 at 11:01
  • I installed apache httpd on ES EC2 and can connect to box using :80. I also opened all TCP ports in security group. My .yml file hasn't changed. Also installed Kibana on same EC2 instance as ES, but cannot connect from box to :5601 – zenith7 May 24 '20 at 23:57
  • Why on 5600? Isn’t search on :9200 port? Also, are u sure you pinging same ip that’s the search is on? And getting results? Strange. Time to reach out search support forums and see. See first link on the answer special reference to network.publish_host – salah-1 May 25 '20 at 11:14
  • Updated the answer, see bolds and first 2 answers -assuming that this is ES related issue and you cornered the firewall issues and can ping and reach box from outside – salah-1 May 25 '20 at 11:19
  • I updated my post with recent findings. I am not trying :5600, I am trying :5601 which is used by Kibana which I also installed on the instance. I also read that "network.host setting sets the bind host and the publish host at the same time". So I shouldn't have to play with network.publish_host as i have already set "network.host 0.0.0.0"? But I will play more with the settings. Yes I can ping, telnet and :80 from the outside, but cannot :9200 or :5601 (only from inside VPC). I rechecked all. – zenith7 May 26 '20 at 00:23
  • Well I'm not sure what's gone on, but it's all working now. I didn't make any config changes since 7 hrs ago when both my ES\Kibana instances were not accessible via 9200\5601 externally. Now it's all magically working from my local box. Wish I knew what made it work as I made no changes. Well thanks anyways for all your persistent help. I'll accept your response as the answer. Cheers. – zenith7 May 26 '20 at 08:19
  • I’m glad it worked out. I seen few times that ES fixes itself or glitches go away :) – salah-1 May 26 '20 at 10:33
0

I had the same issue on AWS. Try using the public DNS or the private IP in lieu of the public IP to connect another ec2 instance in the same VPC.

rocketman
  • 131
  • 1
  • 9
  • In my post I mentioned "On another EC2 instance in same VPC as ES EC2 instance: curl [PUBLIC_IP]:9200 works". But I really want to get it working from my local linux box – zenith7 May 22 '20 at 02:57