0

Database Mysql

I have referred to these links:

Config file

logstash-sample.conf

                # Sample Logstash configuration for creating a simple
                # Beats -> Logstash -> Elasticsearch pipeline.

                                            jdbc {
                jdbc_driver_library => "C:/logstash-6.5.1/logstash-core/lib/jars/x-pack-sql-jdbc-6.5.1.jar"
                jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
                jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
                jdbc_user => "root"
                jdbc_password => ""
                statement => "SELECT * FROM persons"
                jdbc_paging_enabled => "true"
                jdbc_page_size => "50000"
                }

                }

                output{
                elasticsearch { codec => json hosts => ["localhost:9200"] index => "persons" }
                stdout { codec => rubydebug }
                }

Database Name Test

Datable Table persons

Index Persons

Logs

            C:\logstash-6.5.1\bin>logstash -f logstash-sample.conf
            Sending Logstash logs to C:/logstash-6.5.1/logs which is now configured via log4j2.properties
            [2018-11-30T18:10:52,868][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
            [2018-11-30T18:10:52,883][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.1"}
            [2018-11-30T18:10:53,550][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"C:/logstash-6.5.1/bin/logstash-sample.conf"}
            [2018-11-30T18:10:53,559][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
            [2018-11-30T18:10:53,894][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Request GET

http://localhost:9200/persons/_search/?pretty

            {
                "error": {
                    "root_cause": [
                        {
                            "type": "index_not_found_exception",
                            "reason": "no such index",
                            "resource.type": "index_or_alias",
                            "resource.id": "persons",
                            "index_uuid": "_na_",
                            "index": "persons"
                        }
                    ],
                    "type": "index_not_found_exception",
                    "reason": "no such index",
                    "resource.type": "index_or_alias",
                    "resource.id": "persons",
                    "index_uuid": "_na_",
                    "index": "persons"
                },
                "status": 404
            }

Any suggestion is most welcome

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
afeef
  • 4,396
  • 11
  • 35
  • 65
  • 1
    The problem comes from `No config files found in path: C:/logstash-6.5.1/bin/logstash-sample.conf` Where did you specify this path? in `logstash.yml` ? – Val Nov 30 '18 at 12:48
  • Thanks for reply i did save path C:\logstash-6.5.1\config\logstash.yml – afeef Nov 30 '18 at 13:11
  • Please show your logstash.yml file – Val Nov 30 '18 at 13:12
  • please use http://s000.tinyupload.com/index.php?file_id=55258454618782156389 – afeef Nov 30 '18 at 13:12
  • ok it seems you've not modified anything in there right? where is your `logstash-sample.conf` located? and which command do you use to run logstash? – Val Nov 30 '18 at 13:14
  • pipeline.yml path http://s000.tinyupload.com/index.php?file_id=03460491555222688005 – afeef Nov 30 '18 at 13:15
  • logstash-sample.config path C:\logstash-6.5.1\config\logstash-sample.config im new in elastic please bear with me. thanks. – afeef Nov 30 '18 at 13:16

2 Answers2

1

Ok since you're located in the logstash/bin folder and your logstash-sample.conf fiel is located in the logstash/config folder, you need to run logstash like this:

C:\logstash-6.5.1\bin> logstash -f ../config/logstash-sample.conf
                                        ^
                                        |
                               add this relative path
Val
  • 207,596
  • 13
  • 358
  • 360
  • Thanks for the Help logstash-sample.conf logs LOgs http://s000.tinyupload.com/index.php?file_id=54725932227565245733 still i was not able see json data from db and index "persons" . – afeef Nov 30 '18 at 13:25
  • update file name http://s000.tinyupload.com/index.php?file_id=07944564826348286325 with logstash-sample.logs – afeef Nov 30 '18 at 13:30
  • That's looking much better, though, no more missing config error – Val Nov 30 '18 at 13:32
  • Add `--debug` when running logstash and show the new logs, please – Val Nov 30 '18 at 13:33
  • http://s000.tinyupload.com/index.php?file_id=00170330724389692854 file name logstash-sample_1.logs Thanks – afeef Nov 30 '18 at 13:38
  • http://s000.tinyupload.com/index.php?file_id=07555255504900658615 C:\logstash-6.5.1\logs\logstash-plain.log – afeef Nov 30 '18 at 13:41
  • Weird that there's absolutely nothing being logged by the `jdbc` input – Val Nov 30 '18 at 13:43
  • I see that a closing } is missing at the end of your config file, but that's probably a copy/paste issue – Val Nov 30 '18 at 13:44
  • Can you run this `curl -XGET http://localhost:9600/_node/stats?pretty` and share the output you get? – Val Nov 30 '18 at 14:34
  • Thanks for looking into this issue http://s000.tinyupload.com/index.php?file_id=44671467684815196066 – afeef Dec 03 '18 at 06:45
  • ok, Logstash tells us that your config file has a `beats` input and an `elasticsearch` output, but no `jdbc` anywhere. – Val Dec 03 '18 at 06:53
  • Thanks val for reply can you please help what i have done wrong ..thanks. – afeef Dec 04 '18 at 09:08
  • It's weird... are you sure you're running the correct configuration file? The one that runs is one that has the `beats` input, but not the `jdbc` input... Do you have several configuration files maybe, and picking the wrong one? – Val Dec 05 '18 at 14:17
0

Solution that worked

windows

  1. wrong Jdbc path.
  2. mysql driver issue.

Config

                        jdbc {
                        jdbc_driver_library => "C:/logstash-6.5.1/logstash-core/lib/jars/x-pack-sql-jdbc-6.5.1.jar"
                        jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
                        jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
                        jdbc_user => "root"
                        jdbc_password => ""
                        statement => "SELECT * FROM persons"
                        jdbc_paging_enabled => "true"
                        jdbc_page_size => "50000"
                        }

                        }

                        output{
                        elasticsearch { codec => json hosts => ["127.0.0.1:9200:9200"] index => "persons" }
                        stdout { codec => rubydebug }
                        }

Hope it help others.

afeef
  • 4,396
  • 11
  • 35
  • 65