Logstash MalformedCSVError: Illegal quoting in line | how to configure logstash conf to read multi column mutiline CSV?

Question

Already went through CSV::MalformedCSVError: Illegal quoting in line 1 with SmarterCSV and Rescue CSV::MalformedCsvError: Illegal quoting in line n None of them addresses my problem.

I'm facing problem while basic integration of CSV conf with Log Stash:

    input {
      file {
        path => "/softwares/imews_customer_wallet_details.csv"
        start_position => "beginning"
      }
    }
    filter {
      csv {
        columns => ["imiles_wallet_id","creation_date","gcid_cod","total_imiles","is_active"]
        separator => ","
      }
      date {
        match => ["creation_date", "dd-MM-yy"]
        target => "creation_date"
      }
    }
    output {
      elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "ime_customers"
      }
    }

Error Stack prints following line on loop:


    [2020-03-02T13:36:51,968][WARN ][logstash.filters.csv     ][another_test] Error parsing csv {:field=>"message", :source=>"{\"level\":\"DEBUG\",\"timeUTC\":\"Mon Feb 24 08:05:20 UTC 2020\",\"timeUnix\":1582531520800,\"appID\":\"TEST\",\"hub\":\"CI\",\"environment\":\"NONE\",\"host\":\"Yashpal3150153\",\"logMessage\":\"http-outgoing-1 << Content-Type: application/x-amz-json-1.0\",\"threadID\":\"SimpleAsyncTaskExecutor-1\",\"cdll_version\":\"6.5.0\"}\r", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

What am I Missing, what's wrong here?

Update:

CSV:

    125,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid3,100,TRUE
    124,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid2,100,TRUE
    123,IME,"14-11-19","14-11-19",IME,"14-11-19",gcid1,252,TRUE

Update: Pipeline:

     - pipeline.id: another_test
       queue.type: persisted
       path.config: "/softwares/logstash-7.6.0/config/*.conf"

config contains: My conf I mentioned is: cust.conf

Also, I updated the cust.conf to:

    input {
      file {
        path => "D:/softwares/imews_customer_wallet_details.csv"
        start_position => "beginning"
      }
    }
    filter {
      csv {
        columns => ["imiles_wallet_id","creation_date","gcid_cod","total_imiles","is_active"]
        separator => ","
        codec => plain
        convert => {
          "imiles_wallet_id" => "integer"
          "is_active" => "boolean"
          "creation_date" => "date"
          "gcid_cod" => "string"
          "total_imiles" => "integer"
        } 
      }
    }
    output {
      elasticsearch {
        action => "index"
        hosts => ["localhost:9200"]
        index => "ime_customers"
      }
    }

Now the Error is:

    LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`
              create at org/logstash/execution/ConvergeResultExt.java:109
                 add at org/logstash/execution/ConvergeResultExt.java:37
      converge_state at D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339
    [2020-03-04T18:20:05,345][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
    [2020-03-04T18:20:05,397][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<another_test>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in `create'", "org/logstash/execution/ConvergeResultExt.java:37:in `add'", "D:/softwares/logstash-7.6.0/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
    ```

If you check your error message, you'll see that the message that Logstash tries to parse doesn't really look like CSV, it looks like JSON. Are you sure about your input, what files you're reading and their content? — Val, Mar 02 '20 at 12:08
Yes sir, that is the exact Input, In fact I thought the `-`s in the date might cause the issue, that's why I wrapped the dates in double `""` — Yashpal, Mar 02 '20 at 12:12
@Val can you mention the part of error which gives the idea that the input looks like a JSON? — Yashpal, Mar 02 '20 at 12:13
`{:field=>"message", :source=>"{\"level\":\"DEBUG\",\"timeUTC\":\"Mon Feb 24 08:05:20 UTC 2020\",\"timeUnix\":1582531520800,\"appID\":\"TEST\",\"hub\":\"CI\",\"environment\":\"NONE\",\"host\":\"Yashpal3150153\",\"logMessage\":\"http-outgoing-1 << Content-Type: application/x-amz-json-1.0\",\"threadID\":\"SimpleAsyncTaskExecutor-1\",\"cdll_version\":\"6.5.0\"}\r"` — Val, Mar 02 '20 at 12:14
As per my knowledge that a standard error response. Not related to input. — Yashpal, Mar 03 '20 at 19:24
That line comes from the `logstash.filters.csv` filter and is labelled WARN. This means that something coming from an input is not CSV-ready. — Val, Mar 03 '20 at 20:37
@Val But the file contains only lines that I have shown, not even a blank new line at the end, you may try the given input at your system as well, the error should be the same. — Yashpal, Mar 04 '20 at 11:41
How do you start Logstash? Command-line or as a service? In the latter case, do you have other input files in your configuration folder? — Val, Mar 04 '20 at 11:48
Yes, that's my host, I'm using CLI cmd `.\bin\logstash.bat`. Other input file is sample logs file which contains log, and got successfully uploaded to ES, logs can be seen on Kibana. — Yashpal, Mar 04 '20 at 12:53
How do you point your Logstash instance at your configuration file? — Val, Mar 04 '20 at 12:56
Ok, what are all the files you have in `/softwares/logstash-7.6.0/config/*.conf` ? Do you have several `*.conf` files? — Val, Mar 04 '20 at 13:05
Simply start logstash with `bin/logstash -f /softwares/logstash-7.6.0/config/cust.conf` how does it work? — Val, Mar 04 '20 at 13:07
@Val thanks, the error disappeared, the index was added, but no data is shown from the CSV. — Yashpal, Mar 04 '20 at 13:33
One more question, why isn't it executing it from the pipeline? — Yashpal, Mar 04 '20 at 13:34
Ok, the problem was that you have several configuration files (i.e. `*.conf`) and probably several input. When using a pipeline, note that all configuration files are "merged" into a single configuration (with potentially several sources, filters and outputs) — Val, Mar 04 '20 at 13:35
Yeah, that can be the probable reason, But I'm not able to see the data on Kibana. Also @Val, would you take a minute to add this resolution as answer so that I can can mark that correct answer. We did a lot of grooming there :) — Yashpal, Mar 04 '20 at 15:21

score 1 · Accepted Answer · answered Mar 04 '20 at 16:44

By starting Logstash with a pipeline definition /softwares/logstash-7.6.0/config/*.conf you probably have many configuration files that interfere with each other (i.e. several different inputs, filters and outputs). All those configuration files are merged into a single configuration.

Read this article in order to learn how to best organize your Logstash pipelines. In the meantime, you can simply start your Logstash from the command-line like this:

bin/logstash -f /softwares/logstash-7.6.0/config/cust.conf

Logstash MalformedCSVError: Illegal quoting in line | how to configure logstash conf to read multi column mutiline CSV?

1 Answers1