0

I am using logstash 5.6

In my document, I have a subfield "[emailHeaders][reingested-on]", and another field called [attributes], which contains several subfields [string], [double], each of which are arrays. :

{
  "emailHeaders": {
    "reingested-on": ["1613986076000"]
  },
  "attributes": {
    "string": [
      {
        "name": "attributeString1",
        "value": "attributeStringValue1"
      },
      {
        "name": "attributeString2",
        "value": "attributeStringValue2"
      }
    ],
    "double": [
      {
        "name": "attributeDouble1",
        "value": 1.0
      }
    ]
  }
}

If the element [emailHeaders][reingested-on] is present in the document, I want to copy 1613986076000 (ie. the first element of [emailHeaders][reingested-on]) into [attributes][date] as follows:

{
  "emailHeaders": {
    "reingested-on": ["1613986076000"]
  },
  "attributes": {
    "string": [
      {
        "name": "attributeString1",
        "value": "attributeStringValue1"
      },
      {
        "name": "attributeString2",
        "value": "attributeStringValue2"
      }
    ],
    "double": [
      {
        "name": "attributeDouble1",
        "value": 1.0
      }
    ],
    "date": [
      {
        "name": "Reingested on",
        "value": 1613986076000
      }
    ]
  }
}

Note that if [attributes][date] already exists, and already contains an array of name/value pairs, I want my new object to be appended to the array. Also, note that [attributes][date] is an array of objects which contain a date in their [value] attribute, as per the mapping of my ElasticSearch index:

  ...
  "attributes": {
    "properties": {
      ...
      "date": {
        "type": "nested",
        "properties": {
          "id": {"type": "keyword"},
          "name": {"type": "keyword"},
          "value": {"type": "date"}
        }
      }, 
      ...
    }
  },
  ...

I tried the following logstash configuration, with no success:

filter {
  # See https://stackoverflow.com/questions/30309096/logstash-check-if-field-exists : this is supposed to allow to "test" if [@metadata][reingested-on] exists
  mutate {
    add_field => { "[@metadata][reingested-on]" => "None" }
    copy => { "[emailHeaders][reingested-on][0]" => "[@metadata][reingested-on]" }
  }
  if [@metadata][reingested-on] != "None" {
    # See https://stackoverflow.com/questions/36127961/append-array-of-json-logstash-elasticsearch: I create a temporary [error] field, and I try to append it to [attributes][date]
    mutate {
      add_field => { "[error][name]" => "Reingested on" }
      add_field => { "[error][value]" => "[@metadata][reingested-on]" }
    }
    mutate {
      merge => {"[attributes][date]" => "[error]"}
    }
  }
}

But what I get is:

{
  "emailHeaders": {
    "reingested-on": ["1613986076000"]
  }, 
  "error": {
    "name": "Reingested on",
    "value": "[@metadata][reingested-on]"
  },
  "attributes": {
    "string": [
      {
        "name": "attributeString1",
        "value": "attributeStringValue1"
      },
      {
        "name": "attributeString2",
        "value": "attributeStringValue2"
      }
    ],
    "double": [
      {
        "name": "attributeDouble1",
        "value": 1.0
      }
    ]
  }
}

My temporary [error] object is created, but its value is wrong: it should be 1613986076000 instead of [@metadata][reingested-on]

Also, it is not appended to the array [attribute][date]. In this example, this array does not exist, so I want it to be created with my temporary object as first element, as per the expected result above.

PositronBeam
  • 136
  • 1
  • 6
  • To get the value of a field you need to use the sprintf syntax: `add_field => { "[error][value]" => "%{[@metadata][reingested-on]}" }` – apt-get_install_skill Mar 03 '21 at 17:11
  • The merge action wont work even if the field would exist. As the documentation states, merging a hash (the error field) ino an array (attributes.date) is not possible. See https://www.elastic.co/guide/en/logstash/5.6/plugins-filters-mutate.html#plugins-filters-mutate-merge – apt-get_install_skill Mar 03 '21 at 17:16
  • I guess you need to implement this behaviour yourself using the ruby filter. It gives you much more flexibility since you can execute arbitrary ruby code with it. – apt-get_install_skill Mar 03 '21 at 17:19

0 Answers0