How to insert an already created json-format string to Elasticsearch Bulk

Question

In a python script,

I'm trying for elasticsearch.helpers.bulk to store multiple records.

I will get a json-format string from another software, and I want to attach it in the source part

I got the helpers.bulk format by this answer

part of my code:

def saveES(output,name):
    es = Elasticsearch([{'host':'localhost','port':9200}]) 
    output = output.split('\n')
    i=0
    datas=[]
    while i<len(output):
            data = {
                    "_index":"name",
                    "_type":"typed",
                    "_id":saveES.counter,
                    "_source":[[PROBLEM]]
            }
            i+=1
            saveES.counter+=1
            datas.append(data)

    helpers.bulk(es, datas)

I would like to attach a json-format string in [[PROBLEM]]

How can I attach it in? I have tried hard, but it is not output in the correct..

if I use:

"_source":{
"image_name":'"'+name+'",'+output[i]
}

and print data result is:

{'_type': 'typed', '_id': 0, '_source': {'image_name': '"nginx","features": "os,disk,package", "emit_shortname": "f0b03efe94ec", "timestamp": "2017-08-18T17:25:46+0900", "docker_image_tag": "latest"'}, '_index': 'name'}

This result show that combined into a single string.

but I expect:

{'_type': 'typed', '_id': 0, '_source': {'image_name': 'nginx','features': 'os,disk,package', 'emit_shortname': 'f0b03efe94ec', 'timestamp': '2017-08-18T17:25:46+0900', 'docker_image_tag': 'latest'}, '_index': 'name'}

score 0 · Answer 1 · answered Aug 18 '17 at 06:29

0

There is many problems in your code.

You override the value of data in your loop
You don't respect any norms (Pesp8 and stuff)
You are while instead of a comprehension list
You created 2 useless variable
You instantiate your es in your function

Here is your improved code

es = Elasticsearch([{'host':'localhost','port':9200}]) # You don't have to initialise this variable every time you are calling the function but only once.


def save_es(output,es):  # Peps8 convention
    output = output.split('\n') # you don't need a while loop. A comprehension loop will avoid a lot of trouble
    data = [    # Please without s in data
       {
          "_index": "name",
          "_type": "typed",
          "_id": index,
          "_source": {
              "image_name":"name" + name}
        }
        for index, name in enumerate(output)
    ]    
    helpers.bulk(es, data)

save_es(output, es)

Hope this help.

answered Aug 18 '17 at 06:29

mel

2,730
8
35
70

thank you, but I already try that. if I use **"image_name":"name"+output[i]** code, then data is **{'_type': 'typed', '_id': 3, '_source': {'image_name': '"nginx","feat": "foo", "os": 'foo''}, '_index': 'name'}**. However, correct data is **{'_type': 'typed', '_id': 3, '_source': {'image_name': 'nginx', 'feat': 'foo', 'os': 'foo'}, '_index': 'name'}** – haeny Aug 18 '17 at 06:59
Well your 'correct' result is not even a JSON document there is a coma problem. Moreover I should see something like 'image_name':'name' in your JSON following your code and my code, but none have it. – mel Aug 18 '17 at 07:03
sorry. Actual results are very long, so I made a simple correction and it seems that a typo occurred. In simple terms, **"image_name": "name1" + name2** In this case, name1 and name2 are combined into a single string. Is the 'name' value in your code a String? – haeny Aug 18 '17 at 07:16
The `name`value is what is present in your output so I guess a string – mel Aug 18 '17 at 07:18
Yes, `name` is string .. oh .. I think my English skills are not enough, so communication seems to be hard... **two string are merged into a single string**, that is, all the data is stored in image_name as a string. I don't know if I delivered it correctly..... Do you use TeamViewer? I want to show the execution result screen in presentation mode. – haeny Aug 18 '17 at 08:07
Well I'm not on a computer now. Just try to give an example (simplify) of your input and what you expect as a result in data (to be indexed by ES) I will provide you with a working script to process your output. – mel Aug 18 '17 at 08:13

How to insert an already created json-format string to Elasticsearch Bulk

1 Answers1