-1

I have a large json file with a collected of structured location data (coordinates, category etc.) and I need to create key-value pairs in redis for further querying. I have been able to find bits of information here and there but I am struggling to put it all together into something usable.

This is the general structure of the json;

{ 
    "_id" : NumberInt(83412), 
    "contact" : {
        "GooglePlaces" : null, 
        "Foursquare" : "https://foursquare.com/v/caf%C3%A9-vavin/4adcda06f964a520eb3221e3"
    }, 
    "name" : "Café Vavin", 
    "location" : {
        "city" : "Paris", 
        "coord" : {
            "coordinates" : [
                2.3307001590729, 
                48.843868593865
            ], 
            "type" : "Point"
        }, 
        "address" : "18 rue Vavin"
    }
Byron Barkhuizen
  • 1
  • 2
  • 1
    Please specify the structure of your Json data by providing a sample. If you have any code please specify that as well. Did you go through Redis Mass Insertion: https://redis.io/topics/mass-insert? – kayvis Apr 30 '20 at 18:35
  • I've added a sample to the original post. I have had a look at your link and it seems a little beyond what I need to do, but I will see if there's something in ther that can help me. – Byron Barkhuizen Apr 30 '20 at 19:29
  • In your Redis, what will be the key and the value? Is key the `id` from your sample Json and the value be rest of all the other attributes? Or is that the question? What are the queries that you're looking to run on Redis? Based on that you can come up with the key-value in Redis. – kayvis Apr 30 '20 at 19:34
  • That is essentially the question. From what I understand the value will always be 'the rest' in this case. I am lookign to answer this question; "build a key-value Redis database to have a stream of reviews by source, by category and to compute efficiently some statistics such as the average rating and standard deviation". I need to import the massive json file I have, and at the same time create key-value pairs that will allow me to optimally query those values. – Byron Barkhuizen Apr 30 '20 at 20:07
  • This is a multi-step process. Plus there are some 'extra' things that you'll have to handle. Let me try and answer. See if that helps. – kayvis Apr 30 '20 at 20:19

1 Answers1

1

Given that your Json data is nested/multi-level you cannot store them in a Redis Hash directly. So you'd have to serialise the value and store it as a string (other formats would also work, but let's use string for simplicity to start with). This would mean that when you get the value from Redis you'd have to deserialise the Json (stored as string) to the appropriate object in your code. Hence, the key would be the id and the value would be the rest of the Json as a string.

Now, moving on to loading the huge Json into Redis, it is a two-step process:

  1. Parse your input Json file and break it into the format that can be used for 'Redis Mass Insertion' - https://redis.io/topics/mass-insert. You have to extract the value of _id and use it as the key and extract the rest of the Json and use it as the value as per the format shown in the link above.
  2. Run Redis mass insertion with the contents from the previous step as the input. You can add each of the input into a single set in Redis.

Now you can run get queries on the set if you have the id. Remember, Redis is an in-memory cache which is tuned for faster storage and retrieval. You cannot run complex queries on Redis unless you store/duplicate the data in different ways for different querying needs.

Feel free to comment on this answer in case you want me to add more details. There's only so much I could reply in the comments section of the question. Hence I consolidated everything into this answer. I can add more to this if you want me to address anything specific.

More references and samples for mass insert:

kayvis
  • 563
  • 2
  • 9