MySQL nested JSON column search and extract sub JSON

Question

I have a MySQL table authors with columns id, name and published_books. In this, published_books is a JSON column. With sample data,

id | name  | published_books
-----------------------------------------------------------------------
1  | Tina  |  {
   |       |    "17e9bf8f": {
   |       |         "name": "Book 1",
   |       |         "tags": [
   |       |             "self Help",
   |       |             "Social"
   |       |         ],
   |       |         "language": "English",
   |       |         "release_date": "2017-05-01"
   |       |     },
   |       |      "8e8b2470": {
   |       |          "name": "Book 2",
   |       |          "tags": [
   |       |              "Inspirational"
   |       |          ],
   |       |          "language": "English",
   |       |          "release_date": "2017-05-01"
   |       |      }
   |       |   }
-----------------------------------------------------------------------
2  | John  |   {
   |       |     "8e8b2470": {
   |       |         "name": "Book 4",
   |       |         "tags": [
   |       |             "Social"
   |       |         ],
   |       |         "language": "Tamil",
   |       |         "release_date": "2017-05-01"
   |       |     }
   |       |   }
-----------------------------------------------------------------------
3  | Keith |   {
   |       |      "17e9bf8f": {
   |       |          "name": "Book 5",
   |       |          "tags": [
   |       |              "Comedy"
   |       |          ],
   |       |          "language": "French",
   |       |          "release_date": "2017-05-01"
   |       |      },
   |       |      "8e8b2470": {
   |       |          "name": "Book 6",
   |       |          "tags": [
   |       |              "Social",
   |       |              "Life"
   |       |          ],
   |       |          "language": "English",
   |       |          "release_date": "2017-05-01"
   |       |      }
   |       |   }
-----------------------------------------------------------------------

As you see, the published_books column has nested JSON data (one level). JSON will have dynamic UUIDs as the keys and its values will be book details as a JSON.

I want to search for books with certain conditions and extract those books JSON data alone to return as the result.

The query that I've written,

select JSON_EXTRACT(published_books, '$.*') from authors 
   where JSON_CONTAINS(published_books->'$.*.language', '"English"')     
   and JSON_CONTAINS(published_books->'$.*.tags', '["Social"]');

This query performs the search and returns the entire published_books JSON. But I wanted just those books JSON alone.

The expected result,

result
--------
"17e9bf8f": {
    "name": "Book 1",
    "tags": [
        "self Help",
        "Social"
    ],
    "language": "English",
    "release_date": "2017-05-01"
}
-----------
"8e8b2470": {
    "name": "Book 6",
    "tags": [
        "Social",
        "Life"
    ],
    "language": "English",
    "release_date": "2017-05-01"
}

Is there any good reason for not storing the data normalized? — Nico Haase, Feb 18 '19 at 08:56
@NicoHaase It's designed such a way to reduce frequent save/update queries. This is just an example, we had a complex big JSONs which we transform into an object with relations, work with that and do a final save. It's also easy to work with the object because it has it's associated objects already loaded (present inside the JSON itself). If it has to be normalized into multiple tables, then it's much difficult to do save/update. This approach is optimized specifically to our domain. — Kanmaniselvan, Feb 18 '19 at 11:19
@Yep_It's_Me, No. I filtered the results in Ruby after loading the records — Kanmaniselvan, Feb 18 '19 at 11:20
@MuhammadTahirQaiser I've found any solution for this. I filtered the results in Ruby after loading the records. — Kanmaniselvan, Feb 18 '19 at 11:20
it is difficult to extract the json row bcz 'where' clause use to return the row which are pass the conditions. In 'select' clause we are not able to extract the part of json bcz your keys are dynamic (17e9bf8f,8e8b2470) if there is some static keys like id:{17e9bf8f,{}} etc then it is possible to fetch specific row by matching there [0] value. — Ashu, Feb 18 '19 at 11:54

Bill Karwin · Accepted Answer · 2019-02-19T15:53:22.720

There is no JSON function yet that filters elements of a document or array with "WHERE"-like logic.

But this is a task that some people using JSON data may want to do, so the solution MySQL has provided is to use the JSON_TABLE() function to transform the JSON document into a format as if you had stored your data in a normal table. Then you can use a standard SQL WHERE clause to the fields returned.

You can't use this function in MySQL 5.7, but if you upgrade to MySQL 8.0 you can do this.

select authors.id, authors.name, books.* from authors,
  json_table(published_books, '$.*' 
  columns(
    bookid for ordinality,
    name text path '$.name',
    tags json path '$.tags',
    language text path '$.language',
    release_date date path '$.release_date')
  ) as books
where books.language = 'English'
  and json_search(tags, 'one', 'Social') is not null;

+----+-------+--------+--------+-------------------------+----------+--------------+
| id | name  | bookid | name   | tags                    | language | release_date |
+----+-------+--------+--------+-------------------------+----------+--------------+
|  1 | Tina  |      1 | Book 1 | ["self Help", "Social"] | English  | 2017-05-01   |
|  3 | Keith |      2 | Book 6 | ["Social", "Life"]      | English  | 2017-05-01   |
+----+-------+--------+--------+-------------------------+----------+--------------+

Note that nested JSON arrays are still difficult to work with, even with JSON_TABLE(). In this example, I exposed the tags as a JSON array, and then use JSON_SEARCH() to find the tag you wanted.

I agree with Rick James — you might as well store the data in normalized tables and columns. You think that using JSON will save you some work, but it's won't. It might make it more convenient to store the data as a single JSON document instead of multiple rows across several tables, but you just have to unravel the JSON again before you can query it the way you want.

Furthermore, if you store data in JSON, you will have to solve this sort of JSON_TABLE() expression every time you want to query the data. That's going to make a lot more work for you on an ongoing basis than if you had stored the data normally.

Frankly, I have yet to see a question on Stack Overflow about using JSON with MySQL that wouldn't lead to the conclusion that storing data in relational tables is a better idea than using JSON, if the structure of the data doesn't need to vary.

somebody know how could extract the values on the column tags and show them as: tags ------ self Help Social — NeDiaz, Mar 12 '21 at 20:34

score 2 · Answer 2 · answered Feb 18 '19 at 18:58

You are approaching the task backwards.

Do the extraction as you insert the data. Insert into a small number of tables (Authors, Books, Tags, and maybe a couple more) and build relations between them. No JSON is needed in this database.

The result is an easy-to-query and fast database. However, it requires learning about RDBMS and SQL.

JSON is useful when the data is a collection of random stuff. Your JSON is very regular, hence the data fits very nicely into RDBMS technology. In that case, JSON is merely a standard way to serialize the data. But it should not be used for querying.

This answer tell us why people before Edison burns oil to get light..., Solving problems in different ways brings us here, not to stick with old solutions. — Mamrezo, Feb 17 '21 at 09:44

MySQL nested JSON column search and extract sub JSON

2 Answers2

Linked