How to manipulate leaves of a JSON tree

Question

I want to replace rare words with _RARE_ in a JSON tree using JAVA.

My rareWords list contains

late  
populate
convicts

So for JSON below

["S", ["PP", ["ADP", "In"], ["NP", ["DET", "the"], ["NP", ["ADJ", "late"], ["NOUN", "1700<s"]]]], ["S", ["NP", ["ADJ", "British"], ["NOUN", "convicts"]], ["S", ["VP", ["VERB", "were"], ["VP", ["VERB", "used"], ["S+VP", ["PRT", "to"], ["VP", ["VERB", "populate"], ["WHNP", ["DET", "which"], ["NOUN", "colony"]]]]]], [".", "?"]]]]

I should get

["S", ["PP", ["ADP", "In"], ["NP", ["DET", "the"], ["NP", ["ADJ", "_RARE_"], ["NOUN", "1700<s"]]]], ["S", ["NP", ["ADJ", "British"], ["NOUN", "_RARE_"]], ["S", ["VP", ["VERB", "were"], ["VP", ["VERB", "used"], ["S+VP", ["PRT", "to"], ["VP", ["VERB", "populate"], ["WHNP", ["DET", "which"], ["NOUN", "colony"]]]]]], [".", "?"]]]]

Notice how

["ADJ","late"]

was replaced by

["ADJ","_RARE_"]

My code so far is like below:

I recursively iterate over the tree and as soon as rare word is found, I create a new JSON array and try to replace the existing tree's node with it. See // this Doesn't work in below, that is where I got stuck. The tree remains unchanged outside of this function.

public static void traverseTreeAndReplaceWithRare(JsonArray tree){   

        //System.out.println(tree.getAsJsonArray()); 

        for (int x = 0; x < tree.getAsJsonArray().size(); x++)
        {
            if(!tree.get(x).isJsonArray())
            {
                if(tree.size()==2)
                {   
                //beware it will get here twice for same word
                 String word= tree.get(1).toString();  
                 word=word.replaceAll("\"", ""); // removing double quotes

                 if(rareWords.contains(word))
                 {
                 JsonParser parser = new JsonParser();                   

                             //This works perfectly 
                             System.out.println("Orig:"+tree);
                 JsonElement jsonElement = parser.parse("["+tree.get(0)+","+"_RARE_"+"]");

                 JsonArray newRareArray = jsonElement.getAsJsonArray();

                             //This works perfectly 
                             System.out.println("New:"+newRareArray);

                 tree=newRareArray; // this Doesn't work
                 }                 

                }               
                continue;   
            }
            traverseTreeAndReplaceWithRare(tree.get(x).getAsJsonArray());
        }
    }

code for calling above, I use google's gson

JsonParser parser = new JsonParser();
JsonElement jsonElement = parser.parse(strJSON);
JsonArray tree = jsonElement.getAsJsonArray();  

Answer 1

Here's a straight forward approach in C++:

#include <fstream>
#include "JSON.hpp"
#include <boost/algorithm/string/regex.hpp>
#include <boost/range/adaptors.hpp>
#include <boost/phoenix.hpp>

static std::vector<std::wstring> readRareWordList()
{
    std::vector<std::wstring> result;

    std::wifstream ifs("testcases/rarewords.txt");
    std::wstring line;
    while (std::getline(ifs, line))
        result.push_back(std::move(line));

    return result;
}

struct RareWords : boost::static_visitor<> {

    /////////////////////////////////////
    // do nothing by default
    template <typename T> void operator()(T&&) const { /* leave all other things unchanged */ }

    /////////////////////////////////////
    // recurse arrays and objects
    void operator()(JSON::Object& obj) const { 
        for(auto& v : obj.values) {
            //RareWords::operator()(v.first); /* to replace in field names (?!) */
            boost::apply_visitor(*this, v.second);
        }
    }

    void operator()(JSON::Array& arr) const {
        int i = 0;
        for(auto& v : arr.values) {
            if (i++) // skip the first element in all arrays
                boost::apply_visitor(*this, v);
        }
    }

    /////////////////////////////////////
    // do replacements on strings
    void operator()(JSON::String& s) const {
        using namespace boost;

        const static std::vector<std::wstring> rareWords = readRareWordList();
        const static std::wstring replacement = L"__RARE__";

        for (auto&& word : rareWords)
            if (word == s.value)
                s.value = replacement;
    }
};

int main()
{
    auto document = JSON::readFrom(std::ifstream("testcases/test3.json"));

    boost::apply_visitor(RareWords(), document);

    std::cout << document;
}

This assumes you wanted to do replacements on all string values, and only matches whole strings. You could easily make this case insensitive, match words inside strings etc. by changing the regex or regex flags. Slightly adapted in response to the comments.

The full code including JSON.hpp/cpp is here: https://github.com/sehe/spirit-v2-json/tree/16093940