1

I'm dipping my toe into Spring and I've been using the SimpleJdbcTemplate to help reduce the amount of code I need to write but I now have an issue where the exception "java.lang.OutOfMemoryError: GC overhead limit exceeded" is thrown. I've been pulling the categories out of eBay using their web services and I've been inserting each category (about 10'000 records I think) using a call jdbTemplate.update (see code below)

CategoryService:

// If the category already exist (i.e. an error is throuwn as the version must be unique) than do now bother sotring the categories
for(CategoryType ct : categories)
{
    try
    {
        // TODO: Determine why a child ategory can be tied to multiple parents, for now just use the first category in the array
        categoryDao.SaveCategory(ct.getCategoryID(), ct.getCategoryName(), ct.getCategoryParentID()[0]);
    }
    catch(DuplicateKeyException e)
    {
        // No problem here, the version description is the same... just continue as if nothing happened
    }
}

CategoryDaoImpl: (an implementation of the CategoryDao Interface)

@Override
public int SaveCategory(String ebayCategoryId, String ebayCategoryName, String ebayParentCategoryId) 
{
    // Firstly grab the next value in the categoru sequence
    int internalCategoryId = jdbcTemplate.queryForInt(categorySequenceStatement);

    // Insert the new category
    jdbcTemplate.update(insertCategoryStatement, new Object[] {internalCategoryId, ebayCategoryId, ebayCategoryName, ebayParentCategoryId});

    return internalCategoryId;
}

Environment:

  • Spring Framework 3.0.2
  • Oracle Database XE (11g I think!) (using ojdbc6.jar)
  • JDK (jdk1.6.0_26)

I had though of using the batchUpdate method on SimpleJdbcTemplate but I'm unsure of whether there is an underlying issue here.

Any help would be appreciated!

raven-king
  • 1,550
  • 2
  • 18
  • 40
  • 1
    Check this http://stackoverflow.com/questions/1393486/what-does-the-error-message-java-lang-outofmemoryerror-gc-overhead-limit-exceed I guess batchUpdate will be a good option – Dhananjay Jul 27 '11 at 09:40

1 Answers1

1

Stop loading all the categories into memory at once. Process each category as it's loaded. It will be at least an order of magnitude faster and won't cause OOMEs.

Ryan Stewart
  • 126,015
  • 21
  • 180
  • 199
  • hi Ryan and thanks for your reply. Could you please elaborate further with a code sample as I'm not entirely store what you mean. Ebay returns 10'000+ records at a time which will exist in memory post api call. – raven-king Jul 29 '11 at 10:03
  • 1
    @JLove: You said it's a web service, implying it's an HTTP request of some sort. HTTP always (almost) streams. You don't have to wait for the whole response to come back. You can start parsing it as soon as you start receiving it. It's more work to do it that way, but it will absolutely give you a huge performance boost. I'll try to get some sample code together, but it will only be very general. – Ryan Stewart Jul 29 '11 at 15:32