Inserting Multiple Records at a time into Model

Question

I am looking for some best practices when it comes to inserting data into a model, especially when there are a lot of records to be created, so far I am retrieving some XML and saving it into a model

doc = Nokogiri::XML.parse(open(url))
doc.xpath('//xmlns:feed/xmlns:entry[xmlns:title[node()]]').each do |s|

  cid = s.xpath("xmlns:id").text
  email = s.xpath("gd:email/@address").text
  name = s.xpath("xmlns:title").text

  data = Contact.new(
    :cid => cid,
    :email => email,
    :name => name)
  data.save
end

Now this is inserting the records in one by one which is taking too long in my opinion.

I have read that one solution is to use transactions or I could do a single mass insert? My question is which one would I benefit the most from and how would i re format what i already have to each of these? Seeing an example of my current setup into the new setup would benefit me as I would be able to understand it more and actually learn from it

Any help appreciated

Thanks

yes kind of , however this should not be closed as @JeffPaquette supplied a working example for the code presented in question, which differs from the default example provided by the GEM — Richlewis, Apr 04 '13 at 14:16
You should be able to figure that out yourself. In my opinion it should be closed. That this works for your particular situation is not going to be of any extra benefit to other users with the same problem. For them the original question is clear enough. — Mischa, Apr 04 '13 at 14:19
i wasnt sure hence the reason i asked......and how can YOU say it wont benefit any other users, what is easy for you isn't necessarily easy for someone else — Richlewis, Apr 04 '13 at 14:22

score 2 · Accepted Answer · edited May 23 '17 at 11:52

2

Someone else has tackled this, and the solution was to use activerecord-import. See original question...

Details on using activerecord-import are on it's wiki

Edit: Apparently the link button won't duplicate the link as title if you don't highlight text.

doc = Nokogiri::XML.parse(open(url))
data = []
doc.xpath('//xmlns:feed/xmlns:entry[xmlns:title[node()]]').each do |s|

  cid = s.xpath("xmlns:id").text
  email = s.xpath("gd:email/@address").text
  name = s.xpath("xmlns:title").text

  data << Contact.new(
    :cid => cid,
    :email => email,
    :name => name)
end
Contact.import data

edited May 23 '17 at 11:52

Community

1
1

answered Apr 04 '13 at 13:49

Jeff Paquette

7,089
2
31
40

thanks, still unsure how to implement with my setup though, also link provided is broke (i didnt -1 you) could you provide an example please? – Richlewis Apr 04 '13 at 13:52
ok so the example is using 10.times do, i dont know how many records are going to be retrieved, so its a case of n.times...how would i set that? – Richlewis Apr 04 '13 at 14:02
ah i see, thanks, now that helps, i can see whats going on... – Richlewis Apr 04 '13 at 14:10
Wow, massive improvement there on writing to the Model +1 – Richlewis Apr 04 '13 at 14:20
How much improvement did you see? – Jeff Paquette Apr 04 '13 at 14:21
from about 4-5 secs to 1 second, big gains, and that was for around 60 records – Richlewis Apr 04 '13 at 14:22

score 1 · Answer 2 · answered Apr 04 '13 at 16:18

You might also try upsert, which will allow you to insert the records quickly (in some tests, faster than activerecord-import) without first accumulating them in memory:

require 'upsert'
# [...]
doc = Nokogiri::XML.parse(open(url))
Upsert.batch(Contact.connection, Contact.table_name) do |upsert|
  doc.xpath('//xmlns:feed/xmlns:entry[xmlns:title[node()]]').each do |s|
    cid = s.xpath("xmlns:id").text
    email = s.xpath("gd:email/@address").text
    name = s.xpath("xmlns:title").text
    upsert.row(
      :cid => cid,
      :email => email,
      :name => name
    )
  end
end

This works for MySQL, Postgres, and SQLite3.

Inserting Multiple Records at a time into Model

2 Answers2

Linked