4

I am in ongoing development of a simple but processor intensive computer program that I use for scientific research. It would be really helpful to have more processing speed. Right now I'm running Mac OS X 10.5.7 on a 2 GHz Intel Core Duo and my program runs pretty slow. For a while I was running it on a considerably older laptop with Ubuntu and it ran ~20 times faster.

What I'm thinking is that I could hook up over LAN a bunch of cheap second-hand PCs running linux, and use threaded programing to distribute the processing load across the computers. My program is embarrassingly parallel, ie linear increases in the number of cores should lead to near linear increases in computing speed. Also, program size, memory requirements, and HD requirements are all practically nil, and the amount of information to be passed to and from the main routine (between computers) is basically zero.

So what I'm wondering is, what sort of road blocks are likely to stand in the way of implementing this? Should it be pretty straightforward or no? Will it even work? Also, in purchasing the used computers, what factors should I take into account? For my purposes, is a dual core 2GHz ~= to a single core 4GHz? I figure that as a rule of thumb, I should go with the cheapest multicore-processor machines I can find (best processing to cost ratio).

Matt Munson
  • 153
  • 9

3 Answers3

3

Warning: Most knowledge here is theoretical, and needs to be backed up with facts based off specific needs

You probably could/should set up a cluster. If you're going for new systems, the cheapest multicore systems sounds like a good bet (if you go for old ones AVOID PENTIUM IVs, unless you want to heat the room more efficiently than you process. I would consider a dual core 2 ghz system slightly better than a single 4 ghz processor, unless you had a single thread that requires a lot of power (which you don't). I'd note that in general, newer versions of mainstream processors do generally tend to be better than the ones they replace, especially recently since power use has become an issue.If i wanted to get down to the nitty gritty of it, i'd look at the various generations of processors, and benchmarks for them to get a clearer idea of efficiency and power.

You'd probably want to consider booting the worker systems with a minimal os, off standard images via network using PXE to save on storage for each individual system - since your ram requirements are minimal, you could probably have the entire OS in ram, since you're only using it for one thing

You would need to probably have software that is capable of being run on a cluster (IE, take that into account when designing it - maybe with MPI, have the cluster software setup and so on. That being said, i have seen MASSIVE arrays of commodity PCs linked together at a university for a cluster so its plausible

On the other hand, considering massive parallelisation is your main goal, and you're going for massively parallel workloads , you might want to consider alternatives to a x86 cluster - ubuntu has a 48 core cluster that they built out of arm development boards (I'm thinking a cluster of raspberry pi systems would be... interesting, and quite efficient powerwise, if you didn't mind a massive array of odd little inexpensive systems).

Journeyman Geek
  • 133,878
2

Journeyman Geek has very good points about setting up your own cluster but I would also like to add, depending on your workload it may be cheaper to use something like Elastic Map-Reduce. If your workload can be turned in to a Map-Reduce problem you should look in to it. After factoring in the cost of buying the computers, electric, cooling, and maintenance costs it may be cheaper/hr to just run it on amazon ($0.10/hr for the lowest standard tier from Amazon, $0.045/hr if you use the current spot instance price))

If you are willing to do a little more effort you can use your own parallel processing engine and use the micro instance for even cheaper ($0.02/hr for a full instance, $0.007/hr current spot price). Amazon's own Map-Reduce package will not run on it. This can be very useful if your process is cpu bound instead if memory bound as micro instances are faster than the small standard image but have no storage and less ram.

Scott Chamberlain
  • 31,164
0

Another option you may want to consider is running your program inside of BOINC. This has the advantage of an existing system that already has the hard part of distributing and collecting "jobs". This also allows you to get more people to run your program. You can get friends or even random people on the internet to run your program if you advertise it enough. This allows you to have a (potentially) massive cluster of computers to run programs on without the startup cost of lots of small computers and ongoing electricity costs.

Note that I've never programmed on BOINC, so difficulty of doing so is something your going to have to find out. For a starting platform though, I would think the time saved not having to reinvent BOINC would balance the complexity of using it

TheLQ
  • 2,917