Super User
Super User

Questions tagged [parallel-processing]

Running multiple commands (or even computers) in parallel for more efficiency.

114 questions
63
votes
4 answers

How can my computer use 300% PC processing power?

I'm playing around with a script and well I got carried away (it's still running right now and yet my computer has not yet died). I'm running 17 iterations of the script (with the intention of crashing my computer). Looking at the activity monitor…
Frankenmint
  • 805
61
votes
12 answers

Can I run my CPU at 100% usage for a long time?

I have a Dell XPS 15 9570, 32 GB ram, 2.2 ghz and 6 cores. When i use an application (The parsim command on MATLAB) (for more details) then my fan starts running fast , during this my PC doesn't seem to be heating but it never got this noisy in the…
aadil095
  • 737
37
votes
9 answers

Parallelizing rsync

I just moved and discovered after some trial and error that somewhere between my home and my remote server, there is some throttling going on...but the throttling is not very intelligent. It only throttles individual connections. So if I copy one…
stuyguy
  • 611
30
votes
3 answers

How to use Ctrl+C to kill all background processes started in a Bash script?

ArchLinux (Manjaro). I'm running one bash file. It runs 2 processes (commands), using &. But when I press Ctrl+C to stop the program - one process dies, and the other continues to work. How do I stop both processes? Or how do I write a new script…
Ievgen
  • 403
29
votes
7 answers

What's the best way to perform a parallel copy on Unix?

I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network overhead a regular copy cp remote_folder/*…
dsg
  • 1,199
26
votes
7 answers

Running commands in parallel with a limit of simultaneous number of commands

Sequential: for i in {1..1000}; do do_something $i; done - too slow Parallel: for i in {1..1000}; do do_something $i& done - too much load How to run commands in parallel, but not more than, for example, 20 instances per moment? Now usually using…
Vi.
  • 17,755
26
votes
5 answers

Is it possible to combine processing power of 2 computers?

Here are a few questions, I wish you could enlighten me. Is it possible to combine processing power of 2 computers? How do I do it?
Wern Ancheta
  • 6,620
22
votes
0 answers

Jenkins realtime console output for parallel threaded fabric command

We have a Python fabric command that runs in parallel across several hosts, something like this: $ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache This will copy cache to the production servers listed in parallel. There is…
mVChr
  • 405
  • 3
  • 9
19
votes
2 answers

Sort --parallel isn't parallelizing

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are two files, about 3 gigs each, 50% aren't…
Jeremy Kemball
  • 293
  • 1
  • 2
  • 6
17
votes
6 answers

Running multiple concurrent background processes then wait for all of them to terminate, on Windows

Anybody know if there is cmd script that does that and/or a command line tool ready for the job?
Sergio
  • 295
16
votes
5 answers

Is a higher core count or higher clock speed more beneficial to a computer's performance?

With lowering silicon costs and rising consumer needs, manufacturers seem to be pushing one of two things: clock speed and/or core count. With the way things are going, it doesn't seem that clock speed of processors is rising anymore, but the…
Breakthrough
  • 34,847
16
votes
3 answers

netcat as a multithread server

I use netcat to run a simple server like this: while true; do nc -l -p 2468 -e ./my_exe; done This way, anyone is able to connect to my host on port 2468 and talk with "my_exe". Unfortunately, if someone else wants to connect during an open session,…
etuardu
  • 867
15
votes
4 answers

Kill process after it's been allowed to run for some time

I want to limit the time a grep process command is allowed to run or be alive. For example. I want to perform the following: grep -qsRw -m1 "parameter" /var But before running the grep command I want to limit how long the grep process is to live,…
yael
  • 549
11
votes
2 answers

Differentiate CUDA Cores(NVIDIA) and Stream processor(ATI/AMD)

I think the question is answered here but I'm still wondering if what is the difference between the units it is referring to. For example, a GTX 570 has 480 CUDA cores, while the ATI equivalent HD 6970 has roughly 1536 Stream processor. It's quiet…
Mark
  • 197
11
votes
5 answers

Parallel shell loops

I want to process many files and since I've here a bunch of cores I want to do it in parallel: for i in *.myfiles; do do_something $i `derived_params $i` other_params; done I know of a Makefile solution but my commands needs the arguments out of…
math
  • 2,693
1
2 3 4 5 6 7 8