Is there a way to call a Python script from within Node.js and show it's results as soon as they are generated?

Question

I would like to execute a Python Script from within a Node.js app (Electron.js to be specific). I would like to show output as soon as it's generated. The Python script is a large one that takes a lot of time to process but it outputs data regularly.

I've tried this using python-shell and child_process. However when I execute the Python file the output is shown only when the program has ended.

I thought this could be done using shell.on('message',function(){}) or scriptExecution.stdout.on('data',function()) but apparently this is not the case.

Is this possible? How can I do it? Maybe using some other way...

I bet is possible - at least from Python it is. Either those hooks wait till the end, or maybe you need to flush the output on the Python side. I guess former. In Python you can open a process and read stdout from there as it comes. Maybe It's a different technique in node. — antont, Feb 06 '20 at 23:33
@antont thanks for your response, it does work. I will post a full answer to clarify. — loco.loop, Feb 07 '20 at 13:46

score 6 · Accepted Answer · answered Feb 07 '20 at 14:33

As @antont pointed it out, obtaining Python results as soon as they appear on stdout this is easily done using a flush mechanism.

How to do it

I've tested 3 ways to do it:

Inside Python code, pass a key word parameter to print:
```
print('text', flush=True)
```

Inside Python code, using an explicit flush:

import sys
# Do this every time you want to flush
sys.stdout.flush()

When calling the Python executable, give it the option to always flush:
```
python -u scriptName.py
```
_{(see below for two examples using python-shell and child_process.}

Node.js Examples

Using python-shell

The key part of this example is '-u' in pythonOptions: ['-u'], if you remove this option Python won't automatically flush (unless you use methods 1 or 2 from above).

let PythonShellLibrary = require('python-shell');
let {PythonShell} = PythonShellLibrary;
let shell = new PythonShell('/home/user/showRandomWithSleep.py', {
    // The '-u' tells Python to flush every time
    pythonOptions: ['-u']
});
shell.on('message', function(message){
    window.console.log('message', message);
    window.console.log(new Date())
})

Using child_process

The key part of this example is '-u' in spawn(pythonExecutable, ['-u', myPythonScript]), if you remove this option Python won't automatically flush (unless you use methods 1 or 2 from above).

var myPythonScript = "/home/user/showRandomWithSleep.py";
var pythonExecutable = "python";
var uint8arrayToString = function(data) {
    return String.fromCharCode.apply(null, data);
};
const spawn = require('child_process').spawn;
// The '-u' tells Python to flush every time
const scriptExecution = spawn(pythonExecutable, ['-u', myPythonScript]);
scriptExecution.stdout.on('data', (data) => {
    console.log(uint8arrayToString(data));
    window.console.log(new Date())
});

showRandomWithSleep.py, the python file used in the above examples

from random import *
import time
for i in range(5):
    print("showRandomWithSleep.py")
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    print(random())
    time.sleep(random()*5)

Note

I tested the above examples and the results differ a little.

When using python-shell the prints are outputted for every print() line. However, when using child_process the prints are outputted in blocks. I don't know why this happens.