Fastest way to iterate over multiple lists

Question

I'm trying to get 2 giant text files to combine their data into one text file.

One text file containing 200 000 - 500 000 lines of [node_id, x, y, z, temperature] and the other a 203 000 line file which contains vectors.

The idea is to identify the nodes from the one file, to the vectors in the other file, and combine them into a new text file. I've managed to get both files into blocks of 3-6k lines to manage times, but I can only think of:

A for loop on the vectors to get starting and ending coordinates.
A for loop every node to check the coordinate to the box of the vector.
Calculate the distance of the nodes to the vector and select the vectors therein.
get the next vector.

This still creates nestled for loops of 5000 iterations in one for loop, in which another 5000 iterations for loop is nestled. Is there a faster way to do this?

Example of the node file: [node_id, x, y, z, temp]

[21, -10.0, -12.0, 4.0, 160,0]

Example of the vector file: [vector_time, x, y, z, wattage]

[8.83, -9.82, -3.16, 0.05. 150.00]

I've tried:

with open(rf'Node_Temp_result.txt', 'r') as nodeFile:
    nodesInfo = nodeFile.readlines()

with open(rf'laser.txt', 'r') as laserFile:
    laserInfo = laserFile.readlines()
    
for laser in tqdm(range(currentLaserMin, currentLaserNext)):
        local_laser = Split(laserInfo, laser)
        laser_search = Laser(local_laser.col0(), local_laser.col1(), local_laser.col2(), local_laser.col3(), local_laser.col4())
        splits = 0

        x_search_laser = laser_search.laser_x()
        y_search_laser = laser_search.laser_y()
        z_search_laser = laser_search.laser_layer()

        if layer - 1 < z_search_laser <= layer and laser_search.laser_power() > 0:
            laser_next = Split(laserInfo, laser + 1)
            laser_search_next = Laser(laser_next.col0(), laser_next.col1(), laser_next.col2(), laser_next.col3(), laser_next.col4())

            x_start = x_search_laser
            x_end = laser_search_next.laser_x()

            y_start = y_search_laser
            y_end = laser_search_next.laser_y()

            num = 0

            for node in range(currentNodeMin, currentNodeNext):
                local_node = Split(nodesInfo, node)
                node_search = Node(local_node.col0(), local_node.col1(), local_node.col2(), local_node.col3(), local_node.col4())

                x_search_node = node_search.node_x()
                y_search_node = node_search.node_y()
                z_search_node = node_search.node_layer()

                if node_search.node_temp() > 250:
                    if layer - 1 < z_search_node <= layer:
                        if x_start <= x_search_node <= x_end and y_start <= y_search_node <= y_end:

                            crossLocations = cross_finder(x_start, x_end, y_start, y_end)

                            if crossLocations[3] < 0.07:
                                splits += 1
                                'append the node to the vector'

Welcome to SO. Please include an example of your text file and some code (that you've tried; if you have any). And please ask multiple questions in seperate posts to help answers be more focused towards the post — Freddy Mcloughlan, Jun 13 '22 at 07:57
The fastest way to iterate over multiple lists is multiprocessing: https://docs.python.org/3/library/multiprocessing.html — Vitaly Olegovitch, Jun 13 '22 at 07:57
The fastest way to do this job is to read the text files into dataframes. https://www.geeksforgeeks.org/how-to-read-text-files-with-pandas/#:~:text=We%20can%20read%20data%20from,of%20a%20comma%20by%20default. Then merge: https://stackoverflow.com/questions/18792918/combine-two-pandas-data-frames-join-on-a-common-column Then write to text: https://stackoverflow.com/questions/31247198/python-pandas-write-content-of-dataframe-into-text-file — Amiga500, Jun 13 '22 at 08:02
Use a library that is already providing fast functions for reading data and processing it in an efficient representation (more efficient than python lists!), like numpy, pandas, dask, etc. — creanion, Jun 13 '22 at 08:16

Vitaly Olegovitch · Accepted Answer · 2022-06-13T08:34:42.333

Some options that you could consider to do this task are:

using Python multiprocessing, that will allow you to run the tasks in parallel: https://docs.python.org/3/library/multiprocessing.html
using dask dataframes: https://www.dask.org/ It is a very easy to use library with a very good performance.
using pola.rs dataframes: https://www.pola.rs/ Polars has less features that dask and worse documentation, but should have better performances.
using PySpark to run your task in a cluster: https://spark.apache.org/docs/latest/api/python/
use a faster programming language, for example Rust, C or C++.

Fastest way to iterate over multiple lists

1 Answers1