Best way to loop through a list within a sub-loop

Question

First things first, I did a search before posting this question but the answers are so generic that I could not find anything similar to what I'm trying to do. I apologize in advance if this has been answered before and I did not find it.

I have a scenario where I already collected a large list of rows (>10K) from the SQL server and put into an array (List) of strings. These rows are consisted of filenames. Because I already put them on a list, I don't want to query the SQL server again and instead want work with what I already have in memory.

This is the code I'm trying to get right:

'Valid file types for InfoLink1, InfoLink2, InfoLink3

        Dim lstValidImageFormats As New List(Of String)({".JPG", ".JPEG", ".JPE", ".BMP", ".PNG", ".TIF", ".TIFF", ".GIF"})
        Dim lstValidSTLFormats As New List(Of String)({".STL"})
        Dim lstValidSTEPFormats As New List(Of String)({".STP", ".STEP"})

        '////////////////
        '// Components //
        '////////////////

        'We don't check Parts.InfoLink because all formats are allowed in this field

        'Parts.InfoLink1 - File in Infolink1 columns MUST BE images

        For i = 0 To arrStrComponentsInfolink1Values.Count - 1 'We have 10K rows with filenames in this list
            For Each FileExtension As String In lstValidImageFormats
                If arrStrComponentsInfolink1Values.Item(i).EndsWith(FileExtension) = False Then


                End If
            Next

        Next

I'm trying to parse each item (filename) I have in the array arrStrComponentsInfolink1Values and check if the filename DOES NOT end with one of the extensions in the list lstValidImageFormats. If it doesn't, then I'll send the filename to another list (array).

My difficulty here is about how to iterate through each item in arrStrComponentsInfolink1Values, then check if each filename ends with one of the extensions declared in lstValidImageFormats, do what I want to do with the item if it DOES NOT end with one of those extensions, and then proceed to parse the next item in arrStrComponentsInfolink1Values.

I sincerely don't know what's the best way/performance efficient to do that.

My code above is empty because algorithmically I don't know the best approach to do what I want without querying the SQL server again with something like AND filename NOT LIKE '%.JPG' AND filename NOT LIKE '%.JPEG' AND filename NOT LIKE '%.JPE' AND filename NOT LIKE '%.BMP'...

Because I already have the data in the memory in a list, performance would be much better if I could use what I already have.

Any suggestions or material I could read to learn how to do what I'm looking for?

Thank you!

Something like `Dim newList = arrStrComponentsInfolink1Values.Where(Function(x) Not lstValidImageFormats.Contains(IO.Path.GetExtension(x))).ToList()`? You should probably do a [case-insensitive comparison](https://stackoverflow.com/q/3947126/8967612) though. — 41686d6564 stands w. Palestine, Jan 19 '22 at 23:04
@41686d6564 - That was what I've been looking for. An elegant solution! Thank you! — Daniel Santos, Jan 19 '22 at 23:34
So databases have things like indexes that are _really good_ for putting data like this together. Thus the plan to load the entire dataset into memory is probably making things **slower** -- _much_ slower -- rather than speeding things up. — Joel Coehoorn, Jan 19 '22 at 23:41
Anyway, the SQL solution would be to have another table with your valid file types, and then you do a `JOIN` to that table. — Joel Coehoorn, Jan 19 '22 at 23:49

score 1 · Answer 1 · answered Jan 19 '22 at 23:37

I ended up doing this and it worked:

Dim lstValidImageFormats As New List(Of String)({".JPG", ".JPEG", ".JPE", ".BMP", ".PNG", ".TIF", ".TIFF", ".GIF"})
    Dim lstValidSTLFormats As New List(Of String)({".STL"})
    Dim lstValidSTEPFormats As New List(Of String)({".STP", ".STEP"})

    '////////////////
    '// Components //
    '////////////////

    'We don't check Parts.InfoLink because all formats are allowed in this field

    'Parts.InfoLink1 - File in Infolink1 columns MUST BE images

    Dim intExtCounter As Integer = 0

    For i = 0 To arrIntComponentsInfolink1UNRs.Count - 1 'We have 10K rows with filenames in this list
        intExtCounter = 0
        For j = 0 To lstValidImageFormats.Count - 1
            If arrStrComponentsInfolink1Values.Item(i).EndsWith(lstValidImageFormats.Item(j)) = True Then
                intExtCounter += 1
            End If
        Next
        If intExtCounter = 0 Then 'At least one file extension was found
            arrIntComponentsInfolink1UNRsReportSectionInvalidExtensions.Add(i) 'File extension is not in the list of allowed extensions
        End If
    Next

But @41686d6564 answer was the best solution:

Dim newList = arrStrComponentsInfolink1Values.Where(Function(x) Not lstValidImageFormats.Contains(IO.Path.GetExtension(x))).ToList()

Thank you!

score 1 · Accepted Answer · answered Jan 20 '22 at 00:27

Here's how I would tackle this:

Dim invalidFormatFiles = _
    From x In arrStrComponentsInfolink1Values _
    Let fi = New FileInfo(x) _
    Where Not lstValidImageFormats.Contains(fi.Extension.ToUpperInvariant()) _
    Select x

For Each invalidFormatFile In invalidFormatFiles
    ' Do your processing
Next

Best way to loop through a list within a sub-loop

2 Answers2