Asynchronous operation of FileTree files?

Question

Is there a way I can easily make the processing of FileTree files in a smart way in gradle tasks? I basically need to wait for the execution of all files, much like what you can do with GPars, but how do I do this gradle with FileTree?

task compressJs(dependsOn: [copyJsToBuild]) << {
    println 'Minifying JS'

    fileTree {
        from 'build/js'
        include '**/*.js'
    }.visit { element ->
        if (element.file.isFile()) {
            println "Minifying ${element.relativePath}"
            ant.java(jar: "lib/yuicompressor-2.4.6.jar", fork: true) {
                arg(value: "build/js/${element.relativePath}")
                arg(value: "-o")
                arg(value: "build/js/${element.relativePath}")
            }
        }
    }
}

It would be lovely if I could do something like .visit{}.async(wait:true), but my googling turned up nothing. Is there a way I can easily make this multi-threaded? The processing of one element has no effect on the processing of any other element.

score 3 · Accepted Answer · answered Sep 01 '11 at 03:41

3

Before thinking about going multi-threaded, I'd try the following:

Run everything in the same JVM. Forking a new JVM for each input file is very inefficient.
Make the compressJs task incremental so that it only executes if some input file has changed since the previous run.
Run the minifier directly rather than via Ant (saves creation of a new class loader for each input file; not sure if it matters).

If this still leaves you unhappy with the performance, and you can't use a more performant minifier, you can still try to go multi-threaded. Gradle won't help you there (yet), but libraries like GPars or the Java Fork/Join framework will.

answered Sep 01 '11 at 03:41

Peter Niederwieser

121,412
21
324
259

I lose all output unless I specify `fork: true` and run with `ant.java`. I guess I can investigate using gpars directly, but it just seems like this should be something gradle should handle. – Stefan Kendall Sep 01 '11 at 05:20
What do you mean you lose all output? Forking for every input file is horribly inefficient; there must be a better way (I hope). Additionally, judging from the args you pass to yuicompressor, it appears that you overwrite the files in place. That's usually not a good idea; in this case it means that both copyJsToBuild and compressJs will never be up-to-date. Can't you get rid of copyJsToBuild and let the minifier do the copy by choosing different input and output files? As with programming in general, you should concentrate on improving the single-threaded case before going multi-threaded. – Peter Niederwieser Sep 01 '11 at 06:00
With any other invocation of the JAR, output to STDOUT is lost. I posted this as an SO question before; this is the only way to make this work. Regardless of anything else, file processing should be asynchronous. I'd likely get a speedup of 3-8x, given I'm running on 8 cores with a beefy SSD. Right now I'm processor bound, which is dumb. – Stefan Kendall Sep 01 '11 at 13:34
Feel free to submit a feature request. Meanwhile, you should be able to achieve this with a few lines of GPars code. – Peter Niederwieser Sep 02 '11 at 00:38
Done and done. GPars is the best solution for now, and I've submitted a feature request here: http://issues.gradle.org/browse/GRADLE-1778. Linked in case anyone comes here on a search. – Stefan Kendall Sep 02 '11 at 02:24

score 1 · Answer 2 · answered Sep 02 '11 at 03:26

The GPars solution. Note that the compress() function could be modified to properly accept source dir/target dir/etc, but since all my names are consistent, I'm just using the one argument for now. I was able to cut my build time from 7.3s to 5.4s with only 3 files being minified. I've seen build times spiral out of control, so I'm always wary of performance with this kind of behavior.

import groovyx.gpars.GParsPool

buildscript {
    repositories {
        mavenCentral()
    }

    dependencies {
        classpath 'org.codehaus.gpars:gpars:0.12'
    }
}

def compress(String type) {
    def elementsToMinify = []
    fileTree {
        from type
        include "**/*.$type"
    }.visit { element ->
        if (element.file.isFile()) {
            elementsToMinify << element
        }
    }

    GParsPool.withPool(8) {
        elementsToMinify.eachParallel { element ->
            println "Minifying ${element.relativePath}"
            def outputFileLocation = "build/$type/${element.relativePath}"
            new File(outputFileLocation).parentFile.mkdirs()
            ant.java(jar: "lib/yuicompressor-2.4.6.jar", fork: true) {
                arg(value: "$type/${element.relativePath}")
                arg(value: "-o")
                arg(value: outputFileLocation)
            }
        }
    }
}

task compressJs {
    inputs.dir new File('js')
    outputs.dir new File('build/js')

    doLast {
        compress('js')
    }
}

task compressCss {
    inputs.dir new File('css')
    outputs.dir new File('build/css')

    doLast {
       compress('css')
    }
}

Asynchronous operation of FileTree files?

2 Answers2