Had the same problem at work today and sadly can't try my solution now, but try using the directory containing both src and vars as sourcepath. Then you can reference your src and vars subdirectories in your command like this:
groovydoc -sourcepath [PATH_TO_YOUR_LIB] -d doc src\ vars\
If that doesnt work, try referencing each package seperately somewhat like this:
groovydoc -sourcepath [PATH_TO_YOUR_LIB] -d doc src\org.foo.some_package src\org.foo.some_other_package vars\
Alternatively, or as a workaround, you could use the IntelliJ IDE and its "Generate GroovyDoc..."-Tool:
- Start IntelliJ and open any Groovy project (create a new one if there isnt an existing one)
- From the menu bar
Tools > Generate GroovyDoc...
- Choose your source path containing your jenkins pipeline library as
Input directory (containing both src and vars) and any path as Output directory
- Press
Start
- GroovyDoc should be in the specified
Output directory