You could include the target directoryname directly in the command:
for /f "delims=" %%a in ('dir/s/b/a-d "d:\your directory\name\wherever\*.*"') do ( if %%~Za equ 0 ECHO del "%%~a" )
Notes:
There should be no space between the = and " in the delims setting. If you include a space there, then if there is a file named file name.txt, file will be assigned to %%a. Since the output of dir /s/b contains the path, an absolute filename d:\some dir\file name.txt will assign d:\some to %%a and act on that file, even if it is outside of the subtree you intend to scan.
tokens=* simply deletes leading spaces, which won't occur with dir/s/b so is redundant.
I added in an echo to show the filenames which would be deleted. This is a "safe" option - show the names first, so that if there is an error no damage is done. Once you are happy the correct selection has been made, remove the echo keyword to actually delete the files.
Another (but similar) way is
set "targetdir=d:\your directory\name\wherever"
for /f "delims=" %%a in ('dir/s/b/a-d "%targetdir%\*.*"') do ( if %%~Za equ 0 ECHO del "%%~a" )
where targetdir is simply a convenient descriptive name. This is more flexible since you can use code to construct the value in targetdir if you so desire.
Of course, *.* is simply a filemask for all files. If you want to process only the .txt files, then replace *.* with *.txt