0

I am trying to install PySpark in Anaconda on a Windows 10 laptop. I'm running numerous problems, but one is that the CMD interpreter searches %PATH% for files and finds those without extensions before finding those with extension .CMD, e.g., pyspark and pyspark2. The extensionless files are typically Bash files and therefore not meant for CMD (described here). Many such filenames have counterparts with the same filename stem but also with extension .CMD. These are obviously meant for use by the Windows's CMD interpreter.

From web searching, I found that %PATHEXT% determines what extensions to include when searching %PATH%. My %PATHEXT% is .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC. Is there any way to modify %PATHEXT% to exclude extensionless filenames?

user2153235
  • 388
  • 1
  • 11
  • I'm kind of reluctant to advise anyone because I'm new to Python, Anaconda, Spark, and CMD. I will ponder it. How is it possible that I (who knows almost nothing about the above environments) am the first run into this? Is there no way to force CMD.exe to ignore extensionless files when searching %PATH%? P.S. I am perpetually new to CMD as I use it by exception, but have done so for a long time. – user2153235 Jul 21 '23 at 06:16
  • OK, thanks. I have to ponder how deep to troubleshoot this before seeking an alternative to installing everything within Anaconda. If it's something simple I can work around, e.g., not too many scripts with this problem, I might be able to get things working without finding other ways to install PySpark. Conda is supposed to save you from incompatibilities between different versions from different providers. But if it doesn't work and can't be made to work with limited effort, then it's not a solution. – user2153235 Jul 21 '23 at 06:44
  • @Mofi: Did you want to post your last comment as the answer? It does answer the question. Thanks. – user2153235 Jul 21 '23 at 16:32
  • I appreciate what you're saying, Mofi, but you're speaking to the context for the question rather than the question itself. The question was about how to prevent searching for extensionless files in %PATH%, which could prove useful in contexts other than Anaconda's installation of PySpark. About reporting the problem the author, I think I'd need to at least have taken things for a spin before expressing any shortcomings. I'm too new to all of the environments/tools involved. – user2153235 Jul 21 '23 at 20:21
  • As it turns out, it was better not for me to provide feedback to the author because the problem had to do with proper setting of environment variables, as described [here](https://stackoverflow.com/questions/76650431/pyspark-on-anaconda-was-unexpected-at-this-time?noredirect=1#comment135301172_76650431). So it seems that the authors (whoever they are). actually had a working plan for their ecosystem of scripts to work properly on Windows. – user2153235 Jul 22 '23 at 23:56

0 Answers0