This is usually (in my experience!) caused by not targeting the correct build configuration. Within Visual Studio, at the solution level you can go to the Build menu and choose Configuration Manager. Ensure that for all applicable configurations (chosen via the Active Solution Configuration drop-down) the installer project has a tick in the Build column.
Now you need to ensure that when invoking devenv you're passing the appropriate build configuration (i.e. one that has Build ticked for the setup project) as follows:
C:\PathToVisualStudio\devenv.exe /Rebuild Release C:\PathToProject\ProjectName.sln" /Out "PathToProject\vs_errors.txt"
(In this example, Release is the build configuration I'm targeting)
This command also logs the output of Visual Studio to a text file called vs_errors.txt in the same folder as your solution so you can review this to determine if there are any other reasons why the setup project failed to build.