The use of B-frames in a hardware-based encoder implementation, specifically NVENC (be it for both H.264 and HEVC wrappers) will not directly influence the output size of an encoded file, all other factors remaining constant.
B-frames typically increase the decoder resources required to process the output, which is why the Baseline profiles have it disabled.
If you want explicit control over output size(s), then you must set the target bitrate (-b:v), maximum bitrate (-maxrate:v) and buffer size (-bufsize:v) to appropriate values, as per your needs. NVENC's defaults for this are insane, at about ~256 kb/s. You must therefore either set these values above by hand OR explicitly unset it altogether by passing -b:v 0.
On to the question of rate control look-ahead (private option -rc-lookahead:v), this is mostly useful when combined with multiple B-frames (-bf:v n where n > 2 ). In this case, the encoder will dynamically select the number of B-Frames between 0 and the number of B-Frames you specify.
In practice, B-frames are great because they increase image quality, but they consume a lot of bit-rate which negatively impacts the perceptual quality (PQ) of high motion content. This is a factor to consider in some applications, such as live streaming.
Note that any form of weighted prediction cannot be combined with the use of B-frames in NVENC, so weigh your encoding requirements carefully before toggling either option.
For your workload, here are two approaches you can consider (with the caveats on bit rate control as documented above):
1. Constant quality mode in VBR rate control:
For -cq, note that this toggles on constant quality encoding mode when rate control (toggled via -rc:v in NVENC) is set to VBR (Variable bitrate) and its' variants. The scale it uses is logarithmic, with 1 being the highest possible quality mode and 51 being the lowest quality mode. The default setting of 0 implies automatic.
2. Constant quantization parameter rate control method:
This mode requires that rate control mode be explicitly set to constqp, via the private codec option rc:v constqp and the argument -qp:v be present, with valid values starting from 0 to 51. This can be further capped via the minimum and maximum allowable quantization values set via -qmin:v {int} and -qmax:v {int} respectively.
In my testing, I've been able to achieve the best results with method (2) above, ie setting a constant quantizer in effect and selecting the rate control method to constqp, as shown in the two examples below:
(a). Using a quantizer value of 0:
ffmpeg -y -i dnce.mkv -vf 'hwupload_cuda=0,yadif_cuda=0:-1:0,scale_cuda=w=1920:h=1080' -c:v h264_nvenc -rc:v constqp -tune:v hq -preset:v p7 -qp:v 0 -b:v 0 -c:a copy nv-test-7.ts
(b). Using a quantizer value of 19:
ffmpeg -y -i dnce.mkv -vf 'hwupload_cuda=0,yadif_cuda=0:-1:0,scale_cuda=w=1920:h=1080' -c:v h264_nvenc -rc:v constqp -tune:v hq -preset:v p7 -qp:v 19 -b:v 0 -c:a copy nv-test-8.ts
in these cases, the encode with a QP of 0 produced an output of 8.8 GB, whereas the latter, at a QP of 19, produced an output file of 1.3GB.
The output had to be deinterlaced in both cases.
The most confusing result I got was with the constant quality mode, as documented in (1) above:
(a). Using a target quality level of 0 in VBR mode:
ffmpeg -y -i dnce.mkv -vf 'hwupload_cuda=0,yadif_cuda=0:-1:0,scale_cuda=w=1920:h=1080' -c:v h264_nvenc -rc:v vbr -tune:v hq -preset:v p7 -cq:v 0 -b:v 0 -c:a copy nv-test-3.ts
(b). Using a target quality level of 19 in VBR mode:
ffmpeg -y -i dnce.mkv -vf 'hwupload_cuda=0,yadif_cuda=0:-1:0,scale_cuda=w=1920:h=1080' -c:v h264_nvenc -rc:v vbr -tune:v hq -preset:v p7 -cq:v 19 -b:v 0 -c:a copy nv-test-4.ts
in this case, the output sizes for a CQ setting of 0 and 19 were 797 and 790MB respectively.
Your mileage may differ, depending on your input files and the underlying muxer(s) in use, as each muxer in FFmpeg has differing overheads, etc.
Be careful not to mix and match parameters,as my previous answer had, which resulted in skewed observations that did not hold up under further scrutiny.
