Thanks to @henrik for response, but several important headers are missing in his answer. Be careful of that.
Full headers stack:
public function downloadAction() {
    $file = 'path/to/file';
    $response = new \Zend\Http\Response\Stream();
    $response->setStream(fopen($file, 'r'));
    $response->setStatusCode(200);
    $response->setStreamName(basename($file));
    $headers = new \Zend\Http\Headers();
    $headers->addHeaders(array(
        'Content-Disposition' => 'attachment; filename="' . basename($file) .'"',
        'Content-Type' => 'application/octet-stream',
        'Content-Length' => filesize($file),
        'Expires' => '@0', // @0, because zf2 parses date as string to \DateTime() object
        'Cache-Control' => 'must-revalidate',
        'Pragma' => 'public'
    ));
    $response->setHeaders($headers);
    return $response;
}