You need to make a GET request to fileUrl, since permanent "direct" links will not exist.
Directly from these google docs:
Using alt=media
To download files, you make an authorized HTTP GET request to the file's resource URL and include the query parameter alt=media. For example:
GET
https://www.googleapis.com/drive/v3/files/0B9jNhSvVjoIVM3dKcGRKRmVIOVU?alt=media
URLs do not need file extensions for their server to understand what it is you are requesting. However, sometimes servers want an extra query parameter to decide what to send back.
So it also seems you may need to append ?alt=media onto the end of your endpoint url, fileUrl.
Then just use ajax, or a similar requesting method, and send a GET request to the endpoint.
Using the Drive API and JS:
var fileId = '0BwwA4oUTeiV1UVNwOHItT0xfa2M';
var dest = fs.createWriteStream('/tmp/photo.jpg');
drive.files.get({
fileId: fileId,
alt: 'media'
})
.on('end', function () {
console.log('Done');
})
.on('error', function (err) {
console.log('Error during download', err);
})
.pipe(dest);