The short answer: they don't get deleted.
The long answer: 
The managed Path.GetTempFileName() method calls the native Win32API GetTempFileName() method, like this: 
//actual .NET 2.0 decompiled code 
// .NET Reflector rocks for looking at plumbing
public static string GetTempFileName()
{
    string tempPath = GetTempPath();
    new FileIOPermission(FileIOPermissionAccess.Write, tempPath).Demand();
    StringBuilder tmpFileName = new StringBuilder(260);
    if (Win32Native.GetTempFileName(tempPath, "tmp", 0, tmpFileName) == 0)
    {
        __Error.WinIOError();
    }
    return tmpFileName.ToString();
}
The documentation for the native method states:
 Temporary files whose names have been created by this function are not automatically deleted. To delete these files call DeleteFile.
I have found a great article called "Those pesky temp files" (Archived Oct. 2007) that starts from basics and touches some less obvious problems of handling temporary files, like:
- How to make sure the file is deleted (even if the app crashes! hint: 
FileOption.DeleteOnClose and let the kernel deal with it) 
- How to get the correct caching policy for the file, to improve performance  (hint: 
FileAttributes.Temporary) 
- How to make sure the contents of the file stay secure, because:
- the file name is even more predictable with the managed method than with the unmanaged one
 
- the temp file is created, then closed, then you get the path to it (only to open it again), thus leaving a small window of opportunity for malicious code/users to hijack the file. 
 
 
C# Code from article:
using System;
using System.IO;
using System.Security.Permissions;
using System.Security.Principal;
using System.Security.AccessControl;
public static class PathUtility
{
    private const int defaultBufferSize = 0x1000; // 4KB
#region GetSecureDeleteOnCloseTempFileStream
    /// <summary>
    /// Creates a unique, randomly named, secure, zero-byte temporary file on disk, which is automatically deleted when it is no longer in use. Returns the opened file stream.
    /// </summary>
    /// <remarks>
    /// <para>The generated file name is a cryptographically strong, random string. The file name is guaranteed to be unique to the system's temporary folder.</para>
    /// <para>The <see cref="GetSecureDeleteOnCloseTempFileStream"/> method will raise an <see cref="IOException"/> if no unique temporary file name is available. Although this is possible, it is highly improbable. To resolve this error, delete all uneeded temporary files.</para>
    /// <para>The file is created as a zero-byte file in the system's temporary folder.</para>
    /// <para>The file owner is set to the current user. The file security permissions grant full control to the current user only.</para>
    /// <para>The file sharing is set to none.</para>
    /// <para>The file is marked as a temporary file. File systems avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Otherwise, the data is written after the handle is closed.</para>
    /// <para>The system deletes the file immediately after it is closed or the <see cref="FileStream"/> is finalized.</para>
    /// </remarks>
    /// <returns>The opened <see cref="FileStream"/> object.</returns>
    public static FileStream GetSecureDeleteOnCloseTempFileStream()
    {    
        return GetSecureDeleteOnCloseTempFileStream(defaultBufferSize, FileOptions.DeleteOnClose);    
    }
    /// <summary>
    /// Creates a unique, randomly named, secure, zero-byte temporary file on disk, which is automatically deleted when it is no longer in use. Returns the opened file stream with the specified buffer size.
    /// </summary>
    /// <remarks>
    /// <para>The generated file name is a cryptographically strong, random string. The file name is guaranteed to be unique to the system's temporary folder.</para>
    /// <para>The <see cref="GetSecureDeleteOnCloseTempFileStream"/> method will raise an <see cref="IOException"/> if no unique temporary file name is available. Although this is possible, it is highly improbable. To resolve this error, delete all uneeded temporary files.</para>
    /// <para>The file is created as a zero-byte file in the system's temporary folder.</para>
    /// <para>The file owner is set to the current user. The file security permissions grant full control to the current user only.</para>
    /// <para>The file sharing is set to none.</para>
    /// <para>The file is marked as a temporary file. File systems avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Otherwise, the data is written after the handle is closed.</para>
    /// <para>The system deletes the file immediately after it is closed or the <see cref="FileStream"/> is finalized.</para>
    /// </remarks>
    /// <param name="bufferSize">A positive <see cref="Int32"/> value greater than 0 indicating the buffer size.</param>
    /// <returns>The opened <see cref="FileStream"/> object.</returns>
    public static FileStream GetSecureDeleteOnCloseTempFileStream(int bufferSize)
    {
        return GetSecureDeleteOnCloseTempFileStream(bufferSize, FileOptions.DeleteOnClose);
    }
    /// <summary>
    /// Creates a unique, randomly named, secure, zero-byte temporary file on disk, which is automatically deleted when it is no longer in use. Returns the opened file stream with the specified buffer size and file options.
    /// </summary>  
    /// <remarks>
    /// <para>The generated file name is a cryptographically strong, random string. The file name is guaranteed to be unique to the system's temporary folder.</para>
    /// <para>The <see cref="GetSecureDeleteOnCloseTempFileStream"/> method will raise an <see cref="IOException"/> if no unique temporary file name is available. Although this is possible, it is highly improbable. To resolve this error, delete all uneeded temporary files.</para>
    /// <para>The file is created as a zero-byte file in the system's temporary folder.</para>
    /// <para>The file owner is set to the current user. The file security permissions grant full control to the current user only.</para>
    /// <para>The file sharing is set to none.</para>
    /// <para>The file is marked as a temporary file. File systems avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Otherwise, the data is written after the handle is closed.</para>
    /// <para>The system deletes the file immediately after it is closed or the <see cref="FileStream"/> is finalized.</para>
    /// <para>Use the <paramref name="options"/> parameter to specify additional file options. You can specify <see cref="FileOptions.Encrypted"/> to encrypt the file contents using the current user account. Specify <see cref="FileOptions.Asynchronous"/> to enable overlapped I/O when using asynchronous reads and writes.</para>
    /// </remarks>
    /// <param name="bufferSize">A positive <see cref="Int32"/> value greater than 0 indicating the buffer size.</param>
    /// <param name="options">A <see cref="FileOptions"/> value that specifies additional file options.</param>
    /// <returns>The opened <see cref="FileStream"/> object.</returns>
    public static FileStream GetSecureDeleteOnCloseTempFileStream(int bufferSize, FileOptions options)
    {    
        FileStream fs = GetSecureFileStream(Path.GetTempPath(), bufferSize, options | FileOptions.DeleteOnClose);
        File.SetAttributes(fs.Name, File.GetAttributes(fs.Name) | FileAttributes.Temporary);
        return fs;    
    }
#endregion
#region GetSecureTempFileStream
    public static FileStream GetSecureTempFileStream()
    {    
        return GetSecureTempFileStream(defaultBufferSize, FileOptions.None);    
    }
    public static FileStream GetSecureTempFileStream(int bufferSize)
    {
        return GetSecureTempFileStream(bufferSize, FileOptions.None);
    }
    public static FileStream GetSecureTempFileStream(int bufferSize, FileOptions options)
    {
        FileStream fs = GetSecureFileStream(Path.GetTempPath(), bufferSize, options);
        File.SetAttributes(fs.Name, File.GetAttributes(fs.Name) | FileAttributes.NotContentIndexed | FileAttributes.Temporary);
        return fs;
    }
    #endregion
#region GetSecureTempFileName
    public static string GetSecureTempFileName()
    {    
        return GetSecureTempFileName(false);    
    }
    public static string GetSecureTempFileName(bool encrypted)
    {    
        using (FileStream fs = GetSecureFileStream(Path.GetTempPath(), defaultBufferSize, encrypted ? FileOptions.Encrypted : FileOptions.None))
        {    
            File.SetAttributes(fs.Name, File.GetAttributes(fs.Name) | FileAttributes.NotContentIndexed | FileAttributes.Temporary);
            return fs.Name;    
        }
    }
#endregion
#region GetSecureFileName
    public static string GetSecureFileName(string path)
    {    
        return GetSecureFileName(path, false);    
    }
    public static string GetSecureFileName(string path, bool encrypted)
    {    
        using (FileStream fs = GetSecureFileStream(path, defaultBufferSize, encrypted ? FileOptions.Encrypted : FileOptions.None))
        {    
            return fs.Name;    
        }    
    }
#endregion
#region GetSecureFileStream
    public static FileStream GetSecureFileStream(string path)
    {    
        return GetSecureFileStream(path, defaultBufferSize, FileOptions.None);    
    }
    public static FileStream GetSecureFileStream(string path, int bufferSize)
    {
        return GetSecureFileStream(path, bufferSize, FileOptions.None);
    }
    public static FileStream GetSecureFileStream(string path, int bufferSize, FileOptions options)
    {    
        if (path == null)
            throw new ArgumentNullException("path");
        if (bufferSize <= 0)
            throw new ArgumentOutOfRangeException("bufferSize");
        if ((options & ~(FileOptions.Asynchronous | FileOptions.DeleteOnClose | FileOptions.Encrypted | FileOptions.RandomAccess | FileOptions.SequentialScan | FileOptions.WriteThrough)) != FileOptions.None)
            throw new ArgumentOutOfRangeException("options");
        new FileIOPermission(FileIOPermissionAccess.Write, path).Demand();
        SecurityIdentifier user = WindowsIdentity.GetCurrent().User;
        FileSecurity fileSecurity = new FileSecurity();
        fileSecurity.AddAccessRule(new FileSystemAccessRule(user, FileSystemRights.FullControl, AccessControlType.Allow));
        fileSecurity.SetAccessRuleProtection(true, false);
        fileSecurity.SetOwner(user);
        // Attempt to create a unique file three times before giving up.
        // It is highly improbable that there will ever be a name clash,
        // therefore we do not check to see if the file first exists.
        for (int attempt = 0; attempt < 3; attempt++)
        {    
            try
            {    
                return new FileStream(Path.Combine(path, Path.GetRandomFileName()),
                                        FileMode.CreateNew, FileSystemRights.FullControl,
                                        FileShare.None, bufferSize, options, fileSecurity);
            }
            catch (IOException)
            {
                if (attempt == 2)
                    throw;
            }
        }
        // This code can never be reached.
        // The compiler thinks otherwise.
        throw new IOException();
    }
#endregion
}