The following code snippet does what you want:
set "FILE="
pushd "\path\to\folder" || exit /B 1
for /F "eol=| delims=" %%F in ('
    dir /B /A:-D /O:D /T:C "*.*"
') do (
    set "FILE=%%F"
)
if defined FILE del "%FILE%"
popd
The dir command returns files only (/A:-D) sorted by date in ascending order (/O:D) using the creation date (/T:C), which are located in \path\to\folder and match the pattern *.*.
The for /F loop walks through all the files returned by dir and assigns its name to variable FILE, overwriting it in each iteration, hence the final value is the name of the newest file.
The del command finally deletes the found file. The if query covers the case when no files are found in the given location.
Here is a slightly modified variant as recommended by Magoo's comment, which might be a bit more performant than the aforementioned one:
set "FILE="
pushd "\path\to\folder" || exit /B 1
for /F "eol=| delims=" %%F in ('
    dir /B /A:-D /O:-D /T:C "*.*"
') do (
    if not defined FILE set "FILE=%%F"
)
if defined FILE del "%FILE%"
popd
This avoids multiplicately overwriting of variable FILE by querying whether it has already been defined. Note the reversed sort order of dir here.
However, the approach shown in Magoo's answer is still better in performance.