This is a little rushed. I will check back if it makes sense to you.
You should be able to use the standard and built-in RemoveFile feature in MSI to remove log files - if you know the name of the files in question. Then you can add a CreateFolder entry to your installer which creates the logging directory in question and then use that directory property in your RemoveFile entry:
Sample from github:
<Component>
  <File Source="WiX.chm" />
  <Shortcut Id="WixChmShortcut" Directory="ShortcutFolder" Name="WiX Documentation" 
            Icon="WixSetupIcons.ico" IconIndex="0" Advertise="yes" />
  <RemoveFile Id="RemoveWixChw" Name="WiX.chw" On="uninstall" />
</Component>
CreateFolder is a WiX element that you insert to indicate that an empty folder should be created as part of your installation:
   <Directory Id="Mydir" Name="My Directory">
      <Component Feature="MyFeature" >
        <CreateFolder />
        <File Id="NOTEPAD.EXE" Source="$(var.MyReleasePath)\Main Folder\NOTEPAD.EXE" />
      </Component>
   </Directory>
Some further thoughts:
I would not do logging in the installation folder - this folder is supposed to be read-only.
 
I would log elsewhere, somewhere in the UserProfile perhaps, or even the temp folder (and let it be cleaned by whatever mechanism the end users have for that).
 
I would also consider logging to the system's event log instead of to a file, if possible.