System.DateTime comparison ('>') or ('<') does not give what I expected

Question

I want to get files from a list for all the files whose filedate > today's cutOff - so, I have the following codelet

string[] MyFiles = Directory.GetFiles(MyConfig.pathTransmittedFiles, "*.adf")
        .Where(file => new FileInfo(file).LastWriteTime > dtCutOff).ToArray();

I have a file whose LastWriteTime is "{11/3/2015 1:33:26 PM}" being picked up by my collection with dtCutOff == "{11/3/2015 1:33:26 PM}"! So '>' didn't seem to work.

Answer 1

First, I would try running it without the Where clause, just to make sure that all files you expect are indeed part of the initial array returned from Directory.GetFiles. It's entirely possible that date/time comparison is not the source of the discrepancy. It may be more related to the issue Ivan linked to in the question comments, or it may be permission related, or some other thing.

Next, be aware that DateTime violates SRP in that it has a Kind property, which is one of the three DateTimeKind enumeration values. It's either Local, Utc, or Unspecified.

In the case of DateTime.Now, the Kind will be DateTimeKind.Local. File.GetLastWriteTime also returns its value with local kind. Therfore, if you always derive your dtCutOff from DateTime.Now in the manner you showed in the question, then it will almost always be the correct comparison function.

The "almost" stems from the fact that DateTimeKind.Local can actually represent two different kinds under the covers. In other words, there are actually four kinds, but two of them are exposed by one. This is described as "DateTime's Deep Dark Secret" in Jon Skeet's blog post More Fun with DateTime, and is also mentioned in the comments in the .NET Framework Reference Source. In practice, you should only encounter this in the ambiguous hour during a fall-back daylight saving time transition (such as just occurred last Sunday 2015-11-01 in the US).

Now, to the more likely case that your dtCutOff is actually derived not from DateTime.Now, but rather from user input or database lookup or some other mechanism, then its possible that it actually represents the local time in some other time zone than the one on your local computer. In other words, if the dtCutOff has a Kind of DateTimeKind.Utc, then the value is in terms of UTC. If it has a Kind of DateTimeKind.Unspecified, then the value might be in terms of UTC, or the local time zone, or some other time zone entirely.

Here's the kicker: Comparison of two DateTime values only evaluates the value underlying the Ticks property. It does not consider Kind.

Since file times are absolute points in universal time (on NTFS anyway), then you really should use the File.GetLastWriteTimeUtc method, rather than the methods that work in local time.

There are two approaches you could use:

• Load the modified property as UTC, using:

  myResult.modified = File.GetLastWriteTimeUtc(myFile);
  

• Populate dtOffset appropriately.

  • If you're loading from the current time, then use DateTime.UtcNow.
  • If you're loading from other input, ensure the value is converted to UTC to match the input scenario. For example, use .ToUniversalTime() if the value is in terms of the local time zone, or use the conversion functions in the TimeZoneInfo class if the value is in another time zone.

OR

• Change your modified property to be a DateTimeOffset instead of a DateTime.

• Load that using:

  myResult.modified = new DateTimeOffset(File.GetLastWriteTimeUtc(myFile));
  

• Define dtCutOff as a DateTimeOffset, and populate appropriately.

  • If you're loading from the current time, then use DateTimeOffset.UtcNow.
  • If you're loading from other input, ensure the offset is set to match the input scenario. Use TimeZoneInfo functions if you need to convert from another time zone.

DateTimeOffset has many advantages over DateTime, such as not violating SRP. It's always representing an absolute moment in time. In this scenario, it helps to know that comparison operators on DateTimeOffset always reflect that absolute moment. (In other words, it internally adjusts to UTC before doing the comparison.)

Answer 2

Since the files were fed into the queue once a day so the scale of precision is not required down to a millisecond or something. So that one-second TimeSpan difference is acceptable to do the trick and make my case work.

string[] MyFiles = Directory.GetFiles(MyConfig.pathTransmittedFiles, "*.adf")
  .Where(file => new FileInfo(file).LastWriteTime - TimeSpan.FromSeconds(1) > dtCutOff)
  .ToArray();

Now my file with modified date "{11/3/2015 1:33:26 PM}" didn't go into my collection when my cutOffDate is "{11/3/2015 1:33:26 PM}" while my other file with modified date "{11/3/2015 1:33:27 PM}" have successfully passed into my collection as expected!! So, it works and that's how it should work after all these advises! Thanks ye-all.

Answer 3

This code works:

var cutffDate = new DateTime(2015,1,1); // or whatever
var allFiles = Directory.GetFiles(MyConfig.pathTransmittedFiles, "*.adf");
var datedFiles = allFiles.Where(f => (new FileInfo(f)).LastWriteTime > cutffDate);

Update: Since your issue seems to be a precision-related one you could change the comparison to:

const long precision = 10; // vary this as needed
allFiles.Where(f =>
  (new FileInfo(f)).LastWriteTime.ToFileTime()/precision > cutffDate.ToFileTime()/precision);

Alternatively you could use ...LastAccessTime.Ticks/TimeSpan.TicksPerMillisecond In addition to that you may need to convert all DateTime values to UTC (LastAccessTimeUtc and DateTime.UtcNow) to make sure it's not some weird timezone issue

Answer 4

Looks like your Where clause lambda might be incorrect. Try this.

string[] MyFiles = Directory.GetFiles(MyConfig.pathTransmittedFiles, "*.adf").Where(file => file.modified > dtCutOff).ToArray();