It’s a artifact of a case insensitive file system, like that on Windows. Git internally stores branches as files and directories, inside the .git/refs folder.
So, when you create a branch named foo, a file named foo is created inside the refs/heads folder. Since you did it from a checkout command, git also records that foo is now the checked out branch. So, when you try to delete branch foo, git sees that foo is the checked out branch and gives you an error.
Now, when you try to delete branch named Foo, git first checks if a file named Foo exists in the refs/heads folder, and here is where it gets weird:
Since your file system is case insensitive, Foo, as a file name, is equivalent foo. Therefore, the file system reports that path refs/heads/Foo exists, when it’s actually refs/heads/foo. But then, when git checks if Foo is the checked out branch, it sees that Foo is not the checked out branch, because foo is, since git is internally case sensitive, for it "Foo" is not equal to "foo". So, git wrongly determines that: foo is a existent branch, distinct from Foo, and that Foo is not the checked out branch. Therefore, git allows you to delete Foo.