StringBuilder used with PadLeft/Right OutOfMemoryException

Question

All, I have the following Append which I am performing when I am producing a single line for a fixed text file

formattedLine.Append(this.reversePadding ?
                     strData.PadLeft(this.maximumLength) :
                     strData.PadRight(this.maximumLength)); 

This particular exception happens on the PadLeft() where this.maximumLength = 1,073,741,823 [a field length of an NVARCHAR(MAX) gathered from SQL Server]. formattedLine = "101102AA-1" at the time of exception so why is this happening. I should have a maximum allowed length of 2,147,483,647?

I am wondering if https://stackoverflow.com/a/1769472/626442 be the answer here - however, I am managing any memory with the appropriate Dispose() calls on any disposable objects and using block where possible.

Note. This fixed text export is being done on a background thread.

Thanks for your time.

Answer 1

This particular exception happens on the PadLeft() where this.maximumLength = 1,073,741,823

Right. So you're trying to create a string with over a billion characters in.

That's not going to work, and I very much doubt that it's what you really want to do.

Note that each char in .NET is two bytes, and also strings in .NET are null-terminated... and have some other fields beyond the data (the length, for one). That means you'd need at least 2147483652 bytes + object overhead, which pushes you over the 2GB-per-object limit.

If you're running on a 64-bit version of Windows, in .NET 4.5, there's a special app.config setting of <gcAllowVeryLargeObjects> that allows arrays bigger than 2GB. However, I don't believe that will change your particular use case:

Using this element in your application configuration file enables arrays that are larger than 2 GB in size, but does not change other limits on object size or array size:

• The maximum number of elements in an array is UInt32MaxValue.

• The maximum index in any single dimension is 2,147,483,591 (0x7FFFFFC7) for byte arrays and arrays of single-byte structures, and 2,146,435,071 (0X7FEFFFFF) for other types.

• The maximum size for strings and other non-array objects is unchanged.

What would you want to do with such a string after creating it, anyway?

Answer 2

In order to allocate memory for this operation, the OS must find contiguous memory that is large enough to perform the operation.

Memory fragmentation can cause that to be impossible, especially when using a 32-bit .NET implementation.

Answer 3

I think there might be a better approach to what you are trying to accomplish. Presumably, this StringBuilder is going to be written to a file (that's what it sounds like from your description), and apparently, you are also potentially dealing with large (huge) database records.

You might consider a streaming approach, that wont require allocating such a huge block of memory. To accomplish this you might investigate the following:

The SqlDataReader class exposes a GetChars() method, that allows you to read a chunk of a single large record. Then, instead of using a StringBuilder, perhaps using a StreamWriter ( or some other TextWriter derived class) to write each chunk to the output. This will only require having one buffer-full of the record in your application's memory space at one time. Good luck!