1

I am using Tx.Windows nuget package for reading performance counter blg file.

https://github.com/Microsoft/Tx

With small size file, the result is very immediate for below code, but as my file size grow, the time taken for read also increase. With big size file (say 500MB) the time taken for file read is very large.

Is there any way to reduce file reading time?

using System;
using System.Linq;
using System.Reactive.Linq;
using Tx.Windows;

namespace ConsoleApp3
{
class Program
{
    static void Main(string[] args)
    {
        var observable = PerfCounterObservable.FromFile(@"C:\Files\test.blg");

        var grouped = from a in observable
                      group a by new { a.Machine, a.Instance, a.Timestamp } into groups
                      from g in groups.ToArray()
                      select new
                      {
                          groups.Key.Machine,
                          groups.Key.Instance,
                          groups.Key.Timestamp,
                          Counters = g
                      };

        var all = grouped.ToEnumerable().ToArray();

        Console.ReadLine();
    }
}
}

The sample blg file at Microsoft Tx github location,

https://github.com/Microsoft/Tx/blob/master/Traces/BasicPerfCounters.blg

and Here test code,

https://github.com/Microsoft/Tx/blob/master/Test/Tx.Windows.Tests/PerfCounterTest.cs

user584018
  • 10,186
  • 15
  • 74
  • 160
  • Are you sure that the time to read the file is the problem, and not what you're doing with the data? I'm not familiar with Tx, but it looks like that grouping operation could be pretty expensive. And is there a reason you're creating an array in the middle of that operation? That seems unnecessary and wasteful. – JLRishe Jan 27 '18 at 16:12
  • When file size is small, say 10Mb, then processing taking less time – user584018 Jan 27 '18 at 16:22
  • Is there a way I can reduce file reading time by removing grouping – user584018 Jan 27 '18 at 16:23
  • 1
    Have you tried to do performance profiling? It also seems to me, as @JLRishe mentioned, that you have too many points and grouping takes so long. – Artavazd Balayan Jan 27 '18 at 17:03
  • 1
    @user584018 I'd say the first thing to try is to remove that `.ToArray()` in the middle of your query. I don't think it's doing anything useful. _"Is there a way I can reduce file reading time by removing grouping"_ That wouldn't reduce the file reading time, but it might reduce the slowness of your operation. Do you need the grouping? – JLRishe Jan 27 '18 at 17:42
  • @JLRishe & Artavazd Balayan, it's correct that grouping takes long time to perform, but it's required for my work. As line `var observable = PerfCounterObservable.FromFile(@"C:\Files\test.blg");` gives an `observable`, will it be helpful to split it and process by different threads? – user584018 Jan 28 '18 at 12:21
  • @user584018, could you, please, provide the sample of a big file? I want to play with it. Thanks. – Artavazd Balayan Jan 29 '18 at 03:22
  • @Artavazd Balayan, I updated my post and share code link and sample blg file. I can't provide big size file. Sample blg file is around 400KB – user584018 Jan 29 '18 at 04:20
  • @user584018, thanks, played with an example. Do not see any bottle-neck there. Probably need bigger file. – Artavazd Balayan Jan 29 '18 at 15:18
  • @Artavazd Balayan, Is there any way to provide large blg file to you? – user584018 Jan 30 '18 at 07:59
  • @user584018, what about uploading it to file share(Google Drive or anything else) and giving the link? – Artavazd Balayan Jan 30 '18 at 08:12
  • @Artavazd Balayan, I have an idea. We can convert `blg` file into `csv` file using `relog.exe (relog logfile.blg -f csv -o logfile.csv)`. First we have to convert sample file into `csv` and then append a lot data into bottom of csv to grow it around 100 MB. Then finally we can convert `csv` back to `blg` using command (`relog BasicPerfCounters.csv -o logfile.blg`) will will around 500 MB. https://community.microfocus.com/borland/test/silk_performer_-_application_performance_testing/w/knowledge_base/9944/how-can-i-convert-a-windows-performance-monitor-file-from-blg-to-csv – user584018 Jan 30 '18 at 10:11
  • I will try to upload big size blg file to some share, if possible – user584018 Jan 30 '18 at 10:11
  • I created around 200MB `csv` file by appending same data again and again and after converting it to `blg`, it's more than 500MB – user584018 Jan 30 '18 at 10:26
  • @user584018, cool, let me try – Artavazd Balayan Jan 30 '18 at 11:47

0 Answers0