C# - Memory leaking in .NET HttpClient, JsonSerializer or misused Stream?

Any time you allocate a contiguous block of memory >= 85,000 bytes in size, it goes into the large object heap. Unlike the regular heap it isn't compactified unless you do so manually^[1] so it can grow due to fragmentation giving the appearance of a memory leak. See Why Large Object Heap and why do we care?.

As your responseString and contentBytes are usually around 2 MB I would recommend rewriting your code to eliminate them. Instead, asynchronously stream directly from your server and to your JSON file using the relevant built-in APIs like so:

const int BufferSize = 16384;
const bool UseAsyncFileStreams = true; //https://learn.microsoft.com/en-us/dotnet/api/system.io.filestream.-ctor?view=net-5.0#System_IO_FileStream__ctor_System_String_System_IO_FileMode_System_IO_FileAccess_System_IO_FileShare_System_Int32_System_Boolean_

protected async Task DumpAppList()
{
    try
    {
        var httpClient = _httpClientFactory.CreateClient();
        var appList = new List<SteamApiGetAppListApp>();
        int? lastAppId = null;
        do
        {
            // Get the SteamApiGetAppListOuterResponse directly from JSON using HttpClientJsonExtensions.GetFromJsonAsync() without the intermediate string.
            // https://learn.microsoft.com/en-us/dotnet/api/system.net.http.json.httpclientjsonextensions.getfromjsonasync
            // If you need customized error handling see 
            // https://stackoverflow.com/questions/65383186/using-httpclient-getfromjsonasync-how-to-handle-httprequestexception-based-on
            var responseObject = (await httpClient.GetFromJsonAsync<SteamApiGetAppListOuterResponse>(GetAppListUrl(lastAppId)))
                !.response;
            appList.AddRange(responseObject.apps);
            lastAppId = responseObject.have_more_results ? responseObject.last_appid : null;

        } while (lastAppId != null);

        await using var output = new FileStream(Path.Combine(Config.DumpDataPath, DateTime.UtcNow.ToString("yyyy-MM-dd__HH-mm-ss") + ".json.gz"),
                                                FileMode.Create, FileAccess.Write, FileShare.None, bufferSize: BufferSize, useAsync: UseAsyncFileStreams);
        await using var gz = new GZipStream(output, CompressionMode.Compress);
        // See https://faithlife.codes/blog/2012/06/always-wrap-gzipstream-with-bufferedstream/ for a discussion of buffer sizes vs compression ratios.
        await using var buffer = new BufferedStream(gz, BufferSize);
        // Serialize directly to the buffered, compressed output stream without the intermediate in-memory array.
        await JsonSerializer.SerializeAsync(buffer, appList);
    }
    catch (Exception ex)
    {
        Trace.TraceError("skipped...");
    }
}

Notes:

• GZipStream does not buffer its input so there is a chance that streaming to it incrementally can result in worse compression ratios. However, as discussed by Bradley Grainger in Always wrap GZipStream with BufferedStream, buffering the incremental writes using a buffer that is 8K or larger effectively eliminates the problem.

• According to the docs, the useAsync argument to the FileStream constructor

  Specifies whether to use asynchronous I/O or synchronous I/O. However, note that the underlying operating system might not support asynchronous I/O, so when specifying true, the handle might be opened synchronously depending on the platform. When opened asynchronously, the BeginRead(Byte[], Int32, Int32, AsyncCallback, Object) and BeginWrite(Byte[], Int32, Int32, AsyncCallback, Object) methods perform better on large reads or writes, but they might be much slower for small reads or writes. If the application is designed to take advantage of asynchronous I/O, set the useAsync parameter to true. Using asynchronous I/O correctly can speed up applications by as much as a factor of 10, but using it without redesigning the application for asynchronous I/O can decrease performance by as much as a factor of 10.

  Thus you may need to test to see whether, in practice, you get better performance with UseAsyncFileStreams equal to true or false. You may also need to play around with the buffer sizes to get the best performance and compression ratio -- always being sure to keep the buffer smaller than 85,000 bytes.

• If you think large object heap fragmentation may be a problem, see the MSFT article The large object heap on Windows systems: A debugger for suggestions on how to investigate further.

• Since your DumpAppList() method only runs every half hour anyway, you might try compacting the large object heap manually after each run to see if that helps:

   protected override async Task ExecuteAsync(CancellationToken stoppingToken)
   {
       while (!stoppingToken.IsCancellationRequested)
       {
           await DumpAppList();
           GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
           GC.Collect();           
  
           await Task.Delay(RepeatDelay, stoppingToken);
       }
   }
  

• You may want to pass the CancellationToken stoppingToken into DumpAppList().

^[1] Do note that, in Memory management and garbage collection (GC) in ASP.NET Core: Large object heap, MSFT writes:

In containers using .NET Core 3.0 and later, the LOH is automatically compacted.

So my statement about when LOH compaction occurs may be out of date on certain platforms.