0

I currently have a pipeline setup as such:

BufferBlock ==> BatchBlock ==> TransformBlock ==> TransformManyBlock ==> BufferBlock

The BoundedCapacity is set to 2,000. However, if I provide a single item to the BufferBlock, and then call the Complete method on the BufferBlock, it never completes. I tried awaiting the Completion task of the incoming BufferBlock, as well as the Completion task of the outgoing (last) BufferBlock, but it never completes.

My understanding was that calling the Complete method on the first block in the pipeline should indicate that no more data is going to be offered and to process any remaining items through the pipeline. When I add a breakpoint on the await incomingBuffer.Completion line, and check the properties of incomingBuffer via Visual Studio, I see the following:

enter image description here

As seen, it is indeed linked to a target, and is no longer accepting new items, but it is not completed and there is still one item in the Queue. How can I force the BufferBlock to send the remaining items to the BatchBlock when there are no more items being offered. I should point out that when I link the various action blocks (with the LinkTo method) I always pass in a new DataflowLinkOptions class with PropagateCompletion = true.

Also if I set BoundedCapacity to 1 (for every block) then everything works fine. However, in production this pipeline will be processing over 1.5M records, in batches of 1,000, so I don't think having a BoundedCapacity of 1 would work (unless I completely misunderstood its functionality). I also attempted to not even set the BoundedCapacity property on the DataflowBlockOptions class (so it would just use its default value) and this did not work either. I have to explicitly set the BoundedCapacity value to 1 for the pipeline to complete when there is an item in the first BufferBlock; otherwise it just sits there and nothing is completed.

Any help or insight would be greatly appreciated.

Dominick
  • 322
  • 1
  • 3
  • 16
  • Could you include in the question a minimal example that demonstrates the observed behavior? – Theodor Zoulias Dec 02 '21 at 17:03
  • You could take a look at this question: [TPL Dataflow not completing with multiple targets](https://stackoverflow.com/questions/47462495/tpl-dataflow-not-completing-with-multiple-targets), to get an insight about why the last block in the pipeline (a `BufferBlock`) will never complete in your scenario. There is no apparent reason for the first block (also a `BufferBlock`) to not complete though. – Theodor Zoulias Dec 03 '21 at 13:57
  • @Dominick please post your actual code. A block can't Complete until all its outbound items have been consumed. This means that one of the downstream block is blocked for one reason or another. The BatchBlock itself wouldn't block unless it was full *and* the downstream blocks were full as well. This means that one of the TransformBlocks was blocked long enough that the BatchBlock filled up OR that the final BufferBlock is bounded and full – Panagiotis Kanavos Dec 23 '21 at 14:15
  • @Dominick all blocks have input buffers. Transform blocks also have output buffers. This means that neither the start nor the end BufferBlock are necessary. The final BufferBlock is useful if you want to drain the final TransformBlock in a pipeline and process messages after the pipeline completes. – Panagiotis Kanavos Dec 23 '21 at 14:20

0 Answers0