Go Concurrency Patterns: Pipeline

A pattern that processes data in stages, allowing each step to be handled concurrently, optimizing performance and enabling efficient, scalable data pipelines.

In a previous article, we explored the Generator pattern, which allows us to compute values lazily in Go. Now, we’re diving into another concurrency pattern: the Pipeline. This pattern enables us to process data step-by-step through a series of operators. If you’re familiar with Java Streams, which process data by passing each value through a set of filters, we’ll be taking a similar approach but leveraging Go’s concurrency features.

Consider the following Java example:

Stream
 .of(1, 2, 3, 4, 5) // creating the source stream
 .filter(e -> e % 2 == 1) // T1: first operator
 .map(e -> e * 3) // T2: second operator
 .map(e -> e + 1) // T3: third operator
 .forEach(System.out::println); // Terminal operator

In this code, a stream is created from a list of numbers. The stream processes each number sequentially, first filtering out even numbers. For example, when 1 passes through the filter, it's recognized as odd, so it proceeds to the next step where it's multiplied by 3, resulting in 3. This value is then incremented by 1, becoming 4, and finally…