GRUMPS Design: Pipes and Filters

In one sense, pipes and filters are the computational paradigm most reminiscent of classical signal processing.  In circumstances where one input is modified in many ways to produce one output, piping is often the most sensible and flexible solution.  Piping stdout and stderr programs in *nix allows the users to easily execute complicated operations without requiring them to create new programs.  Instead, they can simply string together transformations, finding, manipulating, and redirecting text.  This is similar to audio production, in which a limited number of filters can be used to make a great variety of modifications to an acoustic signal; applying reverb and then noise will sounds substantially different from applying noise first, and then reverb.

Thus, in pipe/filter processing, each permutation of the available filters constitutes operates like a new filter in itself, meaning that every new contribution enriches the entire processing ecosystem.

That said, there are dangers in overstretching this analogy.  Computer pipes and filters encounter issue that direct signal processing does not.  Circuits do not have to deal with error conditions: they will treat all inputs the same.  They do not have to worry about time spent switching between processes: the transition is instantaneous.  They rarely need to navigate a complex push/pull pipeline: each push/pull would be a separate component in itself.

When I was originally designing GRUMPs, I gave consideration to designing some of the components as a pipe/filter set-up.  Because of the flexibility such a system would offer, I assumed that it would enable me to prototype many different kinds of physical set-ups until I came across one that stably opened a worm hole through time.  Some of my initial prototypes processed time flux data in exactly this matter, but I soon came across some issues that prevented me from continuing to work in that paradigm.

The most substantial among these was the increasingly evident need to maintain a global state.  A pipe/filter setup treats each component as structurally independent from one another.  So what do I do when my flux capacitance monitor begun measuring anomalous gravitational waves, forcing me to alter the rate of boson production?  A pipe/filter system can’t handle this condition easily.  Technically, you can design a pipe to take multiple inputs, but that begins to breakdown the purity of the design, complicating the system’s structure in the long run.  Simply put, too many components needed to talk to too many other components in GRUMPS simultaneously. 

Another problem came from the need to dynamically redirect the flow of information.  Technically, one can operate a conditional filter pipeline, but again this breaks the purity of design.  Depending on the nature of information streaming through the black hole, I might need to enable or disable various filters and amplifiers, frequently reconfiguring the data flow multiple times per second.  Ultimately, this began to look more like a fuzzy state diagram, and less like a pipe/filter setup.  My assumptions that I could simply reconfigure and rerun proved consistently wrong, as the setup had to adapt to running conditions.  Filters failed because a large, carefully calibrated system cannot dynamically respond to circumstances when it is designed to process information linearly.

Pipes and filters are a fascinating and profoundly useful paradigm.  But despite their flexibility, it is still a solutions to a very limited subset of problems.  I keep it in my back pocket, but I am less inclined to assume it the best methodology than I used to.