Unix Pipeline resources

Hey hey hey did someone mention the Unix pipelines? You could use some… resources. :)

P.S. Since you tagged “exam” I will definitely not be in the position to answer this; but that hasn’t stopped my urge to yap for the 100th++ time! (On second thought, I want this to be a wiki people can edit – append, modify, or otherwise delete entirely – so… I guess I am posting this as an answer. Sorry :)

(Just don’t tell me you are actually asking about the bitwise OR operator in Java… I actually kind of want to collect good stuff about pipe and IPC)

(Also fun fact: If you search for “pipes” in Piazza, 2/3 posts/threads are mine just about that. This will make it a fourth post… :D)

#0 The manuals (highly technical, but… useful to remember that they exist.)

• GNU Bash bash(1): [man page] [info page]
• The POSIX Bourne shell sh(1): [POSIX.1-2024]

#1 Friendly guides (the ones that are most likely to help)

• Advanced Topics > Pipe: from CS1000, a Unix manual that used to live right here on CSL in 2008 (not factorial)! (Retrieved via Internet Archive as the links now redirect to CSL docs – I know. sadge. i am literally depressed rn csl better bring it back /j)
• Pipes: A Brief Introduction: from the Linux Info Project. Includes a nice history lesson too (the HTML seems a bit messed up but it should be okay.)
• Special characters (search downward for “UNIX pipe”; unfortunately I can’t make a direct link there): you would be very delighted to hear about The Linux Document Project and how they have Bash programming tutorials (this one comes from Advanced Bash-Scripting Guide; there is also Bash Guide for Beginners)
• BashProgramming > Commands and Quoting: explained in three brief sentences, but I thought it’d be obligatory that I include Greycat’s Wiki somewhere. There is a very nice FAQ related to pipe, for instance – namely the subtle fact that pipelines are executed in a subshell. Here are some complements from people other than me to justify this inclusion.
• Watch Brian Kernighan (one the living legends from Bell Labs) explain!!! Here (AT&T archives) (and that first link of this post is secretly a video too… (I will be infinitely grateful for Computerphile inviting him over for talks qvq))
• Bourne Shell Tutorial > Simple Control Flow: very well-paced from what I can tell; written by Bruce Barnett. I know him because I was looking for a C Shell tutorial and stumbled upon this gold (the prologue right after the TOC gave me good chuckles – ah… takes me back to the good old days when I looked into @137 and there weren’t two thousand Piazza posts :) And greycat helped me remember the web site – thanks greycat!

#2.5 Some less directly useful, but more fun resources

• Pipe Practices: I was today years old when I learned that a “UNIX Pipe Card Game” exists. O_O (And oh, oh man, it gets so much crazier… (And I thought Computer Engineering for Babies was crazy enough, wow…)

• Useless Use of Cat (UUOC): This is just something nice to know. (Plus you can learn from the examples!) [wikipedia] [shellcheck SC2002]

#3 Advanced (For the adventurous warriors)

Those of you who love C/Unix…. this is for you. (I am not aware of the size of the set of people who fit the criteria, but I know (as a matter of fact) its cardinality is strictly greater than 0 (I mean, me – I love C/Unix, is that not obvious enough? These are resources for me even if none of you will read them! Ha! >:D))

• “How are Unix Pipes implemented?” on HN links to so many gems at once. But in particular… I like:

  • Pipes, Forks, & Dups: Understanding Command Execution and Input/Output Data Flow by Roslyn McConnell (very, very, very clean diagrams. I don’t think I’ve seen cleaner diagrams than this…)
  • This comment on how Doug McIlory initially conceived the Unix pipeline seems interesting.

• Rethinking the shell pipeline (and related HN thread): for those of you who use Windows, this is for you – because this is precisely what Powershell is about! Most specifically, its “Objected-oriented” approach to pipelines, rather than the text/bytestream/file-based interface in Unix. I don’t want to give you my opinion or tell you what to think; but there are some good arguments for and against this modern re-thinking of shell pipelines both in the blog comments and in HN; and in doing so we are putting Unix pipelines in a bigger context… critical thinking! (If you want to dig an even deeper hole than I did, I heard that the designer of Powershell wrote this thing called Monad Manifesto…)

• Pipes and FIFOs: this is just raw C programming. Here he talks about pipe(2) and mkfifo(2). (You know things are serious from the man page section alone: it’s not a command in section 1 (commands), not a function in section 3 (library subroutines), but a function in section 2 (system calls) – as close as it gets to the OS itself!)

  A note to people who plan to read the man pages in their terminal: you will need to specify the man page section for mkfifo because there is a synonymous page in section 1 (for the mkfifo(1) command). So instead of this: man mkfifo, run this: man 2 mkfifo, or this: man -S 2 mkfifo. (Actually, for some mysterious reason, mkfifo is in section 3 on Debian and Arch Linux. so replace 2 with 3 if that doesn’t work…)

• Since nothing is off-limits in this section (given that I marked it as “Advanced”), I can selfishly direct you to read the abomination I assembled with sort(1) | head(1) | column(1) | sed(1) in @724_f1. One thing I should add: I wouldn’t use cat -n anymore because cat(1) does too many things. But the awk alternative awk '{print NR "\t" $0}' Rob Pike and Brian Kernighan mention in “Program Design in the UNIX Environment” isn’t much prettier IMO. So imagine my delight when I discovered that nl(1) exists (both on Linux and on BSD. So now I use nl -b a (sometimes I squeeze the flag and option, but I’m not sure how portable this form is: nl -ba) and never look back. (yippee! ^^)

• Oh you think I am done? (I kind of wish I am too, it is kind of late now…) But I’ve written even more pipelines – this time relevant to project 1, where I demonstrated two pipelines that basically replicated the whole project: 18:53. (Pipelines are pretty cool if you think about it.)

• Edit (this part is not on Piazza): I explored my posts a little and, as it turns out there is also a pipeline in @785_f1 that decodes Git objects directly from the object database. It’s basically me walking through this part of Scott Chacon’s the Git Book but I managed to squeeze things to (1) fit on one line and (2) use two of my favorite general-purpose programming languages of all time. And you learn how Git object database really is just a simple content-addressible thingy (I mean, we can all thank Linus for having the foresight 20 years ago (or what’s the word, 远见…) but learning about it makes you cool as well, I think!)

P.S. I am hosting a copy on my CSL page (this URL is WRONG on Piazza though I have no intention to correct it unless I need to update), in case you want something nicer to look at. (Idk about CS400, but my write-ups are Creative Commons; so it is open and you are free to share outside of Piazza (as is most of the resources I linked in this wiki-ish post).) If you make edits, I will add it to the page if you email me and I think it’s good. You will be attributed unless you want to remain anonymous.