Shell: How to pipe to another fd?

Recently I found myself wanting to pipe the output of one process into another, but - to make this a bit more interesting, I needed it to go to another fd than 0 (stdin).

So, in my head, I was thinking 3. As an alternative file descriptor to use. Piping the first process' output to the second process' fd 3. Pretty much, I had this in mind :

foo |3 bar

Which, somehow, would mean to put the reading end of the pipe not in fd 0 but fd 3 of bar. Of course, this isn't possible.

Thank you, good night!

Ever heard of the XY problem?

Let's rewind a bit : what am I trying to accomplish here ?

Simple, I basically have an exporter and an importer. Two different processes, and I'd like to pipe the data from the former - written on its stdout - to the later.

The later, which does not have to be reading from its stdin: it actually takes a filename as argument. And there's a pretty good reason for that, too : it will prompt for a password, reading it from its stdin.

(For the record it allows +n as file name where n is a number, which means using fd number n. Hence (maybe?) why I was looking to pipe to a known non-zero fd...)

So it turns out a simple exporter | importer wouldn't work, since, as it happens, the importer could read the data to import, but then when times come to prompt for the password, it would get an EOF on stdin leading to either an error, or an empty password, neither of which are wanted.

The solution you might already been thinking of, which took me longer to remember - probably because I was very much thinking of pipe as a basis for this and had forgotten all about process substitution - that solution is, indeed, process substitution.

importer <(exporter)

Now that should work, right? It is, actually, a piping situation, despite the lack of mention of it in the name. What this does is create a pipe from exporter's stdout into some fd, and put in importer's command-line (argv) the appropriate file name.

Something in the form of /proc/self/fd/3 (or more like 11 or so it seems). I.e. exactly what I was looking for !

Complication #2

Except.

Yeah, it's not that simple. See, I mentioned how importer prompts for a password, to be read from its stdin ? Cool, but it's not the only one !

exporter works just the same, and will too prompt for a password and want to read it from its stdin. So, what happens now ?

Well, it stalls.

Let's assume that importer is smart enough to block when trying to read from its file, or maybe it just asks for the password first of all. Either way, this is a non-issue. What is ?, you ask. The simple fact that both processes can't read their passwords from the "same" stdin, aka my terminal.

We were so close.. what will be needed to cross the line ? SID.

importer <(setsid exporter)

That's it. That was the solution I was looking for. Took me a little while to figure it out, hence me writing this in case this could be of help to anyone, but here it is.

Thank you, good night! :)

What's the explanation, Kenneth?

It works. Do you really need to know why ?

Well, that's bad luck, because I'll be honest : I'm not entirely sure why. If you happen to know (more) please let me know, I'll be curious to find out.

Here's what I got :

• Let's say what exporter does is become a group leader of its own process group, i.e. call setpgid(0, 0), blocks SIGTTOU in order to make its group the foreground process group, i.e. call tcsetpgrp(0, getpid()). (And then unblocks SIGTTOU to be clean.)
• With all that done, it is able to ask for its password without any issue, being in the foreground process group and all, life is well.

However - because there's a but - it wouldn't go as nicely. Specifically, once it got time for importer this time to ask for its password, it would then get a SIGTTOU since it is (no more) in the foreground process group. (I'm assuming here it gets SIGTTOU because it tries to disable echo due to asking for a password, it would of course get a SIGTTIN trying to read from its stdin had it not done so before, or only blocked the former signal and not the later. Not that blocking both would be of any help in this situation anyways.)

Of course all it'd take is to bring it back in the foreground (think fg) and voilà: asks for password and performs importation successfully.

There is no spoon

But here's the thing: that's not what happens here.

What setsid does it actually different : it will fork, thus making sure it's not a progress group leader, and simply call setsid(), which has no reason to fail.

So it succeeds, and.. somehow magic: everything works just fine. exporter is able to access the terminal and ask/read its password, then importer is able to do the same without anyone getting caught in any signal whatsoever.

It's nice.

I'm not sure why it happens, but it's nice nonetheless.

Or is it ? Because having thought about it some more, my understanding of this, is that it happens to work nicely (for me) because of how/when both importer and exporter do what they do, making that they don't collide with one another, but...

....correct me if I'm wrong, but in this case the reason exporter can ask/read for its password is indeed because it is in a new session, has no controlling terminal, and thusly is messing up the controlling terminal of another session.

Turns out it plays nice, and is finished by the time importer needs to ask for its password, something it has no issue doing since it is, properly, the foreground job of its controlling terminal.

So maybe the scenario as I originally described it would have been a proper, or at least more appropriate, way to do things. A scenario where exporter started its own progress group and became the foreground process group, thus effectively grabbing the terminal from importer's process group.

This allows it to ask for its password properly and without risks. Then ideally importer would actually catch SIGTTOU and make itself (its process group) the foreground job, grabbing back the controlling terminal, in order to properly handle its own password input.

Thing is, I'm not sure I know a way to do this from the shell. There's a tool one can get from s6 called s6-setsid which actually has some options to start a new session, background, or foreground process group, even possibly grabbing the terminal in the later case.

That later option matches what I've been describing before, and would lead to the described results. Including importer getting stopped by SIGTTOU when it wants to ask for its password.

And... that's as far as I can get. It's not possible to use that same s6-setsid trick on importer because it would happen too soon and lead to some unwanted blocking, nullifying the effects.

So we're left with having to type fg to bring importer back on the foreground and allow it to ask for its password and finish its importing. Not that big a deal in the end. Or one could always hack into those programs, the beauty of open-source software and whatnot.

Anyhow, that's all I got for today. With all that said, all that's left for me to do now is thank you for reading & wish you a good night, once and for all. :)