TLDR : don't call eof on pipes in Windows! Use read returning zero instead to detect eof.
We observed that semi-randomly pipes would report EOF too soon and we'd get a truncated stream. (this is not the issue with ctrl-Z ; of course you have to make sure your pipe is set to binary).
To reproduce the issue you may use this simple piper :
// piper :
#include
the behavior of piper is this :
<
fcntl.h>
#include <
io.h>
int main(int argc,char *argv[])
{
int f_in = _fileno(stdin);
int f_out = _fileno(stdout);
// f_in & out are always 0 and 1
_setmode(f_in, _O_BINARY);
_setmode(f_out, _O_BINARY);
char buf[4096];
// NO : sees early eof!
// don't ask about eof until _read returns 0
while ( ! _eof(f_in) )
// YES : just for loop here
//for(;;)
{
int n = _read(f_in,buf,sizeof(buf));
if ( n > 0 )
{
_write(f_out,buf,n);
}
else if ( n == 0 )
{
// I think eof is always true here :
if ( _eof(f_in) )
break;
}
else
{
int err = errno;
if ( err == EAGAIN )
continue;
// respond to other errors?
if ( _eof(f_in) )
break;
}
}
return 0;
}
vc80.pdb 1,044,480
redirect input & ouput :
piper.exe < vc80.pdb > r:\out
r:\out 1,044,480
consistently copies whole stream, no problems.
Now using a pipe :
piper.exe < vc80.pdb | piper > r:\out
r:\out 16,384
r:\out 28,672
r:\out 12,288
semi-random output sizes due to hitting eof early
If the eof check marked with "NO" in the code is removed, and the for loop is used instead, piping
works fine.
I can only guess at what's happening, but here's a shot :
If the pipe reader asks about EOF and there is nothing currently pending to read in the pipe, eof() returns true.
Windows anonymous pipes are unbuffered. That is, they are lock-step between the reader & writer. When the reader calls read() it blocks until the writer puts something, and vice-versa. The bytes are copied directly from writer to reader without going through an internal OS buffer.
In this context, what this means is if the reader drains out everything the writer had to put, and then races ahead and
calls eof() before the writer puts anything, it sees eof true. If the writer puts something first, it seems eof false.
It's just a straight up race.
time reader writer
-----------------------------------------------------------
0 _write blocks waiting for reader
1 _eof returns false
2 _read consumes from writer
3 _write blocks waiting for reader
4 _eof returns false
5 _read consumes from writer
6 _eof returns true
7 _write blocks waiting for reader
at times 1 and 4, the eof check returns false because the writer had gotten to the pipe first. At time 6 the
reader runs faster and checks eof before the writer can put anything, now it seems eof is true.
As a check of this hypothesis : if you add a Sleep(1) call immediately before the _eof check (in the loop), there is no early eof observed, because the writer always gets a chance to put data in the pipe before the eof check.
Having this behavior be a race is pretty nasty. To avoid the problem, never ask about eof on pipes in Windows, instead use the return value of read(). The difference is that read() blocks on the writer process, waiting for it to either put some bytes or terminate. I believe this is a bug in Windows. Pipes should never be returning EOF as long as the process writing to them is alive.
No comments:
Post a Comment