tag:blogger.com,1999:blog-5246987755651065286.post709837147673770195..comments2024-02-22T16:15:42.388-08:00Comments on cbloom rants: 03-30-13 - Error codescbloomhttp://www.blogger.com/profile/10714564834899413045noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-5246987755651065286.post-8544758430880557632015-01-09T13:42:00.998-08:002015-01-09T13:42:00.998-08:00Good to know.
Making Win32's ReadFile actuall...Good to know.<br /><br />Making Win32's ReadFile actually robust is *crazy* complicated.<br /><br />I should probably post a snippet that does it some day.<br /><br />I've also never gotten comfortable with Win32's networked file performance.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-36860675232471989092015-01-09T12:42:33.958-08:002015-01-09T12:42:33.958-08:00Just found this thread again and realized I never ...Just found this thread again and realized I never replied. :)<br /><br />In addition to the cases mentioned in the ReadFile docs (pipes, consoles/serial devices) I've also seen it return partial reads on network shares, even when there was no error. In particular, I've seen this on large-block (in the ballpark of 32MB) reads via SMB on Windows XP (a couple years ago). No idea whether that still happens, but I wouldn't be surprised.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-73738433046618069762013-04-06T08:43:42.826-07:002013-04-06T08:43:42.826-07:00"read() will return with less bytes read than..."read() will return with less bytes read than specified in the exact same cases as Windows ReadFile would"<br /><br />Where does this information come from? That's not what the docs say.<br /><br />I don't see anywhere in the Win32 docs that say ReadFile will return less than the # requested except in error cases.<br /><br />Conversely, the POSIX docs specifically say read() can return less than the # requested and do not guarantee under what conditions that is done.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-29850569911572737072013-04-05T16:38:06.819-07:002013-04-05T16:38:06.819-07:00read() will return with less bytes read than speci...read() will return with less bytes read than specified in the exact same cases as Windows ReadFile would (end of file, reading from a pipe/socket that doesn't have any extra data available, reading from a console buffer that has less than the specified amount of bytes you can read).<br /><br />The only extra complication is signals, but as said if you use SA_RESTART (which is the default for all signal handlers) read will always complete on Linux (unless it hits one of the other conditions I just mentioned), again see man 7 signal.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-72547802882298204622013-04-04T19:02:08.710-07:002013-04-04T19:02:08.710-07:00Or something. Bleh, whatever. I used to think it...Or something. Bleh, whatever. I used to think it was neat to learn all the quirks and tricks about each OS and API, but now I just find it ever so tedious.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-65616905334874486522013-04-04T19:00:23.347-07:002013-04-04T19:00:23.347-07:00@Fabian again - I just remember that I specificall...@Fabian again - I just remember that I specifically used "read" in that example, because on UNIX you have to call read() in a loop even without the EINTR issue. read() is allowed to return before the full read is done, even in non-EOF or error conditions.<br /><br />(presumably that doesn't actually happen in practice on normal files, only on network ports and other weird things, but because of the API spec if you don't call it in a loop you could have broken code)<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-84151268516518355582013-04-03T17:08:12.165-07:002013-04-03T17:08:12.165-07:00FWIW, I rewrote the error handling in Iggy a coupl...FWIW, I rewrote the error handling in Iggy a couple weeks ago. Now detailed error codes are only returned if it seems they can usefully be handled; boolean success/fail is returned if that can usefully be handled; otherwise void.<br /><br />All the old error cases are still detected and an error message is sent to a user-installed callback function which is intended to print to debug console.<br /><br />My slogan-y way of thinking about this is along the lines of "if the program can fix it return an error code; if the programmer must fix it print a message".Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-85966038337402287312013-04-03T08:34:31.015-07:002013-04-03T08:34:31.015-07:00Oh yeah, you're right. I was thinking that...Oh yeah, you're right. I was thinking that's terrible practice, but of course it's totally fine for test apps.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-87906258522246297572013-04-01T13:10:03.156-07:002013-04-01T13:10:03.156-07:00"Yeah yeah in real production code of course ...<i>"Yeah yeah in real production code of course you check the error code, but for little test apps you should be able to do: [..]</i><br />And you can. At least on Linux, EINTR only occurs in programs that ask for it (i.e. hook signal handlers and don't specify SA_RESTART). See <a href="http://man7.org/linux/man-pages/man7/signal.7.html" rel="nofollow">man 7 signal</a> for details, specifically the "Interruption of system calls and library functions by signal handlers" section.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-4346322950940250662013-03-31T19:51:33.180-07:002013-03-31T19:51:33.180-07:00cbloom --
Nice link on complexity. This gets to t...cbloom --<br /><br />Nice link on complexity. This gets to the issue:<br /><br />"Early distributed-hypertext projects such as NLS and Xanadu were severely constrained by the MIT-philosophy assumption that dangling links were an unacceptable breakdown in the user interface; this constrained the systems to either browsing only a controlled, closed set of documents (such as on a single CD-ROM) or implementing various increasingly elaborate replication, caching, and indexing methods in an attempt to prevent documents from randomly disappearing. Tim Berners-Lee cut through this Gordian knot by punting the problem in classic New Jersey style."<br /><br />So it's not only true that "worse" can be better, "better" is sometimes just worse masquerading as better.<br /><br />"Better" is stubborn. It assumes that it knows which problems are important. Dangling links turned out to be an acceptable trade-off.Anonymoushttps://www.blogger.com/profile/03640120834994990493noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-43791072093445329642013-03-31T18:34:21.075-07:002013-03-31T18:34:21.075-07:00"My understanding is that Java's exceptio..."My understanding is that Java's exceptions force the programmer to actually do something about the exception"<br /><br />The programming language can't force the programmer to do anything, save add a bunch of boilerplate to satisfy the compiler. Java's checked exceptions are widely considered to have been a bad idea. Googling "checked exceptions" turns up mostly complaints.Anonymoushttps://www.blogger.com/profile/03640120834994990493noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-47992567210686366262013-03-31T17:25:01.584-07:002013-03-31T17:25:01.584-07:00"I thought you were talking more about the mo..."I thought you were talking more about the more game-style allocate yourself a 500M chunk at startup"<br /><br />I partly was, but you're right that's an unusual special case.johnbhttps://www.blogger.com/profile/15875751919051082588noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-78261364806032667152013-03-31T16:54:16.263-07:002013-03-31T16:54:16.263-07:00@jb - yeah okay I basically agree with that. A mo...@jb - yeah okay I basically agree with that. A more general principal is that if your app might fail anywhere, try to do it as soon as possible. Don't run a long compute and then fail a malloc after half an hour - try to preload all your resource acquisition.<br /><br />I thought you were talking more about the more game-style allocate yourself a 500M chunk at startup (regardless of whether you will actually need that much in this session) and then allocate out of that.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-76071120997058282782013-03-31T15:18:48.682-07:002013-03-31T15:18:48.682-07:00Java seems to have a better mechanism to handle er...Java seems to have a better mechanism to handle errors, it's called "exceptions".<br /><br />My understanding is that Java's exceptions force the programmer to actually do something about the exception, if he wants the program to go on. <br />No more "silent error" because the programmer did not checked if the return code is an error message, something which is much too easy to do in C.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-78551446542792597512013-03-31T14:07:53.988-07:002013-03-31T14:07:53.988-07:00"Apps should consume my system resources prop..."Apps should consume my system resources proportionally to how much I use them".<br /><br />Right, but 1) I already said that you can't allocate absolutely everything up front if your dataset size is unknown, and 2) sharing memory between programs is what the virtual memory system is for.<br /><br />Individual programs should still reserve what they need as soon as they can work out how much they need; that way they fail early or not at all instead of failing in the middle of their work and having to either blow up completely or paying the development and maintenance cost of having zero-allocation graceful clean-up code absolutely everywhere.johnbhttps://www.blogger.com/profile/15875751919051082588noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-28067781735583721982013-03-31T13:45:58.817-07:002013-03-31T13:45:58.817-07:00"You should really have fixed memory budgets ..."You should really have fixed memory budgets for everything and just allocate everything at the start, and then you don't have to worry about it."<br /><br />Total disagree. The range of applications where this is appropriate is zero narrow, perhaps only games (and not casual games that the user might play while doing other things).<br /><br />Apps should consume my system resources proportionally to how much I use them.<br /><br />In fact one of the massive fails of the modern era is the way everyone using the standard STL's never return any memory to the OS, because STL's all come with a freelist node allocator that never gives back memory, so memory usage never goes down from peak.<br /><br />Allocation strategies all have different tradeoffs.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-26494811264671822922013-03-31T13:41:44.510-07:002013-03-31T13:41:44.510-07:00WRST the original Worse is Better I found this to ...WRST the original Worse is Better I found this to be a pretty decent discussion of the issue of complexity :<br /><br />http://www.faqs.org/docs/artu/complexitychapter.html<br /><br />As for why EINTR is not useless in reality, see :<br /><br />http://en.wikipedia.org/wiki/PCLSRing<br /><br />http://www.250bpm.com/blog:12<br /><br />http://marc.info/?l=linux-kernel&m=97328060332308&w=2<br /><br />https://issues.apache.org/bugzilla/show_bug.cgi?id=41975<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-87695608317730696252013-03-30T20:03:54.523-07:002013-03-30T20:03:54.523-07:00Isn't EINTR something you have to deal with on...Isn't EINTR something you have to deal with only if you're writing your own signal handlers? It seems like a simplifying assumption if, in your signal handler, you don't have to worry that another part of the process is blocked on a system call.Anonymoushttps://www.blogger.com/profile/03640120834994990493noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-23112448575401561742013-03-30T19:47:10.078-07:002013-03-30T19:47:10.078-07:00"I can't speak to how the author intended..."I can't speak to how the author intended it (the tone is pretty hard for me to parse in that piece), but since then people have definitely taken it as an argument for simplicity of the implementation of the OS/library over all else."<br /><br />It's a very old document, in computer terms, and I think it can't be completely separated from its historical context. It came out of the peculiar Lisp culture that was in decline at that time. Although I agree with it, I think it's ironic that Lisp is held up as "better" -- Lisp was hardly designed as a "diamond-like jewel." Common Lisp and Scheme were late implementations and benefited from a long evolution.Anonymoushttps://www.blogger.com/profile/03640120834994990493noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-1539416609264631372013-03-30T19:25:23.771-07:002013-03-30T19:25:23.771-07:00"look how terse my Hello World program is! ....."look how terse my Hello World program is! ...we still have people around with formalism hard-ons like the type theory nerds who explore this space and espouse the benefits of things like algebraic data types, monoids..."<br /><br />Programming Languages as an academic discipline is so dull... inventing new ways to solve problems that already have solutions... most of the interesting/useful stuff has been known for three decades or more. Functional programming languages become ever more esoteric. To me, a programming language serves a purpose not unlike spoken language. Yeah, English has flaws, and you could invent a better language, according to some definition of better, maybe something along the lines of Esperanto. But the point is what you're trying to say and that you can make yourself understood, not that you're speaking the perfect ideal language -- whatever that would mean.<br /><br />"Unfortunately, because our languages and tools in general don't help with that way of writing code, the effort required to work out a (safe) memory budget and allocate everything up front is massive and so it's overall cheaper to just assume you'll have enough memory and blow up when you find out you're wrong."<br /><br />I don't think that's an unreasonable way to go. If you're targeting a 32-bit architecture, then you might need to care a lot about virtual address space, but physical memory is a fluid thing.<br /><br />Here's some nitty-gritty on memory allocation in Linux:<br /><br />https://news.ycombinator.com/item?id=2544387<br />http://www.etalabs.net/overcommit.html<br /><br />I was going to make the point that the second link calls a "myth," but now I'd feel like a liar if I said it, that your app won't know if the system runs out of memory until a page fault. Obviously it's complicated and depends a lot on the OS you happen to be running on.Anonymoushttps://www.blogger.com/profile/03640120834994990493noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-69091079044333965692013-03-30T17:58:18.170-07:002013-03-30T17:58:18.170-07:00BTW adding on to that comment, it gave me a moment...BTW adding on to that comment, it gave me a moment of clarity :<br /><br />Any time you are writing a function, ask yourself :<br /><br />Will a caller of this function actually realistically write code that takes the error value I return and checks it and does something useful with that information?<br /><br />(eg. will the client write code like<br /><br />int err = whatever(..)<br />if ( err == Eblah )<br /> do X<br />else if ( err = Eshite )<br /> do Y<br />etc.<br /><br />)<br /><br />If so, then yes, sure return an error.<br /><br />But if you are honest with yourself and most callers of the function will just want to do :<br /><br />whatever();<br /><br />or<br /><br />int err = whatever();<br />ASSERT_RELEASE( err == EOK );<br /><br />then don't return an error code.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-57417356598320845212013-03-30T17:54:20.287-07:002013-03-30T17:54:20.287-07:00"Tangential to the main point, but still perh..."Tangential to the main point, but still perhaps useful to you, GCC (and probably clang) have the function attribute "warn_unused_result" that complains when you silently let a return value slip by."<br /><br />Yeah I spotted that in one of my recent gcc ports.<br /><br />If all your functions that shouldn't really ever fail are constantly spewing return values at you, then I think that warning is more annoyance than useful.<br /><br />But if you did as I advise and made most errors be either explosions or handled inside the function, and only return error codes where it is likely and reasonable that the caller actually will write error checking code - then I think this warning is pretty cool and would be a good robustinator.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-21712081808337820772013-03-30T17:52:41.760-07:002013-03-30T17:52:41.760-07:00"First of all, I've always felt that &quo..."First of all, I've always felt that "worse is better" was more of an apology than a design principle."<br /><br />I can't speak to how the author intended it (the tone is pretty hard for me to parse in that piece), but since then people have definitely taken it as an argument for simplicity of the implementation of the OS/library over all else.<br />cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-55804052710735216182013-03-30T16:29:27.146-07:002013-03-30T16:29:27.146-07:00The EINTR thing (sort of) gets more complicated, b...The EINTR thing (sort of) gets more complicated, because different systems and different syscalls do different things. On some platforms, with some sigaction and signal handling configurations, for some syscalls, the system will do the right thing and restart automatically instead of sending EINTR. (c.f. glibc's signal docs and the SA_RESTART sigaction flag)<br /><br />Although I suppose it doesn't matter; you just always write the EINTR handling loop and if it doesn't loop then you haven't lost anything significant.<br /><br />Memory allocation failure (system resource allocation failure in general) is a PITA. You should really have fixed memory budgets for everything and just allocate everything at the start, and then you don't have to worry about it. Of course some programs can't do that because they have a dataset of unknown size and can't process data as a stream with a fixed-size working buffer. In that case you should compute an upper bound on memory use as early as possible and allocate it so that you've only got a handful of places where you have to cope with memory allocation failure. But some programs can't even do *that* because they're doing something with no useful way of putting a bound on memory requirements, and then you get the worst case and have to write all of the code so that you can always get back to a stable internal state, and you can always gracefully save your file and close, all without any allocation (so having fixed, pre-allocated memory to handle all of that stuff is still essential).<br /><br />Unfortunately, because our languages and tools in general don't help with that way of writing code, the effort required to work out a (safe) memory budget and allocate everything up front is massive and so it's overall cheaper to just assume you'll have enough memory and blow up when you find out you're wrong.johnbhttps://www.blogger.com/profile/15875751919051082588noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-46623342282620410992013-03-30T11:02:09.580-07:002013-03-30T11:02:09.580-07:00First of all, I've always felt that "wors...First of all, I've always felt that "worse is better" was more of an apology than a design principle. I think the root issue is that the benefits of formalism often don't pass any Pepsi-challenges (aka, look how terse my Hello World program is!).<br /><br />Fortunately, we still have people around with formalism hard-ons like the type theory nerds who explore this space and espouse the benefits of things like algebraic data types, monoids, and the zoo of monad utility (zippers, iteratee, etc). Of course, the formal versions of these tend to be discovered long after the hackers find the specialized or shitty versions. In other words, these techniques are outbred a la Idiocracy.<br /><br />Tangential to the main point, but still perhaps useful to you, GCC (and probably clang) have the function attribute "warn_unused_result" that complains when you silently let a return value slip by.<br /><br />As for your malloc example, a somewhat useful wrapper is a NOTNULL(ptr) thing that panic-fails if ptr == NULL and return ptr otherwise (or the macro equivalent).won3dhttps://www.blogger.com/profile/09787472194187459747noreply@blogger.com