tag:blogger.com,1999:blog-5246987755651065286.post5555291029743003248..comments2024-02-22T16:15:42.388-08:00Comments on cbloom rants: 09-14-10 - Threaded Stdiocbloomhttp://www.blogger.com/profile/10714564834899413045noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-5246987755651065286.post-69987530120290225332017-08-19T19:26:10.223-07:002017-08-19T19:26:10.223-07:00Another good solution would be to remove the buffe...<br />Another good solution would be to remove the buffering from the stdio FILE and introduce a "FileView" object which has a position and a buffer. Then you can have mutliple FileViews per FILE which are in different threads, but the FileViews themselves must be used only from one thread at a time. Accesses to the FILE are serialized, accesses to the FileView are not. eg : <br /><br />eg. FILE * fp is shared.<br /><br />Thread1 has FileView f1(fp);<br />Thread2 has FileView f2(fp);<br /><br />FileView contains the buffer<br /><br /><br />(of course this is a mess with readwrite files and such; in general I hate dealing with mutable shared data, I like the model that data is either read-only and shared, or writeable and exclusively locked by one thread). <br /><br />This sounds like a problem that can be solved with an RW lock. I'm sure you know (I go to your blog for multithreading problems. You seem to be good at it) an RW lock locks a shared resource for multiple concurrent readers or one exclusive writer. It's something I'm looking to implement in my own CRT-like library. Having a FileView object is much cleaner/simpler and probably more performant than having IO and buffering coupled together.Jesse James Lactinhttps://www.blogger.com/profile/03874699883990324108noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-30654799088679178262010-09-16T21:42:15.444-07:002010-09-16T21:42:15.444-07:00Actually, or maybe it was the same thing again -- ...Actually, or maybe it was the same thing again -- maybe it was just because accesses didn't need to be locked and in the file-IO version I was doing fine-grained fread()s.<br /><br />But it's not like mmap is portable, so.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-49835345445516502672010-09-16T21:40:54.301-07:002010-09-16T21:40:54.301-07:00I definitely have seen speedups from using it on w...I definitely have seen speedups from using it on windows, I forget for what. Maybe my disk database, maybe the netflix prize stuff (although there it's also because you're avoiding needing separate memory for the disk cache).<br /><br />Like, I always figured the speedup would be miniscule--just the savings of touching twice as much memory--but instead I actually saw significant, inexplicable speedups, as if e.g. windows was prefetching when I went through the mmap and not when I went through read/fread.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-52487655265028133402010-09-15T20:19:13.816-07:002010-09-15T20:19:13.816-07:00Yeah portability is just a huge disaster in genera...Yeah portability is just a huge disaster in general. The solution to that is for people to stop buying anything but Intel/Windows , which is the one true platform. I blame all of you who like Linux and iPhones and game consoles for my pain. Quit buying that shit.<br /><br />"I don't think reaching into the hidden innards of FILE is viable"<br /><br />It's about as viable as relying on getc_nolock ;) Actually maybe more viable since I know it works on all MSVC versions since VC6 and that's not true of nolock.<br /><br />"obviously there's always the possibility of mmap."<br /><br />Somebody always has to mention memmap. It's like the Loch Ness monster, the situation where mmap is actually fast, somebody thinks they saw it once but can never reproduce it. I dunno if it works better on UNIX but on Windows it's almost always slower. (not that that matters)cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-6604329955092543452010-09-15T20:11:32.593-07:002010-09-15T20:11:32.593-07:00Anyway, the point is you can't really beat std...<i>Anyway, the point is you can't really beat stdio for byte-at-a-time streaming input, so don't bother. (on Windows, where the disk cache is pretty good).</i><br /><br />Yeah, but this comes up because I write portable code (both my free libs and my work code are supposed to be ultra-portable).<br /><br />I mean, I'm getting bug reports in stb_image from people who compilers that don't allow "//" comments in C, because they're on some exotic embedded platform. And stb_image is where I had this problem and had to write my own buffering layer.<br /><br />I don't think reaching into the hidden innards of FILE is viable, and I don't think fgets_nolock() or whatever are viable portable solutions. Maybe the new Cxx will add this, but it'll be another decade before we can assume it for max-portability.<br /><br />As to whether you can go any faster and the whole disk page cacheing memcpy thing, well, obviously there's always the possibility of mmap.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-79716919513665149442010-09-15T13:23:19.331-07:002010-09-15T13:23:19.331-07:00Interesting!
It looks like it's "_fgetc_...Interesting!<br /><br />It looks like it's "_fgetc_nolock" in my version of the VC headers.<br /><br />VC also has<br /><br />#define _CRT_DISABLE_PERFCRIT_LOCKS<br /><br />and<br /><br />_lock_file()<br /><br />it looks like these appeared in VC2005cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-13689432019499150822010-09-15T12:30:11.952-07:002010-09-15T12:30:11.952-07:00Instead of macrogetc you could also use getc_noloc...Instead of macrogetc you could also use getc_nolock (on windows) or getc_unlocked (on unix).<br /><br />On unix you can easily build a FileLock using flockfile/funlockfile. I suppose there must be something like that on windows too.castanohttps://www.blogger.com/profile/08088335278984724562noreply@blogger.com