tag:blogger.com,1999:blog-5246987755651065286.post5599760124383955511..comments2024-02-22T16:15:42.388-08:00Comments on cbloom rants: Compression of Android Game APK Packages with Oodlecbloomhttp://www.blogger.com/profile/10714564834899413045noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-5246987755651065286.post-70519110713664792052018-03-27T23:45:52.208-07:002018-03-27T23:45:52.208-07:00Great response!
I like the idea of "solid&qu...Great response!<br /><br />I like the idea of "solid" archiving, never thought of that for some reason... <br /><br />Another thought, when combining files into a tar archive, or archives with an index, the files could be in there in any order. I wonder if there is a way to sort files cleverly, in order to improve compression even more?Rattenhirnhttps://www.blogger.com/profile/11110056835578391195noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-89160296355325987162018-03-27T12:03:04.075-07:002018-03-27T12:03:04.075-07:00Yes, my look at compression of whole tars is not m...Yes, my look at compression of whole tars is not meant to suggest that developers can or should do that now. Rather it provides a reference of how small this data could be without the constraints of the current APK system. It's also a possible compression level that the OS or transport layer could achieve with platform support (unpacking the tar at install time, possibly recompressing file by file).<br /><br />For transport over cell networks, minimum size is the main goal, and some time spent unpacking the tar to install is negligible. Oodle decode on modern mobile devices is 500 MB/s or higher. A 50 MB APK takes about 16 seconds to download over LTE in the US (around 3 MB/s), but only 0.1 seconds to unpack the whole thing with Oodle.<br /><br />There are a couple of different issues -<br /><br />1. What can developers do right now within the existing APK system?<br /><br />The best you can do is to take all your data files and put them together in a package like a tar, and then divide that at the seek granularity that you need for random access, maybe 256k or 1M chunks or whatever. (it depends on how you load your data; if you always need to load the whole set of data to run the game, then don't divide at all). As a developer you have to leave your system files out of this pack file. Then compress your pack file with Oodle and name it .mp3 or something so that the APK leaves it uncompressed.<br /><br />This is better than compressing file by file because it combines tiny files together. You could also compress file by file but only merge together your small files. The point is to make well constructed "paging units" that merge data that is usually loaded at the same time.<br /><br />2. There's the separate question of how much better it could be with OS level support, and what is the best thing to do there?<br /><br />The simplest thing would be to replace zlib compression in the APK's with Oodle. Since that decode is done at every app start, the improved decode speed of Oodle would be a nice benefit (on top of the size savings). The next incremental step would be to move to an archive that has "solid" archiving (combining small files into larger compression units) for the small files. Solid archiving is in pretty much every modern archiver (rar, 7z, etc.)<br /><br />Another incremental step would be to compress the system files (exe and java and such) for transmission and then unpack them at install time (but leave the other data as transmitted). This can gain you many megabytes from compressing those large files that are currently transmitted uncompressed.<br /><br />The best of all would be to transmit the game as a compressed tar and unpack at install time.cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-60623727225820067322018-03-27T00:22:24.199-07:002018-03-27T00:22:24.199-07:00This comment has been removed by a blog administrator.Rattenhirnhttps://www.blogger.com/profile/11110056835578391195noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-62281627945706688582018-03-27T00:21:53.186-07:002018-03-27T00:21:53.186-07:00This comment has been removed by a blog administrator.Rattenhirnhttps://www.blogger.com/profile/11110056835578391195noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-2754510124001258962018-03-27T00:21:39.153-07:002018-03-27T00:21:39.153-07:00Hello, all of this is almost completely equally tr...Hello, all of this is almost completely equally true for iOS apps as well, including the archive size comparisons. IPA files are also just Zip, and so is the Windows app package format (.appx if I remember correctly).<br /><br />But only Android does not unpack its apps during installation, which is both good and bad. Good for the users, as they usually take less precious storage space than on iOS devices. This is quite ironic, because storage space is much more previous on iOS devices. It's good for analyses like this, because you can just grab the APKs off a device.<br /><br />However, it is bad, because all data are transparently uncompressed when accessed, which not only can causes a lot of confusion when measuring load times, but also makes it double bad when using an internal archive and forgetting to turn off outer compression.<br /><br />This, somewhat indirectly, brings me to a thing I would really like to see from a modern compression library. Your suggestion to use an inner archive format (i.e. tar) has a few big downsides that make it unusable in practice. <br /><br />It will require decompression at install time. None of the mobile platforms supports this, so it has to happen at the first app start, which is a really bad user experience. It will present as additional load time (no matter what the screen says), and nothing turns users off quicker than load times. Also, platform owners impose (rather squishy) limits on overall load times, so it might even be rejected.<br /><br />Secondly, both the original install (from the platform store) and the uncompressed inner archive will hog previous storage space on the device. There is no way to get rid of the original install.<br /><br />Thirdly, usually you'll want to leave some things compressed up until load time, mostly images and sounds. So the simple inner archive approach might be bad for load times, which again, is bad for everybody.<br /><br />All of these things weigh worse than the improved download time and bandwidth saving. You install an app once, but, hopefully run it many many times.<br /><br />This brings me finally to my wished feature: A compression and archive format that combines the space saving of oodled tars with the ability for somewhat random access decompression. I know that these two goals are somewhat contradicting, but I believe there is a sweeter spot to be found than what we currently have.<br /><br />Finally, I realize that all this is way beyond the scope of your post, but I've had this topic on my chest for some time now and it seemed a good place to unload it and maybe get a discussion going! ;)Rattenhirnhttps://www.blogger.com/profile/11110056835578391195noreply@blogger.com