Artifact bcd7e9b47b469a065e5476c4e6c167edb9af9714

Ticket change [bcd7e9b47b] - Closed ticket 678eb0c20a: Large files make memory usage spike through the roof. plus 2 other changes by drh on 2009-09-16 16:20:55.

D 2009-09-16T16:20:55
J +comment \n\n<hr><i>drh\sadded\son\s2009-09-16\s16:20:55:</i><br>\nNo,\sfossil\sis\s<u>not</u>\spulling\sthe\swhole\srepository\sinto\smemory....\r\n\r\nBut\sit\sdoes\sconstruct\san\sentire\sHTTP\sreply\smessage\sin\smemory.\s\sThat\sinvolves\r\nreading\sin\sthe\s18MB\sblob\sout\sof\sthe\sdatabase\sand\sdecompressing\sit,\r\nthen\sappending\sthat\sblob\sto\sthe\r\nend\sof\sthe\sgrowing\sHTTP\sreply\s(there\sis\s36MB\sfor\syou\salready)\sthen\scompressing\r\nthe\sentire\sHTTP\sreply\susing\sZ-lib.\s\sAs\sthe\sHTTP\sreply\sbuffer\sgrows,\sit\suses\sthe\r\ntrick\sof\srealloc-and-copy\sso\sgrowing\sa\sbuffer\smight\smake\srequire\stwice\sthe\ssize\r\nof\sthe\sbuffer.\s\sEvery\scompress/uncompress\soperation\sinvolves\skeeping\sboth\r\nthe\soriginal\sand\sthe\smodified\scopy\sin\sRAM\sat\sleast\sfor\sa\sshort\swhile.\r\nAdd\sto\sthat\smalloc\sfragmentation\sand\sthe\smemory\sspace\sthat\r\nSQLite\swill\srequest\s(we\sconfigure\sSQLite\sin\sfossil\sto\suse\slots\sof\smemory\r\nbecause\sthat\smakes\sit\sfaster\sand\sbecause\sfossil\sis\sdesigned\sto\srun\son\sa\s\r\nworkstation,\snot\sa\scellphone!)\sand\sit\sis\slittle\swonder\sthat\syou\r\nare\srunning\sout\sof\sRAM.\r\n\r\nThe\sobvious\ssolution\shere\sis\sto\sget\smore\sRAM.\s\sThat\scan't\sbe\shard.\r\nI\suse\s[http://www.linode.com/\s|\sLinode]\sas\smy\sVPS\sprovider\sand\sthe\s\r\n<em>smallest</em>\ssystem\sthey\ssell\sis\s360MB.\r\nI\salso\suse\sfossil\son\s[http://www.he.net/\s|\sHurricane\sElectric].\s\sHE\sis\sa\r\nshared\shosting\sprovider,\snot\sa\sVPS\sprovider,\sbut\sshared\shosting\sis\ssufficient\r\nto\ssupport\sfossil\sand\syou\sdon't\shave\sarbitrary\sRAM\slimits.\r\n\r\nCould\sfossil\sbe\sreengineered\sto\ssupport\s18MB\sbinaries\son\sa\s64MB\sserver?\r\nProbably.\s\sBut\swhy\swould\syou\swant\sto?\s\sRAM\sis\scheap\sand\sgetting\scheaper\r\ndaily.\s\sFossil\sdoes\snot\suse\sthat\smuch\smemory\srelative\sto\sthe\ssize\sof\r\nthe\sfiles\sit\sstores\s-\sperhaps\sa\smultiplier\sof\s3\sor\s4,\sbut\sthat\sis\snot\sso\r\nmuch\sin\sthe\sgrand\sscheme\sof\sthings.\s\sThis\sproblem\shas\sa\svery\ssimple\ssolution:\r\nthrow\smemory\sat\sit\sand\smove\son.
J resolution Not_A_Bug
J status Closed
K 678eb0c20a07f601aa2b11439b7ac49ecaf21e5c
U drh
Z 1d9b67a04d52f36359cc25959be0280b