Artifact 0f9001c9f0d15a71bc48a46351fda01b62d9bbcc:
File cvs2fossil.txt part of check-in [27ed4f7dc3] - Extended pass InitCsets and underlying code with more log output geared towards memory introspection, and added markers for special locations. Extended my notes with general observations from the first test runs over my example CVS repositories. by aku on 2008-02-16 06:46:41.
0000: 0a 4b 6e 6f 77 6e 20 70 72 6f 62 6c 65 6d 73 20 .Known problems 0010: 61 6e 64 20 61 72 65 61 73 20 74 6f 20 77 6f 72 and areas to wor 0020: 6b 20 6f 6e 0a 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d k on.=========== 0030: 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d ================ 0040: 3d 3d 3d 3d 3d 3d 3d 3d 0a 0a 2a 09 4e 6f 74 20 ========..*.Not 0050: 79 65 74 20 61 62 6c 65 20 74 6f 20 68 61 6e 64 yet able to hand 0060: 6c 65 20 74 68 65 20 73 70 65 63 69 66 69 63 61 le the specifica 0070: 74 69 6f 6e 20 6f 66 20 6d 75 6c 74 69 70 6c 65 tion of multiple 0080: 20 70 72 6f 6a 65 63 74 73 0a 09 66 6f 72 20 6f projects..for o 0090: 6e 65 20 43 56 53 20 72 65 70 6f 73 69 74 6f 72 ne CVS repositor 00a0: 79 2e 20 49 2e 65 2e 20 49 20 63 61 6e 2c 20 66 y. I.e. I can, f 00b0: 6f 72 20 65 78 61 6d 70 6c 65 2c 20 69 6d 70 6f or example, impo 00c0: 72 74 20 61 6c 6c 20 6f 66 0a 09 74 63 6c 6c 69 rt all of..tclli 00d0: 62 2c 20 6f 72 20 61 20 73 69 6e 67 6c 65 20 73 b, or a single s 00e0: 75 62 70 72 6f 6a 65 63 74 20 6f 66 20 74 63 6c ubproject of tcl 00f0: 6c 69 62 2c 20 6c 69 6b 65 20 74 6b 6c 69 62 2c lib, like tklib, 0100: 20 62 75 74 20 6e 6f 74 0a 09 6d 75 6c 74 69 70 but not..multip 0110: 6c 65 20 73 75 62 2d 70 72 6f 6a 65 63 74 73 20 le sub-projects 0120: 69 6e 20 6f 6e 65 20 67 6f 2e 0a 0a 2a 09 57 65 in one go...*.We 0130: 20 68 61 76 65 20 74 6f 20 6c 6f 6f 6b 20 69 6e have to look in 0140: 74 6f 20 74 68 65 20 70 61 73 73 20 27 49 6e 69 to the pass 'Ini 0150: 74 43 73 65 74 73 27 20 61 6e 64 20 68 75 6e 74 tCsets' and hunt 0160: 20 66 6f 72 20 74 68 65 0a 09 63 61 75 73 65 20 for the..cause 0170: 6f 66 20 74 68 65 20 6c 61 72 67 65 20 61 6d 6f of the large amo 0180: 75 6e 74 20 6f 66 20 6d 65 6d 6f 72 79 20 69 74 unt of memory it 0190: 20 69 73 20 67 6f 62 62 6c 69 6e 67 20 75 70 2e is gobbling up. 01a0: 0a 0a 09 52 65 73 75 6c 74 73 20 66 72 6f 6d 20 ...Results from 01b0: 74 68 65 20 66 69 72 73 74 20 6c 6f 6f 6b 20 75 the first look u 01c0: 73 69 6e 67 20 74 68 65 20 6e 65 77 20 6d 65 6d sing the new mem 01d0: 6f 72 79 20 74 72 61 63 6b 69 6e 67 0a 09 73 75 ory tracking..su 01e0: 62 73 79 73 74 65 6d 3a 0a 0a 09 28 31 29 20 54 bsystem:...(1) T 01f0: 68 65 20 67 65 6e 65 72 61 6c 20 61 72 63 68 69 he general archi 0200: 74 65 63 74 75 72 65 2c 20 77 6f 72 6b 66 6c 6f tecture, workflo 0210: 77 2c 20 69 73 20 61 20 62 69 74 20 77 61 73 74 w, is a bit wast 0220: 65 66 75 6c 2e 20 41 6c 6c 0a 09 20 20 20 20 63 eful. All.. c 0230: 68 61 6e 67 65 73 65 74 73 20 61 72 65 20 67 65 hangesets are ge 0240: 6e 65 72 61 74 65 64 20 61 6e 64 20 6b 65 70 74 nerated and kept 0250: 20 69 6e 20 6d 65 6d 6f 72 79 20 62 65 66 6f 72 in memory befor 0260: 65 20 67 65 74 74 69 6e 67 0a 09 20 20 20 20 70 e getting.. p 0270: 65 72 73 69 73 74 65 64 2e 20 54 68 69 73 20 6d ersisted. This m 0280: 65 61 6e 73 20 74 68 61 74 20 61 6c 6c 6f 63 61 eans that alloca 0290: 74 65 64 20 6d 65 6d 6f 72 79 20 70 69 6c 65 73 ted memory piles 02a0: 20 75 70 20 6f 76 65 72 0a 09 20 20 20 20 74 69 up over.. ti 02b0: 6d 65 2c 20 77 69 74 68 20 6c 61 74 65 72 20 63 me, with later c 02c0: 68 61 6e 67 65 73 65 74 73 20 70 75 73 68 69 6e hangesets pushin 02d0: 67 20 74 68 65 20 62 6f 75 6e 64 61 72 69 65 73 g the boundaries 02e0: 2e 20 54 68 69 73 0a 09 20 20 20 20 69 73 20 6d . This.. is m 02f0: 61 64 65 20 77 6f 72 73 65 20 74 68 61 74 20 73 ade worse that s 0300: 6f 6d 65 20 6f 66 20 74 68 65 20 70 72 65 6c 69 ome of the preli 0310: 6d 69 6e 61 72 79 20 63 68 61 6e 67 65 73 65 74 minary changeset 0320: 73 20 73 65 65 6d 0a 09 20 20 20 20 74 6f 20 72 s seem.. to r 0330: 65 71 75 69 72 65 20 61 20 6c 6f 74 20 6f 66 20 equire a lot of 0340: 74 65 6d 70 6f 72 61 72 79 20 6d 65 6d 6f 72 79 temporary memory 0350: 20 61 73 20 70 61 72 74 20 6f 66 20 67 65 74 74 as part of gett 0360: 69 6e 67 0a 09 20 20 20 20 62 72 6f 6b 65 6e 20 ing.. broken 0370: 64 6f 77 6e 20 69 6e 74 6f 20 74 68 65 20 61 63 down into the ac 0380: 74 75 61 6c 20 6f 6e 65 73 2e 20 49 6e 69 74 69 tual ones. Initi 0390: 74 69 61 6c 69 7a 65 42 72 65 61 6b 53 74 61 74 tializeBreakStat 03a0: 65 0a 09 20 20 20 20 73 65 65 6d 73 20 74 6f 20 e.. seems to 03b0: 62 65 20 74 68 65 20 63 75 6c 70 72 69 74 20 68 be the culprit h 03c0: 65 72 65 2e 20 49 74 73 20 6d 65 6d 6f 72 79 20 ere. Its memory 03d0: 75 73 61 67 65 20 69 73 20 70 6f 73 73 69 62 6c usage is possibl 03e0: 79 0a 09 20 20 20 20 71 75 61 64 72 61 74 69 63 y.. quadratic 03f0: 20 69 6e 20 74 68 65 20 6e 75 6d 62 65 72 20 6f in the number o 0400: 66 20 69 74 65 6d 73 20 69 6e 20 74 68 65 20 63 f items in the c 0410: 68 61 6e 67 65 73 65 74 2e 0a 0a 09 28 32 29 20 hangeset....(2) 0420: 41 20 6e 75 6d 62 65 72 20 6f 66 20 73 6d 61 6c A number of smal 0430: 6c 20 69 6e 65 66 66 69 63 69 65 6e 63 69 65 73 l inefficiencies 0440: 2e 20 4c 69 6b 65 20 27 73 74 61 74 65 20 65 76 . Like 'state ev 0450: 61 6c 27 20 61 6c 77 61 79 73 0a 09 20 20 20 20 al' always.. 0460: 70 75 6c 6c 69 6e 67 20 74 68 65 20 77 68 6f 6c pulling the whol 0470: 65 20 72 65 73 75 6c 74 20 69 6e 74 6f 20 6d 65 e result into me 0480: 6d 6f 72 79 20 62 65 66 6f 72 65 20 70 72 6f 63 mory before proc 0490: 65 73 73 69 6e 67 20 69 74 0a 09 20 20 20 20 77 essing it.. w 04a0: 69 74 68 20 27 66 6f 72 65 61 63 68 27 2e 20 48 ith 'foreach'. H 04b0: 65 72 65 20 70 6f 74 65 6e 74 69 61 6c 6c 79 20 ere potentially 04c0: 6c 61 72 67 65 20 6c 69 73 74 73 2e 0a 0a 09 28 large lists....( 04d0: 33 29 20 57 65 20 6d 61 69 6e 74 61 69 6e 20 61 3) We maintain a 04e0: 6e 20 69 6e 2d 6d 65 6d 6f 72 79 20 6d 61 70 20 n in-memory map 04f0: 66 72 6f 6d 20 74 61 67 67 65 64 20 69 74 65 6d from tagged item 0500: 73 20 74 6f 20 74 68 65 69 72 0a 09 20 20 20 20 s to their.. 0510: 63 68 61 6e 67 65 73 65 74 73 2e 20 57 68 69 6c changesets. Whil 0520: 65 20 74 68 69 73 20 69 73 20 6e 65 65 64 65 64 e this is needed 0530: 20 6c 61 74 65 72 20 69 6e 20 74 68 65 20 73 6f later in the so 0540: 72 74 69 6e 67 0a 09 20 20 20 20 70 61 73 73 65 rting.. passe 0550: 73 20 64 75 72 69 6e 67 20 74 68 65 20 63 72 65 s during the cre 0560: 61 74 69 6f 6e 20 74 68 69 73 20 69 73 20 77 61 ation this is wa 0570: 73 74 65 64 20 73 70 61 63 65 2e 20 41 6e 64 20 sted space. And 0580: 61 6c 73 6f 0a 09 20 20 20 20 77 61 73 74 65 64 also.. wasted 0590: 20 74 69 6d 65 2c 20 74 6f 20 6d 61 69 6e 74 61 time, to mainta 05a0: 69 6e 20 69 74 20 64 75 72 69 6e 67 20 74 68 65 in it during the 05b0: 20 63 72 65 61 74 69 6f 6e 20 61 6e 64 0a 09 20 creation and.. 05c0: 20 20 20 62 72 65 61 6b 69 6e 67 2e 0a 0a 09 43 breaking....C 05d0: 68 61 6e 67 65 73 3a 0a 0a 09 28 61 29 20 52 65 hanges:...(a) Re 05e0: 2d 61 72 63 68 69 74 65 63 74 20 74 6f 20 63 72 -architect to cr 05f0: 65 61 74 65 2c 20 62 72 65 61 6b 2c 20 61 6e 64 eate, break, and 0600: 20 70 65 72 73 69 73 74 20 63 68 61 6e 67 65 73 persist changes 0610: 65 74 73 20 6f 6e 65 0a 09 20 20 20 20 62 79 20 ets one.. by 0620: 6f 6e 65 2c 20 63 6f 6d 70 6c 65 74 65 6c 79 20 one, completely 0630: 72 65 6c 65 61 73 69 6e 67 20 61 6c 6c 20 61 73 releasing all as 0640: 73 6f 63 69 61 74 65 64 20 69 6e 2d 6d 65 6d 6f sociated in-memo 0650: 72 79 20 64 61 74 61 0a 09 20 20 20 20 62 65 66 ry data.. bef 0660: 6f 72 65 20 67 6f 69 6e 67 20 74 6f 20 74 68 65 ore going to the 0670: 20 6e 65 78 74 2e 20 53 68 6f 75 6c 64 20 62 65 next. Should be 0680: 20 6c 6f 77 2d 68 61 6e 67 69 6e 67 20 66 72 75 low-hanging fru 0690: 69 74 20 77 69 74 68 0a 09 20 20 20 20 68 69 67 it with.. hig 06a0: 68 20 69 6d 70 61 63 74 2c 20 61 73 20 77 65 20 h impact, as we 06b0: 68 61 76 65 20 61 6c 6c 20 74 68 65 20 6e 65 63 have all the nec 06c0: 65 73 73 61 72 79 20 6f 70 65 72 61 74 69 6f 6e essary operation 06d0: 73 0a 09 20 20 20 20 61 6c 72 65 61 64 79 2c 20 s.. already, 06e0: 6a 75 73 74 20 6e 6f 74 20 69 6e 20 74 68 61 74 just not in that 06f0: 20 6f 72 64 65 72 2c 20 61 6e 64 20 74 68 61 74 order, and that 0700: 20 61 6c 6f 6e 65 20 73 68 6f 75 6c 64 0a 09 20 alone should.. 0710: 20 20 20 61 6c 72 65 61 64 79 20 6b 65 65 70 20 already keep 0720: 74 68 65 20 70 69 6c 65 20 66 72 6f 6d 20 66 6f the pile from fo 0730: 72 6d 69 6e 67 2c 20 6d 61 6b 69 6e 67 20 74 68 rming, making th 0740: 65 20 73 70 69 6b 65 73 20 6f 66 0a 09 20 20 20 e spikes of.. 0750: 20 28 32 29 20 6d 6f 72 65 20 6d 61 6e 61 67 65 (2) more manage 0760: 61 62 6c 65 2e 0a 0a 09 28 62 29 20 4c 6f 6f 6b able....(b) Look 0770: 20 69 6e 74 6f 20 74 68 65 20 73 6d 61 6c 6c 65 into the smalle 0780: 72 20 70 72 6f 62 6c 65 6d 73 20 64 65 73 63 72 r problems descr 0790: 69 62 65 64 20 69 6e 20 28 32 29 2c 20 61 6e 64 ibed in (2), and 07a0: 0a 09 20 20 20 20 65 73 70 65 63 69 61 6c 6c 79 .. especially 07b0: 20 28 33 29 2e 20 54 68 65 73 65 20 73 68 6f 75 (3). These shou 07c0: 6c 64 20 73 74 69 6c 6c 20 62 65 20 6c 6f 77 2d ld still be low- 07d0: 68 61 6e 67 69 6e 67 20 66 72 75 69 74 2c 0a 09 hanging fruit,.. 07e0: 20 20 20 20 61 6c 74 68 6f 75 67 68 20 6f 66 20 although of 07f0: 6c 65 73 73 65 72 20 65 66 66 65 63 74 20 74 68 lesser effect th 0800: 61 6e 20 28 61 29 2e 20 46 6f 72 20 28 33 29 20 an (a). For (3) 0810: 64 69 73 61 62 6c 65 20 74 68 65 0a 09 20 20 20 disable the.. 0820: 20 6d 61 70 20 61 6e 64 20 69 74 73 20 6d 61 69 map and its mai 0830: 6e 74 65 6e 61 63 65 20 64 75 72 69 6e 67 20 63 ntenace during c 0840: 6f 6e 73 74 72 75 63 74 69 6f 6e 2c 20 61 6e 64 onstruction, and 0850: 20 70 75 74 20 69 74 0a 09 20 20 20 20 69 6e 74 put it.. int 0860: 6f 20 61 20 73 65 70 61 72 61 74 65 20 63 6f 6d o a separate com 0870: 6d 61 6e 64 2c 20 74 6f 20 62 65 20 75 73 65 64 mand, to be used 0880: 20 77 68 65 6e 20 6c 6f 61 64 69 6e 67 20 74 68 when loading th 0890: 65 0a 09 20 20 20 20 63 72 65 61 74 65 64 20 63 e.. created c 08a0: 68 61 6e 67 65 73 65 74 73 20 61 74 20 74 68 65 hangesets at the 08b0: 20 65 6e 64 2e 0a 0a 09 28 63 29 20 57 69 74 68 end....(c) With 08c0: 20 6c 61 72 67 65 72 20 65 66 66 65 63 74 2c 20 larger effect, 08d0: 62 75 74 20 6d 6f 72 65 20 64 69 66 66 69 63 75 but more difficu 08e0: 6c 74 20 74 6f 20 61 63 68 69 65 76 65 2c 20 67 lt to achieve, g 08f0: 6f 20 69 6e 74 6f 0a 09 20 20 20 20 63 6f 6d 6d o into.. comm 0900: 61 6e 64 20 27 49 6e 69 74 69 61 6c 69 7a 65 42 and 'InitializeB 0910: 72 65 61 6b 53 74 61 74 65 27 20 61 6e 64 20 74 reakState' and t 0920: 68 65 20 70 72 65 63 65 64 69 6e 67 0a 09 20 20 he preceding.. 0930: 20 20 27 69 6e 74 65 72 6e 61 6c 73 75 63 63 65 'internalsucce 0940: 73 73 6f 72 73 27 2c 20 61 6e 64 20 72 65 61 72 ssors', and rear 0950: 63 68 69 74 65 63 74 20 69 74 2e 20 44 65 66 69 chitect it. Defi 0960: 6e 69 74 65 6c 79 20 6e 6f 74 20 61 0a 09 20 20 nitely not a.. 0970: 20 20 6c 6f 77 2d 68 61 6e 67 69 6e 67 20 66 72 low-hanging fr 0980: 75 69 74 2e 20 50 6f 73 73 69 62 6c 79 20 61 6c uit. Possibly al 0990: 73 6f 20 73 6f 6d 65 74 68 69 6e 67 20 77 65 20 so something we 09a0: 63 61 6e 20 73 6b 69 70 20 69 66 0a 09 20 20 20 can skip if.. 09b0: 20 64 6f 69 6e 67 20 28 61 29 20 68 61 64 20 61 doing (a) had a 09c0: 20 6c 61 72 67 65 20 65 6e 6f 75 67 68 20 65 66 large enough ef 09d0: 66 65 63 74 2e 0a 0a 2a 09 4c 6f 6f 6b 20 61 74 fect...*.Look at 09e0: 20 74 68 65 20 64 65 70 65 6e 64 65 6e 63 69 65 the dependencie 09f0: 73 20 6f 6e 20 65 78 74 65 72 6e 61 6c 20 70 61 s on external pa 0a00: 63 6b 61 67 65 73 20 61 6e 64 20 63 6f 6e 73 69 ckages and consi 0a10: 64 65 72 0a 09 77 68 69 63 68 20 6f 66 20 74 68 der..which of th 0a20: 65 6d 20 63 61 6e 20 62 65 20 6d 6f 76 65 64 20 em can be moved 0a30: 69 6e 74 6f 20 74 68 65 20 69 6d 70 6f 72 74 65 into the importe 0a40: 72 2c 20 65 69 74 68 65 72 20 61 73 20 61 0a 09 r, either as a.. 0a50: 73 69 6d 70 6c 65 20 75 74 69 6c 69 74 79 20 63 simple utility c 0a60: 6f 6d 6d 61 6e 64 2c 20 6f 72 20 77 68 6f 6c 65 ommand, or whole 0a70: 73 61 6c 65 2e 0a 0a 09 73 74 72 75 63 74 3a 3a sale....struct:: 0a80: 6c 69 73 74 0a 09 09 61 73 73 69 67 6e 2c 20 6d list...assign, m 0a90: 61 70 2c 20 72 65 76 65 72 73 65 2c 20 66 69 6c ap, reverse, fil 0aa0: 74 65 72 0a 0a 09 09 56 65 72 79 20 66 65 77 20 ter....Very few 0ab0: 61 6e 64 20 73 65 6c 66 2d 63 6f 6e 74 61 69 6e and self-contain 0ac0: 65 64 20 63 6f 6d 6d 61 6e 64 73 2e 0a 0a 09 73 ed commands....s 0ad0: 74 72 75 63 74 3a 3a 73 65 74 0a 09 09 73 69 7a truct::set...siz 0ae0: 65 2c 20 65 6d 70 74 79 2c 20 63 6f 6e 74 61 69 e, empty, contai 0af0: 6e 73 2c 20 61 64 64 2c 20 69 6e 63 6c 75 64 65 ns, add, include 0b00: 2c 20 65 78 63 6c 75 64 65 2c 0a 09 09 69 6e 74 , exclude,...int 0b10: 65 72 73 65 63 74 2c 20 73 75 62 73 65 74 6f 66 ersect, subsetof 0b20: 0a 0a 09 09 4d 6f 73 74 20 6f 66 20 74 68 65 20 ....Most of the 0b30: 63 6f 72 65 20 63 6f 6d 6d 61 6e 64 73 2e 0a 0a core commands... 0b40: 09 66 69 6c 65 75 74 69 6c 0a 09 09 63 61 74 2c .fileutil...cat, 0b50: 20 61 70 70 65 6e 64 54 6f 46 69 6c 65 2c 20 77 appendToFile, w 0b60: 72 69 74 65 46 69 6c 65 2c 0a 09 09 74 65 6d 70 riteFile,...temp 0b70: 66 69 6c 65 2c 20 73 74 72 69 70 50 61 74 68 2c file, stripPath, 0b80: 20 74 65 73 74 0a 0a 09 66 69 6c 65 75 74 69 6c test...fileutil 0b90: 3a 3a 74 72 61 76 65 72 73 65 0a 09 09 49 6e 20 ::traverse...In 0ba0: 74 6f 74 6f 0a 0a 09 73 74 72 75 63 74 3a 3a 67 toto...struct::g 0bb0: 72 61 70 68 0a 09 09 49 6e 20 74 6f 74 6f 0a 0a raph...In toto.. 0bc0: 09 73 6e 69 74 0a 09 09 49 6e 20 74 6f 74 6f 0a .snit...In toto. 0bd0: 0a 09 73 71 6c 69 74 65 33 0a 09 09 49 6e 20 74 ..sqlite3...In t 0be0: 6f 74 6f 0a oto.