e1dbf3186d 2008-02-04 aku: e1dbf3186d 2008-02-04 aku: Known problems and areas to work on e1dbf3186d 2008-02-04 aku: =================================== e1dbf3186d 2008-02-04 aku: e1dbf3186d 2008-02-04 aku: * Not yet able to handle the specification of multiple projects e1dbf3186d 2008-02-04 aku: for one CVS repository. I.e. I can, for example, import all of e1dbf3186d 2008-02-04 aku: tcllib, or a single subproject of tcllib, like tklib, but not e1dbf3186d 2008-02-04 aku: multiple sub-projects in one go. e1dbf3186d 2008-02-04 aku: d2b1d3be25 2008-02-06 aku: * We have to look into the pass 'InitCsets' and hunt for the d2b1d3be25 2008-02-06 aku: cause of the large amount of memory it is gobbling up. de03c8b1ea 2008-02-12 aku: 27ed4f7dc3 2008-02-16 aku: Results from the first look using the new memory tracking 27ed4f7dc3 2008-02-16 aku: subsystem: 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: (1) The general architecture, workflow, is a bit wasteful. All 27ed4f7dc3 2008-02-16 aku: changesets are generated and kept in memory before getting 27ed4f7dc3 2008-02-16 aku: persisted. This means that allocated memory piles up over 27ed4f7dc3 2008-02-16 aku: time, with later changesets pushing the boundaries. This 27ed4f7dc3 2008-02-16 aku: is made worse that some of the preliminary changesets seem 27ed4f7dc3 2008-02-16 aku: to require a lot of temporary memory as part of getting 27ed4f7dc3 2008-02-16 aku: broken down into the actual ones. InititializeBreakState 27ed4f7dc3 2008-02-16 aku: seems to be the culprit here. Its memory usage is possibly 27ed4f7dc3 2008-02-16 aku: quadratic in the number of items in the changeset. 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: (2) A number of small inefficiencies. Like 'state eval' always 27ed4f7dc3 2008-02-16 aku: pulling the whole result into memory before processing it 27ed4f7dc3 2008-02-16 aku: with 'foreach'. Here potentially large lists. 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: (3) We maintain an in-memory map from tagged items to their 27ed4f7dc3 2008-02-16 aku: changesets. While this is needed later in the sorting 27ed4f7dc3 2008-02-16 aku: passes during the creation this is wasted space. And also 27ed4f7dc3 2008-02-16 aku: wasted time, to maintain it during the creation and 27ed4f7dc3 2008-02-16 aku: breaking. 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: Changes: 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: (a) Re-architect to create, break, and persist changesets one 27ed4f7dc3 2008-02-16 aku: by one, completely releasing all associated in-memory data 27ed4f7dc3 2008-02-16 aku: before going to the next. Should be low-hanging fruit with 27ed4f7dc3 2008-02-16 aku: high impact, as we have all the necessary operations 27ed4f7dc3 2008-02-16 aku: already, just not in that order, and that alone should 27ed4f7dc3 2008-02-16 aku: already keep the pile from forming, making the spikes of 27ed4f7dc3 2008-02-16 aku: (2) more manageable. 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: (b) Look into the smaller problems described in (2), and 27ed4f7dc3 2008-02-16 aku: especially (3). These should still be low-hanging fruit, 27ed4f7dc3 2008-02-16 aku: although of lesser effect than (a). For (3) disable the 27ed4f7dc3 2008-02-16 aku: map and its maintenace during construction, and put it 27ed4f7dc3 2008-02-16 aku: into a separate command, to be used when loading the 27ed4f7dc3 2008-02-16 aku: created changesets at the end. 27ed4f7dc3 2008-02-16 aku: 27ed4f7dc3 2008-02-16 aku: (c) With larger effect, but more difficult to achieve, go into 27ed4f7dc3 2008-02-16 aku: command 'InitializeBreakState' and the preceding 27ed4f7dc3 2008-02-16 aku: 'internalsuccessors', and rearchitect it. Definitely not a 27ed4f7dc3 2008-02-16 aku: low-hanging fruit. Possibly also something we can skip if 27ed4f7dc3 2008-02-16 aku: doing (a) had a large enough effect. de03c8b1ea 2008-02-12 aku: de03c8b1ea 2008-02-12 aku: * Look at the dependencies on external packages and consider de03c8b1ea 2008-02-12 aku: which of them can be moved into the importer, either as a de03c8b1ea 2008-02-12 aku: simple utility command, or wholesale. de03c8b1ea 2008-02-12 aku: de03c8b1ea 2008-02-12 aku: struct::list de03c8b1ea 2008-02-12 aku: assign, map, reverse, filter de03c8b1ea 2008-02-12 aku: de03c8b1ea 2008-02-12 aku: Very few and self-contained commands. de03c8b1ea 2008-02-12 aku: de03c8b1ea 2008-02-12 aku: struct::set de03c8b1ea 2008-02-12 aku: size, empty, contains, add, include, exclude, de03c8b1ea 2008-02-12 aku: intersect, subsetof de03c8b1ea 2008-02-12 aku: de03c8b1ea 2008-02-12 aku: Most of the core commands. 812c91bb8d 2008-02-04 aku: de03c8b1ea 2008-02-12 aku: fileutil de03c8b1ea 2008-02-12 aku: cat, appendToFile, writeFile, de03c8b1ea 2008-02-12 aku: tempfile, stripPath, test e1dbf3186d 2008-02-04 aku: de03c8b1ea 2008-02-12 aku: fileutil::traverse de03c8b1ea 2008-02-12 aku: In toto e1dbf3186d 2008-02-04 aku: de03c8b1ea 2008-02-12 aku: struct::graph de03c8b1ea 2008-02-12 aku: In toto e1dbf3186d 2008-02-04 aku: de03c8b1ea 2008-02-12 aku: snit de03c8b1ea 2008-02-12 aku: In toto e1dbf3186d 2008-02-04 aku: de03c8b1ea 2008-02-12 aku: sqlite3 27ed4f7dc3 2008-02-16 aku: In toto
File Annotation