Ryan Olson Veteran  Posts: 213 Joined: 10/9/2006 Location: Austin  User Profile | Jim, As promised here is Cose's response to your thread and emails. Quote Cose - Symdata & profiles: These are Microsoft Access databases. The reason they get sometimes too big is because whenever our apps accesses the databases and perform operations on it, the database access system (Microsoft) will add a lot of additional data to the file for logging and/or recovery purpose. More often an app accessed the database, much bigger it gets until the database gets compacted. Even when deleting data, the data is not actually deleted right away, but is just marked as deleted, and the actual deletion is done when compacting the database (using MS API too). This is the way how these databases work, and we're not too happy about this either. Were compacting the databases periodically through the DataMaintenance process that launches when OT/VT closes. We are talking more & more about a replacement for this database system that has some inconveniences, including the one mentioned by you. Number of bars not matching We’ll need to look into this. One cause might be that since our apps needs to support multiple providers, some of them supports requests by date, some by number of bars, so we’re converting back and forth between number of bars->date interval->number of bars using the treading session that is set for the stock, and maybe something is not quit right there. Now storing 1 minute bars, but store for each timeframe The reason we’re doing it is that all RT data feeds we implement (including OmniData) supports any compression at the server side, and we want to avoid requesting, for example, 60000 1 minute bars is the user wants 1000 60m bars. Out of memory error in the log file: We’ll need to look into this. Not sure it is a but in the code or a limitation. Is the memory low on your machine when you’re collecting this data, which is stored in the memory for all symbols and timeframe when running RT profiles. | |
