OmniTrader Archives
OmniTrader 2009 Technical Support
Out of Memory when getting past Intraday data
2 replies, 1718 viewings
|
OmniTrader Archives
OmniTrader 2009 Technical Support
Out of Memory when getting past Intraday data
|
Last Activity 12/17/2020 7:15 AM 2 replies, 1718 viewings |
|
|
|
| Printer friendly version |
Jim Dean Sage   Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA  User Profile |
Context is important ... I've been trying to gradually build a SymData.otd file that holds the past year's intraday data for a variety of bar-lengths, so that I can archive it, and then re-use it later for backtesting etc without having to reload hundreds of thousands of bars of data later on, for the same period. I'm doing it for the symbols in the Nas-100. It is comprised of eleven different bar-lengths. Re the numbers on the right of the equals-sign ... the 255 days calculates out as 6640 15m bars for 255 days, but the actual number of bars LOADED in the chart when 6640 is input in DataPeriods turns out to be 6616. I don't know why, but I've recorded those "actual avail" bars as the rightmost number. 390m = 255 / 195m = 510 / 130m = 765 / 78m = 1277 / 1274 65m = 1532 / 1510 60m = 1660 / 1655 39m = 2550 / 30m = 3320 / 3309 26m = 3831 / 3818 15m = 6640 / 6616 13m = 7662 / 7545 10m = 9960 / 9923 6m = 16575 / 5m = 19920 / 19844 3m = 33200 / 33072 2m = 49800 / 49607 I've learned that OT doesn't just save the 1-min bars for the historical period (which is the way that Worden does it) ... OT actually calculates each individual bar-length type, and stores it ... thus making SymData.otd MUCH bigger than I would have expected, for this collection of data. So be it. Anyways, I am ALMOST done ... just THREE SYMBOLS for ONE BAR LENGTH away from reaching my goal ... which took the better part of yesterday to accomplish. My SymData.otd file has reached a whopping 727,484 KB ... and all I'm missing is the 2m bars for AAPL, SPLS, and NIHD. The SymData.otd file gets to that size when I start up OT with the Nas list ... but during the time it is loading the data in the symbols, it throws the error. However, every chart ends up visible with all its bars (except the three symbols I mentioned). THEN, when I close out OT (and get another error and log-submission screen), I notice that the SymData.otd file size DROPS to 467,328 ... presumably it has thrown out a bunch of data ???? So, in helping me with this problem, please also *** check with Cose *** re any internal limits. Please expressly pass on the list I provided above for the 11 timeframes and numbers of bars I want. To build this SymData.otd file, I've created several different profiles and loaded them up one at a time: 5m, 26m, 130m 3m, 15m, 78m 2m, 13m, 65m 10m, 30m, 60m ... all also including 255 days, which is the only voted timeframe. The first three of these are the most important to me, since they represent multiples-of-five between each increment. All listed increments divide evenly into a 390-minute day, btw. If it turns out that I've hit some kind of "hard limit", then I'd be OK with leaving out the last profile-set (10,30,60) ... but I don't want to have to restart this incredibly long and boring process unless I *fully* understand the limiting factors. So, please alert Jon to this, and please get Cose involved with the entire text of this posting. Thanks! Another way of asking this, in a nutshell: What is the structure of the SymData.otd file, and what is the most efficient manner for a user to do a one-time accumulation of past intraday data for a variety of bar-lengths, to help with later backtesting in offline situations? PS ... one reason for wanting this is as I mentioned earlier ... to reduce long download-from-server times, repeatedly, when I want to do extended backtesting. ANOTHER reason is that I often work in areas where the internet speed is either very slow or nonexistent. [Edited by Jim Dean on 1/13/2010 8:31 AM]  Attached file : OT.log (37KB - 440 downloads) Attached file : OT.otd (420KB - 438 downloads) | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
ALSO SEE: http://www.omnitrader.com/omnitrader/support/OT2004/forum/thread-view.asp?threadid=4733&posts=1#0 [Edited by Jim Dean on 1/12/2010 10:26 AM] | ||
| Ryan Olson Veteran  Posts: 213 Joined: 10/9/2006 Location: Austin User Profile |
Jim, As promised here is Cose's response to your thread and emails. Quote Cose - Symdata & profiles: These are Microsoft Access databases. The reason they get sometimes too big is because whenever our apps accesses the databases and perform operations on it, the database access system (Microsoft) will add a lot of additional data to the file for logging and/or recovery purpose. More often an app accessed the database, much bigger it gets until the database gets compacted. Even when deleting data, the data is not actually deleted right away, but is just marked as deleted, and the actual deletion is done when compacting the database (using MS API too). This is the way how these databases work, and we're not too happy about this either. Were compacting the databases periodically through the DataMaintenance process that launches when OT/VT closes. We are talking more & more about a replacement for this database system that has some inconveniences, including the one mentioned by you. Number of bars not matching We’ll need to look into this. One cause might be that since our apps needs to support multiple providers, some of them supports requests by date, some by number of bars, so we’re converting back and forth between number of bars->date interval->number of bars using the treading session that is set for the stock, and maybe something is not quit right there. Now storing 1 minute bars, but store for each timeframe The reason we’re doing it is that all RT data feeds we implement (including OmniData) supports any compression at the server side, and we want to avoid requesting, for example, 60000 1 minute bars is the user wants 1000 60m bars. Out of memory error in the log file: We’ll need to look into this. Not sure it is a but in the code or a limitation. Is the memory low on your machine when you’re collecting this data, which is stored in the memory for all symbols and timeframe when running RT profiles. |
|
|
|
|
Administrator
Forum Moderator |
Registered User
Unregistered User |
 E-Mail this thread to a friend |
Toggle e-mail notification |
