OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


  Current location        Thread information  
OmniTrader Archives
OmniTrader 2009 Technical Support
Out of Memory when getting past Intraday data
Last Activity 10/29/2014 8:57 PM
2 replies, 856 viewings

Jump to page : 1
Now viewing page 1 [25 messages per page]
 
back reply
Printer friendly version

^ Top
Jim Dean

Idol
200050010010010025
Posts: 2831

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : Out of Memory when getting past Intraday data
Posted : 1/9/2010 8:46 AM
Post #19220

I've attached the OT.log file. Also the OT.otd file, for grins.

Context is important ...
I've been trying to gradually build a SymData.otd file that holds the past year's intraday data for a variety of bar-lengths, so that I can archive it, and then re-use it later for backtesting etc without having to reload hundreds of thousands of bars of data later on, for the same period.

I'm doing it for the symbols in the Nas-100. It is comprised of eleven different bar-lengths. Re the numbers on the right of the equals-sign ... the 255 days calculates out as 6640 15m bars for 255 days, but the actual number of bars LOADED in the chart when 6640 is input in DataPeriods turns out to be 6616. I don't know why, but I've recorded those "actual avail" bars as the rightmost number.
390m = 255 /
195m = 510 /
130m = 765 /
78m = 1277 / 1274
65m = 1532 / 1510
60m = 1660 / 1655
39m = 2550 /
30m = 3320 / 3309
26m = 3831 / 3818
15m = 6640 / 6616
13m = 7662 / 7545
10m = 9960 / 9923
6m = 16575 /
5m = 19920 / 19844
3m = 33200 / 33072
2m = 49800 / 49607

I've learned that OT doesn't just save the 1-min bars for the historical period (which is the way that Worden does it) ... OT actually calculates each individual bar-length type, and stores it ... thus making SymData.otd MUCH bigger than I would have expected, for this collection of data. So be it.

Anyways, I am ALMOST done ... just THREE SYMBOLS for ONE BAR LENGTH away from reaching my goal ... which took the better part of yesterday to accomplish. My SymData.otd file has reached a whopping 727,484 KB ... and all I'm missing is the 2m bars for AAPL, SPLS, and NIHD.

The SymData.otd file gets to that size when I start up OT with the Nas list ... but during the time it is loading the data in the symbols, it throws the error. However, every chart ends up visible with all its bars (except the three symbols I mentioned).

THEN, when I close out OT (and get another error and log-submission screen), I notice that the SymData.otd file size DROPS to 467,328 ... presumably it has thrown out a bunch of data ????

So, in helping me with this problem, please also *** check with Cose *** re any internal limits. Please expressly pass on the list I provided above for the 11 timeframes and numbers of bars I want. To build this SymData.otd file, I've created several different profiles and loaded them up one at a time:
5m, 26m, 130m
3m, 15m, 78m
2m, 13m, 65m
10m, 30m, 60m
... all also including 255 days, which is the only voted timeframe. The first three of these are the most important to me, since they represent multiples-of-five between each increment. All listed increments divide evenly into a 390-minute day, btw. If it turns out that I've hit some kind of "hard limit", then I'd be OK with leaving out the last profile-set (10,30,60) ... but I don't want to have to restart this incredibly long and boring process unless I *fully* understand the limiting factors.

So, please alert Jon to this, and please get Cose involved with the entire text of this posting. Thanks!

Another way of asking this, in a nutshell:
What is the structure of the SymData.otd file, and what is the most efficient manner for a user to do a one-time accumulation of past intraday data for a variety of bar-lengths, to help with later backtesting in offline situations?


PS ... one reason for wanting this is as I mentioned earlier ... to reduce long download-from-server times, repeatedly, when I want to do extended backtesting. ANOTHER reason is that I often work in areas where the internet speed is either very slow or nonexistent.



[Edited by Jim Dean on 1/13/2010 8:31 AM]

Attached file : OT.log (37KB - 180 downloads)
Attached file : OT.otd (420KB - 158 downloads)

^ Top
Jim Dean

Idol
200050010010010025
Posts: 2831

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Out of Memory when getting past Intraday data
Posted : 1/11/2010 5:48 PM
Post #19229 - In reply to #19220

Could someone please help with this?

ALSO SEE: http://www.omnitrader.com/omnitrader/support/OT2004/forum/thread-view.asp?threadid=4733&posts=1#0

[Edited by Jim Dean on 1/12/2010 10:26 AM]

^ Top
Ryan Olson

Veteran
100100
Posts: 202

Joined: 10/9/2006
Location: Austin

User Profile
 
Subject : RE: Out of Memory when getting past Intraday data
Posted : 1/15/2010 3:45 PM
Post #19304 - In reply to #19220



Jim,

As promised here is Cose's response to your thread and emails.

Quote Cose -

Symdata & profiles:

These are Microsoft Access databases. The reason they get sometimes too big is because whenever our apps accesses the databases and perform operations on it, the database access system (Microsoft) will add a lot of additional data to the file for logging and/or recovery purpose. More often an app accessed the database, much bigger it gets until the database gets compacted. Even when deleting data, the data is not actually deleted right away, but is just marked as deleted, and the actual deletion is done when compacting the database (using MS API too). This is the way how these databases work, and we're not too happy about this either. Were compacting the databases periodically through the DataMaintenance process that launches when OT/VT closes.

We are talking more & more about a replacement for this database system that has some inconveniences, including the one mentioned by you.

Number of bars not matching

We’ll need to look into this. One cause might be that since our apps needs to support multiple providers, some of them supports requests by date, some by number of bars, so we’re converting back and forth between number of bars->date interval->number of bars using the treading session that is set for the stock, and maybe something is not quit right there.


Now storing 1 minute bars, but store for each timeframe

The reason we’re doing it is that all RT data feeds we implement (including OmniData) supports any compression at the server side, and we want to avoid requesting, for example,

60000 1 minute bars is the user wants 1000 60m bars.


Out of memory error in the log file:

We’ll need to look into this. Not sure it is a but in the code or a limitation. Is the memory low on your machine when you’re collecting this data, which is stored in the memory for all symbols and timeframe when running RT profiles.

Jump to page : 1
Now viewing page 1 [25 messages per page]
back reply

Legend    Action      Notification  
Administrator
Forum Moderator
Registered User
Unregistered User
E-Mail this thread to a friend
Toggle e-mail notification


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.