OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


  Current location        Thread information  
OmniTrader Archives
OmniTrader 2009 Technical Support
Seemingly-ridiculous file sizes
Last Activity 10/29/2014 8:57 PM
29 replies, 1742 viewings

Jump to page : 1 2
Now viewing page 2 [25 messages per page]
 
back reply
Printer friendly version

^ Top
Jim Dean

Idol
20005001001002525
Posts: 2771

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/13/2010 8:38 AM
Post #19277 - In reply to #19276

Thank you for the reply, Ryan.

If you'd like an explanation of what I am trying to do ... and also a reminder re another unanswered query ... please see:
http://www.omnitrader.com/omnitrader/support/OT2004/forum/thread-view.asp?threadid=4729&posts=2#0

The reason for query about profiles is because they are SO big ... typically, when COMPRESSED ... 1-3 meg ... and they contain SO little information. I'm in process of creating a LOT of these (see project info), and the cumulative size begins to eat up my HD space that is allocated for drive C.

I'm not asking for help on how to manage my drive. I'm asking for comprehensive info, from Cose, re content of the files, SUCH THAT it makes some sense as to why the files are so large, even when compressed.

My belief is that there is more info in there than what you have listed, since the files are so large.

So, please get an explanation from Cose as to why the files are so large. I have again attached a file which is 64KB that is suppposedly empty of all information that I know how to remove ... yet it is still pretty huge.

Thanks.

[Edited by Jim Dean on 1/13/2010 8:44 AM]

Attached file : aJDprRT-BareDaily.otp (668KB - 130 downloads)

^ Top
Ryan Olson

Veteran
100100
Posts: 203

Joined: 10/9/2006
Location: Austin

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/13/2010 9:04 AM
Post #19279 - In reply to #19277



Jim,

The size of the database is some what out of our control. The database is a Microsoft Access Database. If you open Access and create a new database and add 4 empty fields the size of the database is over 500 KB.

Our profiles will start with more then 4 empty fields; in fact they normally start with upwards of 23 fields with the starting size of 1,600KB.

To put this in perspective - the profile when compacted is about the same as one digital photograph saved to your hard drive (less in some cases today).

To be honest if there is concern about 1 to 3 Megs of compressed non used profiles or the many hundreds of megs a running profile can get to it may be time to upgrade your hardware. OmniTrader does have a minimum requirement of 2 GB of space.

If you need further information on MS Access please visit the Microsoft Access website - http://office.microsoft.com/en-us/access/default.aspx

PS. As for the second issue you are having. We have taken this to Cose and I will reply to the thread the moment I have some information on your request.
^ Top
Jim Dean

Idol
20005001001002525
Posts: 2771

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/13/2010 9:11 AM
Post #19280 - In reply to #19279

OK I've asked enough times for more detailed information re contents. I give up.

Please address the questions about Symdata.otd in the other thread.
^ Top
Ryan Olson

Veteran
100100
Posts: 203

Joined: 10/9/2006
Location: Austin

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/13/2010 9:14 AM
Post #19281 - In reply to #19280



Jim,

Ok, as stated in the prior post I will get you the information the moment it hits my desk. I will follow up this A.M with Cose.
^ Top
Ryan Olson

Veteran
100100
Posts: 203

Joined: 10/9/2006
Location: Austin

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/15/2010 3:45 PM
Post #19303 - In reply to #19237



Jim,

As promised here is Cose's response to your thread and emails.

Quote Cose -

Symdata & profiles:

These are Microsoft Access databases. The reason they get sometimes too big is because whenever our apps accesses the databases and perform operations on it, the database access system (Microsoft) will add a lot of additional data to the file for logging and/or recovery purpose. More often an app accessed the database, much bigger it gets until the database gets compacted. Even when deleting data, the data is not actually deleted right away, but is just marked as deleted, and the actual deletion is done when compacting the database (using MS API too). This is the way how these databases work, and we're not too happy about this either. Were compacting the databases periodically through the DataMaintenance process that launches when OT/VT closes.

We are talking more & more about a replacement for this database system that has some inconveniences, including the one mentioned by you.

Number of bars not matching

We’ll need to look into this. One cause might be that since our apps needs to support multiple providers, some of them supports requests by date, some by number of bars, so we’re converting back and forth between number of bars->date interval->number of bars using the treading session that is set for the stock, and maybe something is not quit right there.


Now storing 1 minute bars, but store for each timeframe

The reason we’re doing it is that all RT data feeds we implement (including OmniData) supports any compression at the server side, and we want to avoid requesting, for example,

60000 1 minute bars is the user wants 1000 60m bars.


Out of memory error in the log file:

We’ll need to look into this. Not sure it is a but in the code or a limitation. Is the memory low on your machine when you’re collecting this data, which is stored in the memory for all symbols and timeframe when running RT profiles.

Jump to page : 1 2
Now viewing page 2 [25 messages per page]
back reply

Legend    Action      Notification  
Administrator
Forum Moderator
Registered User
Unregistered User
E-Mail this thread to a friend
Toggle e-mail notification


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.