OmniTrader Archives
OmniTrader 2009 Technical Support
Seemingly-ridiculous file sizes
29 replies, 3672 viewings
|
OmniTrader Archives
OmniTrader 2009 Technical Support
Seemingly-ridiculous file sizes
|
Last Activity 12/17/2020 7:15 AM 29 replies, 3672 viewings |
|
|
| Printer friendly version |
Jim Dean Sage   Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA  User Profile |
If you'd like an explanation of what I am trying to do ... and also a reminder re another unanswered query ... please see: http://www.omnitrader.com/omnitrader/support/OT2004/forum/thread-view.asp?threadid=4729&posts=2#0 The reason for query about profiles is because they are SO big ... typically, when COMPRESSED ... 1-3 meg ... and they contain SO little information. I'm in process of creating a LOT of these (see project info), and the cumulative size begins to eat up my HD space that is allocated for drive C. I'm not asking for help on how to manage my drive. I'm asking for comprehensive info, from Cose, re content of the files, SUCH THAT it makes some sense as to why the files are so large, even when compressed. My belief is that there is more info in there than what you have listed, since the files are so large. So, please get an explanation from Cose as to why the files are so large. I have again attached a file which is 64KB that is suppposedly empty of all information that I know how to remove ... yet it is still pretty huge. Thanks. [Edited by Jim Dean on 1/13/2010 8:44 AM]  Attached file : aJDprRT-BareDaily.otp (668KB - 418 downloads) | ||
| Ryan Olson Veteran  Posts: 213 Joined: 10/9/2006 Location: Austin User Profile |
Jim, The size of the database is some what out of our control. The database is a Microsoft Access Database. If you open Access and create a new database and add 4 empty fields the size of the database is over 500 KB. Our profiles will start with more then 4 empty fields; in fact they normally start with upwards of 23 fields with the starting size of 1,600KB. To put this in perspective - the profile when compacted is about the same as one digital photograph saved to your hard drive (less in some cases today). To be honest if there is concern about 1 to 3 Megs of compressed non used profiles or the many hundreds of megs a running profile can get to it may be time to upgrade your hardware. OmniTrader does have a minimum requirement of 2 GB of space. If you need further information on MS Access please visit the Microsoft Access website - http://office.microsoft.com/en-us/access/default.aspx PS. As for the second issue you are having. We have taken this to Cose and I will reply to the thread the moment I have some information on your request. | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Please address the questions about Symdata.otd in the other thread. | ||
| Ryan Olson Veteran Posts: 213 Joined: 10/9/2006 Location: Austin User Profile |
Jim, Ok, as stated in the prior post I will get you the information the moment it hits my desk. I will follow up this A.M with Cose. | ||
| Ryan Olson Veteran Posts: 213 Joined: 10/9/2006 Location: Austin User Profile |
Jim, As promised here is Cose's response to your thread and emails. Quote Cose - Symdata & profiles: These are Microsoft Access databases. The reason they get sometimes too big is because whenever our apps accesses the databases and perform operations on it, the database access system (Microsoft) will add a lot of additional data to the file for logging and/or recovery purpose. More often an app accessed the database, much bigger it gets until the database gets compacted. Even when deleting data, the data is not actually deleted right away, but is just marked as deleted, and the actual deletion is done when compacting the database (using MS API too). This is the way how these databases work, and we're not too happy about this either. Were compacting the databases periodically through the DataMaintenance process that launches when OT/VT closes. We are talking more & more about a replacement for this database system that has some inconveniences, including the one mentioned by you. Number of bars not matching We’ll need to look into this. One cause might be that since our apps needs to support multiple providers, some of them supports requests by date, some by number of bars, so we’re converting back and forth between number of bars->date interval->number of bars using the treading session that is set for the stock, and maybe something is not quit right there. Now storing 1 minute bars, but store for each timeframe The reason we’re doing it is that all RT data feeds we implement (including OmniData) supports any compression at the server side, and we want to avoid requesting, for example, 60000 1 minute bars is the user wants 1000 60m bars. Out of memory error in the log file: We’ll need to look into this. Not sure it is a but in the code or a limitation. Is the memory low on your machine when you’re collecting this data, which is stored in the memory for all symbols and timeframe when running RT profiles. |
|
|
|
|
Administrator
Forum Moderator |
Registered User
Unregistered User |
 E-Mail this thread to a friend |
Toggle e-mail notification |
