OmniTrader Archives
OmniTrader 2009 Technical Support
Seemingly-ridiculous file sizes
29 replies, 3668 viewings
|
OmniTrader Archives
OmniTrader 2009 Technical Support
Seemingly-ridiculous file sizes
|
Last Activity 12/17/2020 7:15 AM 29 replies, 3668 viewings |
|
|
| Printer friendly version |
Jim Dean Sage   Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA  User Profile |
That other post (different topic, but same project I am working on): http://www.omnitrader.com/omnitrader/support/OT2004/forum/thread-view.asp?threadid=4729&posts=2#0 Best way to explain it is to ask you to try this: 1. create a new RT profile 2. import Russell 1000 symbol list 3. turn off all patterns 4. turn off all strategies 5. specify data periods of 1m, 10m, 30m, 195m, and Daily ... 1000 bars each 6. do not include any special columns in the FL This profile, on my machine, is 99,556KB. I do NOT understand what is being stored ... nearly 100MB to hold 1000 symbol-names, and some flags and criteria re timeframes and screen settings????? For similar profiles that contain just the SP500 symbols, all else the same ... 2,316KB ... MUCH smaller, but even there ... why in the world does it require 2Meg to store 500 symbol names and a few screen-settings. The fact that support personnel cannot explain this says that someone needs to outline more clearly what the Profile file has IN it. This is very relevant to the tasks I am now, and have been working on for the past week. I'd appreciate a reply ASAP to this queston as well as the one about gathering past intraday data in the Symdata.otd file. Thanks. P.S. if somehow these profiles have been "corrupted" ... what would have caused such HUGE "bloat"??? I would understand it if a simple profile like these was about 5-20K in size, but anything even approaching 1Meg is utterly confusing ... obviously something is being stored there that support (and I) are not aware of. Please help! [Edited by Jim Dean on 1/12/2010 10:27 AM] | ||
| EYEGUY Icon   Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
Are you looking at the profile size while running OT? Or, is the file size you quote long after OT has been exited and OT 2010 maintenance has run? The reason I ask is this. Every night I prepare an OT profile and VT map for my trading group. The OT profile is the last looking for Hot Strategies. When I close out of OT the file size of the profile I have created is often huge, maybe 15,000 KB. Howerver, if I wait for a while the profile is "compressed", for example last night to 1,156 KB. Don't know if that makes any sense, but it is what I have observed. By the way, I should also state that the symbol count of the file is usually around 400 symbols. Tom Helget [Edited by EYEGUY on 1/12/2010 11:47 AM] | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Yes I failed to mention that ... but it is part of my question regarding what the Profile file contains. The 100MB file I mentioned was that size DURING operation. It dropped back to 3.5MB after OT was closed. However 3.5MB seems at least 100x too big to be holding just the symbol names and some flags and settings, in a pattern-less and strategy-less and custom-FL-column-less environment. A better understanding of what is in this file that is so crucial to OT operation will help us all to better manage our operations. Thanks for your additional info. | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
Might I suggest that the file size (depending on the total number of symbols) is set considering the strategies you have available, the patterns you have available, the indicators you have available, the custom charts you have available, etc. creating a potential workspace (like say a spreadsheet) into which data might be "written". When OT closes down, Data Maintainance looks at what was actually used, discards the rest and "compacts" the file. I hope that didn't sound too stupid, but that is how I might go about things, dimensioning a potential workspace and then filling it while in operation subsequently compacting the data (like a .zip file) finally for storage. From all your activity on the websites I would conclude that you might have with all your projects a much more substantial file created than might the average user. Tom Helget [Edited by EYEGUY on 1/12/2010 1:39 PM] | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Yes that makes sense. That's why, in creating the profiles, I started from scratch with all strategies and patters turned off, no charts viewed except for a default candles+volume simple one, etc. Your description is EXACTLY what I would suspect. That's why I'm curious / wierded-out about 2-4Meg files that JUST have the simplest possible setup ... the only "special" thing is the number of symbols, and the specific dataperiod definitions. Try it yourself, if you have time and curiosity about it. Build the smallest profile you possibly can, from scratch ... exit OT, and see how big it is (I have not done this, but I've come close). Then ask, "what are all those bytes holding, anyway?" In fact, this could be a CONTEST. Build and post the SMALLEST RT PROFILE POSSIBLE, and win an oh-so-coveted OT-forum ATTABOY! [Edited by Jim Dean on 1/12/2010 1:41 PM] | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
As I stated, I do that every night. The profile I was referring to has absolutely nothing in it but symbols and maybe two strategies enabled after the Hot Strategy Finder run (all this in realtime) and yet the file size is easily 15,000 KB. Compacting reduces the file to 1,156 KB, a 90% reduction. However, I have a large potential set of possibilites (read previous OT baggage) that I could enable should I elect to. I think OT is just setting up data rows and columns should I decide to employ these possibilities thus obviating the creation of new space each time something is enabled. Tom Helget | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
That if I have a lot of strategies or charts or trade plans out there, unused, in OT2010 folders somewhere, that maybe the profile is including "future hooks" for their use. Hmmm. I sure do hope Cose will clarify all this. | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
What I might suggest is that you take all your custom indicators, systems, chart templates, trade plans, stops, etc and temporarily put them in your Documents folder. Create a new OT profile and see what the size is. I would be willing to bet it would be a lot smaller. Tom Helget [Edited by EYEGUY on 1/12/2010 2:04 PM] | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
[Edited by Jim Dean on 1/26/2010 1:43 PM] | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
Yes, that would be nice. And, just who the heck is "Cose"? Tom Helget | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
PS: He explained to me that Count Dracula has really been "dissed" by the press. Actually was quite an innovative official, if I understood it correctly.. [Edited by Jim Dean on 1/12/2010 2:18 PM] | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
OK, I thought I knew most of the Nivana staff, but I guess I am wrong! But I really go for that "True Blood" stuff too so we should get along just fine. By the way if you have never read "Dracula" by Bram Stoker it really is a fine piece of literature unlike what you might have imagined. Tom Helget | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
| ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Turned off everything I could find. Only Daily bars defined. No symbols loaded. Program would not allow me to remove directories with strategies, templates, etc so I suppose those might have an impact, even though NONE are or have been used in the profile since its creation. The size on my machine was 668KB after closing OT when the profile was first created. Then upon subsequent startup, it became 1280KB with OT alive, and when OT was shut down again after doing absolutely nothing, it held that larger size, and never returned to 668KB. So, well over a million bytes to hold nothing at all. I'd appreciate it if someone would download it and see how big it is on their machine, after Opening it, then immediatedly shutting down OT. Since anyone else will have other stuff in strategies, etc folders than I do, if the file size changes that might indicate the .otp file holds that info or pointers to it, as Tom has suggested. Thanks! [Edited by Jim Dean on 1/12/2010 3:20 PM]  Attached file : aJDprRT-BareDaily.otp (648KB - 438 downloads) | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
As regards your Strategy, Chart Template, etc. directories it might be instructive not to remove the folders but to remove most of the contents and see what happens. I have downloaded the file and will check it out as you request as soon as an analysis I am running concludes. Tom Helget | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
Well here you go! Prior to opening the file I installed the profile in the Profiles folder: After opening your file things were unchanged: Upon closing OT we get: Hope that helps! But I bet it doesn't! Tom Helget [Edited by EYEGUY on 1/12/2010 3:33 PM] Attached file : PRIOR TO OPENING.PNG (11KB - 582 downloads) Attached file : OPENED IN OT.PNG (33KB - 587 downloads) Attached file : AFTER CLOSING OT.PNG (4KB - 573 downloads) | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Vellee intellestink lesults ... Turns out that your 676KB is what my file's final value is, ALSO, when I copied the first archive of that file over, and restarted and exited OT one more time. I've no idea why the last time I tried this it was 1308 and then it dropped back down to 676 ... or why 676 is different than the initial value of 668. Those are distinctions without a difference. I think this "probably" demonstrates that the stuff in other folders does not impact the profile size, since it's unlikely that both of us, prolific as we are in experimentation, have even CLOSE to the same "other stuff" in our OT2010 familiy of folders. Hmm ... just did it a third time ... and final file size is 684KB now. THAT looks like a memory leak ... but it isn't consistent. Ah well ... enuf time wasted on this. Back to collecting historical intraday data for offline testing ... [Edited by Jim Dean on 1/12/2010 3:42 PM] | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
I told you that you wouldn't like my results. Yes, I think I will have to concede that my hypothesis is BS. But it had merit for a brief moment.... Tom Helget | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Cose where are you when we need you??? | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
Start with the BareDaily profile I attached earlier. Now, edit Data Periods to look like this: When you click OK you will probably get an Error Message - there is no reason for this AFAIK ... just OK it and move on ... Now, File > Copy and give this profile a new name ... you should see the periods change from 1min to 5 min in the process ... Finally, close OT. LOOK at the new profile's size ... something around 2048KB on my machine! And no symbols. Or anything else. [Edited by Jim Dean on 1/12/2010 4:05 PM] Attached file : Change data periods ONLY.png (7KB - 561 downloads) | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
As you stated, this just gets wierder and wierder. I'll give my Cose come to the 'Cuse call (I live near Syracuse, NY and natives here like to call the place the 'Cuse). Tom Helget | ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
Here is the file size on a realtime profile containing maybe 400 symbols after initializing realtime but before running the Strategy Wizard: Compare that to the compressed file from yesterday just above it. Tom Helget [Edited by EYEGUY on 1/12/2010 5:12 PM] Attached file : SIZE KEEPS ON GROWING AS I WORK.PNG (4KB - 544 downloads) | ||
| Jim Dean Sage Posts: 3022 Joined: 9/21/2006 Location: L'ville, GA User Profile |
| ||
| EYEGUY Icon Posts: 1543 Joined: 12/12/2003 Location: BALDWINSVILLE, NEW YORK User Profile |
After running Strategy Wizard no further increase in file size was perceived. On shutting down OT, data file maintenance initiated: Here we are working on today's file (100113): And here is the final tidy result: Tom Helget [Edited by EYEGUY on 1/12/2010 5:49 PM] Attached file : DATA MAINTENANCE IN ACTION.PNG (4KB - 549 downloads) Attached file : DATA MAINTENACE ON FILE TODAY.PNG (5KB - 567 downloads) Attached file : AND, AWAY WE GO.PNG (3KB - 568 downloads) | ||
| Ryan Olson Veteran  Posts: 213 Joined: 10/9/2006 Location: Austin User Profile |
All, I would like to try and quickly clear this up. However, I must first ask Jim – has the size of the profile caused any issues or problems that would prompt the inquiry? If so please let us know so that we can look into resolving this. Outside of this the profile database will contain information about symbols, focus list, configuration, symbol list enabled, trading periods times, and will also contain a temporary space for changes that have been made. When your profile is open and data is flowing a few things will happen. The data will modify the fields as required by the profile, it will also be saving all changes to the temporary space. This can get quite large even with only a few dozen stocks. In some extreme cases I have seen this get to many hundreds of Megs. When you close the application OmniTrader will go through the Data Maintenance where it will repair and compact the databases. Doing this remove the temporary information that was stored. Outside of this temporary space there isn’t much more that is removed. I hope this clears things up. |
|
|
|
|
Administrator
Forum Moderator |
Registered User
Unregistered User |
 E-Mail this thread to a friend |
Toggle e-mail notification |
