OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


  Current location        Thread information  
OmniTrader Archives
OmniTrader 2009 Technical Support
Seemingly-ridiculous file sizes
Last Activity 10/29/2014 8:57 PM
29 replies, 1595 viewings

Jump to page : 1 2
Now viewing page 1 [25 messages per page]
 
back reply
Printer friendly version

^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : Seemingly-ridiculous file sizes
Posted : 1/12/2010 10:23 AM
Post #19237

Just spoke with Ryan ... he told me Cose must answer this ... I'm still waiting on an answer to another post (and phone call last week to the other Ryan) from Cose ... so hopefully this will be dealt with at the same time, when Cose gets 5-10min free to help out.

That other post (different topic, but same project I am working on): http://www.omnitrader.com/omnitrader/support/OT2004/forum/thread-view.asp?threadid=4729&posts=2#0


Best way to explain it is to ask you to try this:

1. create a new RT profile
2. import Russell 1000 symbol list
3. turn off all patterns
4. turn off all strategies
5. specify data periods of 1m, 10m, 30m, 195m, and Daily ... 1000 bars each
6. do not include any special columns in the FL

This profile, on my machine, is 99,556KB.
I do NOT understand what is being stored ... nearly 100MB to hold 1000 symbol-names, and some flags and criteria re timeframes and screen settings?????

For similar profiles that contain just the SP500 symbols, all else the same ... 2,316KB ... MUCH smaller, but even there ... why in the world does it require 2Meg to store 500 symbol names and a few screen-settings.

The fact that support personnel cannot explain this says that someone needs to outline more clearly what the Profile file has IN it.

This is very relevant to the tasks I am now, and have been working on for the past week. I'd appreciate a reply ASAP to this queston as well as the one about gathering past intraday data in the Symdata.otd file.

Thanks.

P.S. if somehow these profiles have been "corrupted" ... what would have caused such HUGE "bloat"??? I would understand it if a simple profile like these was about 5-20K in size, but anything even approaching 1Meg is utterly confusing ... obviously something is being stored there that support (and I) are not aware of. Please help!

[Edited by Jim Dean on 1/12/2010 10:27 AM]

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 11:45 AM
Post #19239 - In reply to #19237

Jim:

Are you looking at the profile size while running OT? Or, is the file size you quote long after OT has been exited and OT 2010 maintenance has run?

The reason I ask is this. Every night I prepare an OT profile and VT map for my trading group. The OT profile is the last looking for Hot Strategies. When I close out of OT the file size of the profile I have created is often huge, maybe 15,000 KB. Howerver, if I wait for a while the profile is "compressed", for example last night to 1,156 KB.

Don't know if that makes any sense, but it is what I have observed.

By the way, I should also state that the symbol count of the file is usually around 400 symbols.

Tom Helget

[Edited by EYEGUY on 1/12/2010 11:47 AM]

^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 11:53 AM
Post #19240 - In reply to #19239

Thanks Tom.

Yes I failed to mention that ... but it is part of my question regarding what the Profile file contains.

The 100MB file I mentioned was that size DURING operation. It dropped back to 3.5MB after OT was closed.

However 3.5MB seems at least 100x too big to be holding just the symbol names and some flags and settings, in a pattern-less and strategy-less and custom-FL-column-less environment.

A better understanding of what is in this file that is so crucial to OT operation will help us all to better manage our operations.

Thanks for your additional info.
^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 1:33 PM
Post #19245 - In reply to #19240

Jim:

Might I suggest that the file size (depending on the total number of symbols) is set considering the strategies you have available, the patterns you have available, the indicators you have available, the custom charts you have available, etc. creating a potential workspace (like say a spreadsheet) into which data might be "written". When OT closes down, Data Maintainance looks at what was actually used, discards the rest and "compacts" the file.

I hope that didn't sound too stupid, but that is how I might go about things, dimensioning a potential workspace and then filling it while in operation subsequently compacting the data (like a .zip file) finally for storage.

From all your activity on the websites I would conclude that you might have with all your projects a much more substantial file created than might the average user.

Tom Helget

[Edited by EYEGUY on 1/12/2010 1:39 PM]

^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 1:39 PM
Post #19246 - In reply to #19245

Hi, Tom:

Yes that makes sense. That's why, in creating the profiles, I started from scratch with all strategies and patters turned off, no charts viewed except for a default candles+volume simple one, etc.

Your description is EXACTLY what I would suspect. That's why I'm curious / wierded-out about 2-4Meg files that JUST have the simplest possible setup ... the only "special" thing is the number of symbols, and the specific dataperiod definitions.

Try it yourself, if you have time and curiosity about it. Build the smallest profile you possibly can, from scratch ... exit OT, and see how big it is (I have not done this, but I've come close). Then ask, "what are all those bytes holding, anyway?"

In fact, this could be a CONTEST. Build and post the SMALLEST RT PROFILE POSSIBLE, and win an oh-so-coveted OT-forum ATTABOY!

[Edited by Jim Dean on 1/12/2010 1:41 PM]

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 1:47 PM
Post #19247 - In reply to #19246

Jim:

As I stated, I do that every night. The profile I was referring to has absolutely nothing in it but symbols and maybe two strategies enabled after the Hot Strategy Finder run (all this in realtime) and yet the file size is easily 15,000 KB. Compacting reduces the file to 1,156 KB, a 90% reduction.

However, I have a large potential set of possibilites (read previous OT baggage) that I could enable should I elect to. I think OT is just setting up data rows and columns should I decide to employ these possibilities thus obviating the creation of new space each time something is enabled.

Tom Helget
^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 1:58 PM
Post #19248 - In reply to #19247

I see what you are saying ...

That if I have a lot of strategies or charts or trade plans out there, unused, in OT2010 folders somewhere, that maybe the profile is including "future hooks" for their use. Hmmm.

I sure do hope Cose will clarify all this.
^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 2:03 PM
Post #19249 - In reply to #19248

Jim:

What I might suggest is that you take all your custom indicators, systems, chart templates, trade plans, stops, etc and temporarily put them in your Documents folder. Create a new OT profile and see what the size is. I would be willing to bet it would be a lot smaller.

Tom Helget

[Edited by EYEGUY on 1/12/2010 2:04 PM]

^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 2:12 PM
Post #19250 - In reply to #19249

I intend to try that. Thanks for the suggestion.


[Edited by Jim Dean on 1/26/2010 1:43 PM]

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 2:13 PM
Post #19251 - In reply to #19250

Jim:

Yes, that would be nice.

And, just who the heck is "Cose"?

Tom Helget
^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 2:16 PM
Post #19252 - In reply to #19251

Cose is the developer that is responsible for stuff like database and focus list work and who knows what else (group trader? omniscan?). He's also an admirer of Count Dracula (he's from Romania as I recall). A very nice guy.

PS: He explained to me that Count Dracula has really been "dissed" by the press. Actually was quite an innovative official, if I understood it correctly..

[Edited by Jim Dean on 1/12/2010 2:18 PM]

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 2:20 PM
Post #19253 - In reply to #19252

Jim:

OK, I thought I knew most of the Nivana staff, but I guess I am wrong! But I really go for that "True Blood" stuff too so we should get along just fine. By the way if you have never read "Dracula" by Bram Stoker it really is a fine piece of literature unlike what you might have imagined.

Tom Helget
^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 2:24 PM
Post #19254 - In reply to #19253

Oh, btw ... his name is pronounced "Co-say"
^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 3:17 PM
Post #19255 - In reply to #19254

OK here's the best I could do re creating a "bare minimum" RT profile.

Turned off everything I could find. Only Daily bars defined. No symbols loaded.

Program would not allow me to remove directories with strategies, templates, etc so I suppose those might have an impact, even though NONE are or have been used in the profile since its creation.

The size on my machine was 668KB after closing OT when the profile was first created. Then upon subsequent startup, it became 1280KB with OT alive, and when OT was shut down again after doing absolutely nothing, it held that larger size, and never returned to 668KB.

So, well over a million bytes to hold nothing at all.

I'd appreciate it if someone would download it and see how big it is on their machine, after Opening it, then immediatedly shutting down OT.

Since anyone else will have other stuff in strategies, etc folders than I do, if the file size changes that might indicate the .otp file holds that info or pointers to it, as Tom has suggested.

Thanks!



[Edited by Jim Dean on 1/12/2010 3:20 PM]

Attached file : aJDprRT-BareDaily.otp (648KB - 142 downloads)

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 3:21 PM
Post #19256 - In reply to #19255

Jim:

As regards your Strategy, Chart Template, etc. directories it might be instructive not to remove the folders but to remove most of the contents and see what happens.

I have downloaded the file and will check it out as you request as soon as an analysis I am running concludes.

Tom Helget
^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 3:29 PM
Post #19257 - In reply to #19256

Jim:

Well here you go!

Prior to opening the file I installed the profile in the Profiles folder:





After opening your file things were unchanged:





Upon closing OT we get:





Hope that helps! But I bet it doesn't!

Tom Helget

[Edited by EYEGUY on 1/12/2010 3:33 PM]

Attached file : PRIOR TO OPENING.PNG (11KB - 296 downloads)
Attached file : OPENED IN OT.PNG (33KB - 298 downloads)
Attached file : AFTER CLOSING OT.PNG (4KB - 300 downloads)

^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 3:36 PM
Post #19258 - In reply to #19257

Thank you, Tom. Yes I suppose I could clean out most everything from most everywhere. Thot I would try this first.

Vellee intellestink lesults ...

Turns out that your 676KB is what my file's final value is, ALSO, when I copied the first archive of that file over, and restarted and exited OT one more time.

I've no idea why the last time I tried this it was 1308 and then it dropped back down to 676 ... or why 676 is different than the initial value of 668.

Those are distinctions without a difference.

I think this "probably" demonstrates that the stuff in other folders does not impact the profile size, since it's unlikely that both of us, prolific as we are in experimentation, have even CLOSE to the same "other stuff" in our OT2010 familiy of folders.

Hmm ... just did it a third time ... and final file size is 684KB now.

THAT looks like a memory leak ... but it isn't consistent.

Ah well ... enuf time wasted on this. Back to collecting historical intraday data for offline testing ...

[Edited by Jim Dean on 1/12/2010 3:42 PM]

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 3:46 PM
Post #19259 - In reply to #19258

Jim:

I told you that you wouldn't like my results. Yes, I think I will have to concede that my hypothesis is BS. But it had merit for a brief moment....

Tom Helget
^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 3:58 PM
Post #19260 - In reply to #19259

Oh, but I did like them ... they eliminated a whole class of possibilities that would have taken me a long time to test.

Cose where are you when we need you???
^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 4:04 PM
Post #19261 - In reply to #19260

Here is another interesting offshoot.

Start with the BareDaily profile I attached earlier.

Now, edit Data Periods to look like this:



When you click OK you will probably get an Error Message - there is no reason for this AFAIK ... just OK it and move on ...

Now, File > Copy and give this profile a new name ... you should see the periods change from 1min to 5 min in the process ...

Finally, close OT.

LOOK at the new profile's size ... something around 2048KB on my machine!

And no symbols. Or anything else.

[Edited by Jim Dean on 1/12/2010 4:05 PM]

Attached file : Change data periods ONLY.png (7KB - 278 downloads)

^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 4:14 PM
Post #19262 - In reply to #19261

Jim:

As you stated, this just gets wierder and wierder.

I'll give my Cose come to the 'Cuse call (I live near Syracuse, NY and natives here like to call the place the 'Cuse).

Tom Helget
^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 5:10 PM
Post #19263 - In reply to #19262

Jim:

Here is the file size on a realtime profile containing maybe 400 symbols after initializing realtime but before running the Strategy Wizard:





Compare that to the compressed file from yesterday just above it.

Tom Helget

[Edited by EYEGUY on 1/12/2010 5:12 PM]

Attached file : SIZE KEEPS ON GROWING AS I WORK.PNG (4KB - 262 downloads)

^ Top
Jim Dean

Idol
2000500100100
Posts: 2718

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 5:17 PM
Post #19264 - In reply to #19263

Yes I've seen that too. Clearly there is compression going on. That explains the huge sizes ... sort of. The question is, WHAT is in there?
^ Top
EYEGUY

Icon
100010010010025
Posts: 1327

Joined: 12/12/2003
Location: BALDWINSVILLE, NEW YORK

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/12/2010 5:46 PM
Post #19266 - In reply to #19264

Jim:

After running Strategy Wizard no further increase in file size was perceived. On shutting down OT, data file maintenance initiated:





Here we are working on today's file (100113):




And here is the final tidy result:





Tom Helget

[Edited by EYEGUY on 1/12/2010 5:49 PM]

Attached file : DATA MAINTENANCE IN ACTION.PNG (4KB - 257 downloads)
Attached file : DATA MAINTENACE ON FILE TODAY.PNG (5KB - 283 downloads)
Attached file : AND, AWAY WE GO.PNG (3KB - 269 downloads)

^ Top
Ryan Olson

Veteran
100100
Posts: 202

Joined: 10/9/2006
Location: Austin

User Profile
 
Subject : RE: Seemingly-ridiculous file sizes
Posted : 1/13/2010 8:17 AM
Post #19276 - In reply to #19237


All,

I would like to try and quickly clear this up. However, I must first ask Jim – has the size of the profile caused any issues or problems that would prompt the inquiry? If so please let us know so that we can look into resolving this.

Outside of this the profile database will contain information about symbols, focus list, configuration, symbol list enabled, trading periods times, and will also contain a temporary space for changes that have been made.

When your profile is open and data is flowing a few things will happen.

The data will modify the fields as required by the profile, it will also be saving all changes to the temporary space. This can get quite large even with only a few dozen stocks. In some extreme cases I have seen this get to many hundreds of Megs.

When you close the application OmniTrader will go through the Data Maintenance where it will repair and compact the databases. Doing this remove the temporary information that was stored. Outside of this temporary space there isn’t much more that is removed.

I hope this clears things up.

Jump to page : 1 2
Now viewing page 1 [25 messages per page]
back reply

Legend    Action      Notification  
Administrator
Forum Moderator
Registered User
Unregistered User
E-Mail this thread to a friend
Toggle e-mail notification


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.