OmniTrader Pro Resource Forum
OmniTrader Pro Technical Support
TEST Profile
23 replies, 4375 viewings
|
OmniTrader Pro Resource Forum
OmniTrader Pro Technical Support
TEST Profile
|
Last Activity 9/20/2023 2:25 PM 23 replies, 4375 viewings |
|
|
|
| Printer friendly version |
Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010  User Profile |
I am trying to create a TEST Profile so as to be able to test some of my own Intraday data history. Here are some of my initial questions: The "Using the Test Profile" topic in help file says; "To use the Test Profile, historical data will need to be imported from a source file on your hard drive. There are two methods for importing test profile data. If your data is in the proper format, according to the Rules for Importing Test Data, test data can be treated like user data." 1. Since the topic describes TWO ways of using our data; I want to know if OmniTrader will import data to its own database in both the cases or will it directly use our data IF "our data is in the proper format" ? 2. How can we later delete the imported data; as otherwise the imported data files can keep getting very bigger/ 3. If my CSV data file is updated everyday after Market Close with intraday records for the day appended to historical data; will I need to import everyday even if my CSV files are in PROPER format? 4. If I need to use data for nearly 500 securities; will I need to import all of them separately or is there some way to have all of then imported in one step. with Regards Sanjiv Bansal | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
1. Click the Pencil icon at the top of the Focus List. 2. Select User Data as the Data Source at the top of the Edit window. 3. Use the Drives and Directories window to browse to the location where the data is stored. 4. Open the folder where the data is stored and files that are in the proper format will appear in the Data Found in Selected Directory window. 5. Highlight the files to be added to the Focus List and use the Add button to move the selected files into the Focus List. I took the steps exactly as explained in the help_file; howvever I am getting error and no securites are being imported. The error I am getting is "0 symbols were imported to Easy data. 1 Failed to be added because the symbol size was too large". I think this error should not come as the File name is just NINE character long. The same data file imports fine if I import it via: File»Import»Data File The CSV data file as well image of the error is attached. I will look ofrward to your guidance; It will be a great confidence enhancer for OmniTrader usage; as the date being used in OmniTrader as well the software I am currently using will be same and I will have more confidence in OmniTrader results. With Regards Sanjiv Bansal [Edited by Sanjiv Bansal on 5/24/2012 11:03 PM]  Attached file : NFTY_TEST.zip (138KB - 399 downloads) Attached file : OmniTrader_Trial_Profile_01.png (60KB - 488 downloads) | |||
| Curtis Legend   Posts: 460 Joined: 1/4/2009 Location: San Diego, CA User Profile |
1. After creating the TEST Profile, I then turn off my datafeed, and then create a real time profile (as that is what I always want to test) that I name with "Gather" in the name. I make sure the RT "Gather" profile has the data periods I want to gather and the symbols in the focus list. 2. Exit OT, and go to the Database directory and delete "Symdata.otd". I also delete "SymdataTest.otd". The reason for deleting "Symdata.otd" is that it can get full of bars of data that you don't need in your TEST profile. So by deleting "Symdata.otd" you in effect clean out all old gathered bars from OT, and start over fresh. In the RT profile OT looks to "Symdata.otd" when it gathers bars for historical display in charts. In TEST profiles OT looks to "SymdataTest.otd" for it's historical testing. What we are trying to do, is clean "Symdata.otd", fill it up with the bars you want in your TEST profile, and then move those bars to the "SymdataTest.otd" file so that your TEST profile can acces the bars you collected. 3. Restart OT, and it will create a new empty "Symdata.otd" during the startup. Now connect your data feed in your RT "Gather" profile and wait until all the data has been gathered - usually the symbols icons turn green I think. 4. Open the Database directory and make a copy of the "Symdata.otd" file, naming the copy "SymdataTest.otd". Your TEST profile looks to "SymdataTest.otd" to gather the bars that you see in the TEST profile Charts. 5. Open the TEST profile. 6. Close OT and reopen to the TEST profile (a safety step). Your bars should be loaded now in the TEST profile. Once you have done the above, I have found that OT can better handle updating the TEST profile with new bars on an ongoing basis - but sometimes not and I have to go through the above steps 1-6 to update the bars loaded to the current date. Actually, I do the above steps every time I want to run a TEST profile and don't even bother with OT's update feature for TEST profiles. Gathering bars for 500 symbols can be alot of bars! Hope this helps [Edited by Curtis on 5/20/2012 2:37 PM] | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
Seems like one step forward: Your guidance clears things about where TEST data is stored. And how to do cleanup etc. However; the original source of delayed Intraday data in my case will be ASCII Text files that I already have. This history goes back decently enough (Longer than commercially available sources), and these get updated everyday in the evening. If I can have some efficient approach to import/use this data of nearly 500 or more securites into OmniTrader; I can conveniently test relevence of OmniTrader for my usage. Thanks again for your guidance - taking me ahead. With Regards, Sanjiv Bansal [Edited by Sanjiv Bansal on 5/20/2012 8:12 PM] | |||
| Ryan Olson Veteran Posts: 169 Joined: 12/29/2006 Location: Austin, Tx User Profile |
Thanks Curtis for replying. However, there was a change recently in the Test profiles and our documentation doesn't seem to reflect this change as of yet. Now to use the test profile you can simply click on File > New > Create a new test profile. When you enter the profile you can use your current real time data feed (OmniData, IQFeed, eSiganl). The profile will look as though it is an end of day profile with a "To Do List" visible with one major difference. If you click on the Edit > Data Periods you will see that you have the options for intraday time frames. Now, once you have your data periods set you can add the symbols you wish to run the test on and click your "To Do List". This will download the real time data to store on your computer and you will now be able to use the Test profile on your sample data. I hope this makes it easier for all of you to run your Test moving forward. | |||
| Jim Dean Sage   Posts: 3433 Joined: 3/13/2006 Location: L'ville, GA User Profile |
I know that some work was done about a year ago. At that time, Cose told me that the intention was to dramatically increase the number of historical bars supported, with the minimum goal as I recall of supporting a 500-symbol list of one-minute bars for the entire available history (ie back to Jan 2007) Could you please check on this and either confirm or correct my figures? Please be specific - I have spent many hours fruitlessly trying to discover the limits without success. Thank you! [Edited by Jim Dean on 5/21/2012 2:02 PM] | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
Thanks for your reply. It also takes us one step forward in our understanding of TEST Mode; however..... 1. The help file says: "OmniTrader's Test profile is an advanced feature designed to help you test strategies for profitability over large amounts of historical data. Using imported data from text files in .csv or .txt format, this profile is a powerful, complex feature for strategy testing." Your post seems to suggest that you have improved/added support for using data from the user's current real time datafeed in TEST Profile, it is good. However; I am still looking forward to your guidance about how can I efficently use data from my own ASCII TEXT files in TEST Profile. Your response to the question/problems posted in my original posts in this thread will be helpful. 2. You adviced: "click your "To Do List". This will download the real time data to store on your computer" While in TEST Mode; will the realtime data be downloaded just once or repeatedly/continuously? My view is that since the primary objective of TEST Mode is backtesting; it should not be collecting data repeatedly on its own. (Maybe you can allow user to recollected data manually by selected "Reload Data" in Right-Click Menu). Please guide what is that current approach in the program. 3. (Though not critical in current context) Edit > Data Settings > DataFeed The selection here seems to be Global in OmniTrader; How about having it specific to a profile? Though; I seem to be able to manually import individual security data into TEST Profile; I want to be able to do it efficiently, as explained in help file; your response will help. With Regards Sanjiv Bansal {After posting this message; I tried as suggested by you with eSignal onDemand - with extended Intraday history - the initial experince seems to be wonderful on its own - GREAT. However I still need to use my own ASCII historical files for some specific purposes - eSignal does not seem to have continuous contracts for Indian Futures Market - and I want to keep costs low during the period when I am familiarising myself with OmniTrader - and see its benefits for me. You will agree that sometimes it can take upto an year or more to find something worthwhile in a new pursuit/approach} {Some content not-important to the subject removed} [Edited by Sanjiv Bansal on 5/22/2012 11:33 PM] | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
I updated to the latest Pre-Release today {Pre-Release 2L (2027)}. Still I am coming across the same problem. This time I get TWO Error messages. Images are attached. OTLog has been submitted to Nirvana as suggested in the first error message. Will look forward to your response. With Regards Sanjiv Bansal [Edited by Sanjiv Bansal on 5/24/2012 11:21 PM] Attached file : OmniTrader_Trial_Profile_20120525_01.png (73KB - 1225 downloads) Attached file : OmniTrader_Trial_Profile_20120525_02.png (71KB - 1157 downloads) | |||
| Jim Dean Sage Posts: 3433 Joined: 3/13/2006 Location: L'ville, GA User Profile |
Try removing the dashes and underscores. | |||
| Ryan Olson Veteran Posts: 169 Joined: 12/29/2006 Location: Austin, Tx User Profile |
Jim- While writing this post, I am currently set to 60000 1 minute bars; the system is loading 59831 bars using OmniData Real Time. Cose has said that there are some limitations to the number of bars and that may be related to the data vendor that is being used. On Tuesday when Cose is back I will check to see if he has any solid numbers. Also, to everybody who is interested in test profiles please take the time to join me on June 6, 2012 - 4pm CST when I will be presenting a 30min webinar on the Test Profiles. Register Now | |||
| Jim Dean Sage Posts: 3433 Joined: 3/13/2006 Location: L'ville, GA User Profile |
I need to know the max limits inherent in the CODE, unrelated to vendors. I am using omnidata. Number of one-minute bars I am talking about is MUCH greater than just 60k: 390/day * 250d/y * (jun2012-jan2007) = aprx 550k bars … and … For a list of 500 symbols, that's 268 million total bars. Each bar requires at least 4 bytes * 8 items (ohlcv & time & date & sym), thus 32 * 550k = 34 million bytes for full omnidata 1min bar history for one symbol (ie 33 meg/symbol). For a list of 500 symbols that would be 15 gig of memory. I know that is NOT possible since OT is 32 bit and cannot address more than about 3+ gig, even if it is all avail. As I recall, Cose told me there were some built in database limits but I never found out how that translated into symbols and bars. And there are some hard-coded input limits I suspect. Admittedly this is an extreme scenario - but it's not absurd that someone might want to test 500 symbols in SW over the full 5.5 year avail OData history. I was HOPING that Cose would build in some "paging" yo take care of needs like this - or better yet, revamp the way the data is handled to accommodate processing without database or hardware limitations. I doubt that has happened, or will happen soon. THAT is why I want to know in very explicit terms what the limits are. I'm pretty tired of banging into them. It can get frustrating, and can waste a lot of effort. I hope you can help. Thanks. [Edited by Jim Dean on 5/25/2012 3:40 PM] | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
Thanks for following up on the subject. I am working with OmniTrader with more commitment of time and resources this time to see if it can help in research here. Ease of use of my own delayed intraday data will help me greatly in building confidence in OmniTrader; as the environment/data will be same as the one I am currently using in my present charting/TA software. Please clarify a few things: 1. OmniTrader will only parse the data that you provide. This means that the data isn't stored locally in the OmniTrader symbol databases. There seem to be two files in OmniTrader that might be relevent in the current context: 1. Symbols32.mdb (The file that holds List of all the Symbols, their full names, Shares outstanding, Optionable, etc. etc.) 2. SymDataTest.otd (The file that holds the Datetime_OHLCV data records for various symbols that we have imported into our TEST Profiles). I was having impression from the recent discussion that all the Datetime_OHLCV data records for symbols in TEST profile are stored in the SymDataTest.otd file after the import. Can you please confirm that no data is stored in either of the files. 2. To delete the data you would simply right click on the symbol in the focus list and select remove symbol. I tried the same thing; however filesize of SymDataTest.otd file did not go down after this action even after OmniTrader was restarted. That caused me to wonder will it not result in the SymDataTest.otd filesize keep getting bigger and soon hitting some problem. Please advice as appropriate. 3. If you are using CSV or Txt then you will need to manually update every day. In another post you have written: Any data that is maintained in a flat file or text file will need to manually be updated by removing and re adding. If I am looking forward to maintain 100-500 symbols in the test profile; will it mean that I will need to take 101-501 sets of steps everyday to import the data? Every set of step for importing an individual symbol might involve multiple manual sub-steps to be taken. Does not seem to be viable. I am looking forward to do it in some viable manner. The help file topic "Using the Test Profile" seems to suggest that there is one approach which can be used to do it efficiently. However when I take this approach, I come across the errors as mentioned above. Help file image attached: It will greatly help if you can please provide step-by-step instruction to (1.) Initially populate the Test_Profile with intraday data of 100-500 securities from user's own CSV files (2.) Update it with delayed intraday data from user's own CSV files in the evening everyday. With Regards Sanjiv Bansal [Edited by Sanjiv Bansal on 5/28/2012 1:26 AM] Attached file : OmniTrader_TestProfile_HelpTopic.png (139KB - 1066 downloads) | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
I am looking forward to your response. I understand you will be holding a webinar tomorrow on the subject. It will greatly help to be sure before attending the seminar; how can I do the things as mentioned in the helpfile. With Regards Sanjiv Bansal | |||
| John J Regular Posts: 90 Joined: 12/20/2011 Location: Leduc User Profile |
It's not listed here: http://www.omnitrader.com/Education/Archives.aspx | |||
| Jim Dean Sage Posts: 3433 Joined: 3/13/2006 Location: L'ville, GA User Profile |
This problem re CSV implementation is as yet unresolved and the user is still unable to make OT work. Please continue to work with him. | |||
| Mel Veteran Posts: 235 Joined: 3/18/2006 User Profile |
"Barry Cohen has confirmed that we have a bug in OT 2012 that does not allow you data to be imported. It is still seems to be working in in 2011 and we will have an update that should correct the problem in the next release. John M Garcia" The file name of the symbol data is unimportant. The import dialog box's first field, Symbol, is where you put the name of the symbol in the file being imported. I have written an extensive tutorial on importing test data (when it works) in the Forex section of the OETC forum. Look there for more info if you are interested. Mel | |||
| Barry Cohen Idol Posts: 2277 Joined: 1/1/1900 User Profile |
| |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
Jim Dean; thanks for following it up; I hope again that this time I can start using OmniTrader. Mel; thanks for the update. I did see your note at tradetight yesterday. Using that approach will be good if we are looking forward to populate/update the Test Profile occasionally. For daily updates the other approach mentioned in the helpfile (If possible to implement) might be better. Let's see if it can work. Thanks again for your guidances. With Regards Sanjiv Bansal | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
I viewed this thread again yesterday and noticed what Ryan mentioned a few months ago: OmniTrader will only parse the data that you provide. and in another post he has guided : Any data that is maintained in a flat file or text file will need to manually be updated by removing and re adding. I think, this is what I might be looking for; I will update the Text File everyday in evening and remove and re-add the symbols. However; in the same post he has said If we were in the normal running mode of OmniTrader this would be correct. However, as we are in a test profile you will need to click on File > Import > Data File. Here is a bit of confusion: UserData seems to be availble in TestProfile only. And If I click on File > Import > Data File; then only data for only one security seems to be importable. Further; the facility to "Parse" and directly use my data without importing does not seem to be possible. To Simply solve the problem can you please arrange someone from Support to take my computer on Remote/Teamviewer and walk me through to have it my Intraday Data work without import. With Regards Sanjiv Bansal | |||
| Barry Cohen Idol Posts: 2277 Joined: 1/1/1900 User Profile |
I've reread this thread & want to clarify a few things. Also the issue I previously said was getting fixed was for something else - I apologize. Using Metastock or CSI data formats, you can use Stocks or Test Profiles & import multiple symbol files at the same time. This is done under the Pencil icon when you select User Data for the data source. This type of data cannot be used with intraday data. The user updates the data through another software, then the data is reflected in the charts after refreshing the profile. With CSV data, only Test Profiles can import it. It can be intraday data, but this feature is limited to only 1 symbol imported at a time. I believe the data needs to be imported again as well if it was updated through another software - though I'm not 100% sure on that. To import a CSV file, while in a Test Profile, click File -> Import -> Data File. Type in the symbol & click Resolve. Click the ... button to "Import One File", locate & select it in your drive, then click Ok. The CSV file must have the correct header fields (your file in the first post of the thread does). There is an option to "Import Multiple Files" with CSV, but it doesn't seem to work. I will check & see if it can be fixed any time soon. Though, in order for it to be possible to work, your CSV files would need to also include a Symbol column header with values, otherwise there's no way OT can distinguish what symbol is in each CVS file. So if you have 2 current CSV data files that have this I can take them to development. | |||
| Barry Cohen Idol Posts: 2277 Joined: 1/1/1900 User Profile |
I'm sorry Mel, but there must be some miscommunication here as this issue has not been corrected yet. Could you please attach a current CSV data file that I can use to test this? | |||
| Mel Veteran Posts: 235 Joined: 3/18/2006 User Profile |
The problem is that intraday test profiles (the only kind that are needed) are not importing data properly. The data I import is not current - it is from a disk purchased from a data vendor. The data used to work fine. I had several test profiles loaded with data from the disk, some futures, some FX. Then, awile ago, the data was missing from the profiles. If I try to import the data now, say 1 minute data, I can see the one minute data, but it doesn't appear in any other time frame, e.g. 5 minutes. I sent a ES data file (one minute quotes) to John Garcia. That was the file that prompted the response I posted about the bug to be fixed. As it stands now, I have a large amount of data I want to use to develop strategies, but it is no longer usable. If John no longer has the file, I will resend it. Just let me know. Mel | |||
| Sanjiv Bansal Member Posts: 13 Joined: 2/25/2010 User Profile |
Thanks for your response. I am very happy; now that the problem is on your radar, I can look forward to it solution soon. Your response until the end of third paragraph {....correct header fields (your file in the first post of the thread does).} match with what I have understood about the feature. There are a few things in the last paragraph about which I will put forth my understanding of the things; please correct if it is wrong: 1. For a IMPORTING into a single symbol, there can be one CSV file or there can be multiple csv files. (for one CSV files what you have written in third para seems to be correct; and I have not checked importing multiple CSV files). It seems that OT has designed feature of multiple source csv files for Single symbol to take care of situations when csv data of a symbol is broken into various periodspans. Since the Symbol has been Typedin/resolved as explained by you, there is no need to have a Symbol Column header as the target Symbol is Single only; whether the csv source is single or multiple. For updates naturally we have to reimport. 2. Apart from IMPORTING; there seems to be another feature also; where the CSV files is not imported at all and CSV file is PARSED and used as it is (the CSV file has to be in a proper format as explained in help file, and mentioned by you). In this situation the software seems to recognize the NAME of the CSV file itself as the SYMBOL. Seems to be fine and in this instance again there is no need to have a Symbol column. (This is what I got from the helpfile, Ryan's response; and from actual working of the software - I could go upto the step of seeing the CSV files being recognised; filenames being visible as Symbols - it was after this step that I got the error mentioned in my previous posts). The only thing about which I am not clear in this case is how OmniTrader will know about the "Data_Periodicity" and "Data_Compression" of the data in the CSV file. Since there is no import involved at all; we just have to update our CSV file ourselves. Have not yet been able to figure out how the updated data will get reflected in OmniTrader as have not been able to see it implemented. However it will be fine in whatever way you specify, as this feature is not to be used for situations where user is looking for RT streaming updates. I look forward to update the CSV file once/A_Few_Times a day. Hence refreshing the profile/removing_readding symbols to focuslist/restarting omnitrader is fine with me. It is this feature that seems to be most suitable in my case ; and can be used by users from all over the world for initial exposure to OmniTrader. Helps in smoothly understanding/evaluating the software. In my case it will ensure that the data I am using right now and the one I will see in OmniTrader will be same - enhances confidence. 3. My initial undertanding is that presently there is no feature to import Multiple Symbols in one go. Yes; if this feature is there; then there will be need for some way to specify Symbol. Here I can have the CSV file in any format and naming convention as you specify; hence getting csv files from me may not be required. I will suggest that development may test the feature with multiple CSV files in your proper format. Once you have it working; I can have the CSV files formatted and named as you suggest. I can certainly send files if still needed. It is nice that the problem is getting your attention and I look forward to have it solved sometime soon. May I please have your email address; so that details which are not interesting to other forum users can be sent by email. Will look forward to your response. With Regards Sanjiv Bansal | |||
| Barry Cohen Idol Posts: 2277 Joined: 1/1/1900 User Profile |
1. You're right, the import multiple files is when one symbol is split into many different CSV or text files. There is no possible way to import multiple CSV symbols at the same time. 2. I'm not aware of this feature. If you can get me a couple CSV files for testing I can look further into it. If you use method 1, it should currently work. Though you have to import it again & again any time you have new data. I don't have any current data files to use. Please email me 1 or 2 current CSV files (same format as the file you posted initially is fine) at bcohen@nirvsys.com Mel, if I can get a CSV file from Sanjiv I won't need a data file from you, thanks. |
|
|
|
|
Administrator
Forum Moderator |
Registered User
Unregistered User |
 E-Mail this thread to a friend |
Toggle e-mail notification |
