OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


  Current location        Thread information  
OmniTrader 2017 Upgrade Forums
Trade Plans
No Optimization forward test
Last Activity 12/2/2018 11:56 AM
8 replies, 623 viewings

Jump to page : 1
Now viewing page 1 [25 messages per page]
 
back reply
Printer friendly version

^ Top
LSJ

Legend
1001001002525
Posts: 353

Joined: 8/17/2006
Location: Citrus Springs, FL

User Profile
 
Subject : No Optimization forward test
Posted : 12/4/2017 11:59 AM
Post #42765

My entry for the dumb question contest:

In trying to understand the process What is the forward test using if it follows a backtest with no optimization?

With a 1000 bar BT and a 1000 bar FT, e.g., the results of the FT are significantly different, and always lower performance, than the backtest. If no optimization has been done what exactly is the forward test basing trades on?
^ Top
Jim Dean

Idol
2000500100100100252525
Posts: 2892

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/4/2017 12:07 PM
Post #42766 - In reply to #42765

I haven't observed that "always lower" distinction before ... could be result of your timeframe selection ?

I've always presumed (and depended on!) non-optimized Analysis (and SW) runs to treat BT and FT just as two "bins" of equal significance.
^ Top
LSJ

Legend
1001001002525
Posts: 353

Joined: 8/17/2006
Location: Citrus Springs, FL

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/4/2017 3:13 PM
Post #42773 - In reply to #42765

I may be doing something wrong but I have done many tests where I have "always" observed the lesser performance on the FT. I realize that is not completely scientific but I have never seen otherwise.
I have attached files from one of my many tests. This is a daily 1000 bar BT and FT with no optimization and the relative results of BT and FT are typical of what I get. This is what prompts me to ask what is happening in a FT that is followed by a BT with no optimization. Thanks for any light you can shed.
Attached file : BTFT.jpg (355KB - 56 downloads)
Attached file : Settings.jpg (140KB - 41 downloads)
Attached file : Strats.jpg (107KB - 42 downloads)

^ Top
Jim Dean

Idol
2000500100100100252525
Posts: 2892

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/4/2017 4:00 PM
Post #42776 - In reply to #42773

Could be a couple of reasons for this:

Market during latest 1000 bars (4 yrs) could be less amenable to those Strats than in the prior 1000 bars / 4 yrs, thus showing lower performance. To check, try using the date input instead, and move everything back 4yrs, so the old BT pd is now the new FT pd, and the new BT pd is 4 yrs earlier.

Or ...

If those Strats had been "tuned" a bit too much during development, and if the tuning was done with an emphasis on bars a few years ago and before (not at all uncommon), then the tuning process itself will have introduced "invisible optimization" to the results. The only way to be sure is to test the strats using both BT and FT ranges that are AFTER the strats were first released.

[Edited by Jim Dean on 12/4/2017 4:02 PM]

^ Top
LSJ

Legend
1001001002525
Posts: 353

Joined: 8/17/2006
Location: Citrus Springs, FL

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/4/2017 8:40 PM
Post #42783 - In reply to #42776

Jim, thanks for the response. That helps me to understand that there is nothing squirrely going on internally with the test process as I know you have a lot of expertise in the workings. I have not changed any of the strategies and am only being restrictive with the voting parameters to eliminate trades. So, as you say then I suspect it has a lot to do with the period over which the strats were originally tuned.
I will split up some of the tests to compare periods.
^ Top
LSJ

Legend
1001001002525
Posts: 353

Joined: 8/17/2006
Location: Citrus Springs, FL

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/5/2017 7:32 PM
Post #42801 - In reply to #42783

I have run a backtest with a forward test and a backtest with no forward test with the test periods aligned so that the backtest with no forward test is over the same period of the forward test where there was a backtest.
No optimization was used and all settings are identical. The number of bars in the backtest and forward test are the same(1000).
Comparing the results they are not the same, and not by rounding but by significant enough values that there seems to be something else going on.

It appears OT is processing the tests differently and I am trying to find out why. This apparent anomaly shakes my confidence in these tests until I understand why.

Any explanation would be appreciated.
^ Top
Jim Dean

Idol
2000500100100100252525
Posts: 2892

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/5/2017 9:06 PM
Post #42803 - In reply to #42801

Good test. Barry needs to investigate.
^ Top
Barry Cohen

Sage
5000100100100100252525
Posts: 5497

Joined: 1/19/2004

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/6/2017 10:21 AM
Post #42811 - In reply to #42803

I cannot reproduce what you are seeing using the same strategies & settings. The BT & FT seem very equivalent for me, as they should be with mechanical strategies.

Looking closer at your screenshots, I see that the individual strategies are actually equivalent stats, & it's only the "All Strategies" stats that are suspect. This points to the Strategy Voting settings you are using (button at the bottom of the Strategies list). I would guess that you're using a ranking method or a filter & that's what is throwing it off. Try disabling everything in the Performance tab under Strategy Voting.

^ Top
LSJ

Legend
1001001002525
Posts: 353

Joined: 8/17/2006
Location: Citrus Springs, FL

User Profile
 
Subject : RE: No Optimization forward test
Posted : 12/6/2017 12:26 PM
Post #42818 - In reply to #42765

The screenshots from a previous post were not for the recent 1000 bar test I referred to but good point.
However, if all settings are the same and the data is the same, what is done differently in a FT with no backtest optimization vs. a BT with no optimization over the same data?
I re-ran without any strategy voting and the results are very close now. There is a little over 1% difference in the number of trades and HR% is very nearly the same so that is a completely acceptable result. As a user and not a programmer I don't need to know why the strategy voting is different and am comfortable with the fact that is just is.
Thanks
Attached file : BTFT12_6.jpg (354KB - 51 downloads)
Attached file : BT12_6.jpg (329KB - 36 downloads)

Jump to page : 1
Now viewing page 1 [25 messages per page]
back reply

Legend    Action      Notification  
Administrator
Forum Moderator
Registered User
Unregistered User
E-Mail this thread to a friend
Toggle e-mail notification


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.