OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


Only Forum Moderators, Administrators, and the owner of this message may delete it.

 
Jim Dean

Sage
20001000
Posts: 3022

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Guidelines for Backtesting, Tuning and Evaluat
Posted : 6/28/2010 10:20 AM
Post #20900 - In reply to #20869

Here's something else to consider, to "balance out" the prior comments.

The "prior-history curve-fit boogeyman" WILL BE present, for ANY system or strategy that has been "worked on" at all to improve it. It's almost impossible to avoid some degree of "fit".

We need to step back a bit and realize that to some degree, curve-fitting is just part of the game. Sad, but true.

That is, even if you create an entirely unique new Strategy, with every component invented from scratch, and base it ONLY on your current knowledge about TA and trading logic, and NEVER do any StrategyWizard runs or CB/NN/GA (NClub tools) optimization ... and you have all OT-parameter optimization turned off ... EVEN IN THAT CASE, there will be a small degree of what could broadly be called "curve-fitting."

The reason that this is true is simply due to your extant "common-sense knowledge". For example, "everyone knows" that a MACD(12,26,9) or an RSI(14) or a BolBand(20,2) or an ADX(14) is a "pretty good" setup to work with, when using those indicators. Somewhere, sometime in the past, you or other folks established those "common-knowledge" rules based on observed performance of the indicators on actual charts and/or with actual trading accounts. THEREFORE, any **subsequent** use of those same indicators and param's will be carrying forward that earlier, albeit crude, "curve fit" exercise.

Of course this example is pushing it ... but I think it's healthy to recognize that whatever we do, we do for reasons that are based on some form of prior experience and knowledge. That "past history" does not necessarily repeat itself ... but our memory and thinking process does.

We can help avoid "FOOLING OURSELVES" (that's the true curve-fit "gremlin") by taking reasonable steps when testing and tweaking, and being very VERY careful to eval those tests and tweaks against truly untouched, truly different data.

Prior posts in this thread have provided some broad-brush recommendations as to how to minimize the "fooling myself" pitfalls.

[Edited by Jim Dean on 6/28/2010 10:22 AM]

Deleting message 20900 : RE: Guidelines for Backtesting, Tuning and Evaluat


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.