OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


Only Forum Moderators, Administrators, and the owner of this message may delete it.

 
Jim Dean

Sage
20001000
Posts: 3022

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Guidelines for Backtesting, Tuning and Evaluat
Posted : 6/23/2010 9:30 PM
Post #20857 - In reply to #20855

The second starting-point component affecting testing and tweaking is:

the Focus-List ... this post is part one of two, about them:

The first point is simple ... use a focus-list that is representative of what you intend to trade, when you TEST the strategy.

However, be very careful to AVOID using the SAME Focus-List for your TWEAKING process. The danger is that you may "over-fit" the strategy to a particular list.

Much has been said in other posts about "survivorship bias" ... that is, the danger of using symbols in a present-day index such as the SP500 as the basis for testing and tweaking Strategies on past data, for example over the past ten years.

There is definitely a bias ... how much of a bias is arguable ... but it's definitely true that by using a list of stocks that you know, TODAY, are "high class" ... during years BEFORE the time that they had already been identified as a part of that "class" ... that you are carrying that "future" information back into the past.

I can't take the time to discuss this in depth here, but suffice it to say that this is by no means a "cut and dried" issue. For instance, that very same action might bias AGAINST short trades or a strategy that is supposed to work with vacillating stocks. Furthermore, if the hold time is only a few days or a few weeks, then one of the basic arguments of Technical Analysis is that fundamentals don't matter for such short hold times ... and existence in the SP500 certainly has more to do with fundamentals than technicals.

But, I digress. My point has NOTHING to do with survivorship bias ... although that is a factor which the solution below will serve to ameliorate to some degree, as well.

Too often, a focus list is just based on some very simplistic choice ... "hmm, let's use the SP500 ... or maybe just the SP100, to save time ... or maybe the Russell 1000, to be more exhaustive". The choice is usually made out of CONVENIENCE, rather than a carefully-considered rationale about what KINDS of stocks are best for THIS STRATEGY.

I am a strong, STRONG advocate for doing extensive, pattern-based PRE-FILTERING of stocks, based on price and volume data ONLY ... starting from as big a universe as possible to help eliminate survivorship issues, AND to produce more viable candidates.

How does this work? Again, way way too much to discuss here (we allocate a full multi-session Topic on this in AOTC ;~) ... but let me give you some examples:

1. You want stocks that have RECENTLY DEMONSTRATED adequate liquidity to support the entry and exit of the POSITION SIZE that you intend to use. An OmniScript forumula can be written to filter for this.

2. You want stocks that have RECENTLY DEMONSTRATED reasonable reward-potential (net price movement over typical hold-time) in comparison to the risk-levels (volatility) during that time. An OmniScript forumula can be written to filter for this.

3. IF you are playing "trends", you want stocks that have RECENTLY DEMONSTRATED a tendency to move smoothly as opposed to in big jumps, which rarely can be predicted ahead of time or captured. An OmniScript forumula can be written to filter for this.

4. You want stocks whose RECENT price/share is within an appropriate range to meet your capital limitations, allowing you enough shares to have some flexibility re partial exits or scaling-in, but not requiring huge numbers of shares (making slippage an issue) in order to obtain your desired financial position in that security. An OmniScript forumula can be written to filter for this.

5. You want stocks that have RECENTLY DEMONSTRATED, in the bar-periods you plan to trade, a tendency not to have "long tails" vs body-size, especially in relation to the necessary net-move you are looking for. An OmniScript forumula can be written to filter for this.

... Of course there are more considerations, but these are some biggies.

Each of these can be addressed by an OmniScript filter ... do-able within OT Basic ... but the filters can be more robust (and easier to work with) if you own OT Pro - this allows them to be saved as small compiled functions.

The filters should be used in OmniScan for daily selection of the Focus List you use ... and should be used in the Filter Block for any backtesting.

If you start with a VERY LARGE list ... such as all Optionable stocks ... or the sum of the Russell 3000 + SP1500 + Optionable ... that collection of about 4500 stocks can be filtered down SOLELY by the technical formulae above ... if not enough symbols appear - maybe you should not be trading in that market! ... if too many symbols appear - maybe you should be tightening the parameters of the filters more!

The point is, these filters (in the Filter Block) are applied EVERY DAY in the backtesting runs for THAT DAY ... there is no future bias. All of this is possible WITHOUT OmniScan ... tho OScan will significantly speed up your day-to-day USE of this method.


If this all sounds a bit out of the ordinary ... well, it is. Not many books or articles are written about it. I consider this robust pattern-prefiltering to be a significant "trading edge".

So - the point is ... if your focus list is DYNAMIC in real-life, based on a huge starting point that is logically filtered, then as long as your backtest uses the same logic, you will eliminate a lot of "survivor bias" issues ... AND you will find that your trades work more profitably!

... the next post deals with a second important issue re working with focus lists ...
Deleting message 20857 : RE: Guidelines for Backtesting, Tuning and Evaluat


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.