OmniTrader Forum OmniTrader Forum
forums calendars search
today this week
 
register logon control panel Forum Rules
You are currently browsing as a guest.
You should logon to access more features
A Self-Moderated Community - ALL MEMBERS, PLEASE READ!
Vote for Members who contribute the most to your trading, and help us moderate content within the Forums.


  Current location        Thread information  
OmniTrader 2017 Upgrade Forums
ATM
Macro + Micro Statistics
Last Activity 9/8/2019 11:15 AM
20 replies, 1236 viewings

Jump to page : 1
Now viewing page 1 [25 messages per page]
 
back reply
Printer friendly version

^ Top
mholstius

Veteran
1002525
Posts: 165

Joined: 1/13/2017

User Profile
 
Subject : Macro + Micro Statistics
Posted : 4/30/2018 6:19 AM
Post #44563

For the past month or two, I’ve been using data extracted from 144,000+ trades to develop algorithms that find robust combinations of symbols, strategies, and allocations to use in ATM. I hope to be done in time to present the results at the Bash (only 5 weeks away).

Part of my work required more precise measurement of the time spent in each market state. To make that information available, I’ve put together a spreadsheet that allows you to enter any date range from 1/3/00 to 4/26/18 and see the various statistics for the Macro & Micro market states.

The Statistics are presented on the “Market States” tab of the spreadsheet and the underlying data from the SPY is on the “Data” tab (in case that might be of use to you in your own work).

Simply enter the start and end date range that interests you in E2 and E3 and the tables will display the results ("days" in the tables refers to trading days, not calendar days);



In this example covering 1/3/00 to 4/26/18, there were 480 changes of market state and 29.4% of the time was spent in the Bull and Bull state.

The spreadsheet is attached ("Market State Frequency"). I hope you might find it useful in your own research.

Mark


[Edited by mholstius on 4/30/2018 6:39 AM]

Attached file : Market State Frequency.xlsx (307KB - 89 downloads)
Attached file : 00 snag.png (156KB - 857 downloads)

^ Top
LSJ

Legend
100100100100
Posts: 417

Joined: 8/17/2006
Location: Citrus Springs, FL

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 4/30/2018 6:35 AM
Post #44564 - In reply to #44563

Really helpful. Thanks very much.
^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 4/30/2018 8:15 PM
Post #44567 - In reply to #44563

Mark,
That's fascinating! Thanks for sharing. How did you extract the data of market state by day?

I'd like to run a test of flipping around the macro level bear and bull order to see how many of the Macro Bear's could also have satisfied the Macro Bull definition. I know the setup performs better with the Macro Bear first but I can't help but ask myself why.

Thanks,
Jeff
^ Top
mholstius

Veteran
1002525
Posts: 165

Joined: 1/13/2017

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/1/2018 11:45 AM
Post #44572 - In reply to #44567

Thanks Jeff and Larry…

I’ve attached a pdf file that shows step by step how to get the Market State for every day using OT and ATM ( "Getting Market State For Every Day" ).
There may be a better way, but this method works until Nirvana includes it in some kind of report…

As far as the order…
An interesting point is that the Micro Bull & Micro Bear formulas are NOT mutually exclusive.
They overlap a little and can fire on the same day. This was intentional, to kind of “hedge” things as the market changed. I was planning ahead for the day when ATM can have multiple market states firing at the same time, and not just have one fire in sequence, stopping the process at that point.

Hopefully, that’s on the list for Nirvana in the near future - but I did it to be ready for it if / when it happens.

In the meantime, that’s part of why there’s a difference in the outcome when the order is changed.

I hope the attached pdf is helpful and explains things clearly enough…

Mark

Attached file : Getting Market State For Every Day.pdf (1570KB - 170 downloads)
Attached file : Always Long.ots (788KB - 76 downloads)

^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/1/2018 12:06 PM
Post #44573 - In reply to #44572

Thanks Mark - very helpful!

Just so I'm not missing anything in your description of the process... The "Always Long" strategy is something that you created -- yes?

I can create one, but just thought I'd check first where you got it.

Thanks,
Jeff
^ Top
mholstius

Veteran
1002525
Posts: 165

Joined: 1/13/2017

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/1/2018 12:27 PM
Post #44575 - In reply to #44573

Hmmm...my bad.

I've been using it for so long, I thought it was a standard Nirvana Strategy.

Ah well, I've attached it to this reply. Let me know if it doesn't work OK.

Mark
Attached file : Always Long.ots (788KB - 89 downloads)

^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 11:53 AM
Post #44592 - In reply to #44575

The Always Long strategy and the process you provided worked perfectly. I was able to reproduce the results and will now await the introduction of ATM2 before exploring further (can't wait!).

I did have a couple of interesting observations:
1) It takes 2 runs to collect the market state for all the days between the dates. This seemed strange since I've never seen Mark make a process more complicated than it needed to be.

2) Modifying the strategy to use an orders block with MOC didn't fix it - still needed 2 runs. Changed the orders block to a trade plan (since ATM/AT needs a TP) and it still required 2 runs. (trade plan version attached)

3) The trade plan version produced almost the same results for market state as the orders block version. 18 differences in market state occurred. 17 of the 18 occurred during the BT period. 1 of the 18 was due to an extra day at the end of the run that the TP version captured. Of the 17 differences in the BT period, I couldn't determine why the market state was different other than the Simulator decided that the market state was in fact different. Data was not updated between the runs. The only change was to the Data Periods per the document and the change of the strategy to remove the orders block and introduce the trade plan.

There was one other difference in the Orders Block vs TP Block versions. The G/L was different for a handful due to the number of trades taken was one share up or down.

The base method I used was a brand new install of ATM Macro & Micro. I saved that method as a new one and changed the strategy for each market state. No other changes.

Of the 17 market state differences, most were a flip of a couple of states that oscillated back and forth slightly different during a transition period. Column X of the attached spreadsheet identifies the differences with a "1".

The good news is there was a 100% match between Mark's published Market State Frequency spreadsheet during the forward test period, using an Orders block instead of a Trade Plan block. The Trade Plan block version works equally as well and will produce the same Market State answer for the forward test period.

4) The G/L difference between the 2 methods occurred 8 times. Once during the BT period and 7 times during the FT period. 1 of the FT period differences is due to the one extra day the TP method reported at the end of the run (likely due to the MOC order vs next day close of the Orders block version). The difference of the G/L was due to a different quantity being issued for the order.

In all occasions, the Orders block version reported a purchase of 1 extra share with one exception during the BT period where it purchased 2 extra shares.

Comparison to the Market State Frequency spreadsheet for G/L could not be made. The 6 differences for the forward test period out of 4,608 FT days is a 0.13% difference. Since the focus was the frequency of Market States, I'm ignoring this difference as noise.
Attached file : MS-Trades-Combined.xlsx (901KB - 65 downloads)
Attached file : Always Long - TP.ots (788KB - 52 downloads)

^ Top
Jim Dean

Idol
2000500100100100100
Posts: 2922

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 12:18 PM
Post #44593 - In reply to #44592

FYI re All Longs System ...

Native OT provides what you need:
ABL-S = All Bars Long System
ABS-S = All Bars Short System

So ... no need to use custom OLang System, unless you are doing something special.
^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 12:35 PM
Post #44594 - In reply to #44593

Agreed...

The Always Long strategy uses ABL-S as the system. No special OLang.

I was originally looking to explore Market State order and the impact of that on the MS Frequency. However with ATM2 coming out, I'm holding off on that analysis.

But in the meantime, I thought I'd test out the method Mark used. The result of that testing is above and confirmed the need to do 2 runs with a change to the Data Period setting. It also highlighted a couple of observed differences when changing the strategy from using the Orders block to a Trade Plan block.
^ Top
mholstius

Veteran
1002525
Posts: 165

Joined: 1/13/2017

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 1:34 PM
Post #44597 - In reply to #44594

The difference you see was probably due to different exits in your other strategy.

The only purpose of this is to find the market state every day, so the the process is designed to have an entry every other day on a consistent basis. If another type of exit (trade plan) waits a day or 2, then everything is out of sync from that point forward.

Adding the day at the beginning just forces the second run to make the entries (and find the market state) on the days in between those from the first run.

The market state is based on the underlying (SPY in this case) and the strategy or trade plan has no effect on that. This is just a method to document what those market states are / were each day.

Mark

[Edited by mholstius on 5/3/2018 3:07 PM]

^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 2:22 PM
Post #44598 - In reply to #44597

Mark, thanks. That's the rational I understood for the strategy and why the mechanics of the 2 runs.

I was looking for a way to do it in 1 run even though it is fast. The manual change to the Data Periods is something easily forgotten. So, the TP I created used opening orders as MOO and closing orders as MOC with no conditions. I thought that would force an open and close in the same day and allow the strategy to advance to the next day and do the same -- effectively eliminating the need for 2 runs that I assumed was due to the use of the orders block. I'm apparently wrong here as to why it is every other day instead of every day. Maybe it is a function of the underlying system?

I also agree with you that the market state is based on the underlying symbol (SPY) and that the strategy and trade plan should have no affect. However, my test runs produced results that said, in certain situations during BT, market state is influenced by the construction of the strategy and use of orders block vs TP -- forward test remained unaffected and consistent between the 2 approaches. So, if you ignore the BT range and only look at FT, then all is good.

The test run also showed a minor variance of the allocation routine between the use of the orders block and TP which showed up in quantity and G/L. That may be due to the way the orders are assumed executed and equity calculation. These did occur during FT, but again the incidents of this observation were very small and not relevant to Market State determination.

The only take away I have with this second observation (and it's nothing new) is to always use Trade Plans in the strategies you use with ATM. I think that was also the message from Nirvana that ATM only works with Trade Plans.

Simply sharing results - the 2 run method works to collect market state data for the Forward Test period with great consistency.
^ Top
mholstius

Veteran
1002525
Posts: 165

Joined: 1/13/2017

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 3:15 PM
Post #44599 - In reply to #44598

Interesting...

Yup, I'd tried to build a strategy that entered MOO and exited MOC the same day too with no luck. Guess that's where my "Always Long" came from back in the day (it started with the ABL-S that Jim mentioned).

BTW, Jim - have any ideas on how to do that? Enter and exit on the same day to make this easier?

Mark


^ Top
Jim Dean

Idol
2000500100100100100
Posts: 2922

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 5:02 PM
Post #44601 - In reply to #44599

Trade Plans should not combine MoO with Mkt or MoC or BoO ... but they can use SM, Lim, SLim with MoO. I’ve tested this.

Reason ... MoO orders have to be placed when exchange is closed ... and MoC, Mkt, BoO orders have to be placed when exchange is open.

[Edited by Jim Dean on 5/3/2018 7:21 PM]

^ Top
Barry Cohen

Sage
5000500100
Posts: 5603

Joined: 1/19/2004

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 5:22 PM
Post #44602 - In reply to #44601

I think that was also the message from Nirvana that ATM only works with Trade Plans.


Only if you're AutoTrading.

have any ideas on how to do that? Enter and exit on the same day to make this easier?


You can set the entry in the trade plan as a Market order & set the exits as MoC. The only issue with that is if you were going to AutoTrade it in EOD - since currently AutoTrade only has one submit time.
^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 7:26 PM
Post #44605 - In reply to #44602

Thanks Barry, Mark, and Jim,

I tweaked the TP to have a MKT entry and MOC close. The results were the same:
1) You need to run 2 runs to get all the dates per Mark's original instructions.
2) Running for testing purposes only to obtain the Market State (not for live trading), the TP with MOO/MOC produced the exact same results for MKT/MOC. That includes entry/exit dates on the same day, Market State name the same (and in the same order), and the same G/L.

I ran an additional run changing the Vote block to not look back 1 day. As expected, it made no difference. I suspect it is in the code of the system and doesn't fire a signal every day, rather every other day from the beginning of the data period so the 2 runs are necessary to identify Market State on every day between 2 date points.

I also tried the insane and activated 225 systems. The result was a longer run with 0 trades generated. I'm conceding that it takes 2 runs with the Data Periods change between runs to collect all the Market States defined for a date range. That's probably what Mark had already concluded a long time ago.
^ Top
mholstius

Veteran
1002525
Posts: 165

Joined: 1/13/2017

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 7:48 PM
Post #44606 - In reply to #44605

Well... I was hoping someone had a better idea than mine.

Yup, I spent lots of time trying to come up with something for a one day trade too.

Thanks for trying. This isn't something that's done very often, so the extra time to run it twice isn't a big deal.

Now - on to developing methods for concurrent market states. ;-)

Mark
^ Top
Jim Dean

Idol
2000500100100100100
Posts: 2922

Joined: 9/21/2006
Location: L'ville, GA

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/3/2018 8:02 PM
Post #44607 - In reply to #44606

One day trade requires RT. Several ways of doing it. Must have at least two bars to do Entry and exit the same day. 195 min bars. Mkt or SM or Lim or SL to enter (first bar) - Mkt using Trade Session Boundary Stop 5 min before close to Exit (second bar).

But of course Strat will calc diff with half day bars.

[Edited by Jim Dean on 5/3/2018 8:06 PM]

^ Top
Barry Cohen

Sage
5000500100
Posts: 5603

Joined: 1/19/2004

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/4/2018 12:37 PM
Post #44613 - In reply to #44607

New EOD Dow30 profile with a basic trade plan with market order entries & MoC Fixed Profit & Loss stops.


Attached file : Trades.jpg (142KB - 472 downloads)

^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/4/2018 1:50 PM
Post #44614 - In reply to #44613

Barry,
What's the strategy(s) that you used? or the systems that were active in the strategy?

We aren't looking to use this to trade, just to identify the market state for each day. So, since the Macro/Micro is using the SPY to determine the MS, we've been using it as the only symbol in the profile.

The problem is that you need 2 runs to get the market state for every trading date between 2 dates. Everything that we've tried always produces every other day data (i.e. 1/3/2000, 1/5/2000...). Changing the Data Periods to add one to it and then re-running gives us the 1/4/2000, and 1/6/2000 -- filling in the missing date.

We just haven't been able to construct a strategy against SPY that allows us to do one run instead of 2.

This isn't a big deal. Mark's method works perfect to get what we need and the run time is only a few minutes to pull the data together. So, its not worth the time to figure it out from our perspective.


^ Top
Barry Cohen

Sage
5000500100
Posts: 5603

Joined: 1/19/2004

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 5/4/2018 2:07 PM
Post #44615 - In reply to #44614

I used the OPDemo strategy but added a trade plan to it. It's a good strategy to use if you want lots of trades & don't care about anything else. I just wanted to show that it is possible to enter/exit in the same day in EOD with Port Sim.
^ Top
jpb

Veteran
100
Posts: 122

Joined: 5/11/2005
Location: Brown Deer, WI

User Profile
 
Subject : RE: Macro + Micro Statistics
Posted : 3/21/2019 7:54 PM
Post #46084 - In reply to #44572

I have a lesson learned to share with the process of determining the Market State by day as described in Mark's doc. I've followed it a dozen times and it has worked flawlessly until recently. I had gaps in the days that couldn't be explained by holidays and weekends. I thought for sure something changed and then the light bulb went on.

It turns out, my ATM method had a Trade Filter that used correlation. The degree of correlation varies based on market state. Since I was using the SPY as the only symbol in the list, it always had perfect correlation to itself. So, effectively, the trade was being filtered out and I'd lose the day for purposes of viewing the market state for that day.

To resolve this, I simply deactivated all the filters and the process began to function as it should to identify the MS for all the dates.
Jump to page : 1
Now viewing page 1 [25 messages per page]
back reply

Legend    Action      Notification  
Administrator
Forum Moderator
Registered User
Unregistered User
E-Mail this thread to a friend
Toggle e-mail notification


Nirvana Systems
For any problems or issues please contact our Webmaster at webmaster@nirvsys.com.