OmniVest Developments
OmniVest News
Trade Processor Pre Release 4D is posted
52 replies, 4769 viewings
|
OmniVest Developments
OmniVest News
Trade Processor Pre Release 4D is posted
|
Last Activity 2/20/2018 9:38 AM 52 replies, 4769 viewings |
|
|
| Printer friendly version |
SteveL Veteran   Posts: 189 Joined: 10/11/2012 Location: Boulder, CO  User Profile |
I see each TP's log files. From each TP's View->LogFiles drop down, I see: Local Disk(C:)>ProgramData>NirvanaSystems>TradeProcessor>Logs>MyOVID>MyOVAcctName>2016>Jul. In this folder, I see the following folders: ErrorLogs Reports and Alerts Strategies Symbols TP NOTE: ProgramData is (normally) a hidden folder. Is that why you can't find Logs through FileExplorer? | ||
| Barry Cohen Icon   Posts: 1844 Joined: 10/11/2012 User Profile |
| ||
| Mark S Regular Posts: 56 Joined: 10/11/2012 Location: Barrington, IL User Profile |
| ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
This morning, I started a 4th TP/IBG pair. And over the next hour, I had a couple IBG connections drop during the critical minutes before and after the market opening. I'm running this on a "t2.micro" instance. Over the weekend, I will transition to a larger instance. This t2.micro AWS server appears to be running too close to the edge for reliable service with 4 TP/IBG pairs, and perhaps 3 pairs is also too close to the edge. [Edited by SteveL on 7/30/2016 9:02 AM] | ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
I think logging of Monday's trades dealt my server's last hand. However, I had no execution problems. After I created a new AWS instance and re-loading TP/IBG, all OV positions were reconnected without problem too. BTW, building a new AWS instance is not the only way to address lack of disk capacity (e.g. delete files, etc.) but that's the avenue I chose for a clean platform because I'm surprised the capacity was used, so I can now watch to assure the log file capacity is released. I case you don't know, to check disk capacity, open server's Windows Explorer, right click on your "Local disk (C:)" and click on Properties. If you are out of "Free Space", that's what has caused my server failure. Otherwise, Steve's point of RAM may be something to look at too. Also, I'm running 3 TP/IBG. Have run as many as 4 but not recently. And I am keeping TP Logging to 1 week in each TP. [Edited by BrianD on 7/29/2016 5:09 PM] | ||
| Mark S Regular Posts: 56 Joined: 10/11/2012 Location: Barrington, IL User Profile |
I do see that I have less than 1GB of storage left. [Edited by Mark S on 7/30/2016 8:18 AM] | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
I got a popup telling me the connection was lost. I think it was a TP popup. I'll try to remember to snapshot future popups and post them. I've also gotten IBG popups when a connection is lost. I had a "connection failed" popup from IBG this morning because the newest accounts were connected to the Master/Advisor account and trading was disabled by IB on those client logins (which I've now re-enabled from the Master account). NOTE: IB makes those changes on Friday night after market close. Windows Task Manager is where you see CPU and memory status. Here is a snapshot showing where you open the Task Manager: After Task Manager is opened, select the Performance tab, and select the radio button for the resource you want to see. I've selected Memory: The pagefile is used by the OS to handle over committed memory needs. So, if the pagefile is big enough, then I think, theoretically, things should continue to work. As memory requirements exceed the 1GB available on a micro server, then paging starts happening (things get swapped in and out of the 1GB RAM memory to the pagefile on some storage device which is slower than RAM). You'll see in this snapshot that the 1GB is overallocated. The pagefile has 1.9GB of committed memory. But, there is still plenty of pagefile space left. I suspect that TP and IBG are designed with timeouts and expectations about how fast things like connections should be established. So, when you have too many things running, then the CPU work queue starts getting longer, and things start timing out. That is my interpretation of what happened on Friday morning. So, the problems I saw could have been due to 1 CPU not keeping up, or the pagefile access time is not fast enough. If my analysis is correct, then either a 2nd CPU or more memory will resolve the connection reliability problem. Unfortunately, to get a 2CPU AWS server, you have to move up 2 levels. t2.Micro has 1 CPU and 1GB. t2.Small has 1CPU and 2GB. t2.Medium has 2 CPUs and 4GB. Each step doubles the cost. So, I'll try the first step and see how it goes. In the grand scheme of things, these costs are minimal, and worth avoiding a missed trade entry/exit. [Edited by SteveL on 7/30/2016 9:00 AM]  Attached file : 2016-07-30_7-23-48.png (73KB - 530 downloads) Attached file : 2016-07-30_7-26-10.png (49KB - 517 downloads) | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
I see you've edited your post since I started working on my response. So, obviously you already knew some of what I posted. I believe the CPU usage is MUCH higher (close to 100%) around the times that are configured for the TPs to start doing things (like placing trades). Which makes me think that one other solution might be to set different times for each of the TPs - stagger them 1 minute apart for placing trades. I think I'll try that first, and see how things go. | ||
| Mark S Regular Posts: 56 Joined: 10/11/2012 Location: Barrington, IL User Profile |
I had no idea about the Task Manager capabilities - thank you! I'll try to watch that at 8:30 Monday morning. I thought the idea of staggering trade times made a lot of sense, so I've done that, too. | ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
Also, I doubt there is a lot of computation going on, mostly network and graphics? Would not count on paging to address lack of RAM in windows. I suspect physical RAM and disk space are the things to watch most. Also, monitors eat up capacity. | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
NOTE: I have the submission times staggered by 5 minutes. [Edited by SteveL on 8/1/2016 8:22 AM] Attached file : 2016-08-01_7-12-26.png (21KB - 472 downloads) | ||
| Mark G Member Posts: 41 Joined: 10/11/2012 User Profile |
--Mark G. | ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
Sounds like a bug though. | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
I'll have to work with IB to get that worthless item out of the account. So otherwise, all the other accounts are running fine on a t2.micro. Here were the two errors produced by this failing TP: Problem signature: Problem Event Name: APPCRASH Application Name: TradeProcessor.exe Application Version: 1.4.20.0 Application Timestamp: 573a408c Fault Module Name: KERNELBASE.dll Fault Module Version: 6.3.9600.18340 Fault Module Timestamp: 5736541b Exception Code: e0434352 Exception Offset: 00014878 OS Version: 6.3.9600.2.0.0.272.7 Locale ID: 1033 Additional Information 1: 0ea4 Additional Information 2: 0ea4d10b75a2cc218ace190122c26039 Additional Information 3: b213 Additional Information 4: b213103fad9442fa10ddc5231e319139 Read our privacy statement online: http://go.microsoft.com/fwlink/?linkid=280262 If the online privacy statement is not available, please read our privacy statement offline: C:\Windows\system32\en-US\erofflps.txt Problem signature: Problem Event Name: APPCRASH Application Name: TradeProcessor.exe Application Version: 1.4.20.0 Application Timestamp: 573a408c Fault Module Name: KERNELBASE.dll Fault Module Version: 6.3.9600.18340 Fault Module Timestamp: 5736541b Exception Code: c000041d Exception Offset: 00014878 OS Version: 6.3.9600.2.0.0.272.7 Locale ID: 1033 Additional Information 1: 5faa Additional Information 2: 5faa46876913924e5bf842e8d4040f41 Additional Information 3: 7451 Additional Information 4: 7451ecef86d567f04c612b2fccba3fa8 Read our privacy statement online: http://go.microsoft.com/fwlink/?linkid=280262 If the online privacy statement is not available, please read our privacy statement offline: C:\Windows\system32\en-US\erofflps.txt | ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
| ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
I think the cause of all the failures this past week have been due to that untradeable stock that was a leftover from long ago. I agree with your implication that TP should handle this without crashing. The necessary form has been submitted to IB to remove the symbol (that is no longer tradable) from the account. So, I expect that either tonight or tomorrow it will be removed, and I'll see if the failures stop. NOTE: As of this morning, I'm now running 5 TP/IBG pairs on this t2.micro. | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
NOTE: I can see that I need to specify separate directories for each IBG instance to keep their logs, etc. separate. Possibly/probably an indication that I need a bigger AWS server. NOTE: "Committed Memory" is 2.7GB (on a t2.micro with 1GB of RAM). Also, the previously failing TP again failed when submitting orders because (I assume) the no-longer-tradeable symbol is still in the account. Problem signature: Problem Event Name: CLR20r3 Problem Signature 01: TradeProcessor.exe Problem Signature 02: 1.4.20.0 Problem Signature 03: 573a408c Problem Signature 04: mscorlib Problem Signature 05: 4.6.1055.0 Problem Signature 06: 563c0eac Problem Signature 07: 3fd2 Problem Signature 08: d Problem Signature 09: N3CTRYE2KN3C34SGL4ZQYRBFTE4M13NB OS Version: 6.3.9600.2.0.0.272.7 Locale ID: 1033 Additional Information 1: 5861 Additional Information 2: 5861822e1919d7c014bbb064c64908b2 Additional Information 3: 6ee2 Additional Information 4: 6ee2432e64e99bdcea5e4be767a3b1a2 Read our privacy statement online: http://go.microsoft.com/fwlink/?linkid=280262 If the online privacy statement is not available, please read our privacy statement offline: C:\Windows\system32\en-US\erofflps.txt | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
So, I'm expecting smooth sailing going forward on the t2.small. The one remaining problem is TP (pre-release 4D) does not handle delayed opening orders. It submits the order, gets the response from IB that cash is not available (because the close orders have not yet executed), but apparently never reissues the failed opening orders. Manual intervention is required to place the failed orders after cash has been freed by the close orders. | ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
| ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
| ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
9:30:45 has worked fine as a circumvention. | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
There are 4 times: Between xx:xx:xx and xx:xx:xx Delay opening orders xx:xx:xx Close Orphaned Trades at xx:xx:xx How do you have these configured? Is it: Between 9:30:45 AM and 4:00:00 PM Delay opening orders NOT SELECTED? Thanks. | ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
How does that work with submitting both close and opening orders together when the closing orders need to happen first? | ||
| BrianD Legend Posts: 302 Joined: 2/23/2013 Location: Grand Rapids, MI User Profile |
| ||
| SteveL Veteran Posts: 189 Joined: 10/11/2012 Location: Boulder, CO User Profile |
One more question: Is this an IRA account where the close and open are being submitted at the same time? I'm surprised it works. Maybe (?) the close orders are submitted first by TP, and those close orders have actually executed by the time the opening orders are submitted (perhaps a few milliseconds later?). |
|
|
|
|
Administrator
Forum Moderator |
Registered User
Unregistered User |
 E-Mail this thread to a friend |
Toggle e-mail notification |
