Signal Bender

free & uncensored discussion arena for TheRumpledOne

Moderator: moderators

SignalBender
rank: 50+ posts
rank: 50+ posts
Posts: 108
Joined: Mon May 10, 2010 9:36 pm
Reputation: 0
Gender: None specified

Postby SignalBender » Mon May 24, 2010 11:20 pm

forexbob wrote:i am bad in quoting your answers signalbender, still it is appreciated. ( a little busy on a lot of different things the latest time. )

i fully agree with you, nicky from protrader is very supportive.
and i also have seen bugs disappear overnight.

and in the meantime some WHY link. http://www.why-yachts.com/
as TRO would say: why we do this? because we can.

looking forward to whatever is following in this thread.



ProTrader just stopped streaming quotes and stopped updating bars within the charts. I was working with an M1 chart one minute and the next minute it stops updated the bars. I check all charts including the tick chart and the data simply stopped streaming. Reported the problem in a bug report, but this is the kind of thing that won't allow me to make the transition to ProTrader (fully) at this time.

However, still think that within the next year or two, this will be a very solid platform that should garner a lot of support from Retail Traders. By then, I'll be well on my way into FXall.

As far as WHY is concerned - all I have to say is WOW! My Wife loves it. I wonder if it is technically feasible from a maritime engineering standpoint. Thanks for the WHY info, she looks awesome!

Please add www.kreslik.com to your ad blocker white list.
Thank you for your support.

SignalBender
rank: 50+ posts
rank: 50+ posts
Posts: 108
Joined: Mon May 10, 2010 9:36 pm
Reputation: 0
Gender: None specified

Postby SignalBender » Tue May 25, 2010 2:41 am

Well gang, some bad news.

Just got word that my idea of converting my system from Excel to MQL is simply not feasible. The reason is that the system in its entirety is massive relative to what MQL was designed to do. And, MQL's architecture is simply not programmatic enough to handle the logical relationships necessary to effectively reproduce the functionality necessary. I could give many examples, but here is just one:

The system uses both historical data and real-time data for its baseline calculations that generate Primary Indicators such as TCDs as one example. There are a couple dozen Primary Indicators alone at the core of the Engine. However, above that layer, resides the MetaIndicator layer and that's where the first problem hangs out. MetaIndicators are exactly what the name implies, Indicators of Indicators. So, MQL would need to have a facility for using the Output from Indicator_1 and using it as Input for Indicator_2, and that is just a small example of the MetaIndicator layering.

It is not uncommon in the system to either add 2 Indicator Output values together to derive a 3rd Indicator. Or, to then take the 3rd Indicator Output, add it with 3 other Indicators Output values to derive a 4th Indicator. Or, to then take the 3rd and 4th Indicator Output, run a mathematical calculation or logical statement against it to produce a 5th Indicator and then use that 5th Indicator Output as Input for a series of higher level Indicator Clusters. Nor, is it uncommon to see 10 Indicators producing the 11th, or 20 Indicators producing the 21st and so on.

From my understanding, MT4 starts to slow down when you significantly increase the number of Indicators running simultaneously and it is supposed to have an upper Indicator limit, beyond which you cannot load additional Indicators. In other words, MQL just can't handle the job of processing complex trading systems, fluently.

I've posted a power-point previously about how to integrate Excel into MT4 with no reply, so I won't post an idea I had on a possible multiple EA solution to the complexity problem in MT4, but I will say that the only concept for a solution that I can envision would be to run multiple EAs simultaneously across multiple machines and then pass output from each EA into a final EA designed solely for generating the Order (in essence the Trade Signal EA) on another machine.

So, basically, run an MT4 machine to just under its Indicator limit and classify that machine as EA_Indicator_1. Run the next MT4 machine to just under its Indicator limit and classify that machine as EA_Indicator_2, and so on, until all EA Output is passed to machine name EA_Order, where the trade is actually executed.

Otherwise, it is "Hello World" with a full-blown 32bit OOP Application, bypassing MQL altogether and going straight against the MT4 Server with the API, while building a real-time database with the lower-time interval data for the Excel Engine.

The problem with complex prototype systems, is that they are indeed complex and typically not built on a platform conducive to expansion or portability. Here is a more detailed explanation of the problem.

Let's say I wan to plot a straight linear line on an MT4 chart using this as the first data point, regressing back through the history of bars available:

Harmean(High[i]:High[i+1])+(15%*(High[i]-Low[i])/2)

Now, everyone I talk to says that MQL can easily handle this kind of work. But, this is just a line plot.

Here's the extrapolation that I came up with, including the data points that make up the linear plot:

Harmean(High[i]:High[i+1])
Harmean(High[i+1]:High[i+2])
Harmean(High[i+2]:High[i+3])
Harmean(High[i+3]:High[i+4])
Harmean(High[i+4]:High[i+5])
etc.
etc.
etc.

+

15%*(High[i]-Low[i])/2
15%*(High[i+1]-Low[i+1])/2
15%*(High[i+2]-Low[i+2])/2
15%*(High[i+3]-Low[i+3])/2
15%*(High[i+4]-Low[i+4])/2
etc.
etc.
etc.

Ok, you get the point. Easily done in MQL. But, again, it is just a line. A necessary line, but a just line nonetheless. Now, what happens when I need to place a dynamic price on the chart and then use that numeric value as either a Limit or Stop level. Here's what the output cell's dynamic price level code looks like inside the Excel Engine:

=IF(AE10>=O834,SUM(I828,AE10)/2,SUM(I828,G828)/2)

This is the ultimate problem with using MQL. Each one of the cells in the formula that make up this one dynamic price level, is itself a composite of other cells containing more calculations and to other data.

Note this dynamic price level uses 4 inputs (cells: AE10, O834, I828 and G828). Here are their individual break-outs:

AE10 = High[i+1]
O834 = SUM(K830:K834)/5-0.0005
I828 = G828+0.0025
G828 = E828+0.0025

Further extrapolation:

O834 uses input from the range K830 through K834. I828 and G828 both take input from one additional Indicator respectively with E828 being their dependency. Note that AE10 is the first in the sequence to actually use historical data from the market, the High of One Bar Ago.

Continuing the break-outs:

K830 = IF(CX75>=AR67,AR67,SUM(CX75,DH68)/2)

K831 = IF(D756="Short",SUM(F684,(((((DK76-0.0003)+(AE10+(FF30/10000)))/2)+(AF10+(AJ35*2.75)))/2))/2,(((((DK76-0.0003)+(AE10+(FF30/10000)))/2)+(AF10+(AJ35*2)))/2))

K832 = a value held in memory [I use macros to simulate the storing of values in memory]

K833 = E828+0.0025

K834 = G828+0.0025

I won't break down the details of K830 and K831 for time sake. [No circular references - my system has been debugged!]

And, this is just one lousy price level. We are not even talking about Indicator calculations yet! My goodness, to calculate this single dynamic price level in MQL would take how many lines of code?

There are 18 such dynamic limit level calculations in the system and 18 more entry level calculations, for a total of 36 dynamic variables that are all price level related (only). Here is just 1 of the 18 entry level calculations (they all contain different math and logic):

=IF(D550>0,D550,IF(AND(O539="Short",AR478="Long",AF499="Long",L574="Long",H529<>"Lead Short Off",AI736<>"Short Off",AI737<>"Short Off",AI738<>"Short Off",(AD9-J1151)>=0.002,R630="Short"),J1151,IF(AND(O539="Long",AR478="Short",AF499="Short",L574="Short",H529<>"Lead Long Off",AI736<>"Long Off",AI737<>"Long Off",AI738<>"Long Off",(J1151-AD9)>0.002,R630="Long"),J1151,IF(AND(Q782="Short",O539="Long",AG524="Long"),((AD9+0.0002)+(BP32-0.0015)),IF(AND(Q782="Long",O539="Short",AG524="Short"),((AD9-0.0002)-(BT32-0.0015)),IF(AND(L574="Short",O539="Long",V134="Long",AG524="Long",H529<>"Lead Long Off",AI736<>"Long Off",AI737<>"Long Off",AI738<>"Long Off"),(((AD9+0.0002)+(BP32*35%))+((AD9+0.0002)+((SUM(V135,O582)/2)*(U125*35%))))/2,IF(AND(L574="Long",O539="Short",V134="Short",AG524="Short",H529<>"Lead Short Off",AI736<>"Short Off",AI737<>"Short Off",AI738<>"Short Off"),(((AD9-0.0002)-(BT32*35%))+((AD9-0.0002)-((SUM(V135,O582)/2)*(W125*35%))))/2,0)))))))

I would be sitting here fore for two days straight just typing the break-outs of this cell. So, I won't bore you here. The Indicator logic gets far steeper than these dynamic price level formulas. In fact, I routinely hit the formula length limit in Excel where I end up spreading the entire formula across multiple cell regions, for a single Indicator.

User avatar
forexbob
rank: 150+ posts
rank: 150+ posts
Posts: 247
Joined: Mon Apr 20, 2009 9:52 am
Reputation: 2
Location: Planet Earth
Gender: None specified

Postby forexbob » Tue May 25, 2010 7:47 am

SignalBender wrote:Well gang, some bad news.


Don't tell that it here stops. you have only lost, when you stop trying!

So this needs more thinking outside the box.

stacking mt4 engines, looks not as a good idea to me.
ofcourse there is a reason to convert/move away from excel (limitations?)

The slowdown as mentioned in MT4, i have also seen in the protrader platform.

So maybe building it as a standalone app (gather a bunch of excellent programmers together. i am sure there are which in the past have build excellent apps under many limitations(hd space/memory/cpu-speed)) is an option pricefeed(s) in -->app-->trades out(towards MT4 platform / protrader / FXall -- FIX protocol)

prob not all indicators need visible output, and those who do,
do they all have 2b graphical or also just text based?
The highest Forex rebates: http://www.cashbackforex.info

SmokyByers
rank: <50 posts
rank: <50 posts
Posts: 16
Joined: Wed Sep 12, 2007 5:03 pm
Reputation: 0
Location: Scottsdale
Gender: Male

Postby SmokyByers » Tue May 25, 2010 3:37 pm

Bender,

It seems your best solution would be to connect Excel to MT4 and receive automatic updates at specified intervals into Excel.

Another solution, if your going to get away from MT4 altogether and go the .NET route, here's a nice component that works with Excel:

http://www.gemboxsoftware.com/GBSpreadsheet.htm

You could build a .NET front end and have your intensive calculations running on the Excel back-end. If you're going automated, use API or FIX to hook into your broker. Just a thought. I've played with the GemBox free version and it is a nice component. The free version had severe limitations which I think are overcome by using the paid license, but I decided to go strictly .NET before upgrading.

Good luck.

SignalBender
rank: 50+ posts
rank: 50+ posts
Posts: 108
Joined: Mon May 10, 2010 9:36 pm
Reputation: 0
Gender: None specified

Postby SignalBender » Tue May 25, 2010 6:03 pm

forexbob,

I appreciate the help! Losing is not in my vocabulary. Everything I've done thus far has been fairly beyond the limits of the average Forex trader using only intra-day and swing trading paradigms. So, I wanted to attempt to make the shift into true inter-day trading, to boost the equity curve even more. That's the reason for wanting to make the additions to the system overall.

Successful trading (IMO) is about Risk Mitigation as the first protocol. I designed my system that way and that's how I've been doing things for quite some time. At the lower time-intervals, I feel that I can get more bang for the buck at higher frequency and lower draw profiles over time - resulting in an a significant steepening of the equity curve. I've had to pull myself back from the decision to simply hire a developer to build the application for me from scratch - but then, somebody else would have 7 years worth of highly valued work, regardless of how many NDAs I strap them down with. I don't want the legal headaches.

Over the years, people have take what I shared with them and used their own programing skills to boost what they were doing in the lower time-intervals. Will they admit it? No, probably not - some want others to believe that the ideas that were shared with them, were their own. But, that's ok - because mine was only to help expand the consciousness of what was possible in others and to promote free thinking about what "price" meant in the market place. So, I appreciate that they are successful today, based in some small part on what I shared. (I can always see the signature of my work in the work of others)

Your conceptual solution has some similarities to what I was thinking, so I appreciate your input and encouragement. Thanks!


Smoky,

The Excel to MT4 route is basically a non-starter, for the reasons I wrote about earlier in this thread. Excel is "dumb" when it comes to persistent streaming connections. It won't know that MT4 has populated its missing bars of data, during the last or previous period of disconnection. Excel has no way to back-fill the missing bars after the disconnect/reconnect.

GemBox was an unknown quantity to me until you posted the link, so thanks. I'll look into it to see if it can help me generate the Excel Surrogate Database that connects both to MT4 and to the Signal Engine build in Excel. I've already posted a power-point on the .Net front-end idea, but it is nice to know that someone else came up with the same idea independently.

After doing some thinking about the problem last night, I'm going to open up the Engine and start to analyze potential areas of consolidation and code contraction without losing fidelity, to see if I can get at least an order of magnitude code reduction - it will take at least that much to even think about a full conversion to MQL. If I can restructure the way certain components are calculated or logically treated, then there might be a chance for full conversion.

In the meantime, I like what I'm seeing in the MQL based TCD Indicators in the 5 to 35 pip range on the M5, M15, M30 and H1 charts. Just the TCD Indicators alone are throwing off some pretty consistent trades when accounting for dynamic Omega Fills and H-Scale. So, my what I've been theorizing about for years in the lower time-intervals has been confirmed - that I could build an entire Trading System from just the lower TCD stuff alone, with some adjustment in the rules on how to use Omega, TCD Fills, LocBind and Omega (concepts you already know about).

So, I think it is worthy my time and effort to either learn MQL sufficiently, or find a good (trustworthy) MQL programmer to help with the cause under disclosure agreement.

Again, thanks for the GemBox lead and happy trading!

Please add www.kreslik.com to your ad blocker white list.
Thank you for your support.

SmokyByers
rank: <50 posts
rank: <50 posts
Posts: 16
Joined: Wed Sep 12, 2007 5:03 pm
Reputation: 0
Location: Scottsdale
Gender: Male

Postby SmokyByers » Tue May 25, 2010 7:29 pm

I think what you are trying to do can be done.

I have EA code that parses multi time frame MT4 historical prices to local .csv files, all the way down to m1, which can then be further parsed into Excel in whatever manner necessary (I tried to attach a file, but this site does not allow .csv as attachments). Pretty cool, I could go back to any given minute and view the state of the market at that moment if I wanted to, right now I only scroll hour by hour. It works good for 'manual' backtesting.

You could create something along these lines and use it to 'patch' the holes in your data in the event of a disconnect / reconnect. The problem I've found with this is Excel knowing (and I think you mentioned this) what data is missing, but this could be overcome with some creative VBA coding.

Good luck.

Smoke

SignalBender
rank: 50+ posts
rank: 50+ posts
Posts: 108
Joined: Mon May 10, 2010 9:36 pm
Reputation: 0
Gender: None specified

Postby SignalBender » Tue May 25, 2010 9:46 pm

No doubt, Smoky. I can get the data in the lower time-intervals now and even execute trades directly from Excel against the MT4 Server, using any of the standard lower time-frames. FxDialogue has a product that accomplishes this task. So, I would not have to re-write anything.

The problem with FxDialouge, is the same problem for anyone that attempts to use Excel with streaming data. When Excel gets cut-off from the source, it won't know two things: 1) Which bars of data are missing, and 2) How to retrieve the missing data from a DDE server and populate latent cells before continuing to populate the most recent cell. Excel will continually fall behind the data stream with each and every disconnect. When MT4 disconnects from the trading system, it is still connected to its own real database behind its own firewall. If that disconnect cuts off the MT4 back-end from the various liquidity partners of the Intermediary, it is still driven off of a real database that knows how to reconnect and query all external sources for the missing data since last the last connect.

However, Excel cannot have access to the database that MT4 uses, it cannot intelligently query a DDE Server, nor can it run stored procedures (SQL) against the database that MT4 uses (no network access and no db access privileges). If you take a closer look at the ppt I put together on this, it shows the creation and use of an Excel IRTD Server - different than using Excel as a DDE container only. The IRTD Server would be instantiated by Excel as a COM DLL. This is what I meant by "Excel Surrogate Database" much earlier in this thread.

Apparently, this would entail replacing the current flat-file (dumb) Excel database and then connecting the Signal Engine/Sheet to a (smarter) Excel RTD Server/Sheet via standard Excel linking. Or, use the current Excel Signal Engine to instantiate the IRTD object. In turn, the IRTD Server uses push/pull communication between the Excel Engine and the real-time data source, MT4. Then through some creative .Net programing can code a multi-threaded solution where certain threads handle the processing of the streaming data for placement and calculation within Excel, while another thread handles the actual updating of the streaming data.

But, even so, there is no intelligent dialogue between the IRTD Server and the streaming data source. The IRTD Server has no method for querying the data feed source for missing or latent data and the IRTD Server won't be able to recognize the sequencing of the latent data, even it it could recognize and detect a disconnect. The problem is one of mission critical design violations.

Sure, we can get the lower-time interval into Excel through parsers, right now. But, we'd still be lacking three (3) of the most important giants of good N-tier architecture: Redundancy, Fail-over, Load-Balancing, Back-up and Security. All real-time mission critical systems, have to have these elements, or you invite trouble down-range with poor data control. And, trading the markets with real cash, is as 'mission critical' as it gets, IMO.

I truly wish there was way to keep Excel in the game - but it is not looking good at all for the lower time-interval play.

Thanks for the brainstorming session.

SB

User avatar
tmanbone
rank: 500+ posts
rank: 500+ posts
Posts: 723
Joined: Sun Nov 15, 2009 3:52 am
Reputation: 0
Gender: Male

Postby tmanbone » Wed May 26, 2010 3:05 am

"The simplicity of the markets is it's greatest disguise"

T

SignalBender
rank: 50+ posts
rank: 50+ posts
Posts: 108
Joined: Mon May 10, 2010 9:36 pm
Reputation: 0
Gender: None specified

Postby SignalBender » Wed May 26, 2010 3:27 am

contact...20 miles level...500 kts...hdg 180...

User avatar
forexbob
rank: 150+ posts
rank: 150+ posts
Posts: 247
Joined: Mon Apr 20, 2009 9:52 am
Reputation: 2
Location: Planet Earth
Gender: None specified

Postby forexbob » Wed May 26, 2010 11:24 am

SignalBender wrote:contact...20 miles level...500 kts...hdg 180...


?? flying ??

will reread thread and have a look at your ppt.

personally i would probably do a total custom build (but alas i am a programmer (since the era there was not even a pc) )

but from my simple :
datasignal--->app---->orders
best is to first break it up in how it should/could (can not is not an option)
work and after that see if std modules (excel,..) fits in

i would prob then build it around a DB (mysql,oracle, ..) and yes i admit
excel can be a handy part .
interfacing between excel and DB can for example via excel-db
http://www.excel-db.net/index.htm , but there are prob other tools that fit.

the tradesignals from excel could then go directly, or back to a DB.
and another module picks up the tradesignals from the DB
(maybe even interface between DB and metatrader) (the nicest solution is not always the best (time constraints) )

when i would not program it myself, i would break it up in managable
chunks(which do not have the complete picture) and outsource it to diff
developers. or let the whole build as a proof of concept without meaningfull 'indicators' within the excel engine.

or get some excellent programmers do a NDA, and get better program and support because they will also use the result. (can imagine this is not an option)

2b continued,

Bob
The highest Forex rebates: http://www.cashbackforex.info

Please add www.kreslik.com to your ad blocker white list.
Thank you for your support.


Return to “TheRumpledOne”