FXhackers

V-Entropy: hybrid Entropy Hurst exponent indicator

2013-06-12T06:55:00.004-07:00

This indicator is based on the article Tracing of stock market long term trend by information efﬁciency measures by Virgilijus Sakalauskas & Dalia Kriksciuniene

The entropy-based (EB) indicator is designed by aggregating SE and HE, and is deﬁned by the following expression (4):

EB= 1/2 - (1/2 - HE)*(1-SE)

where HE is the Hurst exponent value and SE is the Shannon’s entropy value.

In case of randomness of the time series, HE=0.5 or SE=1, then the indicator EB is equal to 0.5 and characterizes the time series as the white noise process.

In all the other cases EB indicator can take any value from [0,1] and characterizes persistent series by EB values greater than 0.5, and antipersistent series by values lower than 0.5, similarly to Hurst exponent.

The main difference of aggregated indicator EB from HE is achieved by adding inﬂuence of SE values. If S value approaches to 1, but H value is signiﬁcantly different from 0.5, the compound value of EB will be much nearer to 0.5 than H value. And vice versa, if H value is near 0.5, but S is signiﬁcantly different from 1, the EB value is stronger shifted from 0.5.

The indicator EB aggregating HE and SE as deﬁned by expression (4), can be considered as time series long range dependence measure and applied for deﬁning the revolving moment of time series trend direction.

We further analyze if this measure can be particularly useful for prediction.

The indicator is called V-Entropy because one of the authors of the article is called Virgilijus Sakalauskas.

DOWNLOAD for logged members.

How to trade the market states with discretionary trading?

2013-05-08T08:03:00.003-07:00

I was asked a question how to trade the market states from discretionary perspective. This turned out to be a very complex question. Every school of technical analysis (or price action) incorporates some kind of market state detection method. The basic market state classification comes from Curtis Faith. So maybe it would be a good start to read his book about the turtles method.

However I am not satisfied with that answer. I think that everything depends on the level of understanding of the trader. If an experienced trader is reading those materials he would not ask this question because he will figure out by himself if this is an usefull stuff for him or not.

Howevewer let consider we have been asked by a novice trader. I think that it would be nice if he gets some knowledge of the basic tenets of technical analysis and price action in the first place.

For this reason we made a special group about price action where we have collected some links. The group is called Price action and pattern trading with statistic edge. Here I recommend the law of the charts by Joe Ross. He has his own method of market states identification but you can compare it with our classification that is upgraded with fractal dimension and scales (read Anchor and Trigger).

Basically Joe Ross uses the concepts of ledges and consolidations as ranging patterns. Please read the law of the charts page 8.

What Joe Ross does is that he makes from theoretical point of book very simple connection between the market state and the logical entries. Have a look for his method of Trader's trick entry. After reading this you would have a better understanding after the identification of the market state where the entries are.

This method is quite different from the classical technical analysis. The classical technical analysis methods rely on a very specific patterns that are related with the changes of volatility (heteroscedasticity) and accumulation of orders. When the volatility slows down the market tend to draw the typical technical analysis figures. During this process order start to accumulate and they are triggered afterwards. During this process break - out levels are detected and exploited by market technicians. Every technician has a method of market state identification, the most basic one is to distinguish two states a trending state and range state. When the trending state is detected it is preferred to use moving averages, when the ranging is detected the oscillators are more appropriate tools because they do not lag. However from there many others questions and uncertainties arise that I can't summarize here because it is complex I want to mention that always different scales are examined by the analyst, have a look of the Triple screen method byAlexander Elders.

And finally, last but not least it was revealed the relationship between the market states and with the accumulation of open orders and market states. This is very visible in the EUR/USD. You can check this link. This was intended basically as an upgrade of the Price Volume analysis.

So this is what has been developped here and we hope it is a valid market knowledge you can use in your own research. For the beginners I ask them to start start with Joe Ross as the most simple introduction in price action. It is worth even if you deal with EA trading because EA trading and discretionary trading share a common aspect: the ability to appreciate if your trading method (manual or authomatic) is able to perform on the current market state.

The Real Significance of the Bitcoin Boom (and Bust)

2013-04-12T17:27:00.006-07:00

1 billion vanished in 7 hours! Brutal! Bitcoin market some days ago as within just hours went from 229 to 266, the next 7 hours suffered defeat, even unseen since the craze bubble in Amsterdam on the East India Company, tulips and stock market crashes in Paris and London by 1720.

From $ 226 to $ 105 and if the market capitalization is it reached $ 2 billion, then about $ 1 billion in heaven. By MTGox main market which traded currency announced that the decline is due to problems in handling the large number of transactions.

What is going on?

In this article they have an interesting view on the problem. Provided that some insiders are expecting 1 bitcoin to reach in the future $10 000, $100 000 or even $1 000 000 the current fluctuations are really insignificant.

But still this is a great story to follow. When guys I mentionned this on this forum, this may have been one of the best speculations in your life. Anyway from technical perspective this market is much more prone to analysis by the methods of technical analysis than the traditional markets and currency markets.

However on the other hand a bulgarian technical analyst states that the bitcoin is very risky and he gives some decisive arguments.

As in 2011 the question arises, what would do the merchants who accept payments in Bitcoin? If you make a deal with currency you have no idea if you would be in profit or loss only minutes after the deal.

For the customers pops the question - Buying a product, you may wonder whether you would acquire it ten times its value in other currencies, or extremely profitable.

In this case, it is difficult to hedge on both sides, especially in retail. Thus, the issue of Bitcoin currency touched again to trust.

So a new market arises and technical analyst Karadjov makes his technical estimates:

"Wrote on Apr 08 Bitcoin - where?!, I gave a level of $ 195. Exactly $ 195 turned, dropped to only $ 176, but it went to $ 266, before yesterday's debacle. It is difficult to give just any performance, but in my opinion the currency will drop to at least $ 41 if not much lower. Do not forget that it was also much lower, below $ 2, in a very recent past."

Robust Control Paradigm for Trading

2013-02-22T12:42:00.003-08:00

http://www.engr.wisc.edu/ece/faculty/barmish_ross/ifac_trading_barmish.pdf

Abstract:

The objective of this paper is describe a new paradigm for the trading of equities. In our formulation, the control corresponds to a feedback law which modulates the amount invested I(t) in stock over time. The controller also includes a saturation limit Imax corresponding to a limit on the value at risk. The admissible stock price evolution p(t) over time is modelled as a family P of uncertain inputs against which we seek robust returns. Motivated by the fact that back-testing of candidate trading strategies involves signiﬁcant cost and eﬀort associated with computational simulation over suﬃciently diverse markets, our paradigm involves the notion of synthetic prices and some idealizations involving the volatility of prices and trading liquidity. Our point of view is that a robust performance certiﬁcation in this somewhat idealized market setting serves as a ﬁlter to determine if a trading strategy is worthy of the considerable time and expense associated with full-scale back-testing. The paper also includes a description of a so-called saturation reset controller. This controller is used to illustrate how the model works in practice and the attainment of robustness objectives over various sub-classes of P.

Author: B. Ross Barmish

Here you can watch a video.

Weekly EUR/USD analysis: February 18- 22

2013-02-19T04:13:00.000-08:00

Recently it was a good practice to make the weekly analysis starting from what happened on the markets of Monday, in that way Monday being the leading day of the week. This week I am quite late. However as the market is still in consolidation between 1.33 and 1.34 from the beginning of the week it is worth to post my support and resistence levels.

Resistence:

Major levels: 1.34, 1.35, 1,36

Secondary levels: 1.345,1.355,1365

Support:

Major levels: 1.33, 1.32, 1.31

Secondary levels: 1.325, 1.315

According to dailyfx analitist David Rodriguez the level of 1.32 is a key level rearding the strenght of the trend of EUR/USD. This conforms our analysis from the last week when we mentioned that We have many clusters of support below the market at 1.32 and 1.325.

Technically we have a channel clearly visible on the daily frame. And we are on its lower limit. From technical perspective that would mean that the market is forced to make a choice.

Choice 1: Going back to the channel

Choice 2: Getting out of the linear regression channel

The question is what can be the catalyst of this choice. Based on recent developments this would be some fundamental news. Recently the market on EURUSD was event driven. So More predictable economic event risk can produce the volatility and they would be concentrated on Wednesday and Thursday when US CPI and PPI inflation figures are due and when the minutes will appear from the most recent US Federal Open Market Committee (FOMC) meeting.

Price action and indicators

2013-02-15T11:47:00.000-08:00

"The markets behave much like an opponent who is trying to teach you to trade poorly"

Eckhardt

The price bar is an indicator, it is wrong to separate the price bars from the indicators.

A price bar is sampling the market action in a time period and is indicating:Open,High,Low,Close.

This is an indicator per se. The Japanese candle sticks offer particularly good representations,on the other hand they are wider taking more place than the bar charts.

As Protagoras says "man is the measure of all things". And from human perspective it is important when using discretionary analysis to use the same settings, only in that way you can train your mind with the market patterns.

Price bars are robust because they do not allow you to change their settings inside your time horizon, and in that respect they allow human traders to train them-self to recognize patterns (according to their logical price action strategy).

Most newcomers when they use technical indicators they try to manually and randomly fine tune them on past market history and by doing this they do curve fitting and by this they do not train their mind to recognize patterns as they change the inputs all the time.

(System traders they also do back-testing but machine back testing is precise and they do precise walk forward optimizations trying to avoid data mining bias as much as they can, they use many other validating techniques to challenge their system).

On the other side it is a massive disease to see on public forums screens full of indicators, which settings are changing all the time.

And of course, the best technical analysis authors when they use indicators they do not change their settings. They use the same settings for years. For example the alligator of Bill Williams or the indicators of Dinapoli indicators set, or Elder, there are much examples. What they search for robust indicators which do not produce tremendously different results by small changes in their parameters.

The core idea is to train the mind the relationship between the market and the indicator values on the most deep unconscious level by following some basic framework rules. It is the mind that does the job (transforming the indicator input into actionable trading signals) not the indicators.

In fact this works like training a neural network and this takes a lot of time. It is difficult because humans have many psychological limitations. That is why the education is not only intellectual but also on the emotional level the so called emotional intelligence.

And that can be achieved by human by looking at the same indicators again and again. It is advisable not to have much more than 3 indicators at all.

The Forex Grid on EUR/USD

2013-02-14T13:41:00.004-08:00

I will post here a screenshot about the Forex grid on EUR/USD. The future support and resistance zones are in the order-books. In order to get the whole idea about support and resistance it is important to have a look in the order books not only at the charts.

The prices very often overshoot the true support and resistance levels and that adds a lot of confusion leading to large errors in support and resistance identification relying only on charts.

The charts are often very misleading with lots and lots of fake technical patterns. Very dangerous are the patterns which rely on projections of volatility like projections of contracting of volatility.

Some of the supports and resistances are related with purely statistical limitations of the current price action.

However the practice is difficult, as the price series are fractal even if you put support and residence in random places you will find matches, and a lot of matches.

The primary support and resistance levels are at round numbers at every 100 pips.

At every 50 pips are the secondary support and resistance levels.

At the first shot have a look at the Oanda order book.

At the second shot have a look at the EUR/USD chart with plotted horizontal lines at round numbers.

Weekly EUR/USD analysis: February 11 - 15

2013-02-10T13:49:00.002-08:00

Now I am going to be very brief because I really do not want you to think that is a service I will provide.

I think now that we really again we do need to see the daily candle of Monday in order to see what the market will be for the next week.

As for monday there is 50/50 chance that the market will go to 1.33 or to 1.34.

As the market was correcting last week it is hard to predict where the correction will end.I quite agree with the weekly analysis of forexempire that on EURUSD we are in only buy market.

The recent extreme and directional market movements are closely related with fundamental statements and data. One of the main reasons that we fell last week was related with some precise statements by Mario Draghi after the ECB conference 07.02.2012. He warned that the high value of the euro was a risk to growth in the Eurozone.

However as you can remember the euro began its recent sharp rally following positive remarks by ECB head Mario Draghi about the Eurozone economy.

How can you see this. A possible answer is there is some kind of macroeconomic agenda about how fast the Euro should rise and any sharper upside movement is seen as a threat.

For more information go to beathespread.com

Smooth Commodity Channel Index

2013-02-06T14:59:00.001-08:00

Here is a screen shot of a new Smooth CCI mod. I was looking again at the formula of the CCI, Check this mod too. The CCI smooth is a continuation of the previous release.

Here is the formula of CCI:

CCI = (Typical Price - 20-period SMA of TP) / (.015 x Mean Deviation)

In the previous mod I questionned the constant of 0.015. The CCI was created by Lambert. And he had the idea to use a multiplier 0.015, so as a result 70% to 80 % of the price action would remain within -1oo and +100. However we should focus on what his adea was, and you need to change this number to make fit 70% to 80 % of the price action within the levels -100 and 100.

I think you understand much better this indicator now.However this time we can go one step further and not use a difference between

Typical Price - 20-period SMA of TP but something else:

5-period EMA of TP - 20-period SMA of TP for example. In this way you can get much smoother CCI without adding any complexity to the indicator. What counts is that the major turns are without much lag.

And this indicator allows to build different derivative indicators and possibly to smooth them.

The indicator has three settings:

CCI period - this is the CCI period

CCI Period 1 - This is the period used for the moving average period (we use moving average instead of raw price)

level- the level is used to determine how far the indicator line will go above and below zero, basically this is a scaling parameter. The default is of course 0.015.

Download from here, only for registered users.

Weekly EUR/USD analysis: February 4 - 8

2013-02-03T13:36:00.004-08:00

EUR/USD had a strong upward momentum last week and made a substantial 1.3 % climb hitting a 14 month high. Despite the fact that the movement was a textbook break – out from the consolidation area between 1.325 and 1.34, the sharp rise took the majority of the human traders by surprise. The rise trapped many traders.

From a technical perspective the question is if this pair would show some exhaustion patterns? For this reason the first trading day of the week will be of paramount importance. After such a sharp rise the daily candle of 04.02.2013 will be very important.

The recent market fluctuations were closely related with the fundamental releases. In the weak ahead many fundamentally important events are scheduled to take place, that would lead potentially to a lot of volatility.

Trading outlook:

The trading outlook is difficult this week, the pair is in an uptrend having incredibly strong momentum and fundamentals. This situation is difficult for the majority of human players who are actually trapped in their short positions.

As the pair is strong not only from technical but also from fundamental perspective the best solution would be to exit at the first correction, in that respect a reversal doji after Monday would be great news.

As for the entry in the direction of the trend, I would say that this is difficult if you want to have close stops.

A break – out above the last week close at 1.37 would be a good break - out entry, and potentially a trade for at least 50 pips up to 1.375.

Main Weekly Resistence : 1.38

Main Weekly Support :1.34, 1.35

The Human herd and the Wisdom of the Crowd

2013-01-29T06:50:00.002-08:00

Thanks to some web platforms we can see now over the net the Retail Sentiment.

Why this would be important?

I think that this is important because according to data from 2011: Retail FX is big business. Average daily turnover across the industry from retail traders is approaching the $200bn mark, a sizeable chunk of a $4 trillion a day global market (source).

At a first glance $200bn mark compared to the $4 trillion a day global market does not seem big. However when you combine this information with several other facts:

The information that 98 % of the volume in FX is speculatvie and only 2 % is fundamentally driven.

To move the EURUSD by 0.2 percent, a trade of 200 Mio

USD needs to be made, which requires as little as 5 Mio USD in terms of equity (source).

That means that this retail flow when acts as a herd (and it acts so very often) is able to produce certain market movements and reactions too.

Right now I will give you some screenshot so you can see the momentary picture from retail human herd directional bias.

От Oanda open orders

От Oanda open orders

От Oanda open orders

As you can see from those shots they are samples from the population of the hypothetical human herd.

Yes that exists humans in fact are acting as a herd on the markets, they open positions as a herd and they move their stops at specific places.

That makes the whole technical analysis tools of support and resistence work. I believe that machine order flow can't produce support and resistence by itself in the same way the humans do (this is a pure hypothesis).

From the screen shots you can see that the humans are acting as a herd and roughly the same ratios appear at two independant retail platforms: Oanda and Dukaskopy. Those two ratios are confirmed by the eToro traders insight.

However there is something even more interesting.

The advent of social sharing makes this even more pronounced. In eToro there are leading tradiers (most influential traders) and they are even more bearish than the normal herd.

От Oanda open orders

Check this shot, there you can see the top traders insight.

The most advanced forms of technical analysis schools always have emphasized patterns between retail traiders order flow (uninformed traders) versus the institutional orderflow (informed traders).

You can see for example the VSA (Volume Spread Analysis) and the the articles of Sam Seiden. However they use the limited tools available from the chart (open, high, low close and volume), and the natural support and resistence zones.

Here in this blog post I am trying to show you a direct observation of this phenomenon and to show that the herd activity is real and can be observed directly. The samples from Oanda, and Dukaskopy are sufficiently representive.

In another blog post I may show you how this happens regarding to the price and how the human herd adjusts itself acting like a real herd.

From there it is obvious for me that what masses do matters, but profiting from the herd slaughter I think is way too simplistic. You can check this article. There has been succesful instruments of improving the dynamics of the group. "Matches between traders and recommendations were based on an innovative algorithm designed to optimize information flow within the network. Even this small number of coupons was enough to move the entire network away from dangerously high levels of “groupthink,."

Is there a wisdom of the Crowd? Can you profit from that information and how?

Chaos and complex representations for neuroevolution

2013-01-18T06:17:00.000-08:00

In this article on my blog I summarized my views on the proper way to obtain a robust solution through neroevolution.

Basically I question the whole architecture of the neuron when you are going to use genetic optimization or particle swarm optimization.

I suggest a different architecture, in this architecture we replace the transfer function by more complex mathematical representation and we evolve the weights inside. Doing so we are able to boost the power of the neural network.

The use of the logistic map as a very complex mathematical representation allow to add much more power to the neural net. By tuning the parameters (r) we can add a portion of chaos and by this improve the robustness.

This architecture is useful when no global solution exist and we face multiple local optima. In such case we want to find practical but robust solution.

What is the basic idea of this neuron architecture. The basic idea is that we are not looking for a global optimum at all. Why? Because we know that this global optimum does not exist.

I would refer to the Hitchiker's guide to universe:

"Narrator: There is a theory which states that if ever anyone discovers exactly what the Universe is for (I replace with the market) and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened."

So we know that we have multiple local optima, and no global optimum at all. In fact to find a local optimum is not a problem at all. What we are looking for is to find a stable solution. And that stable solution can be achieved only if you let the genetic optimizer evolve the weights into a complex mathematical space.

So let see what is all about.

The use of genetic algorithm for training SVM models is not something new. Let start from the standard artificial neuron model. However we are going to do something different. We are not going to map the inputs into a multidimensional space. We are going to put the agents into multidimensional space.

The first thing to do is to have a look a the basic structure of the artificial neuron model. I think the relevant Wikipedia article is a very good reference:

"For a given artificial neuron, let there be m + 1 inputs with signals x0 through xm and weights w0 through wm. Usually, the x0 input is assigned the value +1, which makes it a bias input withwk0 = bk. This leaves only m actual inputs to the neuron: from x1 to xm.

The output of kth neuron is:

Where (phi) is the transfer function.

The output is analogous to the axon of a biological neuron, and its value propagates to input of the next layer, through a synapse. It may also exit the system, possibly as part of an output vector.

It has no learning process as such. Its transfer function, weights are calculated and threshold value are predetermined."

So this is the basic neuron model and different learning methods can be applied.

A research paper was proposing a another possible structure making the traditional neuron model more complex."The basic idea is in representing the weights, that belong to interconnections in the neural network, by function. The learning process then doesn't happen directly on the level of weights like it is common in normal neural networks, but instead it happens by changing the function that is used to generate the weight every time when its value is needed."

The resulting neural network is called Hyper neural network. In the paper two approaches are used the representation of a function as a compressed matrix of coefficients used for Inverse Discrete Cosine transformation and representation of a function a a tree composed of elemental operations.

Another possibility is to make something different. We can vary the value of the weight using the formula of a transfer function and by this we may eliminate the use of a transfer function. We can vary and optimize the weight (using transfer function formula).

In the theoretically the biologically inspired neural networks, there the activation function is an abstraction it represents the rate of action potential firing in the cell. However we may not need this abstraction for some practical problems.

By doing this we diverge from the general neural model. The question is what we can gain from that? What are we going to do?

The simplest thing is to use simple transfer functions and vary them (here we will generate a population from the transfer function), then to multiply the result to the input and that is the output.

In the normal model in its simplest form you take the input you multiply by the weight then you input this result within the transfer function to get the response.

However here you can do something different. You can use complex representations and generate populations from them.

You can watch the video in this link to see a practical implementation of approximation of function by genetic algorithm using the Trading Software Metatrader 5. You can read more how the software works on this link.

Then the learning happens on the level of the transfer function. You may vary also the weights too but here we will not do so.

This is the Gaussian function. On the next picture you can see how the genetic algorithm approximate that.

From this the neuron model will look like this:

double perceptron()
{
double weight1 = MathExp(gamma*(-x1*x1-y1*y1));
double weight2 = MathExp(gamma*(-x2*x2-y2*y2));

double input1 = iCustom(TimeFrame,Symbol(),"PFE",x5,true,5,0,0);
double input2 = iCustom(TimeFrame,Symbol(),"PFE",x6,true,5,0,0);
double input3= iCustom(TimeFrame,Symbol(),"PFE",x7,true,5,0,0);
double input3= iCustom(TimeFrame,Symbol(),"PFE",x8,true,5,0,0);

return ( weight1* input1 + weight1* input2 + weight2*input3 + weight2* input4);
}

-We use the genetic algorithm to vary the values of the weights:

x1, x2,y1,y2

-You can modify at will the architecture

-This model is different from the classical neuron model. We do not have a transfer function. By doing this we eliminate the need of transfer function, you can add it but it is not necessary.

However we can use even more complex representations: We can use the logistic map as a complex space and vary the parameters of the logistic map.

That is the equation of the logistic map.

What is interesting for us is the parameter r. If we vary the parameter we will have different behaviour (sensitivity of the initial conditions). I keep it short here but we can get some very interesting stretching-and-folding structures.

This is 3d picture of the logistic map. Logistic map as a kernel function. Why not?

We have a very complex representations from a very simple formula and the properties of the logistic map are well known.

Here we use:

- RECURRENT DIFFERENCE EQUATION: the code is simple however the recurrence is not supported by OpenCL by now.

- TO USE SOME PORTION OF CHAOS: this can be obtained by varying the value of r approximately 3.57 is the onset of chaos, so we can limit from 1 to 3, if we want to add some chaose we can go from 1 to 4 . The use of noise in the neural net training is a common pragmatic approach to increase the robustness of the solution. The x value os from 0.001 to 1 with step size of 0.001 for example. We can vary the number of iterations (limitmap) to as a parameter. As a result we get something terribly complex.

The idea here is that there is not a global optimum and we do not want to find it. It is counter - intuitive.

A sample code of this using the mql4 language (the language is based on C) is like this:

double weight1()
{
double w, j;

w=(r*x*(1-x));
for(j=0;j<=limitmap;j++)
{
w=(r*w*(1-w));
}
return (w);
}

double weight2()
{
double z, i;

z=(r1*x1*(1-x1));
for(i=0;i<=limitmap1;i++)
{
z=(r1*z*(1-z));
}
return (z);
}

double perceptron()
{
//---
{

double p1 = iCustom(NULL,0,"PFE",x5,true,5,0,0);// those are inputs
double p2 = iCustom(NULL,0,"PFE",x6,true,5,0,0);
double p3 = iCustom(NULL,0,"PFE",x7,true,5,0,0);
double p4 = iCustom(NULL,0,"PFE",x8,true,5,0,0);
}

return(weight1()* p1 + weight1()* p2 + weight2()*p3 + weight2()* p4);
}

Why all those complex Artificial intelligence models fail predicting the markets?

2013-01-16T05:08:00.004-08:00

The point as I see it is that predicting the markets using any AI brute force (force method in the sense of relying on the abilities of the algorithm by itself) is a dicey business.

The markets are terribly complex they constantly are looking for balance and falling out of balance. The markets are so terribly complex that you may wander if the complex learning paradigms can beat the buy and hold on one hand and the simple linear regression on the other. Most of the time they can't pass the Monte Carlo simulations.

If that was not the case every AI specialist would find easily his way to riches. The problem is not in the AI methods they work fine on normal problems. The slight nuance is that you are asking them to do impossible things. As they can't let you without an answer they will give you an answer to the question as it is in the book Hitchiker guide to the galaxy. They asked a very complex computer what is the purpous of life and the answer was 42.

So the direct and brute force approach of successful and sound and proven AI do not work consistently enough.

So what can be done. From one hand those guys are hired by the HF quantshops where an edge is found in the speed of execution. Then the edge needs deep infrastructure understanding and market structure understanding that are completely different fields of knowledge. Only by combining this meta knowledge those quant shops succeed.

Other possible way is to look towards econometrics and deep market knowledge. Then you may be able to find a good definition of the problem and apply correctly the model. The economist need to define the problem the AI specialist has to apply correctly the model.

Those are two main models I know that lead to an edge. There may be other ways in order to beat the markets but they are fringe like entropy analysis, market states clustering and chaos theory etc.

So then why people do stay?

One possible answer is that beyond profitability the quest is very interesting per se, you learn new things, it is an excellent "game" for smart people. Eventually you find good on-line friends to work together on projects thinking that the grail is right next on the corner. Everything will be fine unless you allow yourself to make a gross loss.

Start small start smart. Take it easy.

Weekly EUR/USD analysis 14-18

2013-01-14T01:37:00.001-08:00

The EUR/USD is now at the same levels that it was in april 2011. That makes a 9 month high. The last week the Euro was up by 2 %.

After the European conference Mr. Mario Draghi made a statement that he sees signs of euro zone economic stabilization, after those statements the volatility rized sharply and the EUR/USD rised with more than 170 pips. There have been also some postive bond auctions in Spain and Italy.

A daily close above 1.3310 confirmed the current medium uptrend. From technical analysis perpsective we see clearly a bullish bias if the Euro is able to sustain momentum.

Even if we have a clear bullish bias we do not recommend deep stops at this moment much below 1.333 for short term traders. As everything depends on the momentum I waited the beginning of the London session in order to publish this weekly analysis.

Despite the fact that the technical analysis is the domain of what you see and not about what you think I advise extreme caution right now. I think that a short term correction is due from those levels and that the market can't sustain momentum easily. I see a possible correction towards 1.30. From that perspective for medium term traders I can recommend either a very short stop just below 1.320 betting for increase of momentum or a deeper stop below 1.30.

Main Weekly Resistence 1.35

Main Weekly Support 1.322, 1.30

Eureka: genetic programming software

2013-01-05T05:52:00.001-08:00

http://creativemachines.cornell.edu/eureqa

Eureqa (pronounced "eureka") is a software tool for detecting equations and hidden mathematical relationships in your data. Its goal is to identify the simplest mathematical formulas which could describe the underlying mechanisms that produced the data. Eureqa is free to download and use.

This software essentially does the same job as the commercial chaos hunter application however the commercial application also transfers the generated mathematical formulas to the trading plateforms.

Here you need to figure out how to do that on your own.

Anyway the simplest thing that can be done is to model the current trend for example.

Here I would suggest for reading two interesting articles:

What will cause the Singularity? : this is very interesting article it appears that Eureka is an oustanding genetic programming software: “That’s a pretty big deal in the world of evolutionary algorithms”

Product review of Chaos hunter from traders.com : this a basic article about the practical use of the genetic programming software in trading strategies: two methods the definition of a treshold or boolean strategy.

Let me summarize in a simple way the neural nets can do the same job as the genetic programming. However the Neural nets will store their solution as a data set of weights that has no meaning for a human. The genetic software forces that the solution is described by mathematical formulas that a human can understand. And that is why so much hype is there in the scientific community.

This is genetic programming, the algorithm evolve basic equations in order to achieve the best fit. As a result we have equations that we can understand and not weights.

In that way you can understand the results. For example if you do the same thing with a neural net you will get weights and if you run the model you will get predictions but you will never know the idea behind even if basically the prediction is the current close ;).

Weekly EUR/USD analysis 02.01.2013

2013-01-02T08:52:00.004-08:00

EUR/USD is currently in a tight consolidation area between 1.3155 and 1.3310. The 1.3155 level is the support, and the 1.33 level is the resistance.

We can expect for the next week a heavy fundamental flow to force an increase of volatility in the pair as the Fiscal cliff deal is heading to House after Senate vote.

A daily close above 1.3310 would confirm the current medium uptrend. The pair is very likely to test the short term resistence level at 1.33. Any downards movement is expected to be capped at the level 1.3.

A daily close below 1.3 would negate the current upward momentum in the EURUSD.

Long term outlook: flat market with complex consolidation patterns

Medium term outlook: unconfirmed up trend with support levels 1.30 and 1.27

Short term outlook: up trend: support: 1.31 resistence: 1.3355

Asirikuy’s First MT4 Genetic Programming Implementation

2013-01-02T06:15:00.002-08:00

The search come to me because of a link share in linkedin with a link of advanced trading software developpers arguing the use of genetic programming. This is the original link from linkedin.

The bookmark link here is interesting. It is about using genetic programming instead of genetic optimization in building trading strategies.

So what is it about? Let start start with Wikipedia:

"In artificial intelligence, genetic programming (GP) is an evolutionary algorithm-based methodology inspired by biological evolution to find computer programs that perform a user-defined task. It is a specialization of genetic algorithms (GA) where each individual is a computer program. It is a machine learning technique used to optimize a population of computer programs according to a fitness landscape determined by a program's ability to perform a given computational task."

So the question arise who did that in metatrader. So there are some answers.

There is a commercial software: the genetic builder software. This page on their site is very interesting explaining how this stuff works. The site is very informative and you have a clear idea what you are paying for.

And there is asirikuys' implementation. I hope I could see that but the service is available under a paid subscription.

http://mechanicalforex.com/2011/01/asirikuys-first-mt4-genetic-programming-implementation-get-ready-for-coatl-o.html

And this is the Asirikuy's site.

Frame doubling interpolation for 1080p x264 mkv's for superfluid 48 fps

2012-12-30T12:46:00.000-08:00

http://forums.guru3d.com/showthread.php?t=288017

Guys I have found something interesting. This will transform your watching experience thanks to the interpolation you can watch 48 fps instead of 24 frames per second.

"It is great for scenes where the camera pans over the scenery at a constant speed. Fast action scenes tend to be more clear as well. The whole movie gets a very realistic feel to it. It's just awesome. Actually IMAX HD is at 48 fps as well. Beware though, it is not easy to get this working. But once you got it, you never want to go back to 24!"

A friend of mine showed me, it is really amazing stuff.

This forum page is important showing how to change the script regarding the number of cores. If you had at least a quad core you should not have a problem, but even on my modest two core laptop with Intel 2.2 Ghz I get it running.

with: SetMTMode(1,2)

No really it is hard but if you get it running you will transform your film watching experience.

Optimization of trading strategy: is there something wrong Part 2

2012-12-28T04:51:00.000-08:00

Is there something wrong with optimizing trading strategies. I think the whole conception of optimizing may be a wrong paradigm. In fact during optimization we make in the same time the strategies very vulnerable to randomness.

That is the whole point. All trading strategies are very vulnerable to randomness but the problem is that y we do not know to what extent they are vulnerable.

That is why the use of any fancy digital filters or enigmatic time series preprocessing methods: SSA, wavelets, PCA does not add much.

On the other hand the use of different optimization algorithms does not add much too: back propagation, genetic algorithms, particle swarm optimization etc.

So what is the reaction of the big guys. They were investing in infrastructure in order to use the infrastructure edge together with complex data mining methods. And that worked and works. However as many others will do the same the competition will increase in the low latency market. Even more some countries are planning to tax the HFT, in France this is already a reality.Big boys are also using traditional macro strategies but this is another beer.

My point here is that a successful trading strategy needs to balance two pillars:

a. An adequate optimization regarding the market conditions. In practice that means that the strategy needs to learn from the market conditions and not to over - fit the past market conditions. This is a hard stuff.

b. The randomness robust trading strategy. There is a need to secure that the trading strategy is robust regarding the randomness of the market.

There are different ways to achieve it.

- To measure the state of predictability of the market and trade only when the market is sufficiently predictable. This is the core concept of the Spinal Implant EA. In this EA the algorithm takes a trade only if the fractal dimension is below a certain value.

-To apply walk forward testing.

This is a traditional approach and maybe is the state of the art in the trading community. But unfortunately quite a few algorithms pass this test, only to prove later that they are not so good.

-To apply Monte Carlo methods

So I really think that this should be implemented by everybody that seeks consistent results.

The Discussion will be here: only for logged users, sign up for free.

Kathy's trade and comments

2012-12-19T02:52:00.002-08:00

Here I will post a Kathy Lien video about the current situation on 19.12.2012.

She has a long position targeting the break-out (I was talking too about a break-out).

From a fundamental perspective she makes some comments about fiscal cliff and an optimism about Euro crisis and after all she has a long bias.

Anyway, this was the first thread in beathespread, and we somewhat forgot this thread. Now I am reading the posts most of them discretionary analysis. And hey I was right most of the time about big moves coming and Euro going to hell again.

Mostly I was surprised LOL.

Here I want to make a comment and a warning about the current situation.

What Katy does is classical analysis. The classical analysis has two parts.

a. Short Fundamental analysis

Fundamental analysis of the main driving forces and deriving a bias long or short based on the expert and discretionary knowledge.

Here in this case Katy does so and she has a long bias. From what Jaguar is talking there are others who have the same bias. But I really do not have an opinion on it because I am not a fundamental analysis specialist.

b. Technical analysis

She makes a technical analysis and we are clearly in a break-out situation that is so obvious. And last but not least the technical analysis is giving the risk control for placing the exit stops.

My comment is that that is the traditional way to do things. You can add also some more exotic ways of technical analysis as the Elliott waves for example. Here is a screen shot of an Elliott wave analysis with simplified counting (the simplified counting is used in order that everybody could get the reading and not the specialists alone).

So the Elliott waves picture is that we are in a global complex triangulat pattern in EUR/USD that is really hard to predict on the long term but do have some local predictability.

My additional comments

c. Elliott waves (simplified counting)

Here is the chart and of course regarding the position entry it mirrors the normal technical analysis regarding the break-out opportunities.

However in beathespread some more methods ans approaches have been proposed.

The method we propose is that we identify the market conditions locally.

As for the current situation we can just observe and measure the local market conditions. Thr trchnical analysis may be just smoke and mirrors as we jave observed so much sharp V-turns.

So what can be done is to use an EA model that is adpated locally to this market conditions. So if the break - out continues the EA model will follow it. If there is a sharp tern the EA would reverse just in time and follow the reversal.

I think this is what professionals do, I am simply explaining in plain English.

Have a nice day.

Optimization of trading strategy: is there something wrong

2012-12-18T12:30:00.000-08:00

What is a central question of all trading strategies is the optimization. However is there something wrong with the optimization or at least with our perception of optimization of trading strategies.

I think there is something elusive here.

I think that the main problem is that we look at the market as a problem.

Yes, I think so, we think that the market is a problem and if there is a problem we look for a solution. Applying mathematical models for searching a solution is just the right thing to do.

But what if, there is something wrong in our paradigm?

Of course the trading rule with the highest observed performance is likely to perform in the future provided we have sufficient observations and we do properly performance statistics.

However the problem is that the best rules are highly positively biased. Aronson writes in his book the Objective Technical analysis that technical analysis methods "have the potential to be valid knowledge but only if back-tested results are considered in light if randomness (sampling variability) and data-mining bias.

This screen shot is from the book of Aronson Evidence based technical analysis. Here you can see out-of-sample performance deterioration, that means that the rule which performed well in the past did not perform in out-of-sample trading.

Here in the first part is the in-sample data: the data used for data mining or simply put rule back-testing. In this period you see 50 % Return on on investment. However the out of sample is a different thing. The out-of-sample actually means that the data is not used for data mining and is the actual performance.

The reality is more complex as we may assume. There is not only a problem that needs a solution to be found but there is something else too. That explains why the data - mining normally fails in out of sample trading. I will stop here for the moment.

PAMR: Passive Aggressive Mean Reversion Strategy for Portfolio Selection

2012-12-12T01:32:00.001-08:00

PAMR: Passive Aggressive Mean Reversion Strategy for Portfolio Selection by Bin Li, Peilin Zhao, Steven C. H. Hoi, Vivekanand Gopalkrishnan

On this link you can download the Matlab Source code. Otherwise the article is here.

The Bin Li's doctoral dissertation is about "on-line portfolio selection", which intertwines machine learning and portfolio selection. The key issues in the topic is to find effective patterns among markets and then effectively allocate the capital for portfolio trading. He has mainly involved in two types of approaches.

1. He firstly uses nonparametric learning to extract a set, whose patterns are expected to be similar to future pattern. And then, he exploits the patterns via portfolio trading (using Kelly's approach).

2. After realizing that mean reversion is a common pattern among the markets, then he designs learning algorithms to trade on such pattern.

The backtest results of the two approaches on real market are real encouraging according to him.

For other techniques beyond his thesis, he thinks that the main usage of machine learning is to automatically identify (unknown/unknown) patterns among markets. Then feeding the related inputs is critical, such as neural network, or genetic algorithms.

You can contact Ben Li in linkedin.

How can we determine ahead of time when a certain EA would be suitable for upcoming market conditions?

2012-12-11T00:36:00.003-08:00

That a trading holy grail question. Much more than the one million question LOL.

What we can do is to know if it is working now.

Based on that you can evaluate how much draw-down is logical and calculated within the system. You can use a Monte Carlo simulation based on that knowledge. Check the post about Monte Carlo simulations and the appropriate risk control.

You can use market state analysis as an independent tool too. This a discretionary approach based on the best practices among discretionary traders and this is how far this pragmatic field can go by now (in my eyes of course).

And you can use your fundamental knowledge to have some insight. Sometimes the market state is closely related with fundamental information. Here I can give a historical example: the Swiss/Euro EA, this strategy was based on the Swiss Central bank decision setting a minimum exchange rate of 1.20. Until we were under the same circumstances the EA was expected to work and it worked too well for several months doubling the account many times.

And you can use data mining only for predicting market conditions. That is also possible, however it is not an easy task.

Predicting the future performance of an Expert adviser in Metatrader is not an easy task. Before doing this you need to know what is really the worth of the EA model.

Suggested books:

The Evaluation and Optimization of Trading Strategies (Wiley Trading) by Robert Pado
Trading Systems: A New Approach to System Development and Portfolio Optimisation by Emilio Tomasini and Urban Jaekle

Machine learning in trading strategy?

2012-12-10T03:09:00.001-08:00

Question: Does anyone here apply machine learning techinques (such as Neural networks, PCA, local sensitivity hashing) to his trading strategies?

There are two main commercial vendors of AI models for retail traders:
Neuroshell and TradingSolutions.

I think that they are offering a very easy tutorials allowing to grasp the use of those models in trading systems.

On both sites there is a lot of free information about basic models. In trading solution there is even a demo version which gives you several models and gives you the opportunity to make new models.

If I can summarize it in really rough way basically the basis of everything is:

* To select the predictive inputs (mainly technical indicators) and try to predict with them something else output.

* Use the output as a trading signal.

For example:

You have 2 inputs
You use a moving average of period of 20 as input.
You use a second moving average with period 10 as input.

You want to predict the next day close.

So the rule may be if the predicted next day close is higher than the next day open then buy.

If the predicted next day close is lower than the next day open then sell.

You train the model. And you backtest the model over a past unseen data.

So that is the basis and from there start the variations including their use in portfolio.

I have two remarks:

* The problem is that trading with neural networks is not an easy task. My opinion is applying a model over an already viable strategy or idea will allow to squeeze more profit from the idea compared to the rough model.

* Another problem I may say is luck related. Very often you can confuse the optimization with the luck. Very often the results you get are purely due to the luck. Monte Carlo simulations may help a lot.

Please check the book http://www.evidencebasedta.com/ and Monte Carlo
Permutation Evaluation of Trading Systems
http://www.evidencebasedta.com/MonteDoc12.15.06.pdf

I just want to add another remark that by using commercial software within the software is integrated a fool proof.

In Neuroshell for example they are offering to you some default settings that will limit the overfitting. In trading solutions it is the same. If you use your own model or an open source software and you do everything by yourself you need to be carefull what are you doing, for example adding too many weights to the model and too many neurons and hidden neurons will increase the risk to overfit the data and not to learn from the data.

Get real-time feed of live data into Excel

2012-12-09T08:52:00.002-08:00

I have found this list of sources of data for real time feeds of live data. I think it may be useful that is why I share the list here.

Get real-time feed of live data into Excel
Get free historical data into Excel, Matlab or Java
Foreign Exchange, Forex, FX

www.forexite.com (Forex, 1 minute ticks), CSV download
ratedata.gaincapital.com (Forex, tick data), CSV download
www.oanda.com (Forex, daily), CSV download
www.truefx.com (Forex tick data) after login
www.dukascopy.com (Forex, ticks), CSV
Debt/Interest Rates

historical time series
www.euribor-ebf.eu (Interest-Rates: “Euribor, Eonia”), Excel download
www.databank.rbs.com (Exchange Rate, Interest Rate, Consumer Price Inflation (CPI), Commodities: “Brent Spot, Henry Hub Spot, I P E Brent crude futures 1-pos, I P E Natural Gas Index, OPEC Fixing, US EIA Crude Imports, WTI Daily”), Excel download
www.ecb.int Interest-Rates and Government Bonds, web tables and partly Excel download
(German) www.boerse-duesseldorf.de (Bonds, daily), Website table only
(German )www.bundesbank.de German money and interest-rate statistics
current
(German) www.baadermarkets.de (Bonds)
(German) www.westlbmarkets.net (2012-07-12: Just went offline, hopefully online again soon), Swaps, Caps, Floors, CSV download
Stock / Indices

historical time series
finance.yahoo.com/(Stocks, Indices), Excel download
www.google.com/finance, Website table only
www.nasdaq.com (Nasdaq), Excel download
mba.tuck.dartmouth.edu (portfolio data), text download
(German) finanzen.handelsblatt.com
www.investopedia.com, Website table only
www.nasdaq.com, Website table only, intraday, daily
www.standardandpoors.com (S&P 500, daily, yearly)
current
finance.yahoo.com (Stocks, Indices)
www.bloomberg.com (Stocks, Indices)
Energy

EU http://www.epexspot.com/de/marktdaten/intraday-handel (German), Website table only
US http://www.gdfsuezenergyresources.com/index.php?id=33 CSV download
General Historical

(German) www.onvista.de (Stocks, Indices, Warrants, ETFs …, daily Quotes) Website
www.indexmundi.com World-wide econometric data and commodities
www.measuringworth.com US, UK, Japan, China with interest rates, GDP, inflation, gold and silver prices, cool long term, Website table only
Sites with Summaries

scraperwiki.com (The community collects free data from the web and converts it into CSV, etc.)
wikiposit (The author collects free data from the web and converts it into CSV, etc.)
Matlab Trendy (The community collects free data from the web and converts it into plots)
Economic data

World

www.indexmundi.com World-wide econometric data
OECD.StatExtracts includes data and metadata for OECD countries and selected non-member economies.
Google Data e.g. Market Capitalization in GDP: Google Data on Marcet Cap in % of GDP
European Union
Eurostat
www.ecb.int
www.destatis.de Statistisches Bundesamt
United Kingdom
www.statistics.gov.uk
United States
Federal Reserve Economic Data – FRED
www.census.gov
www.bls.gov
www.ssa.gov
www.treasury.gov
www.sec.gov
www.economagic.com (Web, Excel download after login)
www.standardandpoors.com/indices/sp-case-shiller-home-price-indices (Shiller-House Price Index)
www.data.gov Public data of all US government organizations
Live Cams

www.boerse-frankfurt.de (German Dax)
Buy Data:

(I did not try any of these, but they look interesting.)

www.cmegroup.com (derivative prices)
www.ivolatility.com (Implied volatilities)
commodities.euronext.if5.com (Cocoa Future, Robusta Coffee Future, White Sugar Future), 15 min delay, XML
www.eoddata.com (End-of-Day Stock), Excel download and more
www.historicaloptiondata.com (Option price data)
www.livevol.com (Implied volatilities)