A Discussion on Significance, Math, Staticstics & Common Sense

Back in May I submitted a post which discussed significance testing. The basic idea is that if there appears to be a bullish or bearish bias based on a sample set of data then a significance test can help you determine the probability of the perceived bias being due to chance. This is very helpful when deciding whether to factor the results of a certain study into your decision making.

A high confidence level doesn’t mean the past history of bullish or bearish bias will continue at the same rate. It does mean that what occurred in the past was likely due to more than chance.

As an example, on June 12th I wrote a post on the extreme readings of the McClellan Oscillator as measured by Worden Bros. I showed a system that since 1986 (as far back as Worden keeps the data) would have been profitable 17 out of 17 times. The extreme reading led to a bounce the next day. The bounce soon petered out though without breadth improving to the point where the oscillator (as measured by Worden) returned above 0. At this point that system entry is almost a month old and it appears unlikely that is will close profitably. Does this mean the system doesn’t work? Nope. It probably will work well into the future. There is almost certainly an edge. What is certain is that the edge is not 100%.

When considering the results of a study, though, a lot more should be considered than just win %. The line I normally look at first in the results I publish is typically “Avg Trade”. Even if the indicator is only 50% (or less) accurate, do the times when it was right substantially outweigh those when it was wrong?

Depending on the time period you measure, the long-term upward drift of the market typically averages between 0.03% to 0.05% per day. This would equate to $30-$50 on my “$100,000 per trade" studies. If I find a study that averages $200/per day over a 5-10 day period, then that study is likely suggesting an edge.

It’s important to consider several of the other columns as well. Outliers play a large part in the evaluation process. If the system was wrong 10 times for an average loss of 0.5% and right once for a gain of 20%, looking at the “Average Trade” isn’t going to tell the whole story. The outlier has largely skewed the results. For reasons of space and aesthetics I don’t always show the “Max Gain” or “Max Loss” columns. If I do you can be sure I consider them important.

Profit factor is another interesting stat that I consider. It measures how much you need to endure in losses in order to make a certain amount of gains. The formula is Gross Gains / Gross Losses. In general, a system that makes $1000 from $1,200 in profits and $200 in losses is preferable to a system that makes $1000 from $20,000 in profits and $19,000 in losses. Therefore a higher profit factor is generally more desirable.

I normally give some considerations to the columns I show in the results. Readers who want to get the most from the studies should take the time to look at all of the columns.

Also, there are some studies I do that end up with too few instances to derive meaningful statistics from. This does not mean the study isn’t valuable or can’t teach us something. In fact, many times the lack of instances may in itself be a warning sign. If the market is behaving in a way that it either never or only a small number of times has acted in the past, I find that noteworthy. I want to know when the market is in uncharted territory. For me that may signify some extra caution is warranted.

In other cases, even with too few instances for meaningful statistics, the small number of instances found are noteworthy or compelling in some way. An example of this would be the May 22nd “Net New Highs” study. Three instances. All tops. (And a fourth close call that wasn’t.) Instance #4 has turned out pretty bad as well. The May 22nd study wasn’t one that I quantified and factored into the Aggregator, but that doesn’t mean it did provide useful information to consider when determining my market bias and approach.

As a last point, taking a mathematical approach to the market can help to provide a "quantifable edge". It will never be a perfect edge, though. While the math and the calculations may be perfect, the market isn’t. The market is heavily influenced by emotion. A common sense approach to the numbers is necessary. Otherwise, those times when the market acts in a way that is mathematically 99.999% unlikely, (but realistically more common), then trading accounts could suffer incredible drawdowns and damage. The studies may be my guide…but I’m still driving.

------------------------------------------------

For those who would like to see how I use the studies as well as some common sense to construct my market bias, send an email to weekly@quantifiableedges.com to receive a sample of the Quantifiable Edges Weekly Research Letter.