Raccoon

[neil9327]

A couple of years ago I spent the summer writing an automated trading program in the C# programming language, and managed to get it running successfully sending limit orders to the Interactive Brokers API, running a simple mean-revert strategy. While it worked well technically, it didn't make money, instead breaking even. It put on around 200 round-turn trades on $5000 lots on 10 US mid-cap stocks = $1 million of volume - and made the princely sum of $92 profit - which turned into a $300 loss after IB's good value for money trading fees.
Still I don't regret it, and it taught me how to code in C#, which was an interesting learning curve.

I haven't done anything else with it until the past month, when a contract I was working on came to an end, and I once again have some spare time to try some things.
So I thought I would post up a thread here to document some of the ideas I am considering, with a view to possibly developing the program, which I have called "Raccoon", into something more sophisticated, and possibly profitable.

One thing that is clear is that trading the markets profitably is actually quite difficult, and that I'll have to engage in some lateral thinking and creativity if I am going to stand any chance of putting into effect anything worth doing.

 

[neil9327]

Supply vs demand, and what causes a trend.

It is often said that prices move in accordance with supply and demand. That when demand is greater than supply, prices rise, and when demand is less than supply, prices fall. Indeed this appears to be the thinking behind what constitutes a "trend", being, in the case of an up-trend, a sustained period of greater demand than supply, leading to a steady rise in prices.

But I don't think this is quite right, so have an alternative explanation that is similar, but not the same.

The issue with the traditional explanation, in my opinion, is that it explains an up-trend in terms of a sustained period of greater demand than supply. But what is demand? (And what is supply?) In my opinion it is only real demand when a trader places a market order, or when a trader has a limit order in place at the top of the order book about to be executed (in a liquid market). Anything else is for show. Indeed many market participants place limit orders away from the spread and then immediately withdraw them, in order to try to manipulate the market. These don't represent real demand, unless they are placed AT the top.

Therefore at all times in the market, demand must EXACTLY match supply.

So a better way to look at supply and demand, is to say that price, and price changes, are determined by what the price has to be, and has to move to, in order to ensure a continuation of the exact matching of demand, to supply.


Weaver:
I have written some code, which I have named "Weaver" on Microsoft SQL Server to try to simulate this activity.

(SQL Server is a database system. I am a SQL Server developer contractor in the "day job", so understand this technology very well).

It works as follows, with some back-test data I dragged of the API a few months ago:

1. Starting at, say, 2017-07-28 15:00:00.000 UTC. Get from the last close the last price for a single security, say US stock: MPC
Say it was $50.00
2. Advance 1 minute, to 2017-07-28 15:01:00.000
3. Run a stored procedure "pGetActor" that for a range of different types of trader active in the market, what would they do at that point if $50 was still the price. Some would do nothing, some would buy, some would sell. The stored procedure would run with parameter iActorId denoting what type of trader it is supposed to be emulating. The stored procedure would return the positive (if a buy) or a negative (if a sell) volume that would be generated from this trader type in aggregate. Each volume returned is a "actorvolume".
4. The program adds up these actorvolumes generated by the calls of pGetActor, to give the "fVolumeNet" value, the net volume.
If the total is 0 or very close to 0, fix the current price as being the price for this time, because the volumes all match. Advance the clock to the next minute, and go back to 3. for the next minute.
5. If fVolumeNetis greater than zero (or less than zero), then there is some net volume, and according to my theory this means that the price cannot be the current price. So increase (or decrease) the price a single tick, and try again by 3. above with the new price.
With the new price, the actor volumes returned by the calls of pGetActor would also have changed, probably, and therefore adding them up to get fVolumeNet will get a different (probably lower magnitude) value for fVolumeNet.
6. Keep going round the loop, changing the price each time, until the fVolumeNet falls to near zero, or crosses over from positive to negative. This is the price at which volumes are matched, or as good as matched, and according to my theory this is therefore the price that has to result.
Save the price into column cfPriceConclusion and move on to the next minute, and go to 3. above to continue.
Also add up the actorvolumes, but by setting the negative ones of them to positive. And divide by 2. This is VolumeGross, and is the volume that results from the trades that occur at the time from all of the actors.


I've got this bit of code working, with two "Actors", the first "Adam" that always buys at volume 5 regardless of the price, and the second "Brian" who likes to sell at volume: ((@fNumberOfMinutes + 200) * @Fprice * 0.000301)

(where @fNumberOfMinutes is the number of minutes after the start time).

I ran it for a test period from 2017-07-28 15:00:00.000 to 2017-07-28 17:50:00.000 = 2 hours 50 minutes, and its performance was excellent, completing in under 1 second - showing how good SQL Server is as a technology (it ran pGetActor around 8000 times in that period).
And it returned the result of a price that started high (118.650000), and steadily reduced to a final figure of 53.590000
This is correct because "Brian" earlier on needed a higher price to induce him to trade at sufficiently high enough volume to match that from "Adam". But as time went on Brian become a more active seller, and would do so at progressively lower prices.


So my question to myself is, is there any way this code could be developed such that the different actors had specific trading strategies (e.g. day trader tech analysis, day trader high freq, day trader arbitrager, long term, day trade mean-revert etc) with each strategy coded into pGetActor, with parameters to reflect position sizes, strength of signals required before action etc? Then on a period of back-test data I could run the script many times, each time with different parameters, and find the one that results in price and VolumeGross closely matching the real-world time series data. That would allow me to conclude the number and strength of these market participants in the real world. And with this information I could potentially try to predict future market activity on the assumption that these parameters will be the same in the future.


My initial answer is that it is theoretically possible, but it may be possible to model standardised (parameterisable) behaviour of market participants (actors) only for a small percentage of participants, such as beginners who are trading using technical analysis they have read from a book, and even then their actions are likely to be affected by emotions, departing from the book's instructions.
The rest of the actors are likely to be more sophisticated traders using proprietary models that are not in any text books. And of course many will be using artificial intelligence-led approaches that are also not in a book.

Still it would be a big win if I was able to get it to work, particularly in regard to the closing of positions by day traders at the end of the day; if I was able to deduce that, say, a larger number of day traders than usual had long open positions 5 minutes before the close, then I can be sure that there will be a lot of sell orders coming through in that next 5 minutes.

In all but the simplest cases, to get it right is going to be horrendously complicated, and use a great deal of computing power. Perhaps I could utilize some of the cheap graphics cards being discarded by the cyber currency miners post price collapse, to do some of the heavy processing required.
I expect there must be a simpler more elegant way to do the same thing, or nearly the same thing.

 

[neil9327]

pGetActor, above, returns the actorvolume resulting from the actor at a particular price. What it does not return, but could be coded to return, is the first order derivative of this, namely the change in actorvolume that would occur given a unit change in price. I need a noun to describe this, so I'll invent one - "Jinder" (I expect there is an actual name for it).

I am defining "Jinder" to be positive for trend-following types of actor, and negative for mean-reversion types of actor.
1. A trend-following actor who is buying will (as a generalisation) tend to increase his positive actorvolume as prices rise (confirming the trend).
2. And if he is selling, he will increase the magnitude of his (negative) volume as prices go down (confirming the down-trend).
In both cases Jinder is positive.

Conversely a mean-reversion actor trades the other way round:
3. A mean-reversion type of actor who is buying, will tend to increase his positive actorvolume as prices FALL (suggesting that the correction back upwards will be greater)
4. And if he is selling he will increase the magnitude of his (negative) volume as prices go UP (as he expects the prices even more likely to fall back down).
In both cases Jinder is negative.

So arranged a little neater, but in the same order as above:

1. --If volume is positive, and volume would rise if price rises
--, this tends to oppose a mean revert, so Jinder is positive
2. --If volume is negative, and volume would increase in the negative direction rise if price declines
--, this tends to oppose a mean revert, so Jinder is positive

3. --If volume is positive, and volume would rise if price declines
--, this tends to support a mean revert, so Jinder is negative
4. --If volume is negative, and volume would become more negative if price rises
--, this tends to support a mean revert, so Jinder is negative

Jinders from different actors should be added together to generate a "JinderNet" output (in a way analogous to VolumeNet). Or do I have to add their reciprocals? Hmmm too late at night to start thinking about that one. Also have I got their signs the wrong way round for the purpose of adding? More thought required.

A low "JinderNet", at a particular price, represents an area where volume doesn't change much when price changes - and that therefore prices have to move a long way to get the volumes to balance.
Conversely a high "JinderNet" means volumes change sharply as prices change. So prices only have to move a short way to get volume balance.

So:
--Localised areas of high JinderNet magnitude represent regions of support and resistance.
--Localised areas of low JinderNet magnitude are areas the price will move across quickly = higher volatility.

 

[NVP]

excellent work Neil...man i have big respect for people who can programme as well as you obviously can.......looking forward to more of your posts....if you ever feel like drifting to the "dark" side as a forex trader look me up......

 

[neil9327]

excellent work Neil...man i have big respect for people who can programme as well as you obviously can.......looking forward to more of your posts....if you ever feel like drifting to the "dark" side as a forex trader look me up......

Thanks NVP. Forex is a possibility, and is a good trading environment due to its high volume and liquidity. But it has a lot of different types of trader involved, so is perhaps harder to model in the way I am hoping to do with regular stocks.

 

[NVP]

good luck in your journey ........I hope you find what you are looking for.....

 

[neil9327]

Price, is not a thing
********************

The prevailing view of how markets work, certainly from my perspective as at few years ago, is that the "price" of a stock is a real thing in the markets.
A "real thing" might be expected to behave like an object such as a golf ball, and have characteristics like a ball such as having mass, and momentum. So if there is a surge in the number of buyers, this "pushes" on the ball, starting it moving upwards (increasing price), and if this surge stops, the ball will carry on moving upwards, but gradually slow down and stop.
And the same in the reverse direction with the idea of selling pressure.

But going on what I've written earlier, it is clear that price isn't a "thing" at all. It is an "effect", in the same way as a rainbow appears to rise out of the ground at a point some distance away from the viewer, that place it comes from is of no significance other than it happens to be the location where the angle between it, you, and the sun (in the presence of some rain) is the correct angle to produce the coloured light effect.

In the same way, price is what results in a liquid market where that price happens to be that that induces the aggregate total of the buying actions of traders interested in buying the stock, at that exact point in time, to exactly equal the aggregate total of the selling actions of those (different) set of number traders who are interested in selling it.

So price has no momentum, because there is nothing "there" to keep moving, and it has nothing for traders' activity to directly "push" on. There is, metaphorically, no "handle" for traders to pick it up with and carry in any direction, or keep it in one place.


Trends
******
In that case, why do we see "trends" in the market, and why do traders see, in the action of prices, the appearance of "momentum"?
Why do they see that prices typically rise if there is an increase in buying?

The answer to this is that price changes resulting from an increase in buying by buyers, is not defined by this increase in buying. Instead the price action is defined as a result of the behaviour of the corresponding sellers.

In the case of a normal rising "trend" alleged to be caused by rising buy-side volume, what happens is that the extra buyers come in and start buying with greater volume (we will say a sustained volume over a 5 minute period). This means that in order for the volume to balance, the sell-side volume must also rise.
For this to happen, a certain fraction of the prospective sellers, (traders who are considering selling, or going short), must be persuaded at that moment that that higher price is sufficient inducement to make them sell. It must be the case that had the price not risen, they would not have sold (otherwise the "trend" would have not been "caused" by the increased buy-side volume).
So over the course of the 5 minutes the price gradually rises, leading to steadily more sell-side traders deciding to commit to selling, who would not have sold if the price had been a little bit lower.

Eventually the trend comes to an end, and this happens (in the above example) either when the buyers stop their increased buying, and/or the number of sellers who are actively looking to sell right stays high even without a steady increase in price, because sell-side traders think the stock is now overvalued.


Gapping
*******
But a "trend" is not the only thing that can happen when there is sustained increased buying volume. Instead the price can suddenly move upwards, in a fraction of a second. This happens when, for whatever reason, all prospective sell-side traders are interested in selling, but a small price change alone provides insufficient inducement to them to sell.
in this situation the price will move up, or "gap" upwards to a price that IS high enough to induce traders to sell in increasing volume.


So to understand how prices will change as a result of buy-side volume changes, we need to understand what is the "Jinder" (a word that I made up - see an earlier post in this thread) of the sell-side. ("Jinder" is the price change that results from a unit change in volume, and is calculated from knowledge of the aggregate trading strategies of the traders). If the Jinder is low, then you'll get a trending market, and if it is high, the market will be more choppy, with more movements created by "gapping".


Market Makers, and the illusion of stability
********************************************
The presence of the Order Book, with limit orders (normally submitted by market makers) on both sides of the continuous double-auction that constitutes the market in most liquid stocks, gives the impression that prices will tend to stay at one level. The impression given is that in order for prices to move to above the level of these (often large) limit orders, traders must have to submit market orders to a volume level greater than the size of these limit orders.

However this is false.

In practice even a small market order has the ability to move the market, because market makers can and will withdraw their orders, or move them to a higher price (on both sides of the order book) even if the size of the submitted market order is much smaller than the size of that market maker's existing limit orders.

The reason for this is that market makers should in fact been seen simply as "other traders" - they are not there for the good of the market or the good of other traders, by being prepared to offer a nice pair of buy and sell prices.
Prices don't exist - they result from the activity of market participants.
Market makers trading strategies are to induce traders who execute market orders, to buy and sell from them from parts of those limit orders that they have in place on the order book, so that in a near-zero trading cost environment (for them) they can profit from the spread.

While they will tolerate a small amount of mismatch between the buy vs sell volume that their limit orders attract, they don't want an on-going mismatch between their buy and sell positions to persist, because that adds financial risk to their trading, with no corresponding additional profits.
They have a constant conflict between wanting their limit orders to offer good prices to traders so they can create half of a profitable spread position, but at the same time not exposing themselves to the risk of having a large uneven position occur, with the risk this brings from future price changes in the stock.
So market makers' limit orders are a bit like the pot of gold at the end of the rainbow, in that if you start to move towards them, they move away from you.


What if there were no short term traders
****************************************
A stock is a share in the financial performance of a company, giving the holder the right to receive a fraction of the company's profits as dividends, and a cash sum if the company is wound up (solvent) or bought out by another.
Let us consider how the stock market would work if there were no short-term traders present. i.e. the only market participants were long-term investors, those buying, and those selling.

In the absence of traders seeking to achieve short-term "alpha" (outperformance), the only trading in a company's shares will be from and between those who are looking to make a long term investment in a company - those who are looking to buy the shares, and those who, having made their money, are selling them to enjoy the fruits of their long-term investing.
But even in large companies with a lot of different shareholders, people tend to buy and sell in large chunks, rather than continuously.

But from what I have described earlier, buying volume must match selling volume at all times.

As a result of this uneven-ness of desire for buying and selling, resulting as it does from perfectly legitimate new investment and new disinvestment in the stocks (i.e. unrelated to changes in the value of the underlying company), the result is that the price will tend to jump around a lot over the course of the day, as the price has to move sharply to induce other long-term investors who are thinking of selling "today sometime", to sell "today right now".

This kind of volatility is not generally good for the economy because it reduces investor's confidence that they will always be able to get a fair price for their stocks. It also makes it harder, with the prices going all over the place, for the market to work out what a company is worth - so it is harder for a company to, for example, sell newly created stock in the company (a "rights issue") at the correct price. Pricing mis-matches often result, causing the capitalist system to work less efficiently (the Mises school of economics says that free market capitalism is the best system out there, because the clear pricing that it generates means assets are always valued correctly, and in the level of their usefullness to society).

To resolve this problem of volatility, hundreds of years ago, people realised that those market participants who were induced by the lower pricing so as to become the counterparty to the large sale, had the potential to make a short term profit if they subsequently sold their position, gradually, a short while later.
So the "trader" was born, someone who was deliberately prepared to act as a counterparty to those who found they had to sell to someone at a disadvantageous price, and then sell on the item at a later time, hopefully for a profit.


So what was the volatility that this new "trader" was looking at?
There are two sources of it:

1. As mentioned above, where a large seller who wants to sell right now, causes a temporary drop in prices as he does so (the price drop being necessary to induce other long-term investors to buy "right now", as opposed to "some time today".
2. Changes to the underlying value of the company, as evidenced by company announcements, news events, macro global economic events, and the actions and fortunes of competing companies.
In liquid companies there was always a small level of volume at all times from long term investors, and in the absence of 1. above prices would move instantly to the new price being the aggregate market view of long term investors as to the value of the company.
Note I say they will move "instantly". This wasn't really the case back then, due to slow information flow. But if the trading environment was as it was today, with near-instant news flows, the price would certainly move instantly EVEN in the ABSENCE of short-term traders.
(yes, the aggregate activity of a large number of small long-term investors alone, when present in sufficient numbers, will reach a volume equilibrium where buy volume = sell volume at a price reflecting the aggregate opinions of the long-term value of that company).


Technical Analysis
******************
Carrying on from the above, some of the early trading pioneers, such as Richard Wyckoff, who had the luxury of observing the markets when there were far fewer short term traders than there are in today's markets, noticed that the price and volume in markets tended to follow sets of patterns. And that if you could understand these patterns you could operate as a profitable short-term trader. The patterns resulted from the dynamics of the volatility in 1. and 2. above.
These pattens were published - they are "technical analysis".

At first the presence of these new brand of "technical analysts" served a useful function within the financial system because they allowed large long-term traders could buy/sell large amounts of their stocks without having to have the price go low enough to attract other long-term traders to transact "right now" as their counterparty. This was of benefit to the economy as a whole because it tended to reduce the price volatility caused by 1. above, thereby allowing prices to sit more stablely around the "correct" price (that being deduced by fundamental factors in 2. above).
As more traders came into the markets, using technical analysis profitably, more traders were available at any time to prepare to be the counterparty to larger long-term transactions, such that there was competition between them to offer the "best" prices to the long-term market partipitants, the result being that the profits they made reduced, and the corresponding market pricing inefficiencies also reduced.

Fast forward 100 years and into the modern era. The internet.
What is happening nowadays is that there are a lot of short term traders in the market who are not approaching it from the perspective of offering a "service" to long-term traders looking to get out or in. Instead they are looking to trade the markets purely for its own sake.
A great many of these use, or try to use, technical analysis. On their own, in the absence of other short term traders, this technical analysis would make them a profit, and at the same time they would be providing a useful service to society (well the economy) unwittingly, by providing the liquidity and price stabilization who's benefits I have listed above.
But what has happened is that there are more traders using technical analysis now than there are long-term market participants out there who need this liquidity service.
The result of this is that the technical indicators don't work any more because the market is so distorted from what it would have been had there only been long-term traders involved.
The new traders are the ones who lose out, when their methods of technical analysis cause them to lose money, often at a faster rate than if they had chosen trades at random using a coin flip.

Does this mean that technical analysis is dead?
No.
In an environment where, for reasons unknown, there may be few technical analysis traders operating, then it will start to work again, because the dynamics will once again revert to that of 100 years ago with only long-term market participants.

So what I am hoping to do is to see whether price and volume action analysis is able to give me any insights into how many technical-analysis using traders are present in the market place at any one time? And to confine my own trading activities to only those periods of time and markets where they are not operating as much.

Or in other words I should look to only try trading with technical analysis where that provides a real economic benefit to others.

 

[neil9327]

Well I am making good progress on the "Weaver" project described earlier.
Here is a web page showing its output for a day's worth of 1-minute historical data for stock: NVDA for last Thursday.
The blue price line and volume information is the actual historical data for this stock, and the brown price line and volume is the output from the algorithm.

To produce this, I have configured two "Actors", which represent the actions of market participants:

Actor1: This trader takes a long-term view on the prospects of this stock, and thinks its fair market price is $150.
This trader buys if the price drops below $150 (adds positive volume), and sells (adds negative volume) if it rises above $150. With the size of his volume proportional to the difference in this price.
At the top of the page I can add parameters P1_1 which let me increase the fair-value market price above $150, and reduce it below $150.
Then I have parameter P1_2 which controls the multiple of the volume increase with price.

Actor2: This is a day trader who is operating a trend-following strategy. With it he looks at the average price over the past 6 minutes, and compares this with the actual price over the past 3 minutes. This will show whether there is a trend upwards or downwards.
Then he extends that trend line to the current time to give him a @fPriceNowIfThisTrendContinued value, and if the current price is above this, and he can see a rising trend, then this confirms the trend is continuing - so he buys with positive volume.
Conversely if the price is lower than this @fPriceNowIfThisTrendContinued value, this suggests the trend is going the other way, so he sells (negative volume) with volume size proportional to the difference.
(OK this is probably not a fair representation of his behaviour in this scenario - he would probably just not trade. But anyway.)
And again there is parameter P2_2 where you can amplify the effect of this strategy.

So I have run Weaver initially with default parameters. And it produced a graph that was a long way out.

Then I started adjusting the values in the three parameters, in order to get a closer match.
In this case as can be seen:

P1_1=-3
P1_2=0.8
P2_2=5

While the prices do show some correlation, the volumes are a fair way out.

My next step should be to correct the problem with the algorithm above, and make it more representative of the actions of a competent technical analysis trader, and see whether I can more completely model the price and volume time series.