Quant analytics: Filtering signals for portfolio construction

When processing signals for portfolio construction people do a lot of exponentially weighted smoothing and z-scoring.
Exponentially weighted smoothing is a linear filter. Is there any connection between linear filtering and z-scoring?
More in general, is there a reference that reviews the main practical filters, possibly discussing computational issues?

 

==

I’d like to address how one may use linear filters to compute the z-score.

Denote as _n_ the index that, in discrete time, represents a time-point. Say an input series is x[n] and the z-score series is z[n]. I will not address initial conditions.

The z-score is

z[n] = (x[n] – Ex[n]) / sqrt(var(x[n]))

The z-score series may be computed with two recursion equations.

The expected value Ex[n] series may be computed with an ema. Using a finite-difference equation the ema series is generated by the recursion

Ex[n] = p Ex[n-1] + (1-p) x[n]

where the parameter coefficient p = Neff / (Neff + 1), and where Neff is N-effective, the length-scale of the ema. Note that p + (1-p) = 1, so the gain of the ema is unity, which is correct.

Regarding the denominator, sqrt(var(x[n])), let’s forget the sqrt() for a moment. Var(x[n]), in sample form, is

var(x[n]) = 1/(Nb-1) sum_{k=n-Nb+1}^n (x[n] – Ex[n])^2

This equation represents a box-shaped filter on the series (x[n] – Ex[n]). Data for n < n-N is completely dropped, data within the window is uniformly weighted by 1/(N-1).

The recursion for a box B[n] of length Nb is written

B[n] = 1/N b[n] – B[n-Nb]

For variance the b[n] coefficient is 1/(N-1), and b[n] = (x[n] – Ex[n])^2.

The box-length and ema effective lengths need to have similar scales. They cannot be the same because the shapes differ, and the ema is an infinite-impulse response while the box is finite-impulse response. Nonetheless Neff ~ Nb needs to be upheld.

A couple of notes about the recursions:

* both are linear
* both require only two operations for an update
* the ema requires only 1 register for memory, independent of Neff
* the box requires a buffer Nb long to persist the window.

The z-score is then

z[n] = (x[n] – Ex[n]) / sqrt(B[n])

For the very particular, a convexity adjustment can be made to the square-root of the variance.

==

 
Every time I tried to filter a price series, it disgruntled me. What exactly are we trying to filter? Filters are there to filter noise and reveal the underlying state. Of course you can look at everything and try to see them in

y[n] = x[n] + e[n]

format. Nevertheless I recently concluded that this is not a good way of looking at price series (The only thing to my mind that e[n] can represent is perhaps market intraday micro-structure effects).

Suppose there is such thing as x[n] (a level) and we filtered it successfully. What are we going to with it when PNL is measured in terms of y[n]? My comment applies to any type of filter as basic as simple moving average.

That’s why I am using filters only for unobservable things. For example online beta estimation.

I believe all practical linear filters can be represented in state-space representation by increasing the state space dimension. Result is that the filter becomes recursive, hence you don’t need to reprocess the old data as new data comes. EWMA is in such format (An interesting feature of EWMA is that it is optimal for ARIMA(0,1,1) and local level model at the same time).

I don’t know if I have repeated very basic stuff or managed to provide some insight.

—

 

I believe that is relevant only when you would consider deviation from ordinary as a certain muliple of z-score.
There is more rigorous treatment of how deviation of model estimation from observed data in bayesian time series filtering literature.
Please see below (I think) are quite instructive and have further references on the subject for linear (like kalman) or nonlinear filtering problem.

Time Series: Modelling, Computation and Inference, R Prado, M West (2010)
Bayesian Forecasting and Dynamic Models. M West, J Harrison (1997)
A Tutorial on Particle Filtering and Smoothing: Fifteen years later. A Doucet, A Johansen (2008)
Particle Filters and Bayesian Inference in Financial Econometrics. H. Lopes, R Tsay (2011)

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!