Interpreting CUSUM graphs

To interpret the CUSUM graph one needs to look at the slope of the graph, and specifically where slope changes occur. A constant slope is an indication of a stable value in the underlying data despite the presence of noise. In the example given earlier, a number of relatively "constant" slope areas can be identified, and these are shown superimposed on the graph. Points at which the slope changes are the turning points and these have been denoted with vertical lines.

So what do you do with the turning points? We'll this gives you an indication of where to average values from. In the example given, the first identified period is from t=1..14s, and the average for this period is 1.0. For the second period t=15..30s, the average is 6.4, and so on. I haven't shown this but you you could add this graphically to the bottom series as straight lines between the turning points for clarity, at the appropriate y-axis average value.

There is some danger in identifying too many turning points, as you could start reading something into the data which just isn't there. The greater the change in slope, the more convincing the turning point. In this example the turning points near 48, 72 and 84 are the most convincing.

To assist in calculating the average from the graph, one can add a calibration scale/mask which shows the relationship between set slopes and average values. We'll save details on how to do that for a later post though.

Applying the CUSUM to your spreadsheet data

The CUSUM is, simply put the cumulative sum of the differences of your data from its mean value.

i.e. CUSUM(t) = CUSUM(t-1) + data(t)-(mean value)

in spreadsheet for if your data is in range from A2..A32, the cusum could be implemented as follows in cells B2..B32: (Make sure cell B1 is empty equal to zero (0).)

B2=B1+A2-average($a$2..$a$32)

Then copy down to cells B3..B32.

Alternatively you could work out the average in a seperate cell and then absolute reference it.

eg. A1=average($a$2..$a$32)

then B2 would simplify to:

B2=B1+A2-$A$1

In the next post we'll cover the interpretation of the CUSUM.

In the graph shown an example of some noisy data (bottom series) and its corresponding CUSUM is presented. Click here to see the data spreadsheet.

Exponential Filter/Smoothing

#1
The problem...
The problem is you've got noisy time series data. Ideally you would like to plot it on graph to show the average trend or compare it with another variable. BUT its too noisy and clutters the graph if you plot it with markers and lines. With just markers it appears as a scatter cloud?

Aha, then you see that Excel has the option to add a "trendline". But you can't seem to find one that does the data justice -or- its just plain wrong in places. The "moving average" seems the better one of the options, but to get good smoothing you have to average many periods. That has some bad effects: it delays when changes occurs, especially when comparing to another variable (plotted in another series).

Resist the temptation of moving average!!!

A solution ...
A nice easy way to sort out this data is to apply a simple exponential filter or smoothing. There are better, but this is simple and for 80% of the time will work great.

How does it work?

I will illustrate this in the spreadsheet, but the maths is simply like this:

New Smoothed value = (1-factor) * actual data + (factor)*(previous smoothed value)
or:
y_n=(1-k)*x_n + k*y_n-1
y = smoothed value
x = actual (noisy) data
k = smoothing factor, a real number between 0 and 1. 0= no smoothing

How to set this up in a spreadsheet:

1. Estimate a value for the starting smoothed average. I usually use the first raw actual data point or take an average of the first few points.
2. Setup the filter to the right of the actual data points.
3. Reference the smoothing constant "k" as cell using absolute $ reference eg. $b$1. This will allow you to adjust it easily.
4. Copy the filter down so that you have two columns- to the left your original data and to your right the smoothed values.
5. Play and adjust "k" to suit your needs. As a first guess use 0.5. I commonly use anywhere from 0.5 to 0.85. Be wary of values above this as you could be over smoothing, but it does depend on your application.
6. For graphing purposes I suggest keeping your original raw data series (but use markers only, i.e. scatter plot) and then plotting the smoothed trend as a second series (lines only).

http://www.4shared.com/file/11752632/8565b041/exponential_filter.html