One of the most useful pieces of advice I ever got about potency assay development was to think of the assay as a process.  Once I started thinking of an assay as a series of steps with information and material flowing through them, I started utilizing the tools process engineers and six sigma practitioners use to optimize and monitor their processes. A common tool to monitor the performance of a process is the control chart. The topic of this post is to show some examples of how this tool can be beneficial to a potency assay.

Potency assays, and bioassays in particular, tend to drift over time (we have all heard about cell-based assays that seem to change with the cycles of the moon or the seasons). So what?  As long as the assay is passing the acceptance criteria, why should you care?   Let the assay drift around, right?  The problem is that your assay may be slowly moving towards the edge of a cliff and you don’t know it until it falls apart completely.  A process control chart is an easy way to help you monitor critical assay parameters and give you confidence that your assay is stable.

There are many types of process control charts and I won’t discuss all of them here, but the simplest and one of the most useful is the run chart (sometimes called a time-series chart). To build it you simply create a line graph with the parameter of interest on the y-axis (a positive control value in this case) and plot the value of each assay as you go along. Simple enough:

So what can you do with this chart? Well, it is very useful for catching trends over time. Once you have several assays plotted, you can easily detect if your assay changes. For example, something changed after assay 10 in the example above and then it changed back after assay 15.  If you can correlate this pattern with another variable (like your best analyst going on vacation for example), you have learned something important about your assay.  It’s important to note that the acceptance criterion for this parameter may have been around “80” on the chart and thus the assay was deemed acceptable, but without trending this parameter, you would not have seen the pattern.

If all you ever do is make simple run charts of your assay parameters, you have gained a lot of the benefits of control charts. But if you are able to invest a little more effort, you can build on your run chart to make it even more valuable.

But before we get into that, let’s discuss the two different types of variation in processes: common cause and special cause. All processes have some inherent variability due to “noise factors” that we intentionally don’t or can’t control. This is usually thought of as assay-to-assay variability and process engineers call this “common cause”.  Special cause variability is basically everything else. This is variability in your process that occurs when something out of the ordinary happens.

So why is this distinction important?  If you have some data from an assay that includes only common cause variability we can use it to create control limits for your assay. These limits basically tell us the expected day-to-day variation in our assay. Anything outside the these limits is likely due to a special cause and warrants investigation. The limits are usually set at the 99% variance limits around the mean. The top graph in the following figure is an example of such a chart with a special cause at assay 14 (the control limits are shown in red).  It’s immediately obvious that something went wrong with this assay and that you need to investigate it.

Easy enough, but what is that second graph at the bottom?

That graph is a moving range chart. The graph on the top tells you how the parameters tracks against the mean, while the one on the bottom tracks any changes in variability.  This way, you can detect changes in the variance of the assay, even if the mean value of the parameter remains the same.

If a value falls outside the control limit, the chart is telling you that there was a less than 1% chance that this event could happen by chance alone.  This type of indicator is an obvious outlier, but control charts can also be useful in detecting less obvious trends that are also unlikely to happen by chance. For example, it is not very likely to have greater than 8 assays all appear on one side of the mean or the other.  This situation would be flagged on the chart even if all those assays were inside the control limits.  Another common situation is when the data oscillates around the mean for several assays in a row; this could be an indicator that you have more than one population of measurements. There are many more such rules and I won’t discuss them here, but you should be able to find them easily in any reference source on control charts.

I hope I have convinced you that control charts are simple and useful tools to monitor your assay.  But what assay parameters should you trend? Obvious choices would be the value of a positive control that is run in every assay. The parameters of any data fitting models are are also useful to trend (asymptotes, slopes, EC50 etc.). For cell based assays, the number of receptors or other markers on the cell can also be invaluable to collect trending information on. It may even be useful to track the average signal measured at each dose response to look for shifts.

As you can tell, there is probably an unlimited number of things you can track in control charts, and since these days automating this type of data capture can be very easy, why not trend as much as you can? The data may be useful as you look towards future improvements in your procedure.

A word of caution: if you trend too many parameters, the likelihood of one of them showing up as a special cause increases. For example, if you were to trend a 100 different parameters for each assay, and you used 99% control limits for all of them, one of your parameters would appear to be out of control by chance alone with EVERY assay run. So be careful in how you interpret your out of control events.

I’ll leave you with one last thought about control limits. Once an assay starts to drift, it’s very tempting to just adjust the control limits so that the assay appears to be back in control. I strongly discourage this type of adjustment. The control limits should remain fixed unless you have a deliberate reason for changing them. For example, if you make a major change in the assay procedure that has a significant impact on the assay variability, then you may want to consider moving the limits.

Ok, one more last thought. Never ever set your control limits as specifications. They are for you to trend your assay over time and since we always expect the assay to drift, you would be wise to design your assay so that your specifications are much wider than your control limits.  This way, if your assay is out of control you can take action before it fails the specification.

Here are a few links to get you started with learning some more about control charts:
http://en.wikipedia.org/wiki/Control_chart
http://www.isixsigma.com/st/control_charts/
http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.htm

Thanks for reading,

Dan

No tags