One of the most useful pieces of advice I ever got about potency assay development was to think of the assay as a process.  Once I started thinking of an assay as a series of steps with information and material flowing through them, I started utilizing the tools process engineers and six sigma practitioners use to optimize and monitor their processes. A common tool to monitor the performance of a process is the control chart. The topic of this post is to show some examples of how this tool can be beneficial to a potency assay.

Potency assays, and bioassays in particular, tend to drift over time (we have all heard about cell-based assays that seem to change with the cycles of the moon or the seasons). So what?  As long as the assay is passing the acceptance criteria, why should you care?   Let the assay drift around, right?  The problem is that your assay may be slowly moving towards the edge of a cliff and you don’t know it until it falls apart completely.  A process control chart is an easy way to help you monitor critical assay parameters and give you confidence that your assay is stable.

There are many types of process control charts and I won’t discuss all of them here, but the simplest and one of the most useful is the run chart (sometimes called a time-series chart). To build it you simply create a line graph with the parameter of interest on the y-axis (a positive control value in this case) and plot the value of each assay as you go along. Simple enough:

So what can you do with this chart? Well, it is very useful for catching trends over time. Once you have several assays plotted, you can easily detect if your assay changes. For example, something changed after assay 10 in the example above and then it changed back after assay 15.  If you can correlate this pattern with another variable (like your best analyst going on vacation for example), you have learned something important about your assay.  It’s important to note that the acceptance criterion for this parameter may have been around “80” on the chart and thus the assay was deemed acceptable, but without trending this parameter, you would not have seen the pattern.

If all you ever do is make simple run charts of your assay parameters, you have gained a lot of the benefits of control charts. But if you are able to invest a little more effort, you can build on your run chart to make it even more valuable.

But before we get into that, let’s discuss the two different types of variation in processes: common cause and special cause. All processes have some inherent variability due to “noise factors” that we intentionally don’t or can’t control. This is usually thought of as assay-to-assay variability and process engineers call this “common cause”.  Special cause variability is basically everything else. This is variability in your process that occurs when something out of the ordinary happens.

So why is this distinction important?  If you have some data from an assay that includes only common cause variability we can use it to create control limits for your assay. These limits basically tell us the expected day-to-day variation in our assay. Anything outside the these limits is likely due to a special cause and warrants investigation. The limits are usually set at the 99% variance limits around the mean. The top graph in the following figure is an example of such a chart with a special cause at assay 14 (the control limits are shown in red).  It’s immediately obvious that something went wrong with this assay and that you need to investigate it.

Easy enough, but what is that second graph at the bottom?

That graph is a moving range chart. The graph on the top tells you how the parameters tracks against the mean, while the one on the bottom tracks any changes in variability.  This way, you can detect changes in the variance of the assay, even if the mean value of the parameter remains the same.

If a value falls outside the control limit, the chart is telling you that there was a less than 1% chance that this event could happen by chance alone.  This type of indicator is an obvious outlier, but control charts can also be useful in detecting less obvious trends that are also unlikely to happen by chance. For example, it is not very likely to have greater than 8 assays all appear on one side of the mean or the other.  This situation would be flagged on the chart even if all those assays were inside the control limits.  Another common situation is when the data oscillates around the mean for several assays in a row; this could be an indicator that you have more than one population of measurements. There are many more such rules and I won’t discuss them here, but you should be able to find them easily in any reference source on control charts.

I hope I have convinced you that control charts are simple and useful tools to monitor your assay.  But what assay parameters should you trend? Obvious choices would be the value of a positive control that is run in every assay. The parameters of any data fitting models are are also useful to trend (asymptotes, slopes, EC50 etc.). For cell based assays, the number of receptors or other markers on the cell can also be invaluable to collect trending information on. It may even be useful to track the average signal measured at each dose response to look for shifts.

As you can tell, there is probably an unlimited number of things you can track in control charts, and since these days automating this type of data capture can be very easy, why not trend as much as you can? The data may be useful as you look towards future improvements in your procedure.

A word of caution: if you trend too many parameters, the likelihood of one of them showing up as a special cause increases. For example, if you were to trend a 100 different parameters for each assay, and you used 99% control limits for all of them, one of your parameters would appear to be out of control by chance alone with EVERY assay run. So be careful in how you interpret your out of control events.

I’ll leave you with one last thought about control limits. Once an assay starts to drift, it’s very tempting to just adjust the control limits so that the assay appears to be back in control. I strongly discourage this type of adjustment. The control limits should remain fixed unless you have a deliberate reason for changing them. For example, if you make a major change in the assay procedure that has a significant impact on the assay variability, then you may want to consider moving the limits.

Ok, one more last thought. Never ever set your control limits as specifications. They are for you to trend your assay over time and since we always expect the assay to drift, you would be wise to design your assay so that your specifications are much wider than your control limits.  This way, if your assay is out of control you can take action before it fails the specification.

Here are a few links to get you started with learning some more about control charts:
http://en.wikipedia.org/wiki/Control_chart
http://www.isixsigma.com/st/control_charts/
http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.htm

Thanks for reading,

Dan

No tags Hide

In a previous post on this topic, I discussed ways of preventing edge effects in plate-based potency assays. But how do you know you have them in the first place?   Sometimes it can take a long time and many runs to figure out that these problems exist. At that point you may have generated results for several samples and by tampering with the assay you lose connection to historical data or at the very least cause the previous results to be looked at with suspicion.

Instead of waiting for bias to appear in your data, I advocate doing edge effect studies early in assay development. The results you find can be invaluable in determining your plate layout, assay testing strategy and assay analysis.

There are several different ways you can characterize your assay for edge effects during development. A simple and useful experiment is to run a “heatmap” on your plate. This type of experiment is as simple as picking a single dose response, usually from the reference standard, and applying it to every well on the plate. The data is then graphed using a color code to visually pick up patterns in the results and/or statistical tests on the row and column data can be performed.

An important part of this experiment is to pick a suitable dose response. Ideally this response should come from the part of the dose response curve with the steepest slope (see the red circle in the image below). This is to ensure that you will have the optimum sensitivity to small changes in signal.

The resulting data from the experiment can be visualized using a graph such as this one:

As you can clearly see in this "heatmap" the center wells of the plate are lower in signal than the outside of the plate.

Most plate based assays will have some sort of edge effect due to the asymmetrical nature of 96 well plates. It isn’t always clear if the effect will actually have a significant impact on the results of the assay.  If the differences in signal between rows and columns are much smaller than the variability of the signal itself, then the effect may not be noticeable in the final results. One way to determine if this is the case is to do a two-way  ANOVA for statistical significance.  In this case, the ANOVA clearly shows that the effect is significant across both rows and columns (p value is less than 0.05):

Two-way ANOVA: Response versus Row, Column

Source  DF        SS       MS     F      P
Row      7   9088182  1298312  7.87  0.000
Column  10   4776165   477616  2.90  0.004
Error   70  11540595   164866
Total   87  25404942

S = 406.0   R-Sq = 54.57%   R-Sq(adj) = 43.54%

Heat map experiments generate a lot of information about your assay, but it only deals with detecting an edge effect in a single dose. Since the ultimate readout of these assays, the potency, relies on the entire dose response range, you may want to do some further characterization of the edge effect by running the reference standard in multiple sample positions on several plates.  The graph below shows the results of such a study where the reference standard was placed in every row of five plates (positions 2-8) and a potency was generated relative to the reference in position 1. The expected potency is 1.0 since every position contains the reference standard. It is clear that in this assay, the potency is severely biased by the position effect.  It is much higher in the middle of the plate.  This type of analysis can also indicate if there is a bias in the variance of the response. In this case, positions 4 and 5 clearly have much less variability than the others.

As you can see, there is much information about your assay that can be gained from just a few experiments early in the development process. Once you have this information, the assay protocol can be designed to prevent a bias in your results. But that’s a topic for another post.

Thanks for reading,

Dan

No tags Hide

I haven’t seen a comprehensive list of contract providers for bioassays anywhere so I decided to create my own.  This is a list I will be keeping of companies that do outsourced potency assays for GMP release (as listed on their websites).

Leave me a comment if you have others to add.

• Althea (www.altheatech.com)
• BioAnaLab (www.bioanalab.com)
• BioOutsource (www.biooutsource.com)
• BioReliance (www.bioreliance.com)
• Charles River (www.criver.com)
• Clean Cells (www.clean-cells.com)
• Covance (www.covance.com)
• IBA Analytics (www.iba-analytics.com)
• Marin Biologic (www.marinbio.com)

No tags Hide

Another nice reference I found:

http://www.google.com/url?sa=t&source=web&ct=res&cd=4&ved=0CBkQFjAD&url=http%3A%2F%2Fwww.nihs.go.jp%2Fdbcb%2FBiologics_forum%2FBioforum-4%2F4-DrPetricciani.pdf&rct=j&q=potencyassay.com&ei=GZf1So2THcfj8Qbvt5DzCQ&usg=AFQjCNGt72BARy6jI5WbcU03CUrUU-zf6w

No tags Hide

I ran across this PDF by Susan Kirchner at CDER.  It’s a nice overview of why we need run potency assays for biotherapeutics.

http://www.usp.org/pdf/EN/meetings/bioassayWorkshop/session1f.pdf

No tags Hide

If you work in the potency assay field long enough you will eventually run into problems with edge effects.  Sometimes they are blatantly obvious, and in some assays you might not notice them at first, but over time they show up as a bias in your potency results.

What causes edge effects?  There is not always a straight forward answer to this question and even when you know the answer it’s not always possible to prevent edge effects.

In cell based assays, edge effects are often caused by evaporation in the outer wells of the plate.  This leads to changes in salt concentrations in the cell medium, affecting cell metabolism in those wells.  They can also be caused by temperature gradients when the plate is placed in a heated incubator;  the outer wells tend to heat up faster than the inner wells.  Often, this will affect cell attachment and adhesion.

Lundholt et. al. write about an elegant solution here:  http://jbx.sagepub.com/cgi/content/abstract/8/5/566.  They investigated several different ways to prevent uneven cell-seeding of their plates and they came up with a simple solution.  They simply pre-incubated the plates at room temperature until the cells adhered, and then transferred them to the incubator.

Anther common way of preventing edge effects in 96 well plates is to use various humidity and temperature chambers.  One such example can be found here:  http://www.btc-bti.com/product_literature/stabilitychamber.htm.  There are also special gas permeable membranes that can be attached to the cell plate to help prevent evaporation.

Immunoassays can also suffer from edge effects.  Because of the shorter incubation times of these assays, evaporation is probably not the culprit, but instead they are often due light exposure and temperature gradients.  These causes can be exacerbated in assays where plates are stacked on top of each other during incubations.

Like in cell-based assays, common ways to prevent edge effects in immunoassays include temperature controlled chambers and plate seals.  In this case, cheaper plate seals can be used since gas exchange isn’t necessary in assays without cells.  Goetz et. al. discusses some of these solutions here:  http://gateway.nlm.nih.gov/MeetingAbstracts/ma?f=102268104.html

These are some of the physical causes and preventions of edge effects in potency assays.  However, in come cases, none of these help remove the bias from your assay, and your only recourse is to deal with it using statistical measures.  This will be the topic of future post.  I also want to try to convince you that waiting for these effects to show up during testing isn’t necessary, instead I will write about ways to discover the bias of edge effects during assay development.

Thanks for reading,
Dan

No tags Hide

Welcome to Potencyassay.com

I’ve been working in the analytical development field at a major pharmaceutical company for the last ten years or so.  Most of that time, I’ve been involved with development of various kinds of potency assays.  I have looked around the internet several times for a resource where people in the bioassay and potency assay community can share ideas and best practices.  I haven’t found such a place yet.  So instead, I thought I would start this blog and share my experiences and ideas about these often difficult assays.

The topics I plan on covering in this space include:

• Bioassays
• Immunoassays
• In vivo potency models
• Assay development practices
• Assay optimization
• Design of Experiments
• Data analysis and statistics for potency assays
• Laboratory Automation
• Six sigma and other process optimization methodologies as they relate to potency assays

I would also like to invite all of you to comment on my posts or write a guest post if you’re an expert on a related topic.  Just leave a comment and I will get in touch with you.

I also want to point out that the opinions that are posted on this blog are strictly my own and are in no way endorsed by or affiliated with my employer.

Thanks for visiting and I hope you come back soon.

Dan

No tags Hide