The Regulatory Clinic - March 1997
Handling Aberrant Results in the Pharmaceutical Laboratory
Dear Clinic:
I have been a supervisor in a pharmaceutical analytical laboratory for some time. Although there has been much said since Barr about OOS results, I am interested in proper handling of aberrant results.
An aberrant result is an unexpected result which is not out of specification. For example, if we analyze a true solution by HPLC for a soluble analyte and get results of 99.9%, 99.7%, 99.8% and 108.9%, I would consider the 108.9% result aberrant, although not out-of-specification (90% to 110%).
An "assignable cause" can not always be determined. We can sometimes investigate the aberrant result to death, but there are many laboratory errors which leave no evidence.
If an analysis is performed from a homogenous composite sample, I expect all replicate analyses to agree within the precision of the method, the precision of the method being determined by method validation data or system suitability criteria. If the replicate analyses do not meet the precision requirement, it would seem reasonable to assume that some error or malfunction has occurred, even if an assignable cause cannot be found.
A recent FDA inspector was uncomfortable with this logic, and preferred that a "number" be assigned. When pressed, this investigator said "I like 2%." I am afraid, however, that if I expect all composite analyses to replicate within "2%" that I will have to defend where this value came from. On the other hand, if I do not have a written criteria, then I am guilty of using "scientific judgment".
It is my experience that FDA will always consider that scientific judgment is simply arbitrary result selection based on no reasoning other than financial motivation. Can you help?
Signed, Simply Aberrant, St. Louis, MO
Dear "Aberrant":
Let me begin by commending you on such a good question. As a scientist who spent many years in research laboratories and then found himself responsible for high-paced pharmaceutical QC and Analytical R&D laboratories, I can certainly empathize with your situation. Based on my own experience, and the 2% numerical precision reference and the HPLC analogy you have described, I assume we are specifically discussing routine HPLC ASSAYS.
I would like to begin with several clarifications to ensure the terminology means the same to both of us. An aberrant result is a response obtained from a series of repetitive events or measurements of the same homogeneous sample that appears to differ, or not belong to, the complete set of all observations. An aberrant result need not lie within the limit expression of a pre-determined specification. In your example of the HPLC measurement series, if the 108.9% result happened to be 115.9 %, it would still be an aberration when compared to the series from which it originated (i.e., 99.9, 99.7, and 99.8%). It would simply be an aberrant result that was also out-of-specification (OOS).
In addition, as we assess your question, I am assuming that we are not discussing trace analyses (e.g., cross-contamination or cleaning validation studies) or statistically-based tests (e.g., content uniformity, blend uniformity, dissolution, packaging/batch uniformity, etc.). I am also assuming that we are discussing small molecular entities and not biotechnologically-derived products or proteins, etc.
Throughout this discussion, I will preferentially use the following terms, not because I am attempting to alter the terminology you have chosen, but to ensure that my response is the least ambiguous that I can make it:
Sample Solution(s)/Preparation(s): I believe this term is synonymous to what you called the "true solution". The sample solution/preparation is a solution which results from a single-assay preparation made from a single homogeneous sample.
Homogeneous Sample: A homogeneous sample is the material used to prepare sample solutions/preparations. This may be a homogeneous grind of 20 tablets in a mortar, the liquid from a single bottle, a composite liquid sample combined into a single container and mixed, the semi-solid material from a single tube, etc. In other words, the homogeneous sample is the "mother lode" from which the analyst will remove a small portion to prepare a sample solution/preparation for the actual analysis.
Test Value: A value obtained from a single measurement on a single sample solution/preparation.
Test Result: The final representative value of the analyte calculated as the average of multiple test values in a multiple sample preparation assay scheme. If only a single sample assay preparation scheme is chosen, this value is the same as the Test Value.
Questionable Result: A test result that does not exceed specification limits but is nonetheless unexpected, due to the history and knowledge of a particular test method and product.
The logic you have chosen regarding the use of the "method's" validated precision seems reasonable in that you might expect a series of replicate measurements to agree within the precision of said "method". However, the question immediately raised is this:
Which type of precision are you discussing?
I might expect the same sample solution/preparation to exhibit similar precision (let us say the relative standard deviation (RSD) of replicate measurements) as that of the system precision when running duplicate injections of the same sample solution/preparation containing your analyte(s) of interest (typically ca. 0.5%).
I might expect multiple sample solutions/preparations from the same homogeneous sample to exhibit similar precision in assay results as the validated repeatability of the method when running duplicate injections of duplicate sample preparations/solutions using the same analyst and instrument (typically ca. 1-3%).
I might expect multiple sample solutions/preparations from the same homogeneous sample to exhibit similar precision in assay results as that of the method's validated intermediate precision when having samples assayed by different analysts on different days.
I might expect multiple preparations from the same homogeneous sample to exhibit similar precision in assay results as that of the method's ruggedness when having portions from the same homogeneous sample assayed in different laboratories using different analysts on different days.
In our experience FDA inspectors are primarily concerned with the Intra-Method Precision (or Repeatability) of the method, (probably since the USP general chapter <1225> Validation of Compendial Methods, clearly states that, For most purposes, repeatability is the criterion of concern in USP analytical procedures.), along with assurance that system suitability has been performed. This is also the most logical point of view. If you look at the "typical" precision values of the system and the intra-method precisions, as described above, you see that 2.0% is a scientifically reasonable value of precision for current HPLC technology coupled with a trained analyst. In addition, consider that the USP derived values of 2.0% (five replicate injections) and 3.0% (six replicate injections) for the system suitability precision requirement originates from data obtained over many years from many different sources.
In the average pharmaceutical laboratory, the responses obtained from duplicate injections in an HPLC assay should generally be within 2.0% of each other. Although the FDA inspector with whom you spoke may have had no idea as to why he or she "liked two percent," this value does have basis in scientific fact through the combination of years of data and propagation of error. The inspector probably "liked two percent" because he or she had seen it used as the criterion in many other industry laboratories for HPLC type analyses. As stated above, this value represents a good compromise between the precision of the system and the repeatability associated with the technology.
It should be pointed out, however, that for "dirty" samples (e.g., viscous creams and ointments, etc.), this requirement may have to be relaxed. When dealing with samples such as these, the 2.0% requirement simply may not be achievable. You may have to accept a higher number based on your sample quality. Your experience with such samples should allow you to choose a number that is scientifically sound and reasonable (I suggest not going much higher than 3% between the responses from duplicate injections) as well as defensible to the next FDA inspector. Often an inspector will accept issues and explanations that are non-standard if you can "Show Me The Data".
However, if you cannot achieve these two qualities with the combination of sample cleanliness and methodology (i.e., scientific soundness and reasonable acceptance of the quality of the data), it may be time to re-develop the method. Re-development may also have practical value, especially as it pertains to stability studies. For example, if your method's average recovery is only 95%, you may have to artificially apply an expiry that is too short or erroneously justify a product active ingredient overage.)
What follows is a description of the quality scheme I prefer to assure good data quality for the type of analysis we are discussing:
On the other hand, if the two values are within 2%, which is usually the case, then the responses should be averaged and a test value obtained. This takes care of the "system precision" type of QC check. If this were a single sample preparation scheme, then the test value would also be the test result and if it met specifications, it would be acceptable and you would have to live with it.
Repeating this procedure for a second sample solution/preparation provides information again regarding the "system precision" but in addition, will provide some "repeatability" information as well. If the duplicate injection responses meet the criteria, then calculate the second test value for the sample. The two test values should be within 2 to 3.0% of each other to achieve a valid final test result. The two test values must also be within specification. If all three criteria have been met (i.e., precision between duplicate injections, duplicate preparations, and all test values within specification), then the average of the test values will produce a final, reliable test result that provides a high degree of confidence that the sample analyte value obtained is very near to the true sample value. If the sample is truly representative of the batch, then your test result will represent the average value of the batch to a very high degree of confidence.
The advantage with this scheme is that defensible, high quality results are obtained. The disadvantage is that the labor is twice that of a single sample solution/preparation scheme.
In some cases, a single sample solution/preparation with a single sample injection scheme is absolutely necessary due to time and financial considerations (e.g., process validations with very large sample quantities and/or methods with very long sample run times). Although there is nothing unreasonable about this scheme, this technique is not sensitive to detecting system and analyst precision errors. The disadvantage here is that you may get a questionable result that does not allow for further investigation because it lies within your specification range. The tenets of the Barr decision do not allow us to continue to test without reason, and the reason can no longer be based upon scientific "intuition" (even though it is this very intuition -- culled from life experience -- that keeps us from sticking our fingers into empty light bulb sockets, running into moving traffic, avoiding suspicious looking people on the street, etc.!). Therefore, your result is not OOS, but your experience with the product and test method tells you that it is unlikely or questionable.
Questionable results become troublesome when someone has to attempt to perform product stability trend analyses, statistical process control analyses, or any other type of basic historical data evaluation. Avoiding these type of results is the primary reason I prefer the duplicate preparation with duplicate injection scheme described above.
On the other hand, if you do get an OOS result, then you need to deal with it in accordance with your internal procedures (another story altogether).
Also, it is wise to re-inject the system suitability solution standard periodically (check standard) to ensure system performance (analogous to manufacturing equipment performance qualification, PQ) has not been compromised since the original system suitability was demonstrated.
Outlier tests may also be applied to microbiological assay results, although they are not quite as virtuous due to the poor precision of biological methodology when compared to chemical and physical measurements. This indicates that an aberrant result obtained in a chemical or physical experiment is more likely to be truly aberrant (a mistake, anomaly, or cumulative bias due to indeterminate error), whereas in a microbiological test, it is more likely to be a representation of the subject under experiment (a true result).
Despite these facts, the Barr decision has effectively denied the use of outlier testing for questionable and/or OOS results obtained from chemical tests and assays, while having approved the routine usage of the USP outlier test for microbiological assays. Although the use of an outlier test may be done and may be the most appropriate way to handle an aberrant datum, please be aware that you should not rely solely on the use of these tests for invalidating such results while you are working in an FDA-regulated industry. While you may perform an outlier test, the outlier test result may only be used to support other systems for the handling of analytical data. Hence, it is desirable to develop a "no guess work" approach quality system. If you do this, it should obviate future difficulties you may have in these areas.
If you need more guidance please e-mail me at accureg@regulatory.com and we can work through specific examples with your data.
I hope this information was helpful. Good luck, and thank you for visiting The Regulatory Forum.
Robert V. Sarrio, Senior Associate
AccuReg, Inc.
Return to The Regulatory Forum
Write to the Clinic at accureg@regulatory.com
Copyright (c) 1997 AccuReg, Inc.
7501 Northwest 4 Street, Suite
210
Plantation, Florida 33317, USA
Telephone: 954-641-6400 Fax: 954-641-6410
Email: accureg@regulatory.com
Contact webmaster
Copyright © 2001 AccuReg, Inc.
All rights reserved.
No portion of this site may be reproduced without express written consent of
AccuReg, Inc.