An investigation into the standardisation

of HBA1c  assays using a common calibrator.

 

                   

SA AACB QC Subcommittee:   J.Calleja1, J.Gill, J.Booth, L.Bourne  J.Bouras,  J.Crocker, L.Howe, T.Kuss, C,Malavazos, D.Robertshaw, E.Whitham.

 

Contact Address :      John Calleja:    RCPA AACB Chemical Pathology QAP Flinders Medical Centre Bedford Park SA. 5042

 

1.0              Introduction

 

Diabetes has become the Western World’s fastest growing disease. In Australia alone, it is the sixth leading cause of death, with an estimated 800,000 Australians being affected ( 50% of which are undiagnosed )1 . Of equal concern to mortality rate though, are the debilitating long-term complications of diabetes such as retinopathy, nephropathy, micro-vascular disease and neuropathy; which significantly affect the patients quality of life and place ever increasing burdens on the  health dollar.

 

In 1993, a ray of hope was afforded to patients afflicted with diabetes, via the Diabetes Control and Complications Research Group.  The Group showed that intensive treatment regimes significantly delay the onset and progression of diabetic complications by between 39 to 70%.2  Then in 1997, the American Diabetes Association, in accordance with these studies, formalised recommendations for diabetes therapy and put forward Guidelines for HBA1c control ratings.3 The consequent implications of these recommendations for the laboratory, were and are,  the implicit requirements of long term availability to accurate (traceable to the DCCT Method)  and precise ( CV’s of 2 – 3 %)  measurements of HBA1c between laboratories. 

 

Standardisation of HBA1C assays is one means of realising tighter performances and has been previously addressed by several authors. They demonstrated that improvement to both accuracy and precision between methods can be achieved when common calibrators are used. But closer examination of their methods prove them to be somewhat cumbersome to apply6,7.  The IFCC Working party on HBA1c Standardisation is also focused on this area, but its initiatives centre on selecting a reference method and producing a primary reference material.8   Australia has representation on this committee via Mr Ian Goodall from the Austin Repatriation Hospital in Victoria .

 

In South Australia , the SA Ministerial Health Advisory Group on Diabetes –  formed a working party in 1998 and undertook their own studies with the aim of standardising assays between laboratories 4. This was due to the view by some South Aust-ralian clinicians and endocrinologists, that methods between laboratories were giving unacceptably diverse performances.

 

In response the South Australian laboratory community, whilst co-operating with the Ministerial Group, undertook its own investigations into possible standardisation strategies that could be used. It did so via the SA AACB QC Sub-committee (SAQCC) , a committee made up of representative scientists from laboratories in both the private and public sector, who meet monthly on quality control issues. To some degree this process was considered to be an easier issue to address in SA (in contrast to the wider Australian community) because in S.A. the majority of HBA1c testing is performed in Adelaide and  each laboratory has representation on the SAQCC.

 

The SAQCC’s initial investigations focussed on the use of a primary calibrator to standardise QAP results, across the methods used in their labs. This was followed by an analysis of the feasibility of possible secondary calibrator value assignment protocols and their large scale practicalities. Data modelling studies were also undertaken to predict the improvement that could be gained, or further error that could be introduced between laboratories, by such techniques.

 

Whilst initial improvements to overall CV’s within the SA Group were apparent, no appreciable improvements were noted after data for the DCA2000’s were removed.  Ultimately it devolved that secondary cal value assignment can not be easily applied to all methods currently in use ( DCA2000) and in addition the inherent complexities involved  in achieving adequately consistent secondary cal value assignment practices across laboratories long term, would possibly lead to further error introduction, via the process.

 

Accordingly  this report  discusses the studies undertaken via the sub-committee in conjunction with the RCPA AACB QAP Group and the issues we explored.  We conclude that whilst we have every good reasons to seek improvement in inter-laboratory accuracy & precision considering the DCCT requirements and the ADA Position statement, that this would best be served, not by secondary calibrator reassignment from a common calibrator within individual laboratories;  but by collaborating with instrument/reagent manufacturers to provide calibration materials and methods referenced to a common primary calibrator and designated reference methods.

 

 

 

2.0              Materials & Methods.

 

Primary reference material manufactured by the Netherlands based EQA provider, SKZL, was issued to twelve laboratories on the SAQCC. The material (referred to as the ERL, in this articel) had an assigned HBA1c value of 7.6%, traceable to the DCCT method of Goldstein2. It was made available curteousy of the RCPA-AACB Chemical Pathology QAP Group. 

 

The laboratories were asked to run the ERL as a routine sample alongside QAP samples 1 and 2, from cycle 12 in the RCPA-AACB Glycohaemoglobin Program. Of note also is that the QAP samples themselves also had DCCT assigned target values of 12.0% and 5.3% respectively. The range of analysers used by the participating laboratories included the DCA2000 (n=5), Bio-Rad Variant 1 (n=3), Primus CLC330 (n=1), Waters HPLC (n=1), Pharmacia HPLC (n=1) and the Lancer Manual (n=1).

 

3.0              Results & Discussion

 

The results achieved for the QAP samples and the ERL are shown tabulated in table 1. Here they  have been grouped in terms of analyser and QAP sample results are shown in both the un-standardised and standardised formats. Standardisation was achieved  by using the following simple calculation [ standardised_result = achieved_result  x ( assigned ERL_value / achieved ERL_value ) ] .

 

Basic descriptive statistics (mean, median, sd and CV%) were derived for each group of results (standardised and non-standardised) and for each of the QAP samples and these were compared. Notable improvements in the median recoveries for both samples is observed in relation to the  assigned  target values on standardisation. The overall between instrument CV also showed significant improvement for the higher  concentration QAP sample, however no appreciable improvement in CV was observed for the lower concentration sample. 

 

 

Table 1.  Results for ERL and QAP samples 12-01 &12-02

Standardised (Std) and Non-Standardised.

 

 

Table2.  Results for ERL and QAP samples 12-01 &12-02

Standardised (Std) and Non-Standardised (DCA2000 Excluded).)

 

In a true laboratory setting however, the DCA2000  results cannot be re-standardised  to ERL values by applying a calibration set-point modification or by applying a slope and/or off-set correction,  as the instrument is closed to this type of end-user adjustment. Accordingly we further investigated the issue of standardisation versus non-standardisation by excluding the DCA2000 data from our original results and recalculating the statistics for comparative purposes (Table 2).

 

Interestingly, the overall ability of the group tested to recover QAP target values shows no significant improvement when the DCA2000 data is removed. In addition  only a marginal  improvement  in between-instrument CV,  is noted for the low concentration QAP sample.  

 

 

4.0       Status Quo of HBA1c testing in Australia

 

When contemplating the issue of  large scale between laboratory re-standardisation , the status quo of testing in Australia also needs careful consideration.  This should be done in the context of which methods are in use, their relative performances and whether they are amenable to end-user re-standardisation in practical terms.  Some insight into these issues can be gained from examining reports and results from the RCPA-AACB Glycohaemoglobin Program.  

 

 

 

4.1          Instruments in use:

 

Figure 1, an End-of-Cycle Summary Report from the RCPA-AACB Glycohaemoglobin Program, reveals that from a total of 176  participating laboratories, 56 are  DCA2000 users. This represents approximately 30% of all laboratories. As mentioned earlier, none of these laboratories  can have their accuracy base re-aligned by application of a slope, offset or calibrator set-point modification and therefore need to be excluded from consideration for re-standardisation in the context of this article. Of the 120 laboratories remaining in the program, 37 are the BioRad Variant 1 users.  These instruments therefore represent a group of 30% of the remaining  total of re-calibratable laboratories. 

 

In addition two further technically relevant aspects of the Bio-Rad Variant need consideration in the context of interlaboratory re-standardisation. The first relates to a need for daily calibration which has implications which are further discussed in section 5 (b) and the second relates to the Variants use of a single point calibration, which has a set-point of around 9%.

 

This aspect is important in the context of  the extent to which  re-standardisation can improve inter-laboratory accuracy at levels significantly lower than the routine calibrator set-point (eg. at levels of around 5.0). This is especially the case for laboratories or instrument groups that have significant bias at the lower end of the analytical range. An example of this aspect can be seen by the insignificant impact that re-standardisation had on the recovery for QAP sample 12-02, for the three Variant laboratories  that took part in the SAQCSC Study (Fig 2) and demonstrate a low end positive bias.

 

4.2          Inter-lab CV’s.

 

Figure 2, a regression of all HBA1c data from cycle 15, shows a spread of accuracy bases across laboratories, with an overall inter-laboratory CV of 6.44% (estimated from the Sy  statistic) and for which a 3SD limit was applied to the data. To put this into perspective, this represents a standard deviation (SD) of 0.58 at a HBA1c level of 9.0% (or +/- 1.16, for the 95% C.I.) which is highly indicative of a need for standardisation if truly reflective of  between laboratory performances. This is especially the case in the context of  the precision and accuracy requirements  established in the DCCT Study2.

 

However,  although our motives for considering standardisation are consequently well founded, we should also assess  and carefully consider the realistic gains that we might expect to make in this area. This can be accomplished by using  a representative “already well standardised group” (in terms of instrumentation, reagents, calibrators and manufacturer recommended procedures) and examining their inter-laboratory performances, as a model.

 

A  group that has a significantly large number of users on the QAP Program and is already  well standardised (in the context mentioned) is the Bio-Rad Variant group.  This group has a median within-lab CV of 2.6%, which represents an SD of 0.13 at a HBA1c level of 5.1 (or +/- 0.26, for a 95% C.I.) and an SD of  0.36, at a HBA1c level of 13.8 (or +/- 0.72, for a  95% C.I.)  From an inter-laboratory perspective however, when a regression of all Variant data is performed , we obtain a between-lab SD of  0.34 9, which at a HBA1c level of  9%, equates to +/- 0.68 at the 95% C.I. or an inter-lab  CV of  3.77%. This level of inter-laboratory variability therefore is probably reflective of the realistic level of standardisation we can expect to achieve by introducing a between lab standardisation protocol. 

 

 

 

Fig.1  RCPA-AACB Glycohaemoglobin Program - Cycle 11

            End-of-Cycle Report – Statistical Summary data

 

 

 

Fig.2  RCPA-AACB Glycohaemoglobin Cycle 15

            Regression of all data (with 3 SD limit applied)

5.0        Examination of possible Secondary Cal. value Assignment approaches and Protocols.

 

Fundamental to our goal of  having  HBA1c accuracy traceable to a common calibrator, were three key considerations. Firstly which common primary reference material (PRM)  to use. Secondly whether to use the selected  PRM as a routine day-to-day calibrator or to use it to ascribe secondary values to routine calibrators used by laboratories.  Thirdly  to employ a protocol which is practical, technically robust  and economical. These issues are discussed below:

 

(a)           Given that at the time no primary reference material for HBA1c was yet available from the IFCC, we elected to consider the ERL material, manufactured by the Dutch SKZL Group, for this purpose. As noted above we chose the Level 2, ERL which has a HBA1c value of 7.6%, assigned by the DCCT method of Goldstein2.

 

(b)                 When considering whether to use the ERL material as a routine calibrator, or alternatively to use it to assign secondary values to routine calibrators, we firstly determined which methods were most frequently in routine use amongst laboratories. From the Glycohaemoglobin QAP Program - Cycle 11, we identified the Bio-Rad Variant I to be this instrument,  with n=37 laboratories from a total of 120. (the DCA2000 was excluded for reasons outlined above).  A recent check on the spread of methods in use in the most recent cycle (Cycle 15) shows that this situation has not changed significantly. Consequent to this determination, we ruled out the first option  of using the ERL as a routine calibrator, as  the Bio-Rad Variant 1 requires daily calibration and  the additional costs involved if this option were to be pursued would therefore be prohibitive.  Consequently  we modelled our goal of  common standardisation  via the alternative option, which entails assigning secondary values to routine calibrators, from the PRM.

 

(c)                 In terms of practicality, technical robustness and economy we considered the relative merits of four possible protocols for secondary calibrator value re-assignment from a PRM, for the Bio-Rad Variant 1. As a minimum, the

protocol selected would need to be performed at least once for every  lot.no change of column/reagent/calibrator that occurred. In the authors previous laboratory this equated to an interval of once every 2 to 3 months. The formulae proposed for the re-assignment process for each protocol are shown in Fig.3

 

·         Protocol-1 Involves running the ERL as a sample (in quadruplicate) in one analytical run, removing the

lowest and highest values and averaging the two remaining central values.  The protocol has the advantages of being easy and economical to perform, but suffers from the disadvantage of  only compensating for error associated with “with-in run” variation. It could therefore end up introducing bias, by not accounting for random between-day variation. Ultimately this could add to between lab variation.

 

·         Protocol-2 This is essentially the same as Protocol-1, but involves the additional steps of repeating the process

over four other days.  The protocol would therefore compensate for both within and between run variation. It suffers however, from the disadvantages of  being  time consuming and potentially more costly due to  the potential for a four fold increase in the amount of ERL required. On this issue, the factors associated with possible approaches to ERL handling also need consideration. For instance, three possible handling approaches might include (I) making the ERL up fresh before each run, (ii) using the same aliquot over the 4 successive assays or (iii) using frozen aliquots of ERL made from the one preparation. .It is believed  that inconsistencies at this level may impact on contributing to increased inter-laboratory variation due to the inherent-bias’s of each approach.

 

·         Protocol-3 This involves running a set of patient samples (n= 20 –> 30) with an even spread of HBA1c values

in the range of 6 to 14%, over two analytical runs. In the first run the routine calibrator would be used for calibration of the assay (run-a). The run would then be repeated using the ERL as the calibrator for the assay (run-b). Results for run-a would then be plotted against those from run-b and the resulting regression line equation would be used to re-assign the calibrator value.

 

This protocol has the advantages in that it uses real patient response data in the reassignment process as well as the primary reference material. As for protocol-1 however, it suffers from the disadvantage of  only compensating for error associated with “with-in run” variation.  It is also relatively time consuming, complex and costly (ie. consider cost/test x 60 additional tests) 

 

·         Protocol-4 This protocol is essentially the same as Protocol-3, but involves the additional steps of repeating the process over four other days.  The regression plot would therefore combine the data accumulated over the four runs.  In addition to the advantages noted in protocol–3, this protocol has the added advantage of compensating for both within and between run variation

 

It suffers from similar disadvantages in terms of complexity, cost and time consumption as protocol-3, but to a greater degree . In addition the issues of ERL handling between runs (as noted for protocol-2) must also be considered.

 

 

Overall then , considering that time, cost and process complexity are key issues of concern to all laboratories we could not for-see any broad acceptance of  the protocols, with the exception of protocol-1 and possibly protocol-2.   

 

Protocol

Secondary Calibrator value re-assignment.

 

 

Protocol-1

Reassigned = Inst.Cal.Value x ( ERL.Assigned.Value / ERL.Achieved.Value)

Protocol-2

Reassigned = Inst.Cal.Value x ( ERL.Assigned.Value / ERL.Achieved.Value from 4 runs )

Protocol-3

Reassigned = ( Inst.Cal.Value – intercept run-a vs. run-b ) x ( 1 / slope run-a vs. run-b )

Protocol-4

Reassigned = ( Inst.Cal.Value – intercept run-a vs. run-b  (4days data) ) x ( 1 / slope run-a vs. run-b  (4days data) )

 

Fig 3.      The formulae devised for the re-assignment process for each secondary calibrator value re- assignment protocol 1 -> 4. (Note: Inst.Cal.Value = The value assigned to the instruments regular calibration material)

 

 

6.0        Prediction of Expected outcomes using data-modelling

 

An area of interest to our group was to also determine the error we might actually end-up introducing (both within and between our laboratories), as a result of re-assigning calibrator values from a primary calibrator. We modelled this for the Bio-Rad Variant 1, using the first protocol and took into consideration, the known between day assay CV ( estimated here conservatively as a CV=2%, the expected CV of a good laboratory), the insert value of the Variant calibrator, the insert value of the ERL material and the  values we might expect to achieve if we assayed the ERL as a sample four times in a routine assay.

 

Figure 4, shows the range of re-assigned variant calibrator value assignments we might achieve ( min= 8.43, max= 9.14 avg = 8.77 ) based on the average, minimum and maximum ERL recoveries ( 7.49 and 8.11 respectively)  we could expect. These are in turn  based on the known between run CV of the assay.

 

Extending this model further, we estimated the level of imprecision we could  potentially introduce to the process,  by  examining the influence of the minimum, maximum and average re-assigned variant calibration values on a theoretical set of patient data. A range of patient values (n=35) have been plotted in figure 5 under the three different calibrator re-assignments. The within run imprecision component has also been added to these graphs by means of error bars (representing a conservative within run CV of 2%).

 

 

                          Error Modelling.

 

1.  Known between day assay CV                            2.0%  (SD=0.156 at HBA1c = 7.8)

2.  Variant calibrator value                                      9.0

3.  ERL (DCCT value)                                              7.6

4.  Simulated achieved ERL values                          7.8,7.8,7.9,7.7

 

        - Average of 2 central values                            7.8

 

5.  Allow for influence of between day CV on the value achieved  for ERL value, to estimate

     the min. and max. ERL values achievable, based on the know  “day-to-day” CV.

 

      -  Max.= Avg +2sd  = 7.8 + ( 2 x 0.156)   = 8.11    }

      -  Min =  Avg -2sd   = 7.8 –  (2 x 0.156)   =  7.49    }    Expected range of ERL Cal  values

 

6. Calculate the range of achievable re-assigned variant Calibrator values, based on the Min, 

    Max and Average ERL values;

 

      -  Max  =  9.0 x ( 7.6 /  7.49 )       =  9.14        }

      -  Avg  =   9.0 x ( 7.6 /  7.80 )       =  8.77        }   <- Expected range of  re-assigned

      -  Min   =  9.0 x ( 7.6 /  8.11 )       =  8.43        }       Variant Calibrator values

.

          Fig. 4   Modelling error associated with secondary calibrator value assignment

 

 

 

 

Fig 5.  Modelled results for n=36 patient results, using the 3 extreme calibrator    

       Reassignment values (min,max,avg) , 2sd error bars represent within- run CV 

 

As can be seen from this graph, at a HBA1c level of  9.0 %, the HBA1c values could range from 8.24 – 9.66% (or the equivalent of a between run CV of approximately 3.94%). At a HBA1c level   of  7.0%, the HBA1c values could range from 6.42 – 7.58% (or a between-run CV of  around 4.14%). This therefore represents a significant step back-wards in the context of the possible inflation of  with-in lab CV’s, from levels  of around 2.0 to 2.5%,  to levels of  up-to 4.14% for the BioRad Variant

 

 

 

 

Having put this model forward however, it is acknowledged that what is represented here is an extreme depiction of the spread of values that could be achieved;  as the distribution of errors  most  likely to be achieved would be Gausian in nature.  However the model presented still serves to illustrate the limitations that may be expected from this approach.

 

From a between laboratory perspective, when we examine the performance of the BioRad Variant 1 group on a recent Glycohaemoglobin QAP program, we note a between lab CV, of approximately 3.77%9,  which was determined at a level of  9.0%. This CV is not dissimilar to that achieved in our above model.  Protocol-1 could  therefore could not be expected to yield much  improvement to inter-laboratory  CV for already well controlled groups but could be expected to probably improve performances between various instrument groups.

 

In  the balance therefore, Protocol-1 could realise some improvements to overall between laboratory CV’s but not necessarily to specific instrument user groups  and could quite considerably inflate individual within-lab CV’s to undesirable levels, due to the introduction of medium-to-long term systematic shifts. It is acknowledged however  that  steps to reduce error associated with day-to-day variation  in the secondary calibrator re-assignment process (such as in protocol-2), would clearly play a role in improving performances further

 

 

 

7.0        Conclusions

 

The SAQCC in conjunction with the RCPA-AACB Chemical Pathology QAP, has responded to a call for tighter performances for HBA1c testing by undertaking a detailed investigation into the relative merits of  common standardisation. In our study we noted good improvement to inter-laboratory bias when a common standard was used.  This was no where near as significant however when results for the DCA2000 were excluded. This is an important consideration given that the DCA2000, is a closed systems as far as end-user re-standardisation is concerned, yet makes up a considerable portion of methods in use.

 

Regrettable also was that none of the protocols examined would show adequate corrective influence for intercept bias,  for methods that employed only a one point calibration (eg Bio-Rad Variant-1 ). Such methods would only be compensated for in the area of slope bias. Only methods that employ  true two-point calibrations can achieve intercept correction by the protocols  suggested.  This may restrict the expected corrective outcomes between methods, considering the number of methods on the QAP Program that show an intercept bias and also use a single point calibration in their processes.

 

For some laboratories lot number changes for reagent, columns and calibrators occur relatively frequently at intervals as often as three to four times a year, especially amongst those employing just-in-time management practices with respect to reagent stocks. This means that secondary calibrator re-assignment would need to occur rather frequently. Associated with this, would be the need to consider the added costs, the requirement for close supervision of  lot changes and  the requirement for staff to develop and pass on skills in this area. In addition such an approach, due to its tediousness and relative complexity may indeed lead to the introduction of further between lab bias, through the introduction of undesirable systematic shifts if not performed consistently and vigilantly by laboratories.

 

There is also the added challenge of convincing all laboratories to participate in such a venture. Even in the best case scenario where all-labs subscribe to this idea, such an exercise would take time to implement and benefits would not be realised for some-time.. It would also be a considerable challenge to expect long term ongoing compliance in this area for laboratories concerned.

 

Not withstanding the above challenges however, the international community has shown good clinical cause to standardise HBA1c assays to a common accuracy base, in order to realise the many-fold benefits of better diabetic patient management. An alternative, more robust and realistic means of achieving this would be to discuss the issue of standardisation with all the instrument and reagent manufacturers and cultivate and promote the need for the standardisation of  their assays against a common primary calibrator, traceable to the DCCT method of  Goldstein. This would also be a potential fix for instruments such as the DCA2000.

 

Therefore for all the above stated reasons it is the opinion of the SAQCC that the alternative of industry involvement and consultation,  would most certainly realise our goals of common standardisation and would do so without all of the inherent pitfalls, of venturing on the path of unilateral secondary calibrator value re-assignments across all laboratories.

 

 

 

 

 

 

10.0  Bibliography.

 

1                    Diabetes Australia Website: Diabetes

 http://www.diabetesaustralia.com.au/

 

2                    The Diabetes Control and Complications Research Group.

The effect of Intensive Treatment of Diabetes on the Development and Progression of Long-Term Complications in Insulin-Dependent Diabetes Mellitus.

The New England Journal of Medicine – Sept 30, 1993 – Vol.329, No 14

 

3                    American Diabetes Association.

ADA Position Statement.

 – Diabetes Care 20 ( Supp.1) S5-S11. 1997

 

4                    SA Diabetes Strategy Laboratory Working Group, South Australia. G.Phillipov,P.Phillips, K.Rowland, N.Walmsley, E.Whitham, R.Abbott, G.White 

Standardisation of Glycohaemoglobin Reporting.

ADS & ADEA Annual Scientific Meeting 23-25 August 2000 Cairns.

 

5                     Bodor, GS, Little, RR, Garrett, N, Brown, W, Goldstein, DE, Nahm, MH
Standardization of glycohemoglobin determinations in the clinical laboratory: three years of experience

Clin Chem 1992 38: 2414-2418

 

6          Weykamp, CW, Penders, TJ, Muskiet, FA, van der Slik, W
Effect of calibration on dispersion of glycohemoglobin values determined by 111 laboratories using 21 methods
Clin Chem 1994 40: 138-144

 

7.         Little, RR, Wiedmeyer, HM, England, JD, Wilke, AL, Rohlfing, CL, Wians, FH, Jr, Jacobson, JM, Zellmer, V, Goldstein, DE
Interlaboratory standardization of measurements of glycohemoglobins
Clin Chem 1992 38: 2472-2478

 

8.                  IFCC Website:

http://www.ifcc.org/divisions/SD/projects.htm

 

9.         Inter-lab CV derived for Variant 1 Group, on the RCPA-AACB Chemical Pathology QAP Glycohaemoglobin Program – Cycle 15, as determined from the Sy statistical measure, from a regression of  41 Variant 1 laboratories.