Validação de método in vitro para irritação cutânea

Prévia do material em texto

Regulatory Toxicology and Pharmacology 117 (2020) 104725
Available online 5 August 2020
0273-2300/© 2020 Elsevier Inc. All rights reserved.
Me-too validation study for in vitro skin irritation test with a reconstructed 
human epidermis model, KeraSkin™ for OECD test guideline 439 
Juhee Han a, Seolyeong Kim b, Su-Hyun Lee b, Jin-Sik Kim c, Yu Jin Chang d, Tae-Cheon Jeong e, 
Mi-Jeong Kang e, Tae-Sung Kim f, Hae Seong Yoon f, Ga Young Lee f, SeungJin Bae a,**, 
Kyung-Min Lim a,* 
a College of Pharmacy, Ewha Womans University, Seoul, Republic of Korea 
b Biosolution Co, Seoul, Republic of Korea 
c COSMAX Korea, Seongnam-si, Republic of Korea 
d Korea Conformity Laboratories, Incheon, Republic of Korea 
e College of Pharmacy, Yeungnam University, Gyeongsan, Republic of Korea 
f National Institute of Food and Drug Safety Evaluation, Ministry of Food and Drug Safety, Osong, Republic of Korea 
A R T I C L E I N F O 
Keywords: 
KeraSkinTM 
Reconstructed human epidermis 
OECD TG 439 
Reproducibility 
Predictive capacity 
Skin irritation 
A B S T R A C T 
We conducted a me-too validation study to confirm the reproducibility, reliability, and predictive capacity of 
KeraSkin™ skin irritation test (SIT) as a me-too method of OECD TG 439. With 20 reference chemicals, within- 
laboratory reproducibility (WLR) of KeraSkin™ SIT in the decision of irritant or non-irritant was 100%, 100%, 
and 95% while between-laboratory reproducibility (BLR) was 100%, which met the criteria of performance 
standard (PS, WLR≥90%, BLR≥80%). WLR and BLR were further confirmed with intra-class correlation (ICC, 
coefficients >0.950). WLR and BLR in raw data (viability) were also shown with a scatter plot and Bland-Altman 
plot. Comparison with existing VRMs with Bland-Altman plot, ICC and kappa statistics confirmed the compat-
ibility of KeraSkin™ SIT with OECD TG 439. The predictive capacity of KeraSkin™ SIT was estimated with 20 
reference chemicals (the sensitivity of 98.9%, the specificity of 70%, and the accuracy of 84.4%) and additional 
46 chemicals (for 66 chemicals [20 + 46 chemicals, the sensitivity, specificity and accuracy: 95.2%, 82.2% and 
86.4%]). The receiver operating characteristic (ROC) analysis suggested a potential improvement of the pre-
dictive capacity, especially sensitivity, when changing cut-off (50% → 60–75%). Collectively, the me-too vali-
dation study demonstrated that KeraSkin™ SIT can be a new me-too method for OECD TG 439. 
1. Introduction 
The European Commission has prohibited animal experiments for 
cosmetics in 2013 and has been working on developing alternative test 
methods to replace animal testing based on the ‘3R’s Principle’ (Adler 
et al., 2011). Since the EU ban on animal testing for cosmetics, more 
than 37 countries worldwide have legally prohibited animal experi-
ments for the development of cosmetics (Akbarsha and Mascarenhas, 
2019). In addition, there is an increasing demand for new approach 
methodologies (NAMs) to evaluate the safety of numerous chemicals on 
human health in various sectors (Parish et al., 2020). In vitro skin irri-
tation test, OECD TG 439, has been developed to replace OECD TG 404 
(OECD, 2002) in which rabbits are used as a test species to evaluate the 
skin irritancy of chemicals and cosmetic products (Kose et al., 2018; 
Park et al., 2018). OECD TG 439 uses a reconstructed human epidermis 
(RhE) (OECD, 2015a) and can be used as a stand-alone to identify UN 
GHS No category chemical, i.e., non-irritant or in combination with 
other replacement methods, such as OECD TG 435 “In Vitro Membrane 
Barrier Test Method for Skin Corrosion” (OECD, 2015b) to further 
classify the hazard on skin in the framework of Integrated Approach on 
Abbreviations: OECD, The Organisation for Economic Co-operation and Development; TG, Test guideline; PS, Performance Standards; VRM, Validated Reference 
Method; SIT, skin irritation test; WLR, Within-laboratory reproducibility; BLR, Between-laboratory reproducibility; RhE, Reconstructed human Epidermis; CV, cell 
viability; I, irritant; NI, non-irritant; GHS, Globally Harmonized System of Classification and Labelling of Chemicals. 
* Corresponding author. 
** Corresponding author. 
E-mail addresses: sjbae@ewha.ac.kr (S. Bae), kmlim@ewha.ac.kr (K.-M. Lim). 
Contents lists available at ScienceDirect 
Regulatory Toxicology and Pharmacology 
journal homepage: www.elsevier.com/locate/yrtph 
https://doi.org/10.1016/j.yrtph.2020.104725 
Received 4 March 2020; Received in revised form 25 June 2020; Accepted 2 July 2020 
mailto:sjbae@ewha.ac.kr
mailto:kmlim@ewha.ac.kr
www.sciencedirect.com/science/journal/02732300
https://www.elsevier.com/locate/yrtph
https://doi.org/10.1016/j.yrtph.2020.104725
https://doi.org/10.1016/j.yrtph.2020.104725
https://doi.org/10.1016/j.yrtph.2020.104725
http://crossmark.crossref.org/dialog/?doi=10.1016/j.yrtph.2020.104725&domain=pdf
Regulatory Toxicology and Pharmacology 117 (2020) 104725
2
Testing and Assessment (IATA) for Skin Corrosion and Irritation (OECD, 
2017). 
Performance standards (PS) of OECD TG 439 for the assessment of 
similar and modified RhE-based test methods, OECD guidance docu-
ment 220 (OECD, 2015b) was developed to encourage the introduction 
of new ‘‘me-too’’ RhE models and methods. At the time of original 
adoption of OECD TG 439 (OECD, 2010), EpiDerm™ SIT and EpiSkin™ 
SIT were only two validated reference methods (VRMs). Since then, 
various me-too skin irritation tests like SkinEthic™ RhE SIT and Labcyte 
EPI-MODEL24 SIT were developed, validated and approved for OECD 
TG 439 in 2013. In 2019, OECD TG 439 was updated with two additional 
me-too test methods, epiCS® and Skin+® (OECD, 2019a; 2019b). 
KeraSkin™ is a newly developed RhE using human primary kerati-
nocytes to evaluate skin irritation of chemicals (Jung et al., 2014). 
Several studies have demonstrated that KeraSkin™ can be applied to the 
skin toxicity in various settings (Choi et al., 2014; Hwang et al., 2018; 
Jung et al., 2014; Kim et al., 2017; Lee et al., 2016, 2017). Here, we 
conducted a me-too validation test to confirm the reproducibility, and 
predictive capacity of KeraSkin™ SIT as a me-too method’ of OECD TG 
439 in accordance of OECD TG 439 PS with 20 reference chemicals. 
Additional 46 chemicals were tested to further evaluate the predictive 
capacity of KeraSkin™ SIT. 
In addition to the evaluation of WLR, BLR and predictive capacity 
based on proportion of concordance of decisions, various statistical 
methods have been used to assess the WLR and BLR, and compatibility 
to existing VRMs (Alépée et al., 2016; Lim et al., 2019a, 2019b). 
Intra-class Correlation Coefficient (ICC) is a commonly used indicator to 
assess reproducibility and is an estimate of the fraction of the total 
measurement variability caused by individual appreciator variation 
(Szklo and Nieto, 2014). It was also used in the analysis of WLR and BLR 
(Hirschmann et al., 2011). ICC values less than 0.5 are indicative of poor 
reliability, values between 0.5 and 0.75 indicate moderate reliability, 
values between 0.75 and 0.9 indicate good reliability, and values greater 
than 0.90 indicate excellent reliability (Koo and Li, 2016). 
The Bland-Altman plot is also recommended for comparative studies 
by showing the relationship between the mean and the difference in 
values obtained from each test result. It is very useful in examining 
patterns of inconsistencies between measurements (Atkinson, 1998). 
Bland-Altman plot shows three horizontal lines parallel to the x-axis, 
representing the mean differences between the measured values, and 
±95% limit of agreement. Bland-Altman plot was recommended that 
95% of the data points should lie within ±95% limit of agreement 
(Giavarina, 2015). In addition, Cohen’s kappa statistics is often used to 
measure the concordance between two observers for normal scales 
(Berry et al., 1988; Blackman, 2000). Kappa statistics can vary from − 1 
to 1 (Blackman, 2000; Landisand Koch, 1977). The kappa value less 
than 0.00 indicates poor, 0.00–0.20, slight, 0.21–0.40, fair and 
0.41–0.60, moderate, 0.61–0.80, substantial and 0.81–1.00, almost 
perfect concordance (Landis and Koch, 1977). 
For the assessment of optimal cutoffs, receiver operating character-
istic (ROC) curve analysis is an effective method to assess quality or 
performance (Park et al., 2004; Yang et al., 2017). In addition, to pro-
vide 95% confidence interval for predictive capacity with a small sample 
number, Wilson’s score confidence interval was used (Alépée et al., 
2016). With the various statistical analysis, we could demonstrated that 
KeraSkin™ SIT exhibits good reproducibility and predictive capacity 
that can meet the criteria of OECD TG 439 PS, which could contribute to 
reducing animal tests by providing with another new method to replace 
the rabbit skin irritation test. 
2. Material and methods 
2.1. Chemicals and reagents 
The 20 reference chemicals as suggested in the OECD TG 439 per-
formance standards were used as described in Table 1. 20 chemicals 
consisted of 10 non-irritants (top 10 in Table 1) and 10 irritants (bottom 
10 in Table 1) according to the UN GHS category, of which 4 were solids 
and 16, liquids. Also, 46 chemicals (19 solids and 27 liquids) were 
selected from previous validation study reports and papers to further 
Table 1 
20 reference chemicals recommended from GD 220 published in 2015. 
No. Chemicals CASN UN GHS Cat. Physical 
Status 
Storage 
condition 
Manufacturer Cat No. Purity 
(%) 
Non-classified 
chemicals 
1 1-Bromo-4-chlorobutane 6940-78- 
9 
No Cat. liquid RT Sigma B60800 99.0 
2 Diethyl phthalate 84-66-2 No Cat. liquid RT Sigma 524972 99.9 
3 Naphthalene acetic acid 86-87-3 No Cat. solid RT Sigma N0640 97.0 
4 Allyl phenoxy-acetate 7493-74- 
5 
No Cat. liquid RT Sigma W203807 99.8 
5 Isopropanol 67-63-0 No Cat. liquid RT Sigma W292907 100.0 
6 4-Methyl-thio-benzaldehyde 3446-89- 
7 
No Cat. liquid RT Sigma 222771 98.9 
7 Methyl stearate 112-61-8 No Cat. solid RT Sigma 85769 99.2 
8 Heptyl butyrate 5870-93- 
9 
No Cat. (Optional 
Cat. 3) 
liquid RT Sigma W254908 99.4 
9 Hexyl salicylate 6259-76- 
3 
No Cat. (Optional 
Cat. 3) 
liquid RT Sigma W520306 99.5 
10 Cinnamaldehyde 104-55-2 No Cat. (Optional 
Cat. 3) 
liquid RT Sigma W228613 99.1 
Classified 
chemicals 
11 1-Decanol 112-30-1 Cat. 2 liquid RT Sigma 239763 99.1 
12 Cyclamen aldehyde 103-95-7 Cat. 2 liquid RT TCI I0377 97.8 
13 1-bromohexane 111-25-1 Cat. 2 liquid RT Sigma B68240 99.3 
14 2-Chloromethyl-3,5-dimethyl-4- 
methoxypyridine HCL 
86604- 
75-3 
Cat. 2 solid RT Sigma 535508 ≥99.9 
15 Di-n-propyl disulphide 629-19-6 Cat. 2 liquid RT Sigma 149225 97.8 
16 Potassium hydroxide(5% aq.) 1310-58- 
3 
Cat. 2 liquid RT Sigma 417661 – 
17 Benzenethiol, 5-(1,1-dimethylethyl)- 
2-methyl 
7340-90- 
1 
Cat. 2 liquid RT ACROS 279960500 93.9 
18 1-Methyl-3-phenyl-1-piperazine 5271-27- 
2 
Cat. 2 solid RT AK scientific A083 99.0 
19 Heptanal 111-71-7 Cat. 2 liquid RT Sigma W254002 97.7 
20 Tetrachloroethylene 127-18-4 Cat. 2 liquid RT Sigma 371696 100.0 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
3
assess predictive capacity. In total, 66 chemicals composed of 23 solids 
and 43 liquids (21 irritants and 45 non-irritants) were tested in further 
evaluating the predictive capacity of KeraSkin™ SIT. Participating lab-
oratories (Biosolution Co, Seoul, Korea, COSMAX Korea, Seongnam-si, 
Korea and Korea Conformity Laboratories, Incheon, Korea) tested all 
the substances under the blind conditions. 
2.2. KeraSkin™ SIT protocol 1.5 
Overall procedure of KeraSkin™ SIT was described in Fig. 1. Original 
protocol of KeraSkin™ SIT was developed in 2012, which was estab-
lished through adopting the existing VRMs of OECD TG 439. Five times 
of revisions were made to optimize KeraSkin™ SIT protocol and to 
correct mis-predictions for some PS chemicals with respect to washing 
procedure (employment of squeeze bottle and cotton-swab), removal of 
mesh application, treatment volume (from 30 to 40 μL) and treatment 
time (from 40 min to 30 min). 
Viscous substance sometimes made the mesh stick to the tissue even 
after the washing process, so the mesh was removed. Instead the treat-
ment volume was increased (30 μL→40 μL) to evenly distribute the 
substance on the tissue and the treating time (45 min→30 min) was 
shortened to compensate the increased treatment volume. Post- 
incubation time of 42 h is the same as the existing VRMs. 
Briefly, the KeraSkin™ model, which passed the quality control 
[optical density values of negative control (NC) by MTT (3-(4,5-dime-
thylthiazol-2-yl)-2,5-diphenyltetrazolium bromide) assay: 0.5–1.2, 
ET50: 6.9.0–14.0 h, IC50 for sodium dodecyl sulfate (SDS) 1.5–4.7 mg/ 
mL and histological examination] of the manufacturer (Biosolution Co. 
Seoul, Korea) was delivered to the testing labs in a 24-well format on an 
agarose gel at about 25 ◦C. Actual QC data for the 13 batches used for the 
main study was NC OD: 0.9 ± 0.1, ET50: 10.7 ± 1.5, IC50: 3.3 ± 0.4 
(Mean ± SD). It was confirmed through an oversea shipment that Ker-
aSkin™ was stable up to 56 h of delivery as determined by quality 
control criteria of NC OD and IC50 for SDS (data not shown). Upon 
receipt of the shipment, culture medium provided by the manufacturer 
was pre-warmed in a 37 ◦C thermostat for 30 min. As for the pre- 
incubation step, 900 μL of the pre-warmed medium was added to each 
well of a 6-well plate using micropipette and the RhE model insert was 
carefully transferred to a well using forceps. Then the well-plate was pre- 
incubated at 37 ◦C, 5% CO2 for 22 ± 2 h. To conduct the experiment, 40 
Fig. 1. KeraSkinTM SIT protocol 1.5. Overall KeraSkin™ skin irritation test (SIT) procedure. CV, cell viability, NI, non-irritant. 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
4
μL of liquid substance or solution, or 40 mg (with 40 μL DPBS) of solid 
substance was topically applied on the upper epithelial surface of the 
model insert (0.6 cm2) after the pre-incubation. The tissue was incu-
bated again (37 ◦C, 5% CO2 condition) for 30 ± 1 min (Fig. 2). Then the 
tissue was washed to remove the test substances and further incubated 
(37 ◦C, 5% CO2 condition) for 42 ± 2 h. The resulting tissue viability was 
determined by MTT assay. 
2.2.1. Washing 
Washing procedure for the 6-wells should be done within 20 min. 
DPBS for washing should be kept warm in a 37 ◦C thermostat before use. 
The treated tissue was removed from a 37 ◦C, 5% CO2 incubator and 
DPBS was flowed on the side wall of the insert using a poly wash bottle, 
and when DPBS was fully filled in the insert, DBPS in the insert was 
poured to a beaker at once. Washing was repeated 10 times in total. 
However, if residual material (i.e. oils or highly viscous liquid materials, 
and solid materials with gelation) was still observed after 5 times of 
wash, a cotton swab was used to wipe off the residual materials. After 
washing, DPBS outside of the insert were removed using a sterile gauze. 
2.2.2. MTT assay 
After post-incubation, the media inside and outside of inserts were 
removed with a pipette. Inserts were transferred into 24-well plates, 
prefilled with 200 μL of 0.4 mg/mL MTT per a well. Then, 100 μL of MTT 
solution was further added to the inside of insert. Then the plate was 
covered with aluminum foil to protect from the light and placed in an 
incubator (37 ◦C, 5% CO2) for 3 h ± 5 min. Afterincubation, MTT so-
lution remaining was removed from the inside/outside of the insert 
using 200 μL pipette and the tissues were transferred to a new 6-well 
plate containing 1.9 mL/well of isopropanol. Then additional 100 μL 
of isopropanol was added to the inside of the insert. Plate was covered 
from the light using aluminum foil and shaken on a plate shaker for 3 h 
± 5 min to extract formazan. After formazan extraction, the extracts 
inside and outside of the insert were collected and mixed well by 
pipetting such that formazan crystals were dissolved clearly. 250 μL of 
extract per a well was transferred into a 96 well plate, for the optical 
density (OD) measurement at 570 nm. 
2.2.3. Detecting and correcting interference from colored chemicals and 
MTT reducers 
In order to prevent the possible interference with the MTT reading by 
colored chemicals, i.e., ‘direct staining’ or ‘MTT reacting’ chemicals, 
were pre-checked. Though, when a test substance is determined to be an 
irritant, the correction shall be not necessary since correction always 
over-predict the classification of skin irritancy by subtracting the cell 
viability. Accordingly, the color interference corrections may be 
reserved as optional for the test chemicals with significant concerns. 
Overall scheme for correcting color interference could be advised as 
described in Fig. 2. 
Colored test substance may interfere with OD measurement of MTT 
formazan by directly staining the tissue. 0.5 mL of deionized water and 
40 μL (liquid) or 40 mg (solid) of the test chemical were added to each 
Eppendorf tube and the mixture was incubated in a cell incubator (37 ±
1 ◦C, 5 ± 1% CO2, 95% RH) for 30 min. The color changes were 
observed, and if there was a significant color change, correction for color 
interference with the viability value calculated from OD of the staining 
control [viability value of a tissue incubated with phenol red free-DMEM 
instead of MTT solution (%NSCliving)] was necessary. 
Fig. 2. Color interference post-checking scheme for KeraSkin™ SIT. CV, cell viability, %TT, Uncorrected viability value, %NWliving, viability value of a tissue 
incubated with phenol red free-DMEM instead of MTT solution, %NSMTT, viability value of reactivity control obtained with OD from the test done with freeze-killed 
tissues, %NSCKilled, viability value of freeze-killed tissue incubated with phenol red free-DMEM instead of MTT solution. 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
5
Table 2 
Within- and between-laboratory reproducibility of the Keraskin™ SIT test results. 
Fig. 3. Scatter plot of 3 runs for 20 PS reference chemicals by 3 laboratories. Cell viability data from the KeraSkin™ SIT. Scatter plot with the mean± ​ SD (error bar) 
of 3 runs for each lab. Biosolution (BS), KCL and COSMAX. 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
6
A test substance with MTT reacting activity may reduce MTT directly 
and form formazan alone. 0.5 mL of MTT solution (0.4 mg/mL) and 40 
μL (liquid) or 40 mg (solid) of the test chemical were added to an 
Eppendorf tube and the mixture was incubated in a cell incubator (37 ◦C, 
5% CO2, 95% RH) for 1 h. The color change was observed and if the 
color of MTT solution changes to blue/purple, correction for color 
interference with the viability value of reactivity control obtained with 
OD from the test done with freeze-killed tissues (%NSMTT) was 
necessary. 
If a test substance was both direct-staining and MTT-reacting, then 
the correction needs to be done for both direct-staining and MTT reac-
tivity, but the color interference would be subtracted twice. To resolve 
this, another staining control with freeze-killed tissues [viability value of 
freeze-killed tissue incubated with phenol red free-DMEM instead of 
MTT solution (%NSCKilled)] is necessary. 
2.2.4. Prediction model for the KeraSkin™ SIT 
According to the EU and the GHS classification, an irritant is pre-
dicted if the mean relative tissue viability of minimum three replicate 
tissues exposed to the test chemical is reduced to 50% or below of the 
mean viability of the negative controls. The test chemical was defined as 
a non-irritant if the tissue viability was higher or equal to 50%. Other-
wise, it was determined as an irritant. Since 50% cut-off value is adopted 
by all existing VRMs of OECD TG 439, KeraSkin™ SIT protocol has been 
optimized to 50% cut-off. 
2.2.5. Acceptance criteria 
A test is regarded as acceptable unless the results do not fall into the 
criteria for a re-test, which is adopted for KeraSkin™ SIT from OECD TG 
439, 
1) If absorbance value of the negative control chemical is less than 0.7 
or exceeds 1.6. 
2) If a cell viability value of the positive control chemical is higher than 
40%. 
3) In negative control chemical, positive control chemical, and test 
chemical: When the standard deviation of the cell viability value of 
treated tissue in replications exceeds 18%. 
4) If the average cell survival rate of the chemical-treated-well is be-
tween 45% and 55% (≥45% and ≤55%, borderline chemicals). 
If the re-test gives borderline value again, the concordance of the 
decision should be considered. If the original and re-test are concordant 
in decision, additional re-test is not necessary. If the decision is different, 
then the third test should be conducted and majority vote approach 
should be taken for final decision. Acceptance criteria for KeraSkin™ SIT 
were established referring to existing VRMs of OECD TG 439. 
2.3. Statistical analysis for the reliability of the KeraSkin™ SIT 
2.3.1. Data collection and chart 
Cell viability data were obtained for 20 reference chemicals from a 
total of three repetitions of KeraSkin™ SIT by three laboratories (total 9 
runs of data for a chemical were obtained). 46 additional chemicals were 
tested by two laboratories, Biosolution and KCL (One or two runs of data 
for a chemical was obtained). All participating laboratories performed 
the tests and quality assurance in the spirit of Good Laboratory Practice 
(GLP). The cell viability was shown in the table as mean± ​ SD. Also, the 
results were plotted on a scatter plot to show the data distribution at a 
glance, with the cut-off line to identify incorrect decisions. 
2.3.2. Within-laboratory and between-laboratory reproducibility 
Within-laboratory reproducibility (WLR) is an evaluation of the 
consistency of the results of three replicate tests conducted by a labo-
ratory and between-laboratory reproducibility (BLR) is an evaluation of 
the consistency between the three laboratories using the mean value of 
results from different independent laboratory tests of the reference 
chemical. WLR and BLR for 20 chemicals were presented as a 
percentage. 
WLR and BLR were also expressed in confidence intervals using 
Wilson’s score confidence interval method to consider the uncertainty in 
the value of the point estimate presented. ICC and Bland-Altman 
methods were also used for the reproducibility analysis. The formula 
for the ICC is, 
ICC=
Vb
VT
=
Vb
Vb + Ve 
in which Vb = variance between individuals, VT = total variance, which 
includes both Vb and Ve [Ve = unwanted variance (“error”)] (Szklo and 
Nieto, 2014). There are two types of ICC: Single measures ICC is an index 
for the reliability of the ratings for one, typical, single rater while 
average measures ICC is of different raters averaged together. (Shrout 
and Fleiss, 1979) The average measures ICC is always higherthan the 
single measures ICC. (Bartko, 1976) 
2.3.3. Correlation with validated reference methods 
In order to determine the compatibility of KeraSkin™ SIT with 
validated reference methods (VRMs) of OECD TG 439; EpiDerm™ SIT, 
EpiSkin™, and LabCyte EPI-MODEL24 SIT, ICC analysis, Bland-Altman 
plot and Cohen’s kappa analysis were used to evaluate the agreement. 
2.4. Predictive capacity of the KeraSkin™ SIT 
2.4.1. 20 chemicals and 46 additional chemicals for the evaluation of 
predictive capacity 
The mis-predicted test/total was used to determine the predictive 
capacity (Alépée et al., 2016) where in all test results of each laboratory 
were used. The judgment are classified as irritant (I) and non-irritant 
(NI) and concordance of the reference values and test results was 
shown in 2 × 2 contingency table. The result of 2 × 2 contingency table 
judgment was calculated with various predictive power indicators 
(Cooper and Saracci, 1979). Predictive capacity was calculated as an 
index of sensitivity, specificity, and accuracy. The results were presented 
as point estimates and 95% confidence intervals were calculated using 
Wilson’s score confidence interval. According to the OECD TG 439 
Performance Standards, the sensitivity, specificity, and accuracy shall be 
80%, 70%, and 75%, respectively or higher. 
The ROC curve was used to evaluate the predictive capacity of Ker-
aSkin™ SIT. The additional predictive capacity assessment consisted of 
Table 3 
Intra-class correlation coefficient of the test results of 3 repetitions in 3 labs. 
Intra-class 
correlation 
coefficient (ICC) 
95% confidence 
interval 
Lower 
bound 
Upper 
bound 
WLR BS Single 
measures 
.971 .940 .987 
Average 
measures 
.990 .979 .996 
KCL Single 
measures 
.976 .949 .989 
Average 
measures 
.992 .983 .996 
COSMAX Single 
measures 
.957 .912 .981 
Average 
measures 
.985 .969 .994 
BLR Single 
measures 
.961 .921 .983 
Average 
measures 
.987 .972 .994 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
7
Fig. 4. Bland-Altman plots for evaluating the BLR between two laboratories. Bland-Altman plots for a) BS and KCL, b) BS and COSMAX, c) KCL-COSMAX. Dotted 
lines indicate 95% confidence limits. Blue circle represents liquid chemicals and red circle, solids. (For interpretation of the references to color in this figure legend, 
the reader is referred to the Web version of this article.) 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
8
66 chemicals with 46 chemicals added to the existing 20 chemicals. 66 
chemicals have unequal numbers of test results. To correct this, a 
weighted approach analysis was also performed for the analysis of 
predictive capacity. 
3. Results 
3.1. Within-and between-laboratory reproducibility 
To investigate whether KeraSkin™ SIT met the criteria of “me-too” 
test method evaluation stated in the performance standards (PS) for 
OECD TG 439, 20 reference chemicals were subject to three repeated 
tests by three laboratories (Table 2 and Fig. 3). Overall decisions were 
mostly consistent within and between laboratories except for Di-n- 
propyl disulphide for which Lab 3 incorrectly predicted once. Within- 
laboratory reproducibility (WLR) and between-laboratory reproduc-
ibility (BLR) in cell viability and decisions were also presented with a 
scatter plot which showed small scatters of viability data for each 
chemical except for Di-n-propyl disulphide (No. 5). In terms of concor-
dance in decision of “non-irritant” or “irritant”, Lab 1 and Lab 2 showed 
100% (20/20) and Lab 3, 95% (19/20) WLRs while BLR of three labo-
ratories was 100% (20/20). These results met the criteria of WLR ≥ 90% 
and BLR ≥ 80% of OECD TG 439 PS. 
ICC was also calculated to evaluate WLR and BLR of cell viability 
results of KeraSkin™ SIT. Both the single measures and average mea-
sures ICC of WLR for BS, KCL and COSMAX, and BLR were all above 0.9 
(Table 3), suggesting an excellent level of WLR and BLR of KeraSkin™ 
SIT. Also, Bland-Altman plot were drawn to evaluate BLR as shown in 
Fig. 4, which suggests that most of values fell within 95% agreement 
limit, and the difference between the measured values was close to zero, 
regardless of the physical states. Interestingly it could be seen that mean 
CV values aggregated around 0 or 100%, reflecting an excellent iden-
tification of irritants from non-irritants. 
3.2. Correlation of KeraSkin™ SIT with validated reference methods of 
OECD TG 439 
KeraSkin™ SIT was compared with other existing validation refer-
ence methods (VRMs) of OECD TG 439 using ICC, Bland-Altman plot 
and Kappa statistics. As shown in Table 4, the average ICC between 
KeraSkin™ SIT and Episkin™ SIT was 0.972 (95% CI: 0.929–0.989), 
EpiDerm™ SIT, 0.966 (95% CI: 0.914–0.986), and LabCyte EPI- 
MODEL24 SIT, 0.993 (95% CI: 0.983–0.997), suggesting an excellent 
correlation. Also, the Bland-Altman plot was distributed close to zero, as 
shown in Fig. 5. There was no difference between the liquids and the 
solids but viability values of KeraSkin™ SIT tended to be lower than 
VRMs. Especially, 4-methyl-thio-benzaldehyde and Di-n-propyl disul-
phide were outliers for Episkin™-KeraSkin™, and EpiDerm™- and 
LabCyte EPI-MODEL24-KeraSkin™ pairs, reflecting that KeraSkin™ SIT 
may be more sensitive. In the Kappa statistics (Table 5), Episkin™- 
KeraSkin™ showed 0.608, i.e., substantial concordance while Epi-
Derm™- and LabCyte EPI-MODEL24-KeraSkin™ were 0.828, almost 
perfect concordances. 
3.3. Predictive capacity of KeraSkin™ SIT for 20 reference chemicals 
The predictive capacity of KeraSkin™ SIT, i.e., sensitivity, specificity 
and accuracy was evaluated with three repeated tests for 20 PS chem-
icals by three laboratories (Table 6). With all 180 individual runs, mis- 
predicted test/total was calculated. The result showed the sensitivity 
of 98.9%, specificity of 70%, and accuracy of 84.4%. Wilson’s 95% 
confidence interval of sensitivity was 0.940–0.998, specificity was 
0.599–0.785 and accuracy was 0.784–0.890. 
3.4. Color interference 
All three participating laboratories performed the correction pro-
cedure for color interference (Fig. 2) for all 20 reference chemicals to 
pre-check the possibility of direct staining and MTT reactivity. No 
correction was necessary for the 20 PS chemicals in Lab 1 and Lab 2, but 
in Lab 3, decision of a chemical, Benzenethiol, 5-(1,1-dimethylethyl)-2- 
methyl was changed from non-irritant to irritant due to a substantial 
MTT reactivity as shown in Table 7. 
3.5. Predictive capacity of KeraSkin™ SIT for 66 chemicals 
Additional 46 reference chemicals (Table 8) with in vivo Draize score 
available were selected and tested to further evaluate the predictive 
capacity of KeraSkin™ SIT. Overall, data for 66 chemicals (46 + 20 PS 
chemicals) was obtained (Fig. 6). KeraSkin™ SIT showed a sensitivity of 
95.2%, specificity of 82.2% and accuracy of 86.4% for those 66 chem-
icals, which met OECD PS criteria. The predictive capacity was further 
calculated for liquids (43) and solids (23) separately, which also satis-
fied the criteria (liquids; sensitivity of 100%, specificity of 73.1% and 
accuracy of 83.7%, and solids; 75%, 94.7%, 91.3%) (see Table 9). 
The ROC curve analysis was conducted with viability data for 66 
chemicals. As shown in Fig. 7, cut-off value of 60–80% was optimum for 
solids while 45–75% was for liquids. Overall, optimal cut-off values 
were determined to be 60–75%. The AUC was estimated to be 0.7235, 
whichrepresents a reasonable level of predictive capacity (Park et al., 
2004; Youngstrom, 2014). Also, the weighted approach analysis was 
conducted to correct unequal number of test results across chemicals, 
which showed the sensitivity 94.7%, specificity 82.2%, and accuracy 
86.2% (Table 10). 
4. Discussion 
This study was conducted to determine if the KeraSkin™ SIT satisfies 
the performance standards (PS) of OECD TG 439. The assessment of 
reproducibility and predictive capacity of the KeraSkin™ SIT test 
method was performed using 20 PS chemicals and additional 46 
chemicals were tested to further assess the predictive capacity. Ker-
aSkin™ SIT has achieved a WLR of 95%–100% in all three participating 
Table 4 
Intra-class correlation coefficient of the test results between the KeraSkin™ SIT 
and validated reference methods. 
KeraSkin™ SIT Intra-class correlation 
coefficient (ICC) 
95% confidence 
interval 
Lower 
bound 
Upper 
bound 
Episkin™ Single 
measures 
.945 .867 .978 
Average 
measures 
.972 .929 .989 
EpiDerm™ SIT Single 
measures 
.934 .841 .973 
Average 
measures 
.966 .914 .986 
LabCyte 
EPI- 
MODEL24 
SIT 
Single 
measures 
.986 .966 .995 
Average 
measures 
.993 .983 .997 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
9
Fig. 5. Bland-Altman plots for evaluating the agreement between KeraSkin™ SIT and existing VRMs of OECD TG 439. Bland-Altman plots for a) EpiSkin™-Ker-
aSkin™ SIT, b) EpiDerm™ SIT-KeraSkin™ SIT, c) LabCyte EPI-MODEL24 SIT- KeraSkin™ SIT. Dotted lines indicate 95% confidence limits. Blue circle represents 
liquid chemicals and red circle, solids. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.) 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
10
laboratories and a BLR of 100% in the three participating laboratories. 
Predictive capacity of the KeraSkin™ SIT (sensitivity, specificity and 
accuracy: 98.9%, 70%, 84.4%) for 20 reference chemicals was also 
better or equal to the PS criteria (sensitivity, specificity and accuracy: 
≥80%, ≥70% and ≥75%). With additional chemicals, predictive ca-
pacity was a sensitivity of 95.2%, a specificity of 82.2% and an accuracy 
of 86.4% for 66 chemicals (20 PS chemicals + 46 additional chemicals). 
There were 4 false predictions for the non-irritant chemicals. 1- 
bromo-4-chloroethylene (no. 11), 4-methyl-thio-benzaldehyde (no. 
16), cinnamaldehyde (no. 20) were incorrectly predicted in all three 
laboratory. This is considered to be due to the characteristic of the 
materials or limitation of RhE SITs since other VRMs like EpiDerm™ SIT 
and LabCyte EPI-MODEL24 SIT methods also predicted them incorrectly 
(Kandárová et al., 2009; Katoh and Hata, 2011; Spielmann et al., 2007). 
Interestingly, KeraSkin™ SIT can predict the irritant, Di-n-propyl 
disulphide, correctly except for one run by Lab 3. Di-n-propyl disul-
phide, along with 1-decanol, is Category 2 chemical and it was 
frequently predicted as a false negative in VRMs, suggesting that Ker-
aSkin™ SIT may be more sensitive than existing VRMs. By correctly 
identifying Di-n-propyl disulphide, the sensitivity of KeraSkin™ SIT was 
higher than VRMs for 20 reference chemicals, exceeding the sensitivity 
Table 5 
Cohen’s Kappa statistics of the results between the KeraSkin™ SIT and the 
validated reference methods in OECD TG 439. 
comparison Kappa coefficient P- value 
EpiSkin™ SIT .608 .007 
EpiDerm™ SIT .828 .000 
LabCyte EPIMODEL24 SIT .828 .000 
Table 6 
Predictive capacity of KeraSkin™ SIT for 20 reference chemicals in OECD TG 
439 PS. 
PS criteria Predictive capacity 
BS (60) KCL (60) COSMAX (60) Total(180) 
Sensitivity ≥80% 100% 100% 96.7% 98.9% 
Specificity ≥70% 70% 70% 70% 70% 
Accuracy ≥75% 85% 85% 83.3% 84.4% 
Lab1; Biosolution Co., Lab2; Korea Conformity Laboratories, Lab3; COSMAX. 
Table 7 
Color Interference check for 20 PS reference chemicals in COSMAX. 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
11
criteria of PS, which was confirmed again with 66 chemicals (sensitivity 
of 95.2%). 
Comparison of KeraSkin™ SIT with VRMs of OECD TG 439 revealed 
that KeraSkin™ SIT agrees well with VRMs with respect to irritant de-
cisions and raw data (viability). Various statistical analyses were 
conducted, such as ICC, Bland-Altman plot and Kappa statistics. Pair- 
wise comparison with all VRMs using these statistical methods all sug-
gests a moderate or high correlation between KeraSkin™ SIT and 
existing VRMs. Interestingly, KeraSkin™ SIT was more similar to Epi-
Derm™ and LabCyte EPI-MODEL24 than Episkin™ as determined by 
Table 8 
List of 46 substances for further evaluation. 
No Chemicals CASN In vivo Class (In 
vivo Cat.) 
Physical 
Status 
Storage 
condition 
Manufacturer Cat No. Purity 
(%) 
1 Phenylethyl alcohol (2-Phenylethanol) 60-12-8 NI (NC) liquid RT Sigma 77861 99.7 
2 Isopropyl myristate 110-27-0 NI (NC) liquid RT Sigma 172472 98.0 
3 Triethylene glycol 112-27-6 NI (NC) liquid RT Sigma T59455 99.3 
4 4,4′-Methylenebis (2,6-di-tert-butylphenol) 118-82-1 NI (NC) solid RT Sigma 277924 99.6 
5 Benzyl benzoate 120-51-4 NI (NC) liquid RT Sigma B6630 99.8 
6 3-Chloronitrobenzene (1-Chloro-3-nitrobenzene) 121-73-3 NI (NC) solid RT Sigma 218758 99.8 
7 Isopropyl palmitate 142-91-6 NI (NC) liquid RT Sigma W515604 99.9 
8 Dodecanoic acid (Lauric acid) 143-07-7 NI (NC) solid RT Sigma W261408 98.0 
9 Sodium bicarbonate 144-55-8 NI (NC) solid RT Sigma S5761 100.0 
10 4-Amino-1,2,4-triazole (4-Amino-4H-1,2,4-triazole) 584-13-4 NI (NC) solid RT Sigma A81803 99.8 
11 3,3′-Dithiodipropionic acid 1119-62-6 NI (NC) solid RT Sigma 109010 99.8 
12 Silane A-1430 ((3-Chloro propyl) trimethoxy silane) 2530-87-2 NI (NC) liquid RT Sigma 440183 97.5 
13 2,5-Dimethyl-4-oxo-4,5-dihydrofuran-3-yl acetate (4- 
Acetoxy-2,5-dimethyl-3(2H)-furanone) 
4166-20-5 NI (NC) liquid RT TCI A2192 98.1 
14 Di-propylene glycol 25265-71- 
8 
NI (NC) liquid RT Sigma D215554 99.0 
15 Polyethylene glycol 400 25322-68- 
3 
NI (NC) liquid RT TCI N0443 – 
16 Dipropylene glycol monobutyl ether (DPnB) 
(Di(propylene glycol) butyl ether, mixture of isomers) 
29911-28- 
2 
NI (NC) liquid RT Sigma 484237 99.1 
17 Propylene glycol 57-55-6 NI (NC) liquid RT Sigma P4347 100.0 
18 1-(4-Chlorophenyl)-3-(3,4-dichlorophenyl)urea 101-20-2 NI (NC) solid RT Sigma 105937 98.9 
19 Erucamide 112-84-5 NI (NC) solid − 18 ◦C Sigma 280577 88.4 
20 Benzyl salicylate 118-58-1 NI (NC) liquid RT Sigma 84260 99.9 
21 3.3-Dimethylpentane 562-49-2 NI (NC) liquid RT ACROS 152660050 99.6 
22 Tetrabromophenol blue 4430-25-5 NI (NC) solid RT Sigma 199311 80.0 
23 2-Ethylhexyl 4-methoxycinnamate 5466-77-3 NI (NC) liquid RT Sigma 78848 98.9 
24 Sodium bisulphite 7631-90-5 NI (NC) solid RT Sigma 243973 – 
25 2-(Formylamino)-3-thiophenecarboxylic acid 43028-69- 
9 
NI (NC) solid RT Sigma S935344 100.0 
26 Methyl laurate (at 37 ◦C, liquid) 111-82-0 NI (Cat 3) liquid RT Sigma 234591 99.9 
27 Benzophenone-3 131-57-7 NI (NC) solid RT Sigma H36206 99.5 
28 3-Chloro-4-fluoronitrobenzene (below 25 ◦C, solid) 350-30-1 NI (NC) solid RT Sigma 233234 99.2 
29 Cyclohexadecanone (below 37 ◦C, solid) 2550-52-9 NI (NC) solid RT Symrise 600359 – 
30 2-Phenylhexanenitrile3508-98-3 NI (Cat 3) liquid RT IFF 190979 – 
31 2-Ethylhexyl-4-aminobenzoate 26218-04- 
2 
NI (NC) solid 4 ◦C Santacruz SC-496103 96.0 
32 Capryl-isostearate 209802- 
43-7 
NI (NC) liquid RT Nikko chemical S028795 100.0 
33 Diisopropyl sebacate 7491-02- 
03 
NI (NC) liquid RT Nippon Fine 
Chemicals 
– 100.0 
34 Barium sulfate 7727-43-7 NI (NC) solid RT/ 
hygroscopic 
ACROS 222515000 99.2 
35 10% Xanthan gum (in D.W.) 11138-66- 
2 
NI (NC) gel RT/ 
hygroscopic 
Sigma 43708 – 
36 1-Bromopentane 110-53-2 I (Cat 2) liquid RT Sigma 117811 99.5 
37 Nonanoic acid (at 37 ◦C, liquid) 112-05-0 I (Cat 2) liquid 2–8 ◦C Sigma N5502 97.0 
38 Decanoic acid (at 37 ◦C, liquid) 334-48-5 I (Cat 2) liquid RT Sigma W236403 99.3 
39 Butyl methacrylate 97-88-1 I (Cat 2) liquid RT Sigma 235865 99.4 
40 alpha-Terpineol 98-55-5 I (Cat 2) liquid RT Sigma W304506 97.9 
41 Butyric acid 107-92-6 I (Cat 1B) liquid RT Sigma B103500 99.8 
42 Heptanoic acid 111-14-8 I (R34) liquid RT Sigma 75190 99.9 
43 Octanoic acid (Caprylic acid) 124-07-2 I (Cat 1B/1C) liquid RT Sigma C2875 99.9 
44 25% Cetrimonium chloride (in D.W.) 112-02-7 I (Cat 2) liquid RT Sigma 292737 – 
45 Ferric chloride 7705-08-0 I (R34) solid RT Sigma 157740 97.0 
46 Piroctone olamine 68890-66- 
4 
I (Cat 2) solid 4 ◦C TCI P2178 99.8 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
12
ICC and Kappa statistics. More importantly, Bland-Altman plot revealed 
that KeraSkin™ SIT appears to be more sensitive overall than existing 
VRMs, which can explain the higher sensitivity than VRMs. These points 
could not be seen with a simple point estimation of predictive capacity, 
reflecting the utility of statistical methods for evaluating and comparing 
me-too tests with existing gold standards as also exemplified by the 
recent studies (Alépée et al., 2016; Lim et al., 2019a). 
Color interference is a potential issue for all colorimetric assay-based 
in vitro tests (Joo et al., 2019). We also confirmed that decision can be 
changed for a non-irritant with color interference after correction. 
However, the correction for color interference was seldom necessary 
since color-interfering chemicals persistent through vigorous washing 
steps are not common and irritant decisions are not affected by 
correction. One run with one chemical needed correction among 180 
runs in the main study. And among 46 chemicals tested for additional 
predictive capacity assessment, 10 chemicals showed concerns of color 
interference but correction did not change the decision (data not 
shown). Against this background, we suggest that correction for color 
interference may be reserved only for non-irritants as a post-check 
rather than doing pre-check for all chemicals as shown in Fig. 2. This 
will significantly save costly RhE tissues for color checking. 
Collectively, we demonstrated that KeraSkin™ SIT showed good 
reproducibility and predictive capacity, meeting the criteria of OECD TG 
439 PS. The results of the validation study which was conducted in 
accordance of OECD TG 439 PS, support that KeraSkin™ SIT is 
compatible with other existing VRMs of OECD TG 439 in terms of 
reproducibility, predictive capacity and applicability domain. There-
fore, we believe that KeraSkin™ SIT may be considered as a me-too 
method similar to RhE SIT VRMs of OECD TG 439. 
Funding sources 
This research was supported by grants from the Ministry of Food and 
Drug Safety in 2018 (18182MFDS463) and National Research Founda-
tion (NRF) funded by Ministry of Science and ICT (MSIT) 
(2018R1A5A2025286). 
Fig. 6. KeraSkin™ SIT viability data for 66 chemicals composed of 23 solids and 43 liquids. Red color indicates falsely predicted substances. Dotted line indicates 
cut-off (50%) and blue line represents 60%, an optimum cut-off suggested by ROC analysis. (For interpretation of the references to color in this figure legend, the 
reader is referred to the Web version of this article.) 
Table 9 
Predictive capacity of KeraSkin™ SIT for 66 chemicals. 
Total (66) Liquid (43) Solid (23) PS 
I NI I NI I NI 
I 20 1 17 0 3 1 
NI 8 37 7 19 1 18 
Total 66 43 23 
Sensitivity 95.2% 100% 75% 80% 
Specificity 82.2% 73.1% 94.7% 70% 
Accuracy 86.4% 83.7% 91.3% 75% 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
13
Fig. 7. ROC curve for 66 substances and 23 solids and 43 liquids. The receiver operating characteristic (ROC) curve was drawn with cell viability data obtained with 
KeraSkin™ SIT. Grey color is the current cut-off, 50% and yellow color is the optimum cut-off value suggested from the analysis. (For interpretation of the references 
to color in this figure legend, the reader is referred to the Web version of this article.) 
Table 10 
Comparison of weighted and unweighted predictive capacity of KeraSkin™ SIT. 
Predictive capacity Unweighted PS criteria Weighted 
Total (66) Liquid (43) Solid (23) Total (66) Liquid (43) Solid (23) 
Sensitivity 95.2% 100% 75% ≥80% 94.7% 99.3% 75% 
Specificity 82.2% 73.1% 94.7% ≥70% 82.2% 73.1% 94.7% 
Accuracy 86.4% 83.7% 91.3% ≥75% 86.2% 83.5% 91.3% 
J. Han et al. 
Regulatory Toxicology and Pharmacology 117 (2020) 104725
14
Declaration of competing interest 
The authors declare that they have no known competing financial 
interests or personal relationships that could have appeared to influence 
the work reported in this paper. 
Acknowledgements 
This research was supported by grants from the Ministry of Food and 
Drug Safety in 2018 (18182MFDS463) and National Research Founda-
tion (NRF) funded by Ministry of Science and ICT (MSIT) 
(2018R1A5A2025286). 
References 
Adler, S., et al., 2011. Alternative (non-animal) methods for cosmetics testing: current 
status and future prospects—2010. Arch. Toxicol. 85, 367–485. 
Akbarsha, M.A., Mascarenhas, B., 2019. Cosmetic Regulation and Alternatives to Animal 
Experimentation in India. Alternatives to Animal Testing. Springer, Singapore, 
pp. 57–62. 
Alépée, N., et al., 2016. Multi-laboratory evaluation of SkinEthic HCE test method for 
testing serious eye damage/eye irritation using solid chemicals and overall 
performance of the test method with regard to solid and liquid chemicals testing. 
Toxicol. Vitro 34, 55–70. 
Atkinson, G., 1998. Statistical methods for assessing measurement error (reliability) in 
variables relevant to sports medicine. Sports Med. 26, 217–238. 
Bartko, J.J., 1976. On various intraclass correlation reliability coefficients. Psychol. Bull. 
83, 762. 
Berry, K.J., et al., 1988. A generalization of Cohen’s kappa agreement measure to 
interval measurement and multiple raters. Educ. Psychol. Meas. 48, 921–933. 
Blackman, N.J.M., 2000. Interval estimation for Cohen’s kappa as a measure of 
agreement. Stat. Med. 19, 723–741. 
Choi, J., et al., 2014. Skin corrosion and irritation test of sunscreen nanoparticles using 
reconstructed 3D human skin model. Environ. Toxicol. 29. 
Cooper 2nd, J., Saracci, R., 1979. Describing the validity of carcinogen screening tests. 
Br. J. Cancer. 39, 87. 
Giavarina, D., 2015. Understanding bland altman analysis. Biochem. Med. 25, 141–151. 
Hirschmann, M., et al., 2011. The position and orientation of total knee replacement 
components: a comparison of conventional radiographs, transverse 2D-CT slices and3D-CT reconstruction. J. Bone Joint Surg. 93, 629–633. 
Hwang, J.-h., et al., 2018. Investigation of dermal toxicity of ionic liquids in monolayer- 
cultured skin cells and 3D reconstructed human skin models. Toxicol. Vitro 46, 
194–202. 
Joo, K.M., et al., 2019. Development and validation of UPLC method for WST-1 cell 
viability assay and its application to MCTT HCE eye irritation test for colorful 
substances. Toxicol. Vitro 60, 412–419. 
Jung, K.-M., et al., 2014. KeraSkin™-VM: a novel reconstructed human epidermis model 
for skin irritation tests. Toxicol. Vitro 28, 742–750. 
Kandárová, H., et al., 2009. In vitro skin irritation testing: improving the sensitivity of 
the EpiDerm skin irritation test protocol. Altern. Lab. Anim. 37, 671–689. 
Katoh, M., Hata, K., 2011. Refinement of LabCyte EPI-MODEL24 skin Irritation test 
method for adaptation to the requirements of OECD test guideline 439. Altern. Anim. 
Test. Exp. 16, 111–122. 
Kim, K., et al., 2017. Anti-pigmentary effect of (-)-4-Hydroxysattabacin from the marine- 
derived Bacterium Bacillus sp. Mar. Drugs 15, 138. 
Koo, T.K., Li, M.Y., 2016. A guideline of selecting and reporting intraclass correlation 
coefficients for reliability research. J. Chiropr. Med. 15, 155–163. 
Kose, O., et al., 2018. Evaluation of skin irritation potentials of different cosmetic 
products in Turkish market by reconstructed human epidermis model. Regul. 
Toxicol. Pharmacol. 98, 268–273. 
Landis, J.R., Koch, G.G., 1977. The measurement of observer agreement for categorical 
data. Biometrics 33, 159–174. 
Lee, M., et al., 2017. Alternatives to in vivo draize rabbit eye and skin irritation tests with 
a focus on 3D reconstructed human cornea-like epithelium and epidermis models. 
Toxicol. Res. 33, 191–203. 
Lee, S.-H., et al., 2016. Evaluating the micronucleus induction potential for the 
genotoxicity assay using the human skin model, KeraSkin TM. J Soc. Cosmet. Sci. 
Korea 42, 211–216. 
Lim, S.-E., et al., 2019a. Statistical analysis of the reproducibility and predictive capacity 
of MCTT HCE™ eye irritation test, a me-too test method for OECD TG 492. Regul. 
Toxicol. Pharmacol. 107, 104430. 
Lim, S.E., et al., 2019b. Me-too validation study for in vitro eye irritation test with 3D- 
reconstructed human cornea epithelium, MCTT HCE(TM). Toxicol. Vitro 55, 
173–184. 
OECD, 2002. Test No. 404 : Acute Dermal Irritation/Corrosion. OECD Publishing, paris, 
p. 13. 
OECD, 2010. Test no. 439: in vitro skin irritation. In: OECD. OECD Publishing, Paris. 
OECD, 2015a. In Vitro Skin Irritation: Reconstructed Human Epidermis Test Method, vol. 
439. OECD Publishing, paris. 
OECD, 2015b. No. 220 performance standards for the assessment of proposed similar or 
modified in vitro reconstructed human epidermis (RhE) test methods for skin 
irritation testing as described in TG 439. In: OECD. OECD Publishing, Paris. 
OECD, 2017. Guidance document on an integrated approach on testing and assessment 
(IATA) for skin corrosion and irritation. In: OECD. OECD Publishing, Paris. 
OECD, 2019a. Test No. 439: In Vitro Skin Irritation: Reconstructed Human Epidermis 
Test Method. OECD Publishing, Paris. 
OECD, 2019b. Reports of the Peer Reviews of the epiCS and Skin+ Test Methods in View 
of Their Inclusion in Test Guideline 439 on In Vitro Skin Irritation. 
Parish, S.T., et al., 2020. An evaluation framework for new approach methodologies 
(NAMs) for human health safety assessment. Regul. Toxicol. Pharmacol. 112, 
104592. 
Park, J., et al., 2018. Mixture toxicity of methylisothiazolinone and propylene glycol at a 
maximum concentration for personal care products. Toxicol. Res. 34, 355–361. 
Park, S.H., et al., 2004. Receiver operating characteristic (ROC) curve: practical review 
for radiologists. Korean J. Radiol. 5, 11–18. 
Shrout, P.E., Fleiss, J.L., 1979. Intraclass correlations: uses in assessing rater reliability. 
Psychol. Bull. 86, 420. 
Spielmann, H., et al., 2007. The ECVAM international validation study on in vitro tests 
for acute skin irritation: report on the validity of the EPISKIN and EpiDerm assays 
and on the Skin Integrity Function Test. Altern. Lab. Anim. 35, 559–601. 
Szklo, M., Nieto, F.J., 2014. Epidemiology: Beyond the Basics. Jones & Bartlett 
Publishers. 
Yang, H., et al., 2017. Prevalidation trial for a novel in vitro eye irritation test using the 
reconstructed human cornea-like epithelial model, MCTT HCE™. Toxicol. Vitro. 39, 
58–67. 
Youngstrom, E.A., 2014. A primer on receiver operating characteristic analysis and 
diagnostic efficiency statistics for pediatric psychology: we are ready to ROC. 
J. Pediatr. Psychol. 39, 204–221. 
J. Han et al. 
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref1
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref1
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref2
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref2
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref2
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref3
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref3
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref3
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref3
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref4
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref4
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref5
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref5
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref6
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref6
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref7
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref7
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref8
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref8
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref9
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref9
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref10
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref11
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref11
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref11
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref12
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref12
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref12
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref13
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref13
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref13
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref14
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref14
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref15
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref15
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref16
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref16
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref16
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref17
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref17
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref18
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref18
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref19
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref19
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref19
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref20
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref20
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref21
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref21
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref21
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref22
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref22
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref22
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref23
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref23
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref23
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref24http://refhub.elsevier.com/S0273-2300(20)30151-3/sref24
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref24
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref25
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref25
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref26
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref27
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref27
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref28
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref28
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref28
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref29
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref29
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref30
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref30
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref31
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref31
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref32
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref32
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref32
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref33
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref33
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref34
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref34
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref35
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref35
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref36
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref36
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref36
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref37
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref37
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref38
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref38
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref38
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref39
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref39
http://refhub.elsevier.com/S0273-2300(20)30151-3/sref39
	Me-too validation study for in vitro skin irritation test with a reconstructed human epidermis model, KeraSkin™ for OECD te ...
	1 Introduction
	2 Material and methods
	2.1 Chemicals and reagents
	2.2 KeraSkin™ SIT protocol 1.5
	2.2.1 Washing
	2.2.2 MTT assay
	2.2.3 Detecting and correcting interference from colored chemicals and MTT reducers
	2.2.4 Prediction model for the KeraSkin™ SIT
	2.2.5 Acceptance criteria
	2.3 Statistical analysis for the reliability of the KeraSkin™ SIT
	2.3.1 Data collection and chart
	2.3.2 Within-laboratory and between-laboratory reproducibility
	2.3.3 Correlation with validated reference methods
	2.4 Predictive capacity of the KeraSkin™ SIT
	2.4.1 20 chemicals and 46 additional chemicals for the evaluation of predictive capacity
	3 Results
	3.1 Within-and between-laboratory reproducibility
	3.2 Correlation of KeraSkin™ SIT with validated reference methods of OECD TG 439
	3.3 Predictive capacity of KeraSkin™ SIT for 20 reference chemicals
	3.4 Color interference
	3.5 Predictive capacity of KeraSkin™ SIT for 66 chemicals
	4 Discussion
	Funding sources
	Declaration of competing interest
	Acknowledgements
	References