Prévia do material em texto

Classification of Rockburst in Underground Projects:
Comparison of Ten Supervised Learning Methods
Jian Zhou, Ph.D.1; Xibing Li2; and Hani S. Mitri3
Abstract: Rockburst prediction is of crucial importance to the design and construction of many underground projects. Insufficient
knowledge, lack of characterizing information, and noisy data restrain rock mechanics engineers from achieving optimal prediction results.
In this paper, a data set of 246 rockburst events was examined for rockburst classification using supervised learning (SL) methods. The data
set was analyzed with 8 potentially relevant indicators. Eleven algorithms from 10 categories of SL algorithms were evaluated for their ability
to learn rockburst, including linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), partial least-squares discriminant
analysis (PLSDA), naïve Bayes (NB), k-nearest neighbor (KNN), multilayer perceptron neural network (MLPNN), classification tree (CT),
support vector machine (SVM), random forest (RF), and gradient-boosting machine (GBM). The data set was randomly split into two
parts: training (70%) and test (30%). A 10-fold cross-validation (CV) method was applied during modeling, and an external testing set was
employed to validate the prediction performance of the SL models. Two accuracy measures for multiclass problems were employed:
classification rate and Cohen’s Kappa. The accuracy analysis, together with Cohen’s kappa and a nonparametric statistical test for the
rockburst data set, revealed that the best models for the prediction of rockburst were GBM and RF when compared with other learning
algorithms. DOI: 10.1061/(ASCE)CP.1943-5487.0000553. © 2016 American Society of Civil Engineers.
Author keywords: Underground project; Rockburst; Supervised learning; Classification; Cross-validation; Nonparametric statistical test.
Introduction
Rockburst is a dynamic, spontaneous, uncontrolled geological dis-
aster that occurs in underground structures such as mines, hydro-
power caverns, and tunnels (Ortlepp 1997; Wang and Park 2001;
Zhou et al. 2012). The occurrence of rockbursting is attributed to
the release of accumulated energy in the rock in a violent manner
(Cook et al. 1966; Kaiser et al. 1997; Blake and Hedley 2003).
Because rockbursts occur suddenly and intensely, rock particles
can be ejected with a velocity of 8–50 m=s (Ortlepp 1993); they
usually cause considerable damage to infrastructure and/or equip-
ment and may even cause fatal injuries (Ortlepp 2005; Zhou et al.
2012). Many hard rockburst–prone mines in China, Canada, South
Africa, the United States, Australia, Sweden, and other countries,
as well as some deep buried civil tunnels in Switzerland, China, and
Peru, have suffered from rockbursts to various degrees. For exam-
ple, in 1960 a rockburst in the Witwatersrand Mines in South Africa
caused 435 deaths; this was the worst accident in South African
mining history (Durrheim 2010).
In China, according to incomplete statistics from the period
2001–2007, many deep-mine hazards due to rockburst were re-
ported, accompanied by more than 13,000 accidents and at least
16,000 casualties (Zhou et al. 2012). Zhang et al. (2012a) reported
an extremely intense rockburst that occurred in the drainage tunnel
of the Jinping II Hydropower Station at a depth of 2,330 m in 2009.
It caused seven deaths and one injury, as well as complete destruc-
tion of the tunnel boring machine (TBM). Nowadays, rock exca-
vations can reach great depths, and with increasing mining
activities worldwide, the problem of rockbursting is likely to get
worse (Shi et al. 2010; Li et al. 2011b). Thus, the prediction of
rockbursts is very important for disaster prevention and control.
Although it is very difficult to accurately predict rockburst dur-
ing excavation, valuable results have been reported in the past
several decades by a number of authors on a variety of rockburst
aspects, such as triggering mechanism, microgravity method, re-
bound method, drilling-yield test, microseismicity, and probabilis-
tic methods (Zhang and Fu 2008; Zhou et al. 2012). Extensive
rockburst research has been conducted in South Africa, Canada,
Australia, China, and many other countries (Ortlepp 2005). In par-
ticular, since their inception in Johannesburg in 1982 the symposia
on rockbursts and seismicity in mines (RaSiM) have provided a
platform for the exchange of information on both practical and fun-
damental aspects of rockbursts and seismicity-related problems
(Ortlepp 2005; Cai 2013). These collective efforts have greatly
improved understanding of rockbursting. Rockburst classification
prediction, however, is a complex and nonlinear procedure that is
influenced by model and parameter uncertainty. Therefore, under-
standing and predicting rockbursts are still considerable challenges
for underground projects.
Considerable research effort in South Africa, Australia, Canada,
and China has been devoted to understanding the rockburst
phenomenon (Ortlepp 2005; Potvin 2009; Zhou et al. 2012). As
pointed out, different methods of estimating and predicting rock-
burst, such as stress criteria classification, in situ testing, pre-
liminary and qualitative judgment prediction, and computational
1School of Resources and Safety Engineering, Central South Univ.,
#932 Lushan South Rd., Changsha 410083, China; Visiting Scholar, Dept.
of Mining and Materials Engineering, McGill Univ., 3450 University St.,
Montreal, QC, Canada H3A 0E8 (corresponding author). E-mail:
csujzhou@hotmail.com
2Professor, School of Resources and Safety Engineering, Central South
Univ., #932 Lushan South Rd., Changsha 410083, China. E-mail: xbli@
mail.csu.edu.cn
3Professor, Dept. of Mining and Materials Engineering, McGill Univ.,
3450 University St., Montreal, QC, Canada H3A 0E8. E-mail: hani.mitri@
mcgill.ca
Note. This manuscript was submitted on April 24, 2015; approved on
September 30, 2015; published online on January 6, 2016. Discussion per-
iod open until June 6, 2016; separate discussions must be submitted for
individual papers. This paper is part of the Journal of Computing in Civil
Engineering, © ASCE, ISSN 0887-3801.
© ASCE 04016003-1 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000553
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000553
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000553
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000553
mailto:csujzhou@hotmail.com
mailto:csujzhou@hotmail.com
mailto:xbli@mail.csu.edu.cn
mailto:xbli@mail.csu.edu.cn
mailto:xbli@mail.csu.edu.cn
mailto:xbli@mail.csu.edu.cn
mailto:xbli@mail.csu.edu.cn
mailto:hani.mitri@mcgill.ca
mailto:hani.mitri@mcgill.ca
mailto:hani.mitri@mcgill.ca
mailto:hani.mitri@mcgill.ca
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
intelligence, have been proposed (Zhou et al. 2012; Liu et al. 2013).
With the increasing availability of in situ data, data-mining tech-
niques can be used for the identification of sites prone to rockburst
occurrence. Although a large number of evaluation models have
been described and compared by several authors, it remains to be
seen which models perform best given particular circumstances.
Moreover, relationships between variables in rockburst events are
almost always highly nonlinear and extremely complicated. Still,pre-
dictors were sorted from most to least important. Fig. 7(b) shows
an example for the GBM Model A using function varImp() in
the caret package and displaying the relative importance of var-
iables for each of the eight predictor variables. Not surprisingly,
it demonstrates that Wet was the most sensitive factor among the
indicators for the prediction of rockburst classification. The in-
dicator MTS takes second place for sensitivity. The indexes of
SCF, D, and B1 were a bit sensitive. UTS, UCS, and B2 were
not as sensitive. These results demonstrate that Wet was the most
relevant predictor among the indicators for predicting rockburst
classification.
Contributions
The key contributions of this research are summarized in three
aspects. First, in previous work the data sets were small (usually
only a few dozen groups involved in rockburst event modeling).
Here a new comprehensive database comprising 246 rockburst
events was developed that will be beneficial for future studies
and research in the field. Second, there have been a number of
representative studies on rockburst prediction; however, in these
studies all the data were artificially (not randomly) separated
into training and testing sets and model parameters were usually
fixed by trial and error or empirically, so model performance
was not reliable or stable. Unlike previous studies, here the
original rockburst data set with known classes was randomly
divided into two subsets (training and test) and two types of
model sensitivity analysis (tuning the model parameters by
10-fold CV and determining variable importance from the
ROC analysis) were implemented. Third, despite the fact that
supervised learning (SL) techniques have been widely used
for rockburst prediction, they have not been compared for rock-
burst estimation. Again, unlike other studies, this one presented
a systematic assessment of 10 SL algorithms for rockburst clas-
sification. To the best of the authors’ knowledge, the results of
using QDA, PLSDA, DT, KNN, NB, and GBM to assess rock-
burst have not been fully documented by any other researchers
to date.
Table 10. Classification Results for the Original Data Set Using Some Empirical Criteria
Empirical method Equation
Classification criteria Predictive
accuracy (%)None Light Moderate High
Russenes criterion (1974) σθ=σc ≤ 0.2 0.2–0.3 0.3–0.55 >0.55 45.12
Stress concentration factor (Wang et al. 1998) σθ=σc 0.7 46.75
Depth (Zhang et al. 2010) D 700 25.82
UCS (Zhang et al. 2010) UCS 180 29.27%
Rock brittleness coeffieient B1 (Peng et al. 1996; Wang et al. 1998) σc=σt >40 40–26.7 26.7–14.5 22 21.14
Strain energy storage index (Kidybinski 1981) Wet 5.0 51.22
Burst proneness index (Singh 1989) Wet 15.0 25.61
GB50487-2008 σc=σmax >7 4–7 2–4 0.75 52.85
Note: S ¼ tanhf½0.1648ðσθ=σcÞ3.064ðB1Þ−0.4625ðWetÞ2.672�ð1=3.6Þg.
Fig. 7. Variable importance assessment in Model A for predicting rockburst: (a) RF method; (b) GBM method
© ASCE 04016003-15 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
Summary and Conclusions
Models for predicting rockburst can be valuable tools in under-
ground mining and civil engineering projects. This study compared
SL models for predicting rockburst. Eight variables (D,MTS, UCS,
UTS, SCF, B1, B2, andWet) were measured, and 11 base models for
10 SL approaches for predicting rockburst were selected. A data set
of 246 rockburst events compiled from recent published research
was used to construct the SL models. This data set was evaluated
for significant differences between models using Friedman’s non-
parametric test and Nemenyi’s post hoc tests with α ¼ 0.05. Based
on the analysis results, the following conclusions can be drawn:
• Unlike previous studies, this study employed the 10-fold CV
strategy to select appropriate parameters for tuning the data set
to ensure good generalization capability; it also standardized
work in the field by discovering hidden relationships and knowl-
edge in a complete data set; the use of multiple splitting into TS
and PS was needed for a reliable model comparison;
• None of the SL classification models should be utilized blindly
because none of them provide fully automatic classification;
none was competent enough to exclusively classify these rock-
burst events;
• Nonparametric tests (Friedman, Iman-Davenport, and Nemenyi)
could be used in analysis of multiple data sets and in comparison
of the 10 SL algorithms; according to the nonparametric tests,
the best models, in order of performance quality, were GBM/RF,
SVMR, NB, KNN, ANN, CT, SVML, QDA, LDA, and PLSLDA;
• The comparisons indicated that Model A, consisting of eight
variables input with the RF and GBM methods, was more reli-
able for evaluating rockburst than other models; RF and GBM
demonstrated that indicator Wet was the most relevant predictor
of rockburst classification, followed by indicators MTS, SCF,
and D; and
• For the 10 SL techniques, TS performance (in terms of accu-
racy) ranged [0.477–0.687] across the 11 models, whereas
PS performance (in terms of accuracy) ranged [0.438–0.766];
the predictive accuracy of the empirical criteria methods was
21.14–52.85% of the original data and less than 60% of the fil-
tered data; it is obvious, judging from the results for predictive
accuracy, that empirical methods cannot generate satisfactory
predictions for rockburst events.
Acknowledgments
This research is partially supported by the National Natural Science
Foundation Project of China (Grant Nos. 41272304 and
11472311), the Innovation Driven Plan of Central South University
(Grant No. 2015CX005) of China and by Project Grant No. 1343-
76140000022 supported by the Scholarship Award for Excellent
Doctoral Student of Ministry of Education of China. The authors
would like to express thanks to these foundations. The first author
would like to thank the Chinese Scholarship Council for financial
support toward his joint Ph.D. at McGill University, Canada.
Supplemental Data
Table S1 is available online in the ASCE Library (http://www.
ascelibrary.org).
References
Adoko, A. C., Gokceoglu, C., Wu, L., and Zuo, Q. J. (2013). “Knowledge-
based and data-driven fuzzy modeling for rockburst prediction.” Int. J.
Rock Mech. Mining Sci., 61, 86–95.
Arditi, D., and Pulket, T. (2005). “Predicting the outcome of construction
litigation using boosted decision trees.” J. Comput. Civ. Eng., 10.1061/
(ASCE)0887-3801(2005)19:4(387), 387–393.
Arditi, D., and Pulket, T. (2010). “Predicting the outcome of construction
litigation using an integrated artificial intelligence model.” J. Comput.
Civ. Eng., 10.1061/(ASCE)0887-3801(2010)24:1(73), 73–80.
Bai, M. Z., Wang, L. J., and Xu, Z. Y. (2002). “Study on a neutral network
model and its application in predicting the risk of rock burst.” China
Saf. Sci. J., 12(4), 65–69.
Barton, N., Lien, R., and Lunde, J. (1974). “Engineering classification of
rock masses for the design of tunnel support.” Rock Mech. Rock Eng.,
6(4), 189–236.
Berrueta, L. A., Alonso-Salces, R. M., and Heberger, K. (2007).
“Supervised pattern recognition in food analysis.” J. Chromatogr. A,
1158(1–2), 196–214.
Blake, W., and Hedley, D. G. F. (2003). Rockbursts: Case studies from
North Americanhard-rock mines, Society for Mining, Metallurgy,
and Exploration, New York, 121.
Breiman, L. (2001). “Random forests.” Mach. Learn., 45(1), 5–32.
Breiman, L., and Cutler, A. (2004). “Random forests.” Dept. of Statistics,
Univ. of California, Berkeley, 〈http://www.stat.berkeley.edu/~breiman/
RandomForests/cc_home.htm〉 (Sep. 30, 2014).
Breiman, L., Friedman, J., Stone, C. J., and Olshen, R. A. (1984).
Classification and regression trees, CRC Press, Boca Raton, FL, 368.
Cai, M. (2013). “Principles of rock support in burst-prone ground.”
Tunnelling Underground Space Technol., 36, 46–56.
Castro, L. M., Bewick, R. P., and Carter, T. G. (2012). “An overview of
numerical modelling applied to deep mining.” Innovative numerical
modelling in geomechanics, R. Azevedo, ed., CRC Press/Taylor &
Francis, London, 393–414.
Chen, D. F., Feng, X. T., Yang, C. X., Chen, B. R., Qiu, S. L., and Xu, D. P.
(2013). “Neural network estimation of rockburst damage severity based
on engineering cases.” Proc., Sinorock2013 Symp., Taylor & Francis,
London, 457–463.
Chen, H. J., Li, N. H., Ni, D. X., and Shang, Y. Q. (2003). “Prediction of
rockburst by artificial neural network.” Chinese J. Rock Mech. Eng.,
22(5), 762–768.
Chou, J. S. (2012). “Comparison of multilabel classification models to
forecast project dispute resolutions.” Expert Syst. Appl., 39(11),
10202–10211.
Chou, J. S., and Lin, C. (2013). “Predicting disputes in public-private
partnership projects: Classification and ensemble models.” J. Comput.
Civ. Eng., 10.1061/(ASCE)CP.1943-5487.0000197, 51–60.
Chou, J. S., Tsai, C. F., and Lu, Y. H. (2013). “Project dispute prediction
by hybrid machine learning techniques.” J. Civ. Eng. Manage., 19(4),
505–517.
Clark, M. (2013). “An introduction to machine learning: With applications
in R.” 〈http://www3.nd.edu/~mclark19/learn/ML.pdf〉 (Jan. 5, 2014).
Cohen, J. (1960). “A coefficient of agreement for nominal scales.” Educ.
Psychol. Meas., 20(1), 37–46.
Cook, N. G. W., Hoek, E., Pretorius, J. P. G., Ortlepp, W. D., and Salamon,
M. D. G. (1966). “Rock mechanics applied to the study of rockbursts.”
J. S. Afr. Inst. Min. Metall., 66(3), 435–528.
Demšar, J. (2006). “Statistical comparisons of classifiers over multiple data
sets.” J. Mach. Learn. Res., 7, 1–30.
Ding, X. D., Wu, J. M., Li, J., and Liu, C. J. (2003). “Artificial neural net-
work for forecasting and classification of rockbursts.” J. Hohai Univ.
(Nat. Sci.), 31(4), 424–427.
Dong, L. J., Li, X. B., and Peng, K. (2013). “Prediction of rockburst clas-
sification using random forest.” Trans. Nonferrous Met. Soc. China,
23(2), 472–477.
Durrheim, R. J. (2010). “Mitigating the risk of rockbursts in the deep hard
rock mines of South Africa: 100 years of research.” Extracting the sci-
ence: A century of mining research, J. Brune, ed., Society for Mining,
Metallurgy, and Exploration, Littleton, CO, 156–171.
Feng, X. T., andWang, L. N. (1994). “Rockburst prediction based on neural
networks.” Trans. Nonferrous Met. Soc. China, 4(1), 7–14.
Feng, X. T., Webber, S., Ozbay, M. U., and Wang, Y. J. (1996). “An expert
system on assessing rockburst risks for South African deep gold mines.”
J. Coal Sci. Eng., 2(2), 23–32.
© ASCE 04016003-16 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
http://ascelibrary.org/doi/suppl/10.1061/%28ASCE%29CP.1943-5487.0000553/suppl_file/Supplemental_Data_CP.1943-5487.0000553_Zhou.pdf
http://www.ascelibrary.org
http://www.ascelibrary.org
http://dx.doi.org/10.1016/j.ijrmms.2013.02.010
http://dx.doi.org/10.1016/j.ijrmms.2013.02.010
http://dx.doi.org/10.1061/(ASCE)0887-3801(2005)19:4(387)
http://dx.doi.org/10.1061/(ASCE)0887-3801(2005)19:4(387)
http://dx.doi.org/10.1061/(ASCE)0887-3801(2005)19:4(387)
http://dx.doi.org/10.1061/(ASCE)0887-3801(2010)24:1(73)
http://dx.doi.org/10.1061/(ASCE)0887-3801(2010)24:1(73)
http://dx.doi.org/10.1007/BF01239496
http://dx.doi.org/10.1007/BF01239496
http://dx.doi.org/10.1016/j.chroma.2007.05.024
http://dx.doi.org/10.1016/j.chroma.2007.05.024
http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm
http://dx.doi.org/10.1016/j.tust.2013.02.003
http://dx.doi.org/10.1016/j.eswa.2012.02.103
http://dx.doi.org/10.1016/j.eswa.2012.02.103
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000197
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000197
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000197
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000197
http://dx.doi.org/10.3846/13923730.2013.768544
http://dx.doi.org/10.3846/13923730.2013.768544
http://www3.nd.edu/~mclark19/learn/ML.pdf
http://www3.nd.edu/~mclark19/learn/ML.pdf
http://www3.nd.edu/~mclark19/learn/ML.pdf
http://www3.nd.edu/~mclark19/learn/ML.pdf
http://dx.doi.org/10.1016/S1003-6326(13)62487-5
http://dx.doi.org/10.1016/S1003-6326(13)62487-5
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
Fisher, D. H. (1936). “The use of multiple measurements in taxonomic
problems.” Ann. Eugenics, 7(2), 179–188.
Friedman, J. H. (2001). “Greedy function approximation: A gradient boost-
ing machine.” Ann. Stat., 29(5), 1189–1232.
Garzón, M. B., Blazek, R., Neteler, M., de Dios, R. S., Ollero, H. S., and
Furlanello, C. (2006). “Predicting habitat suitability with machine
learning models: The potential area of Pinus sylvestris L. in the Iberian
Peninsula.” Ecol. Modell., 197(3–4), 383–393.
Ge, Q. F., and Feng, X. T. (2008). “Classification and prediction of rock-
burst using AdaBoost combination learning method.” Rock Soil Mech.,
29(4), 943–948.
Gill, D. E., Aubertin, M., and Simon, R. (1993). “A practical engineering
approach to the evaluation of rockburst potential.” Proc., 3rd Int. Symp.
on Rockbursts and Seismicity in Mines, R. P. Young, ed., A.A. Balkema,
Rotterdam, Netherlands, 63–68.
Gong, F. Q., and Li, X. B. (2007). “A distance discriminant analysis method
for prediction of possibility and classification of rockburst and its
application.” Chinese J. Rock Mech. Eng., 26(5), 1012–1018.
Gong, F. Q., Li, X. B., and Zhang, W. (2010). “Rockburst prediction of
underground engineering based on Bayes discriminant analysis
method.” Rock Soil Mech., 31(Suppl.1), 370–377.
González-Rufino, E., Carrión, P., Cernadas, E., Fernández-Delgado, M.,
and Domínguez-Petit, R. (2013). “Exhaustive comparison of colour
texture features and classification methods to discriminate cell catego-
ries in histological images of fish ovary.” Pattern Recognit., 46(9),
2391–2407.
Grinand, C., Arrouays, D., Laroche, B., and Martin, M. P. (2008).
“Extrapolating regional soil landscapes from an existing soil map:
Sampling intensity, validation procedures, and integration of spatial
context.” Geoderma, 143(1), 180–190.
Grouven, U., Bergel, F., and Schultz, A. (1996). “Implementation of linear
and quadratic discriminant analysis incorporating costs of misclassifi-
cation.” Comput. Methods Programs Biomed., 49(1), 55–60.
Guelman, L. (2012). “Gradient boosting trees for auto insurance loss cost
modeling and prediction.” Expert Syst. Appl., 39(3), 3659–3667.
Guo, C. B., Zhang, Y. S., Deng, H. K., Su, Z. G., and Sun, D. S. (2011).
“Study on rockburst prediction in the deep-buried tunnel at Gaoligong
Mountain based on the rock proneness.”Geotech. Invest. Surv., (10), 8–13.
Haykin, S. (1999). “Neural networks: A comprehensive foundation.” 2nd
Ed., Prentice Hall, NJ.
Heal, D., Potvin, Y., and Hudyma, M. (2006). “Evaluating rockburst dam-
age potential in underground mining.” Proc., 41st U.S. Symp. on RockMechanics (USRMS), D. P. Yale, et al., eds., ARMA, Virginia, 322–329.
Hoek, E., and Brown, E. T. (1980). Underground excavation in rock,
Institute of Mining and Metallurgy, London.
Hothorn, T., Hornik, K., and Zeileis, A. (2006). “Unbiased recursive par-
titioning: A conditional inference framework.” J. Comput. Graph. Stat.,
15(3), 651–674.
Hou, F. L. (1989). “Criterion and prevention measures on rockburst in cir-
cular tunnel.” Application on rock mechanics engineering, Knowledge
Press, Beijing, 195–201.
Iman, R. L., and Davenport, J. M. (1980). “Approximations of the critical
region of the Friedman statistic.” Commun. Stat. Theory Methods, 9(6),
571–595.
Jain, P., Garibaldi, J. M., and Hirst, J. D. (2009). “Supervised machine
learning algorithms for protein structure classification.” Comput. Biol.
Chem., 33(3), 216–223.
Jia, Y. R., and Fan, Z. Q. (1991). “Hydraulic underground cavern medium
of rockburst mechanism and criterion.” Water Power, (6), 30–34.
Jiang, L. F. (2008). “Study on prediction and prevention of rockburst in
Anlu tunnel.” Master’s thesis, Southwest Jiaotong Univ., Chendu,
China.
Jiang, Q., Feng, X. T., Xiang, T. B., and Su, G. S. (2010). “Rockburst char-
acteristics and numerical simulation based on a new energy index: A
case study of a tunnel at 2, 500 m depth.” Bull. Eng. Geol. Environ.,
69(3), 381–388.
John, G. H., and Langley, P. (1995). “Estimating continuous distributions in
Bayesian classifiers.” Proc., 11th Conf. on Uncertainty in Artificial
Intelligence, P. Besnard and S. Hanks, eds., Morgan Kaufmann,
Montreal, 338–345.
Kaiser, P. K., McCreath, D. R., and Tannant, D. D. (1997). Canadian
rockburst research program 1990–1995, Vol. 1–6, CAMIRO Mining
Division, Sudbury, ON, Canada.
Kang, Y. (2006). “Research on relevant problems about failure mechanism
of surrounding rock in deep buried tunnel.” Ph.D. thesis, Chongqing
Univ., Chongqing, China, 118–120.
Karatzoglou, A., Smola, A., Hornik, K., and Zeileis, A. (2004). “Kernlab–
An S4 package for Kernel methods in R.” J. Stat. Software, 11(9), 1–20.
Kidybinski, A. (1981). “Bursting liability indices of coal.” Int. J. Rock
Mech. Mining Sci. Geomech. Abstr., 18(4), 295–304.
Kohavi, R. (1995). “A study of cross-validation and bootstrap for accuracy
estimation and model selection.” Proc., 14th Int. Joint Conf. on Arti-
ficial Intelligence IJCAI’95, Vol. 2, Morgan Kaufmann, San Francisco,
1137–1143.
Kotsiantis, S. B. (2007). “Supervised machine learning: A review of clas-
sification techniques.” Informatica, 31(3), 249–268.
Kuhn, M. (2008). “Building predictive models in R using the caret
package.” J. Stat. Software, 28(5), 1–26.
Kuhn, M. (2012). “Caret” package (R Package Version 5.15-023),
R Foundation for Statistical Computing, Vienna, Austria.
Kuhn, M., and Johnson, K. (2013). Applied predictive modeling, Springer,
New York.
Lai, C. C. (2007). “An empirical study of three machine learning methods
for spam filtering.” Knowl. Based Syst., 20(3), 249–254.
Landis, J., and Koch, G. (1977). “The measurement of observer agreement
for categorical data.” Biometrics, 33(1), 159–174.
Lavecchia, A. (2015). “Machine-learning approaches in drug discovery:
Methods and applications.” Drug Discovery Today, 20(3), 318–331.
Le-Thi-Thu, H., et al. (2011). “A comparative study of nonlinear machine
learning for the “in silico” depiction of tyrosinase inhibitory activity
from molecular structure.” Mol. Inf., 30(6–7), 527–537.
Li, L. (2009). “Study on scheme optimization and rockburst prediction in
deep mining in Xincheng gold mine.” Ph.D. thesis, Univ. of Science and
Technology, Beijing.
Li, M. X., Zhang, G., and Li, Y. G. (2011a). “Experimental study on rock-
burst proneness based on fractal method.” 〈http://www.paper.edu.cn/
download/downPaper/201102-5〉 (Nov. 20, 2013).
Li, S. L. (2000). “Study on rockburst proneness and strata control technol-
ogy for deep mines with hard rock.” Ph.D. thesis, Northeastern Univ.,
Shenyang, China.
Li, X. B., Yao, J. R., and Gong, F. Q. (2011b). “Dynamic problems in deep
exploitation of hard rock metal mines.” Chinese J. Nonferrous Met.,
21(10), 2552–2563.
Li, X. F., and Xie, C. J. (2005). “Research on prevention of rock burst
in deep high-stress area of Fankou deposit.” Mining Res. Dev., 25(1),
76–79.
Liang, Z. Y. (2004). “Study on the prediction and prevention of rockburst in
the diversion tunnel of Jinping II hydropower station.” Master’s thesis,
Chengdu Univ. of Technology, Chendu, China, 61–62.
Liaw, A., and Wiener, M. (2002). “Classification and regression by
randomforest.” R News, 2(3), 18–22.
Liu, J. P. (2011). “Studies on relationship between Microseism time-space
evolution and ground pressure activities in deep mine.” Ph.D. thesis,
Northeastern Univ., Shenyang, China.
Liu, Z., Shao, J., Xu, W., and Meng, Y. (2013). “Prediction of rock burst
classification using the technique of cloud models with attribution
weight.” Nat. Hazards, 68(2), 549–568.
Luengo, J., García, S., and Herrera, F. (2009). “A study on the use of
statistical tests for experimentation with neural networks: Analysis of
parametric test conditions and nonparametric tests.” Expert Syst. Appl.,
36(4), 7798–7808.
Mahfouz, T., and Kandil, A. (2011). “Litigation outcome prediction of
differing site condition disputes through machine learning models.”
J. Comput. Civ. Eng., 10.1061/(ASCE)CP.1943-5487.0000148, 298–308.
Martin, C., Kaiser, P., and McCreath, D. (1999). “Hoek-Brown parameters
for predicting the depth of brittle failure around tunnels.”Can. Geotech. J.,
36(1), 136–151.
Mashford, J., Marlow, D., Tran, D., and May, R. (2010). “Prediction of
sewer condition grade using support vector machines.” J. Comput.
Civ. Eng., 10.1061/(ASCE)CP.1943-5487.0000089, 283–290.
© ASCE 04016003-17 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
http://dx.doi.org/10.1111/j.1469-1809.1936.tb02137.x
http://dx.doi.org/10.1214/aos/1013203451
http://dx.doi.org/10.1016/j.ecolmodel.2006.03.015
http://dx.doi.org/10.1016/j.patcog.2013.02.009
http://dx.doi.org/10.1016/j.patcog.2013.02.009
http://dx.doi.org/10.1016/j.geoderma.2007.11.004
http://dx.doi.org/10.1016/0169-2607(95)01705-4
http://dx.doi.org/10.1016/j.eswa.2011.09.058
http://dx.doi.org/10.1198/106186006X133933
http://dx.doi.org/10.1198/106186006X133933
http://dx.doi.org/10.1080/03610928008827904
http://dx.doi.org/10.1080/03610928008827904
http://dx.doi.org/10.1016/j.compbiolchem.2009.04.004
http://dx.doi.org/10.1016/j.compbiolchem.2009.04.004
http://dx.doi.org/10.1007/s10064-010-0275-1
http://dx.doi.org/10.1007/s10064-010-0275-1
http://dx.doi.org/10.18637/jss.v011.i09
http://dx.doi.org/10.1016/0148-9062(81)91194-3
http://dx.doi.org/10.1016/0148-9062(81)91194-3
http://dx.doi.org/10.18637/jss.v028.i05
http://dx.doi.org/10.1016/j.knosys.2006.05.016
http://dx.doi.org/10.2307/2529310
http://dx.doi.org/10.1016/j.drudis.2014.10.012
http://dx.doi.org/10.1002/minf.v30.6/7
http://www.paper.edu.cn/download/downPaper/201102-5
http://www.paper.edu.cn/download/downPaper/201102-5
http://www.paper.edu.cn/download/downPaper/201102-5
http://www.paper.edu.cn/download/downPaper/201102-5
http://www.paper.edu.cn/download/downPaper/201102-5
http://dx.doi.org/10.1007/s11069-013-0635-9
http://dx.doi.org/10.1016/j.eswa.2008.11.041
http://dx.doi.org/10.1016/j.eswa.2008.11.041
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000148
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000148
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000148
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000148
http://dx.doi.org/10.1139/t98-072
http://dx.doi.org/10.1139/t98-072
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000089
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000089
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000089
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000089
Ministry of Water Resourcesof the People’s Republic of China. (2008).
“Code for engineering geological investigation of water resources
and hydropower.” GB50487–2008, China Planning Press, Beijing.
Mitri, H. S. (2007). “Assessment of horizontal pillar burst in deep hard rock
mines.” Int. J. Risk Assess. Manage., 7(5), 695–707.
Mitri, H. S., Hassani, F. P., and Kebbe, R. (1993). “A strain energy approach
for the prediction of rockburst potential in underground hard rock
mines.” Proc., 1st Canadian Symp. Numerical Modelling Applications
in Mining and Geomechanics, H. S. Mitri, ed., McGill Univ., Montréal,
228–239.
Mitri, H. S., Tang, B., and Simon, R. (1999). “FE modelling of mining-
induced energy release and storage rates.” Trans. SAIMM J., 99(2),
103–110.
Ni, F. L. (2011). “Neural network and its application to predict rock burst.”
Chongqing Archit., 10(7), 33–36.
Ortlepp, W. D. (1993). “High ground displacements velocities associated
with rockbursts damage.” 3rd Int. Symp. Rockbursts and Seismicty in
Mines, Balkema, Rotterdam, Netherlands, 101–106.
Ortlepp, W. D. (1997). Rock fracture and rockbursts: An illustrative study,
South African Institute of Mining and Metallurgy, Johannesburg, South
Africa, 98.
Ortlepp, W. D. (2005). “RaSiM Comes of age—A review of the contribu-
tion to the understanding and control of mine rockbursts.” Controlling
Seismic Risk—Proc., 6th Int. Symp. on Rockburst and Seismicity in
Mines, Y. Potvin and M. Hudyma, eds., Australian Centre for Geome-
chanics, Nedlands, Australia, 3–20.
Palmström, A. (1996). “Characterizing rock masses by the RMi for use in
practical rock engineering. Part 2: Some practical applications of the rock
mass index (RMi).” Tunnel. Underg. Space Technol., 11(3), 287–303.
Peng, Z., Wang, Y. H., and Li, T. J. (1996). “Griffith theory and rock burst
of criterion.” Chinese J. Rock Mech. Eng., 15, 491–495.
Pino-Mejías, R., Cubiles-de-la-Vega, M. D., Anaya-Romero, M., Pascual-
Acosta, A., Jordán-López, A., and Bellinfante-Crocci, N. (2010). “Pre-
dicting the potential habitat of oaks with data mining models and
the R system.” Environ. Modell. Software, 25(7), 826–836.
Potvin, Y. (2009). “Strategies and tactics to control seismic risks in mines.”
J. S. Afr. Inst. Mining Metall., 109(3), 177–186.
Qin, S. W., Chen, J. P., and Wang, Q. (2009). “Research on rockburst pre-
diction with extenics evaluation based on rough set.” Proc., RaSiM7
(2009): Controlling Seismic Hazard and Sustainable Development of
Deep Mines, C. A. Tang, ed., Rinton Press, Princeton, NJ, 937–944.
Qiu, S. L., Feng, X. T., Zhang, C. Q., and Wu, W. P. (2011). “Development
and validation of rockburst vulnerability index (RVI) in deep hard rock
tunnels.” Chinese J. Rock Mech. Eng., 30(6), 1126–1141.
R Development Core Team. (2013). R: A language and environment for
statistical computing, R Foundation for Statistical Computing, Vienna,
Austria.
Ridgeway, G. (2007). “Generalized boosted models: A guide to the gbm
package.” 〈http://cran.r-project.org/web/packages/gbm/index.html〉
(Jan. 5, 2014).
Russenes, B. F. (1974). “Analysis of rock spalling for tunnels in steep valley
sides.” M.Sc. thesis, Norwegian Institute of Technology, Trondheim,
Norway, 247 (in Norwegian).
Ryder, J. A. (1988). “Excess shear stress in the assessment of geologically
hazardous situations.” J. S. Afr. Inst. Mining Metall., 88(1), 27–39.
Sakiyama, Y., et al. (2008). “Predicting human liver microsomal stability
with machine learning techniques.” J. Mol. Graphics Modell., 26(6),
907–915.
Salamon, M. D. G. (1984). “Energy considerations in rock mechanics:
Fundamental results.” J. S. Afr. Inst. Mining Metall., 84(8), 233–246.
Sharan, S. K. (2007). “A finite element perturbation method for the predic-
tion of rockburst.” Comput. Struct., 85(17–18), 1304–1309.
Sheskin, D. J. (2006). Handbook of parametric and nonparametric statis-
tical procedures, 4th Ed., Chapman & Hall/CRC, FL.
Shi, X. Z., Zhou, J., Dong, L., Hu, H. Y., Wang, H. Y., and Chen, S. R.
(2010). “Application of unascertained measurement model to prediction
of classification of rockburst intensity.” Chinese J. Rock Mech. Eng., 29,
2720–2727.
Simon, R. (1999). “Analysis of fault-slip mechanisms in hard rock mining.”
Ph.D. thesis, McGill Univ., Montreal, 130–136.
Singh, S. P. (1987). “The influence of rock properties on the occurrence and
control of rockbursts.” Mining Sci. Technol., 5(1), 11–18.
Singh, S. P. (1988). “Burst energy release index.” Rock Mech. Rock Eng.,
21(2), 149–155.
Singh, S. P. (1989). “Classification of mine workings according to their
rockburst proneness.” Mining Sci. Technol., 8(3), 253–262.
Son, H., Kim, C., and Kim, C. (2011). “Automated color model-based con-
crete detection in construction-site images by using machine learning
algorithms.” J. Comput. Civ. Eng., 10.1061/(ASCE)CP.1943-5487
.0000141, 421–433.
Stephens, D., and Diesing, M. (2014). “A comparison of supervised clas-
sification methods for the prediction of substrate type using multibeam
acoustic and legacy grain-size data.” PLoS One, 9(4), e93950.
Su, G. S., Zhang, Y., and Chen, G. Q. (2010). “Identify rockburst grades for
Jinping hydropower station using Gaussian II process for binary clas-
sification.” Proc., 2010 Int. Conf. on Computer, Mechatronics, Control
and Electronic Engineering (CMCE 2010), Vol. 2, IEEE Press, NJ,
364–367.
Sun, H. F., Li, S. C., Qiu, D. H., Zhang, L. W., and Zhang, N. (2009).
“Application of extensible comprehensive evaluation to rockburst pre-
diction in a relative shallow chamber.” Proc., RaSiM7 (2009): Control-
ling Seismic Hazard and Sustainable Development of Deep Mines,
C. A. Tang, ed., Rinton Press, Princeton, NJ, 777–784.
Tan, Y. A. (1992). “Rockbursting characteristics and structural effects of
rock mass.” Sci. China, 35(8), 981–990.
Tan, Y. A., Sun, G. Z., and Guo, Z. (1991). “A composite index Krb
criterion for the ejection characteristics of the burst rock.” Scientia
Geologica Sinica, 2, 193–200.
Tang, C. A. (1997). “Numerical simulation of progressive rock failure and
associated seismicity.” Int. J. Rock Mech. Mining Sci., 34(2), 249–261.
Tang, S. H., Wu, Z. J., and Chen, X. H. (2003). “Approach to occurrence
and mechanism of rockburst in deep underground mines.” Chinese J.
Rock Mech. Eng., 22(8), 1250–1254.
Tao, Z. Y. (1988). “Support design of tunnels subjected to rockbursting.”
Rock mechanical power plants, Romana, ed., Balkema, Rotterdam,
Netherlands, 407–411.
Tesfamariam, S., and Liu, Z. (2010). “Earthquake induced damage classi-
fication for reinforced concrete buildings.” Struct. Saf., 32(2), 154–164.
Therneau, T., and Atkinson, E. (1997). “An introduction to recursive
partitioning using the rpart routine.” Technical Rep. No. 61, Section of
Biostatistics, Mayo Clinic, Rochester, MN.
Turchaninov, I. A., et al. (1972). “State of stress in the upper part of the
Earth’s crust based on direct measurements in mines and on tectono-
physical and seismological studies.” Phys. Earth Planet. Inter., 6(4),
229–234.
Vapnik, V. N. (1995). The nature of statistical learning theory, Springer,
New York.
Venables, W. N., and Ripley, B. D. (2002).Modern applied statistics with S,
4th Ed., Springer, New York, 495.
Veredas, F. J., Luque-Baena, R. M., Martín-Santos, F. J., Morilla-Herrera, J. C.,
and Morente, L. (2015). “Wound image evaluation with machine
learning.” Neurocomputing, 164, 112–122.
Wang, G. Y., Zhang, S. X., and Ren, G. F. (2005). “Analysis and prediction
of rock burst in deep mining of Tonglushan copper-iron ore.” Mining
Saf. Environ. Prot., 32(5), 20–22.
Wang, J. A., and Park, H. D. (2001). “Comprehensive prediction of rock-
burst based on analysis of strain energy in rocks.” Tunnel. Underg.
Space Technol., 16(1), 49–57.
Wang, J. L., Chen, J. P., Yang, J., and Que, J. S. (2009). “Method of
distance discriminant analysis for determination of classification of
rockburst.” Rock Soil Mech., 30(7), 2203–2208.
Wang, X. F., Li, X. H., Gu, Y. L., Jin, X. G., Kang, Y., and Li, D. X. (2004).
“Application of BP neural network intoprediction of rockburst in tun-
neling.” Proc., 2004 Int. Symp. on Safety Science and Technology,
China Science Press, Shanghai, China, 617–621.
Wang, Y. C., Shang, Y. Q., Sun, H. Y., and Yan, X. S. (2010). “Research and
application of rockburst intensity prediction model based on entropy
coefficient and ideal point method.” J. China Coal Soc., 35(2), 218–221.
© ASCE 04016003-18 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
http://dx.doi.org/10.1504/IJRAM.2007.014094
http://dx.doi.org/10.1016/j.envsoft.2010.01.004
http://cran.r-project.org/web/packages/gbm/index.html
http://cran.r-project.org/web/packages/gbm/index.html
http://cran.r-project.org/web/packages/gbm/index.html
http://cran.r-project.org/web/packages/gbm/index.html
http://dx.doi.org/10.1016/j.jmgm.2007.06.005
http://dx.doi.org/10.1016/j.jmgm.2007.06.005
http://dx.doi.org/10.1016/j.compstruc.2006.08.084
http://dx.doi.org/10.1016/S0167-9031(87)90854-1
http://dx.doi.org/10.1007/BF01043119
http://dx.doi.org/10.1007/BF01043119
http://dx.doi.org/10.1016/S0167-9031(89)90404-0
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000141
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000141
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000141
http://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000141
http://dx.doi.org/10.1371/journal.pone.0093950
http://dx.doi.org/10.1016/S0148-9062(96)00039-3
http://dx.doi.org/10.1016/j.strusafe.2009.10.002
http://dx.doi.org/10.1016/0031-9201(72)90005-2
http://dx.doi.org/10.1016/0031-9201(72)90005-2
http://dx.doi.org/10.1016/j.neucom.2014.12.091
http://dx.doi.org/10.1016/S0886-7798(01)00030-X
http://dx.doi.org/10.1016/S0886-7798(01)00030-X
Wang, Y. H., Li, W. D., Li, Q. G., Xu, Y., and Tan, G. H. (1998). “Method
of fuzzy comprehensive evaluations for rockburst prediction.” Chinese
J. Rock Mech. Eng., 17(5), 493–501.
Weihs, C., Ligges, U., Luebke, K., and Raabe, N. (2005). “klaR: Analyzing
German business cycles.” Data analysis and decision support, D. Baier,
R. Decker, and L Schmidt-Thieme, eds., Springer, Berlin, 335–343.
Wiles, T. D. (2002). “Loading system stiffness—A parameter to evaluate
rockburst potential.” Proc., 1st Int. Seminar on Deep and High Stress
Mining, Australian Centre for Geomechanics, Perth, Australia, 10.
Wiles, T. D. (2005). “Rockburst prediction using numerical modelling:
Realistic limits for failure prediction accuracy.” Proc., 6th Int. Symp.
on Rockbursts and Seismicity in Mines (RaSiM 6), Australian Centre
for Geomechanics, Perth, Australia.
Wold, S., Sjostrom, M., and Eriksson, L. (2001). “PLS-regression: A basic
tool of chemometrics.” Chemom. Intell. Lab. Syst., 58(2), 109–130.
Wu, X., et al. (2007). “Top 10 algorithms in data mining.” Knowl. Inf. Syst.,
14(1), 1–37.
Xia, B. W. (2006). “Study on prediction and forecast of geologic disaster in
highway tunnel construction.” Master’s thesis, Chongqing Univ.,
Chongqing, China.
Xiao, X. P. (2005). “A study on the prediction and prevention of rockburst
traffic tunnel of Jinping II hydropower station.” Master’s thesis,
Chengdu Univ. of Technology, Chengdu, China.
Xu, L. S., and Wang, L. S. (1999). “Study on the laws of rockburst and its
forecasting in the tunnel of Erlang Mountain road.” Chinese J. Geotech.
Eng., 21(5), 569–572.
Xu, M. G., Du, Z. J., Yao, G. H., and Liu, Z. P. (2008). “Rockburst pre-
diction of Chengchao iron mine during deep mining.” Chinese J. Rock
Mech. Eng., 27(S1), 2921–2928.
Yang, J. L., Li, X. B., Zhou, Z. L., and Lin, Y. (2010). “A fuzzy assessment
method of rock-burst prediction based on rough set theory.”Metal Mine,
6, 26–29.
Yi, Y. L., Cao, P., and Pu, C. Z. (2010). “Multi-factorial comprehensive
estimation for Jinchuan’s deep typical rockburst tendency.” Sci. Tech-
nol. Rev., 28(2), 76–80.
Yu, X. Z. (2009). “Highway tunnel geological disaster prediction and the
development of treatment measures database management system.”
Master’s thesis, Chongqing Univ., Chongqing, China.
Zar, J. H. (2010). Biostatistical analysis, 5th Ed., Prentice Hall, Upper
Saddle River, NJ, 241.
Zhang, B. (2007). “Study on security and stability of deep buried highway
tunnel based on anisotropic theory.” Ph.D. thesis, Chinese Academy of
Sciences, Wuhan, China.
Zhang, C. Q., Feng, X. T., Zhou, H., Qiu, S. L., and Wu, W. P. (2012a).
“Case histories of four extremely intense rockbursts in deep tunnels.”
Rock Mech. Rock Eng., 45(3), 275–288.
Zhang, C. Q., Zhou, H., and Feng, X. T. (2011). “An index for estimating
the stability of brittle surrounding rock mass: FAI and its engineering
application.” Rock Mech. Rock Eng., 44(4), 401–414.
Zhang, G. C., Gao, Q., Du, J. Q., and Li, K. K. (2013). “Rockburst criterion
based on artificial neural networks and nonlinear regression.” J. Central
South Univ. (Sci. Technol.), 44(7), 2977–2981.
Zhang, J. F. (2010). “Study on prediction by stages and control technology
of rockburst hazard of Daxiangling highway tunnel.” Master’s thesis,
Southwest Jiaotong Univ., Chendu, China.
Zhang, J. J., and Fu, B. J. (2008). “Rockburst and its criteria and control.”
Chinese J. Rock Mech. Eng., 27(10), 2034–2042.
Zhang, J. J., Fu, B. J., Li, Z. K., Song, S. W., and Shang, Y. J. (2012c).
“Criterion and classification for strain mode rockbursts based on five-
factor comprehensive method.” Proc., 12th ISRM Int. Congress on Rock
Mechanics, Harmonising Rock Engineering and the Environment,
Q. Qian and J. Zhou, eds., Taylor & Francis Group, London, 1435–1440.
Zhang, L. W., Zhang, D. Y., and Qiu, D. H. (2010). “Application of exten-
sion evaluation method in rockburst prediction based on rough set
theory.” J. China Coal Soc., 35(9), 1461–1465.
Zhang, L. X., and Li, C. H. (2009). “Study on tendency analysis of rock-
burst and comprehensive prediction of different types of surrounding
rock.” RaSiM7 (2009): Controlling Seismic Hazard and Sustainable
Development of Deep Mines, C. A. Tang, ed., Rinton Press, Princeton,
NJ, 1451–1456.
Zhang, Y. L., Liu, X., and Hu, Z. Q. (2007). “Rock burst forecast based on
artificial neural network in underground engineering.” Hunan Nonfer-
rous Metal, 23(3), 1–4.
Zhang, Z. L. (2002). “Study on rockburst and large deformation of Xuefeng
mountain tunnel of Shaohuai highway.”Master’s thesis, Chengdu Univ.
of Technology, Chengdu, China.
Zhao, H. B. (2005). “Classification of rockburst using support vector
machine.” Rock Soil Mech., 26(4), 642–644.
Zhao, X. F. (2007). “Study on the high geo-stress and rockburst of the
deep-lying long tunnel.” Master’s thesis, North China Univ. of Water
Resources and Electric Power, Zhengzhou, China.
Zhou, J., Li, X. B., and Mitri, H. S. (2015a). “Comparative performance of
six supervised learning methods for the development of models of hard
rock pillar stability prediction.” Nat. Hazards, 79(1), 291–316.
Zhou, J., Li, X. B., and Shi, X. Z. (2012). “Long-term prediction model of
rockburst in underground openings using heuristic algorithms and
support vector machines.” Saf. Sci., 50(4), 629–644.
Zhou, J., Shi, X. Z., Dong, L., Hu, H. Y., and Wang, H. Y. (2010). “Fisher
discriminant analysis model and its application for prediction of
classification of rockburst in deep buried long tunnel.” J. Coal Sci.
Eng. (China), 16(2), 144–149.
Zhou, J., Shi, X. Z., and Li, X. B. (2015d). “Utilizing gradient boosted
machine for the prediction of damage to residential structures owing
to blasting vibrations of open pit mining.” J. Vib. Control, 1–12.
Zhou, K. P., and Gu, D. S. (2004). “Application of GIS-based neural net-
work with fuzzy self-organization to assessment of rockburst tendency.”
Chinese J. Rock Mech. Eng., 23(18), 3093–3097.
Zhu, Y. H., Liu, X. R., and Zhou, J. P. (2008). “Rockburst prediction analy-
sis based on v-SVR algorithm.” J. China Coal Soc., 33(3), 277–281.
Zubelewicz, A., and Mroz, Z. (1983). “Numerical-simulationof rock burst
processes treated as problems of dynamic instability.” Rock Mech. Rock
Eng., 16(4), 253–274.
© ASCE 04016003-19 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
http://dx.doi.org/10.1016/S0169-7439(01)00155-1
http://dx.doi.org/10.1007/s10115-007-0114-2
http://dx.doi.org/10.1007/s10115-007-0114-2
http://dx.doi.org/10.1007/s00603-011-0218-6
http://dx.doi.org/10.1007/s00603-011-0150-9
http://dx.doi.org/10.1007/s11069-015-1842-3
http://dx.doi.org/10.1016/j.ssci.2011.08.065
http://dx.doi.org/10.1007/s12404-010-0207-5
http://dx.doi.org/10.1007/s12404-010-0207-5
http://dx.doi.org/10.1177/1077546314568172
http://dx.doi.org/10.1007/BF01042360
http://dx.doi.org/10.1007/BF01042360many models for predicting rockburst can be valuable and efficient
tools for work in mining and geological engineering.
Supervised learning (SL) has steadily become more mathemati-
cal and more successful in applications over the past 20 years (Jain
et al. 2009; Chou et al. 2013; Chou and Lin 2013; Stephens and
Diesing 2014; Veredas et al. 2015). The use of SL algorithms for
the development of descriptive and predictive data-mining models
has become widely accepted in mining and geotechnical applica-
tions, promising powerful new tools for practicing engineers. Based
on these considerations, the main objective of this study was to
illustrate and compare the suitability of different SL algorithms for
the prediction of rockburst in underground projects. To achieve this,
a research methodology was developed for comparing the perfor-
mance of different SL algorithms, including linear discriminant
analysis (LDA), quadratic discriminant analysis (QDA), partial
least-squares discriminant analysis (PLSDA), naïve Bayes (NB),
k-nearest neighbor (KNN), multilayer perceptron neural network
(MLPNN), classification tree (CT), support vector machine (SVM),
random forest (RF), and gradient-boosting machine (GBM). These
algorithms were specifically chosen because they are increasingly
used in civil engineering yet have not been compared with one an-
other exhaustively, and because of their open-source availability.
The remainder of this paper is organized as follows. The “Back-
ground” section comprehensively reviews the literature on artificial
intelligence and its application to predicting rockburst. The “Meth-
odology” section presents the rockburst database and evaluation
methods, providing a theoretical perspective on the classification
models and data information adopted for subsequent investigation.
SL model development and parameter optimization based on the
rockburst database are then presented. In the “Results and Discus-
sion” section, results are analyzed by performance criteria and non-
parametric statistical testing.
Background
Predicting rockburst in underground projects has been actively re-
searched for more than 50 years. Various techniques are used to
assess rockburst; although similar, they have different taxonomies.
The reported techniques can be grouped as empirical, numerical
simulation, and statistical artificial intelligence–based. A compre-
hensive literature review for each of these domains is provided in
this section.
In the literature, various empirical methods have been proposed
and often applied in practice, such as the Turchaninov criterion
(Turchaninov et al. 1972), the Barton criterion (Barton et al. 1974),
the Russense criterion (Russenes 1974), the Hoek criterion (Hoek
and Brown 1980), the strain energy storage index Wet (Kidybinski
1981), the energy release rate and excess shear stress method
(Ryder 1988), the Tao Zhenyu criterion (Tao 1988), the burst en-
ergy release index (Singh 1988), the Erlang mountain method (Xu
and Wang 1999), the Hou criterion (Hou 1989), the composite in-
dex criterion (Tan et al. 1991), the burst potential index (Mitri et al.
1999), the local energy release rate (Wiles 2005), the excavation
vulnerability potential (Heal et al. 2006), the potential stress failure
(Mitri 2007), the rockburst vulnerability index (Qiu et al. 2011),
and the five-factors comprehensive criterion (Zhang et al. 2012c).
These methods or criteria have been employed with local monitor-
ing data and laboratory tests to study the mechanical characteristics
of rockbursts.
Some studies have suggested that rockbursts can be predicted by
comparing the stiffness of the bursting rock to the stiffness of the
host rock (Gill et al. 1993; Mitri et al. 1993). Empirical models give
very good results. Their most prominent advantage lies in their sim-
ple form and convenience for engineering applications. Also, they
are often derived from the interpolation of curves and barely have a
physical meaning. Their main weakness is their limited validity in
some cases, giving rise to doubts concerning their effectiveness.
These empirical approaches are open to improvement because they
are based on limited collected data.
Meanwhile, the vast majority of work related to understanding
and predicting rockburst has been undertaken by means of numeri-
cal simulation (Zubelewicz and Mroz 1983; Mitri et al. 1993; Tang
et al. 1997; Sharan 2007; Castro et al. 2012). In terms of strainburst,
rockburst potential can be assessed and evaluated via an excavation
scheme using the failure approaching index (FAI) (Zhang et al.
2011), the burst potential index (BPI) (Mitri et al. 1999), the brittle
shear ratio (BSR) (Castro et al. 2012), the local energy release rate
(LERR) (Wiles 2002), the local energy release density (LERD)
(Wiles 2002; Jiang et al. 2010), or the energy release rate (ERR)
(Salamon 1984). If there is a controlling structural surface, strain-
structural surface-slip rockburst or fault-slip burst may occur. The
former can be evaluated using FAI, the bursting potential ratio
(BPR) (Simon 1999), and LERR, whereas the latter can be evalu-
ated using excess shear stress (ESS) (Ryder 1988), BPR, and the
out-of-balance index (OBI) (Simon 1999).
Because of the lack of appropriate energy criteria, absolute rock-
burst classification cannot be obtained from the rockburst of energy
that is given by FAI, ESS, LERD, LERR, or ERR. This problem
is waiting for the statistical analysis of a large number of project
examples to be undertaken in future work. Each numerical method
has its strengths and weaknesses. However, the estimation of reli-
able model input parameter values is an increasingly difficult task.
It is therefore essential that full consideration be given to the avail-
ability of realistic input data before applying sophisticated numerical
methods.
Numerous approaches to rockburst prediction have been devel-
oped based on various SL techniques during recent decades. For
example, some studies apply a single learning technique, such as
artificial neural networks (ANNs) (Feng andWang 1994; Feng et al.
1996; Chen et al. 2013), distance discriminant analysis (DDA)
(Gong and Li 2007), support vector machine (SVM) (Zhao 2005),
Bayes discriminant analysis (BDA) (Gong et al. 2010), or Fisher
linear discriminant analysis (LDA) (Zhou et al. 2010). On the other
hand, some systems are based on hybrid (Zhou et al. 2012; Adoko
et al. 2013; Liu et al. 2013) or ensemble (Ge and Feng 2008; Dong
et al. 2013) learning techniques. Table 1 summarizes the primary
studies in rockburst prediction that use SL methods. These studies
approached the problems of rockburst, but never completely solved
them. A certain method may be favorable for some cases but not
good enough for other cases. Notably, accuracy varies 66.5–100%,
which is an extremely large deviation in rockburst prediction.
Methodology
Supervised Learning Methods
Several studies have compared multiple SL techniques
(e.g., Kotsiantis 2007; Wu et al. 2007; Jain et al. 2009;
© ASCE 04016003-2 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wongResaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
Arditi and Pulket 2010; Mahfouz and Kandil 2011; Son et al. 2011;
Chou 2012; Chou and Lin 2013; Stephens and Diesing 2014;
Lavecchia 2015; Veredas et al. 2015; Zhou et al. 2015a). Based on
these studies and the focus here on rockburst classification, ten
classification techniques were considered in the current study and
compared with respect to their predictive performance. They were
LDA (Fisher 1936; González-Rufino et al. 2013), QDA (Grouven
et al. 1996; González-Rufino et al. 2013), PLSDA (Wold et al.
2001), KNN (Lai 2007; Berrueta et al. 2007), NB (Wu et al. 2007;
Kotsiantis 2007), CART (Breiman et al. 1984; Therneau and
Atkinson 1997; Garzón et al. 2006; Hothorn et al. 2006), RF
(Breiman 2001), MLPNN (Haykin 1999; Pino-Mejías et al. 2010),
SVM (Vapnik 1995; Zhou et al. 2012), and GBM (Friedman 2001;
Arditi and Pulket 2005; Guelman 2012; Zhou et al. 2015d). These
techniques share certain characteristics that make them interesting
to the current analysis: (1) they are increasingly used; (2) some of
them have been used in rockburst classification tasks with good
results and are known to enable the analysis of more complex
nonlinear relationships; (3) they have efficient implementations;
(4) they use different classifiers to reduce the uncertainty of the
results that might be related to the algorithm that each classifier
uses; and (5) they produce models that allow fast classification
processing. For a more in-depth discussion, the reader is referred
to the relevant references.
Data Sources and Data Description
To measure the performance of the developed SL approaches, this
study used data from 246 cases of rockburst events collected from
the original Zhou et al. (2012) database and 30 other studies. These
sources are reliable and include references published over the
period 1991–2013, approximately 70% between 2008 and 2013.
The updated database contains more detailed information and sev-
eral recently completed underground projects. Details are presented
in Table S1 (available at http://www.ascelibrary.org), which lists the
main database parameters. The general database contains data on
more than 20 underground engineering projects and 246 rockburst
events. In general, the projects chosen experienced the most sig-
nificant rockburst activity. In addition, they come with well-
documented records and published reports that analyze probable
causes and mechanisms. The database here is based on one devel-
oped by Zhou et al. (2012) modified with the addition of recent
projects from the literature and other sources. Also, an effort
was made to fill in Zhou et al.’s missing data fields.
The post-1991 distribution of rockburst data used in this study is
shown in Fig. 1(a) as a pie chart illustrating the proportion of the
four types of rockburst in underground engineering: none (43
cases), low (78 cases), moderate (81 cases), and high (44 cases).
The box plot of the original data set is given in Fig. 1(b). For most
of the data groups, the median is not in the center of the box, which
indicates that the distribution of most of them is not symmetric. In
addition, all dependent variables have some outliers except UTS
and σt (uniaxial tensile strength of the rock) for high and moderate
rockburst types, MTS or σθ (maximum tangential stress around the
excavation) and D (depth) for high rockburst types, stress concen-
tration factor (SCF or σθ=σc) (Martin et al. 1999) for high and low
rockburst types, andUCS and rock brittleness index B2½B2 ¼ ðσc −
σtÞ=ðσc þ σtÞ� for none types. In Fig. 1(c), the scatter plot matrix in
the lower panel demonstrates the pairwise relationship between
parameters, with corresponding correlation coefficients shown in
the upper panel and the marginal frequency distribution for each
parameter shown on the diagonal. It can be observed that the
parameter Wet (strain energy storage index) is notably correlated
with MTS.
R is a popular open-source software for statistical computing
and data visualization available for most mainstream platforms
(R Development Core Team 2013). All data processing in this
study was performed using R software (version 3.01). The R lan-
guage and environment for statistical computing has continued
to gain acceptance among geologists (e.g., Grinand et al. 2008;
Tesfamariam and Liu 2010; Chou and Lin 2013; Zhou et al.
2015a, d). The platform adopted to develop the 10 SL approaches
was a personal computer with the following features: an Intel (R)
Core i5-4200 CPU running at 2.90 GHz and 8 GB RAM, the
Windows 8 operating system, and the R development environment.
Because R provides the most common SL classification algo-
rithms, it was the common platform for all of the classification
methods, which use the respective R packages. The packages nec-
essary for each model and the functions used to build the models
are summarized in Table 2. Further details about input parameters,
implementation, and references can be found in the R documenta-
tion. Each R package implements its own performance measure-
ment procedures, which cannot be directly compared. Scaling of
the input-output data was generally required prior to processing.
Table 1. Studies on SL Rockburst Classification with Influence Factors and Accuracy Values
Algorithm D σθ σc σt Wet σθ=σc σc=σt Kv E β Accuracy (%) Data Reference
DDA — — — — X X X — — — 100 15 Gong and Li (2007)
BDA — — — — X X X — — — 100 21 Gong et al. (2010)
FDA — — — — X X X — — — 100 15 Zhou et al. (2010)
SVM — X X X X — — — — — 100 16 Zhao (2005)
v-SVR — X X X X X X — — — 93.75 45 Zhu et al. (2008)
HSVM X X X X X X X — — — 66.67–90 132 Zhou et al. (2012)
ANFIS — X X X X X X — — — 66.5–95.6 174 Adoko et al. (2013)
ANN — X X X X — — X X X 100 10 Feng and Wang (1994)
— X X X X — — — — — 72.2 18 Chen et al. (2003)
AdaBoost — X — — X X X — — — 87.8–89.9 36 Ge and Feng (2008)
RF — X X X X — — — — — 100 46 Dong et al. (2013)
CM — X X X X X X — — — 90–100 162 Liu et al. (2013)
Note: ANFIS = adaptive neuro fuzzy inference system; AdaBoost = adaptive boosting; B1 = rock brittleness index, B1 ¼ σc=σt; CM = cloud model; D =
depth, m; E = Young’s modulus; HSVM = heuristic algorithms and support vector machines; Kv = rock mass intact coefficient; MTS or σθ = maximum
tangential stress of surrounding rock, MPa; NA = not available; SCF or σθ=σc = stress concentration factor;UCS or σc = uniaxial compressive strength of rock,
MPa; UTS or σt = uniaxial tensile strength of rock, MPa; v-SVR = v-support vector regression; Wet = strain energy storage index; β = separation angle
between the strike of the main joint set and the maximal principal stress, degrees.
© ASCE 04016003-3 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
http://ascelibrary.org/doi/suppl/10.1061/%28ASCE%29CP.1943-5487.0000553/suppl_file/Supplemental_Data_CP.1943-5487.0000553_Zhou.pdf
http://www.ascelibrary.org
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
Fig. 1. Data visualization: (a) pie chart showing the distribution of observed rockburst cases; (b) box plot of each variable for the four rockburst
groups; (c) pairs plot of rockburst samples for the 8 parameters
Table 2. R Packages and Functions Used to Run the SL Models
Model R package R function
Tuning
parameters Tuning range
LDA MASS lda None None
QDA MASS qda None None
PLSDA pls pls ncomp ncomp ¼ f1,2; : : : ; tg
KNN caret knn K K ¼ f1,3,5,7,9,11,25,51,101g
NB klaR naiveBayes usekernel usekernel¼ fTRUE; FALSEg
DT rpart ctree cp cp ¼ f0; 0.0202; 0.0404; 0.0606; 0.0808; 0.101; 0.121; 0.141; 0.162; 0.182g
MLPNN nnet nnet decay decay ¼ f0,1 × 10−4; 0.000237; 0.000562; 0.00133; 0.00316; 0.0075; 0.0178; 0.0422; 0.1g
H 1X —
C X X X X X X — X
D X X X X X — — —
E — X X X X X X —
F — X X X X X — X
G X — — — X X X —
H X — — — X X — X
I — — — — X X X —
J — — — — X X — X
K — X X X X — — —
© ASCE 04016003-5 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
where xii = cell count in the main diagonal; n = number of exam-
ples; C = number of class values; and xi, xi are column and row
total counts, respectively.
Landis and Koch (1977) proposed a scale to describe the
degree of concordance associated with the Kappa statistic:
−1.000–0.000 = poor; 0.000–0.200 = slight; 0.210–0.400 = fair;
0.410–0.600 = moderate; 0.610–0.800 = substantial; and 0.810–
1.000 = almost. Obviously, Cohen’s Kappa ranges from −1 (total
disagreement) to 0 (random classification) to 1 (perfect agreement).
A Kappa below 0.4 is an indication of poor agreement, whereas a
Kappa of 0.4 and above is an indication of good agreement (Landis
and Koch 1977; Sakiyama et al. 2008).
Within-Class Classification Metrics
To evaluate the performance of each classification, a confusion
matrix was used, and its recall, precision, and F-measure were cal-
culated (Mahfouz and Kandil 2011; Chou and Lin 2013; Lavecchia
2015):
Recalli ¼
�
xii
xþC
�
× 100% ¼
�
xiiP
C
i¼1 xiC
�
× 100% ð3Þ
Precisioni ¼
�
xii
xþC
�
× 100% ¼
�
xiiP
C
i¼1 xiC
�
× 100% ð4Þ
F-measure ¼ 2Recall × Precision
Recallþ Precision
ð5Þ
Proposed Model Validation Methods
Model validation methods include substitution, holdout, and cross-
validation (CV), which includes leave-one-out CV, leave-more-out
CV, and k-fold CV (Chou et al. 2013; Chou and Lin 2013; Kuhn
and Johnson 2013). Several adjustable “tuning parameters” used by
the SL algorithms to optimize classification performance were ex-
amined using 10-fold CV in terms of computation time and vari-
ance, with 10 the number of folds recommended by Kohavi (1995)
when comparing the performance of machine learning algorithms
(Le-Thi-Thu et al. 2011; Chou and Lin 2013; Clark 2013). Thus, in
the present work a 10-fold CV procedure was used in the construc-
tion of each model for each possible parameter configuration. In
this procedure, compounds of the TS were randomly divided into
10 subsets. Nine subsets were used as novel TS to develop each SL
model, and the holdout set was used to “predict” the performance of
the fitted model. This process was repeated 10 times on different
TSs until every instance was used exactly once for testing. Finally
the CVestimate of overall accuracy was calculated by simply aver-
aging the 10 individual accuracy measures. This procedure was
used for the selection of parameters and to avoid overfitting of the
SL models. The PS was not used in the development of the model,
but was used to test the predictive power of the model as completed.
The predictive models were constructed using selected variables
and TSs and then applied to PSs, as shown in Fig. 2.
SL Model Development and Parameter Optimization
Most of the SL methods, except LDA and QDA from discriminant
classifiers, include at least one tuning parameter to avoid either over-
fitting or underfitting. The train function from the R caret package
(Kuhn 2012) creates a grid of tuning parameters for a number of
classification routines, allowing a single consistent environment for
training each SL algorithm and tuning their associated parameters.
After assessing the optimal parameters, the entire TS was used to
build the final rockburst prediction model. The term “rockburst” re-
fers to the classification task. A desired tune length variable can be
passed to the train function in the caret package (Kuhn 2012). Op-
timal values for tuning parameters were selected using 10-fold CV
based on the original TS, with the original test removed completely
from the CV process. Tuning parameters were considered optimized
based on the classification models that achieved the highest overall
classification during the CV process. This means that the one with
the highest accuracy was found and thus an optimal solution could be
searched. Specific details on tuning parameters used by the 10 SL
algorithms examined in this study are discussed in this section:
• LDA: performed using the lda() function from the class package
in R (Venables and Ripley 2002); as described in González-
Rufino et al. (2013), the LDA classifier does not need hyper-
parameter tuning;
• QDA: implemented in the class package in R (Venables and
Ripley 2002); as described in González-Rufino et al. (2013),
the QDA classifier does not need hyperparameter tuning;
• PLSDA: performed in the R pls package (Wold et al. 2001) and
requires ncomp to be specified; ncomp represents the appropri-
ate number of latent variables to be used in the model (Kuhn
2008); three components are good for the rockburst prediction
of Model A [Fig. 3(a)];
• KNN: requires selection of K, representing the number of near-
est neighbors to consider in the classification; it also uses Eu-
clidean distance; KNN training and prediction were carried out
using the knn() function in the R class package (Venables and
Ripley 2002); for Model A, the optimal number of nearest
neighbors was determined to be 5 by 10-fold CV [Fig. 3(b)];
Table 4. Standard Rockburst Classifications
Classification Failure characteristics (Zhou et al. 2012) Failure characteristics (Russenes 1974)
High (H) The surrounding rock bursts severely and is suddenly thrown out
or ejected into the tunnel, accompanied by a strong burst and a
roaring sound, air spray, and storm phenomena with continuity.
Rapid expansion to the deep surrounding rock occurs
Severe rockfalls from roof and walls begin immediately after
blasting. Slabs pop from the floor, or the floor may heave.
Considerable overbreaks and deforming of the periphery occur.
Rock noises of gunshot strength may be heard
Moderate (M) The surrounding rock is deformed and fractured and a
considerable number of rock chips are ejected. Loose and
sudden destruction occurs, accompanied by crisp crackling often
in the local cavern of surrounding rock
Considerable slabbing and loosening of rock occur, and a
deformed periphery tends to develop with time. Strong cracking
noises from the rock are heard
Low (L) The surrounding rock is deformed, cracked, or rib-spalled. There
is a weak sound and no ejection phenomenon
Cracking and loosening of rock occur, and light noises emerge
from the rock
None (N) There are no sounds of rockburst and no rockburst activity There are no stability problems caused by rock stresses and no
noises from the rock
© ASCE 04016003-6 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
• NB: implemented using the klaR package (Weihs et al. 2005);
either a kernel density estimate or a normal density estimate can
be used (John and Langley 1995); usekernel = TRUE imple-
ments a nonparametric kernel density estimation technique to
establish priors for each input; alternatively, usekernel = FALSE
assumes that the marginal distributions are normal; for this
application, using Model A as an example, a kernel density es-
timate was a better choice [Fig. 3(c)];
• CT: The recursive partitioning (rpart) classification tree
(Therneau and Atkinson 1997) was used in this study; the
rpart() function is used in Rwith the class-splitting functionand
considers different parameter configurations for the value of
complexity cp; it runs a 10-fold CV to average and raise the
reliability of the resulting tree [Fig. 3(d)];
• RF: implemented from the randomForest package (Liaw and
Wiener 2002) to train RF, which involves finding optimum va-
lues for a number of classification trees (ntree) and a number of
variables (mtry) randomly selected at each split in the tree-building
process; overall classification accuracy is more sensitive to mtry
and not much effected by ntree (Breiman and Cutler 2004), so
ntree is fixed at a default value of 500 andmtry is tested for t values,
where t is the number of input layers in each classification setup;
ntree ¼ 500 for each model, andmtry ¼ 5 in each node is obtained
for Model A [Fig. 3(e)].
• ANN: implemented from the multilayer perceptron (MLP)
using the nnet function from R’s nnet package (Venables and
Ripley 2002); ANN tunes the number of hidden neurons H
in the range 1Results from the Independent Test Set
The performance (in terms of accuracy) of PS fell into the range
[0.438–0.766], as shown in Table 6. Fig. 5 shows box plots of
the performance of different PS predictors. Obviously, the RF pre-
dictor achieved the highest average accuracy rate (71.18%), fol-
lowed by GBM and SVMRadial with average accuracy rates of
69.33 and 64.62%, respectively. PLSDA had the lowest accuracy
with a rate of 48.86%. As for Kappa value, the predictive accuracy
Table 6. Classification Performance Metrics across 11 Models
Model Type Metric LDA QDA PLSDA KNN NB CT RF ANN SVML SVMR GBM
A TS Accuracy 0.549 0.516 0.541 0.544 0.539 0.592 0.670 0.637 0.565 0.571 0.687
Kappa 0.358 0.327 0.348 0.357 0.359 0.422 0.539 0.495 0.392 0.392 0.562
PS Accuracy 0.500 0.563 0.484 0.641 0.625 0.609 0.734 0.578 0.578 0.609 0.750
Kappa 0.321 0.415 0.304 0.511 0.496 0.459 0.638 0.431 0.422 0.466 0.659
B TS Accuracy 0.488 0.551 0.514 0.539 0.556 0.592 0.676 0.627 0.532 0.576 0.682
Kappa 0.273 0.369 0.310 0.350 0.382 0.422 0.548 0.482 0.336 0.406 0.555
PS Accuracy 0.484 0.500 0.484 0.625 0.594 0.609 0.734 0.500 0.594 0.609 0.766
Kappa 0.298 0.332 0.304 0.489 0.453 0.459 0.637 0.333 0.442 0.465 0.681
C TS Accuracy 0.564 0.555 0.575 0.555 0.545 0.592 0.660 0.632 0.559 0.592 0.670
Kappa 0.381 0.377 0.393 0.378 0.364 0.422 0.523 0.489 0.375 0.432 0.535
PS Accuracy 0.531 0.563 0.531 0.609 0.625 0.609 0.734 0.641 0.641 0.578 0.734
Kappa 0.363 0.411 0.362 0.467 0.495 0.459 0.639 0.515 0.508 0.429 0.641
D TS Accuracy 0.559 0.527 0.560 0.532 0.608 0.597 0.677 0.621 0.549 0.582 0.671
Kappa 0.374 0.336 0.377 0.349 0.449 0.427 0.554 0.469 0.356 0.407 0.542
PS Accuracy 0.563 0.578 0.531 0.656 0.625 0.609 0.719 0.656 0.563 0.656 0.734
Kappa 0.404 0.432 0.365 0.532 0.494 0.459 0.616 0.537 0.400 0.530 0.639
E TS Accuracy 0.532 0.550 0.533 0.554 0.556 0.592 0.642 0.608 0.517 0.544 0.643
Kappa 0.323 0.367 0.333 0.375 0.381 0.422 0.497 0.455 0.317 0.364 0.505
PS Accuracy 0.547 0.484 0.438 0.625 0.563 0.609 0.750 0.578 0.563 0.672 0.734
Kappa 0.376 0.313 0.225 0.486 0.408 0.459 0.659 0.428 0.397 0.557 0.636
F TS Accuracy 0.547 0.516 0.570 0.536 0.550 0.592 0.625 0.614 0.548 0.554 0.622
Kappa 0.348 0.319 0.384 0.351 0.375 0.422 0.477 0.460 0.358 0.366 0.470
PS Accuracy 0.563 0.578 0.516 0.609 0.594 0.609 0.750 0.578 0.594 0.563 0.672
Kappa 0.396 0.433 0.332 0.471 0.450 0.459 0.657 0.429 0.441 0.404 0.554
G TS Accuracy 0.521 0.511 0.477 0.576 0.561 0.564 0.621 0.675 0.554 0.626 0.682
Kappa 0.308 0.320 0.252 0.402 0.381 0.380 0.467 0.550 0.366 0.484 0.557
PS Accuracy 0.547 0.516 0.453 0.578 0.672 0.563 0.703 0.594 0.594 0.672 0.672
Kappa 0.376 0.346 0.242 0.419 0.550 0.390 0.595 0.451 0.446 0.557 0.550
H TS Accuracy 0.532 0.510 0.495 0.565 0.550 0.571 0.638 0.624 0.575 0.609 0.682
Kappa 0.326 0.321 0.269 0.378 0.367 0.404 0.490 0.482 0.402 0.459 0.554
PS Accuracy 0.594 0.531 0.469 0.578 0.641 0.609 0.703 0.656 0.625 0.625 0.672
Kappa 0.433 0.369 0.252 0.425 0.507 0.464 0.595 0.536 0.487 0.493 0.552
I TS Accuracy 0.494 0.523 0.500 0.571 0.550 0.592 0.559 0.625 0.558 0.579 0.615
Kappa 0.260 0.324 0.274 0.388 0.367 0.423 0.384 0.475 0.378 0.406 0.465
PS Accuracy 0.500 0.500 0.453 0.578 0.625 0.563 0.656 0.625 0.578 0.656 0.656
Kappa 0.294 0.317 0.230 0.422 0.489 0.390 0.530 0.491 0.414 0.529 0.528
J TS Accuracy 0.500 0.510 0.488 0.593 0.555 0.586 0.560 0.592 0.565 0.576 0.610
Kappa 0.269 0.318 0.253 0.428 0.371 0.414 0.389 0.425 0.384 0.402 0.458
PS Accuracy 0.516 0.531 0.500 0.672 0.609 0.563 0.656 0.594 0.578 0.625 0.703
Kappa 0.314 0.369 0.294 0.552 0.462 0.390 0.529 0.444 0.418 0.489 0.596
K TS Accuracy 0.537 0.538 0.559 0.542 0.603 0.597 0.637 0.598 0.571 0.609 0.642
Kappa 0.330 0.345 0.369 0.361 0.445 0.427 0.493 0.428 0.391 0.459 0.502
PS Accuracy 0.547 0.609 0.516 0.594 0.641 0.609 0.766 0.594 0.531 0.625 0.734
Kappa 0.376 0.425 0.333 0.448 0.510 0.459 0.681 0.442 0.354 0.487 0.638
Note: Bold values indicate that Kappa is more than 0.4 (desirable).
© ASCE 04016003-10 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
of the CT, RF, ANN, SVMRadial, and GBM models for TS in
model calibration data was moderate to substantial. The accuracy
of all modeling techniques (except LDA, QDA, and PLSDA) for
the model evaluation PS was moderate to substantial on the basis of
the academic point system, and moderate according to the scale of
concordance.
From Figs. 4 and 5 and Table 6, it can be observed that the pre-
dictions were very unstable along the different SL algorithms, and
none of the models was found to be excellent in respect to every test
measure. However, GBM and RF , particularly GBM, performed
better than KNN, NB, CT, ANN, SVML, and SVMR. There was
no significant difference in terms of generalization performance. In
addition, LDA performed the worst in terms of TS across various
subsets of attributes and its predictive performance was worse than
that of the learners of QDA and PLSDA in most cases. PLSDA
performed the worst in terms of PS across various subsets of attrib-
utes, and its predictive performance was worse than that of the
learners of LDA and QDA in most cases. When 11 models were
compared for their predictive accuracy on the test data sets, it was
found that predictive performance was similar to but in most cases
better than the quality of fit.
The results from various models were also compared with the
subsets of attributes. It was observed that Model B performed the
best and that predictive performance was comparable and even bet-
ter when a significant number of attributes were removed from the
original data sets. This means that rockburst classification and
attribute selection were successfully applied. Among the 10 SL al-
gorithms, two ensemble versions, GBM and RF, were found to be
the best whereas the nonlinear classification models (SVM, ANN,
NB, QDA, KNN) had slightly better performance and reliability
than the linear classifiers (LDA, PLSDA). All nonlinear SL meth-
ods produced better, more generalizable results than traditional
linear discriminant function analysis. Because machine learners did
not strongly differ in their classification accuracies, the SL method
employed is the prerogative of the investigator.
For quantifying the accuracy of the 11 classifiers in distinguishing
among four rockburst classes, a confusion matrix was investigated
using 64 independent reference samples. Three metrics—precision,
recall, and F-measure—were calculated for each rockburst class us-
ing Model A (Table 7). Precision is the ability to classify instances
correctly, whereas recall is the ability to classify as many instances as
possible. The F-measure offers a global description that considers
both precision and recall. Table 7 clearly shows that the precision,
recall, and F-measure of each classifier in rockburst class detection
of the test set yielded large deviations in results (precision = 35.00–
100.00%, recall = 38.89–100.00%, F-measure = 18.18–95.24%).
For the none category, the best performance was obtained with
the RF method (precision, recall, and F-measure 100, 90.91, and
95.24%, respectively). Precision was above 70% for all classifiers
except CT (50%). Also, the high category showed relatively low
precision (53.33–73.33%), recall (56.25–90.00%), and F-measure
(64.00–73.33%). However, precision, recall, and F-measure for the
Fig. 5. Box plot distributions of the test set by 11 classifiers with results from the independent test set: (a) classification accuracy; (b) Cohen’s Kappa
© ASCE 04016003-11 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
tsr
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
high category were very low (below 75%). Table 7 shows that the
crucial issue was the unsatisfactory recognition rate of the light cat-
egory. In particular, the confusion matrix proved that the most
prominent source of error was the misclassification of light as none
and moderate. This was likely the effect of overlapping classifica-
tion rules and the small number of samples in this class. Some SL
methods substantially outperformed others for this classification
problem. Obviously GBM and RF were both capable of achieving
high accuracy for all classes despite the heavily unbalanced data set.
In general, accuracy and Kappa performances did not differ
much. The behavior of both metrics was quite similar, taking into
account that the differences in Kappa are usually lower because of
the compensation of random success that it offers. RF produced the
best outcome in terms of classification accuracy and Kappa for all
PS. Although GBM showed the highest TS classification accuracy
for Models B, C, E, F, I, J, and K (Table 6), it was the most com-
putationally intensive technique and took the longest to train. Its
high classification accuracy likely resulted from the computationally
intensive back-propagation process, during which feature weights
were modified according to an iterative algorithm. As noted by
Zhou and Gu (2004), however, the accuracy of rockburst analysis
and assessment is entirely dependent on the reliability of the raw
mechanics data and the rationality of mathematical models. More-
over, every approach has its strengths and weaknesses, Table 8
summarizes the advantages and disadvantages of the various SL
methods. In particular, in terms of model implementation and com-
putational cost, model training with ANN and GBM was found to
be the most time-consuming.
Statistical Test for Comparing SL Classification
Algorithms
When the comparison included more than two classifiers over
multiple data sets, the study followed the recommendation of
Table 8. Friedman Test Rankings for the 11 Algorithms Considered
Type Model LDA QDA PLSDA KNN NB CT RF ANN SVML SVMR GBM
TS A 11 7 10 8 6 4 2 3 9 5 1
B 11 7 10 8 6 4 2 3 9 5 1
C 7 9.5 6 9.5 11 4.5 2 3 8 4.5 1
D 8 11 7 10 4 5 1 3 9 6 2
E 10 7 9 6 5 4 2 3 11 8 1
F 9 1 6 10 8 4 1 3 5 7 2
G 9 10 11 5 7 6 4 2 8 3 1
H 9 10 11 7 8 6 2 3 5 4 1
I 1 9 10 5 8 3 7 1 6 4 2
J 10 9 11 2 8 4 7 3 6 5 1
K 11 10 8 9 4 6 2 5 7 3 1
Average rank 9.27 9.50 8.91 7.23 7.18 4.59 2.91 2.91 7.27 4.86 1.27
PS A 10 9 11 3 4 5.5 2 7.5 7.5 5.5 1
B 10.5 8.5 10.5 3 6.5 4.5 2 8.5 6.5 4.5 1
C 10.5 9 10.5 6.5 5 6.5 1.5 3.5 3.5 8 1.5
D 9.5 8 11 4 6 7 2 4 9.5 4 1
E 9 10 11 4 7.5 5 1 6 7.5 3 2
F 9 7.5 11 4.5 6 4.5 1 7.5 10 3 2
G 9 10 11 7 3.5 8 1 5.5 5.5 3.5 2
H 8 10 11 9 4 7 1 3 5.5 5.5 2
I 9.5 9.5 11 6.5 4.5 8 1.5 4.5 6.5 3 1.5
J 10 9 11 2 5 8 3 6 7 4 1
K 9 5.5 11 7.5 3 5.5 1 7.5 10 4 2
Average rank 9.45 8.73 10.91 5.18 5.00 6.32 1.55 5.77 7.18 4.36 1.55
Note: The numbers 1–11 are performance rankings according to the Friedman test; bold indicates the best performance values.
Table 7. Classification Performance in Terms of Within-Class Classification Metrics for Model A with SL Classifiers on the Validation Data Set
Category Metrics
SL method
LDA QDA PLSDA KNN NB CT RF ANN SVML SVMR GBM
None Precision (%) 70.00 80.00 70.00 80.00 90.00 50.00 100.00 90.00 70.00 90.00 100.00
Recall (%) 53.85 47.06 50.00 80.00 75.00 83.33 90.91 69.23 70.00 81.82 83.33
F-measure (%) 60.87 59.26 58.33 80.00 81.82 62.50 95.24 78.26 70.00 85.72 90.91
Low Precision (%) 36.84 10.53 36.84 52.63 52.63 36.84 68.42 57.89 57.89 52.63 68.42
Recall (%) 38.89 66.67 35.00 52.63 52.63 70.00 81.25 47.83 44.00 52.63 72.22
F-measure (%) 37.84 18.18 35.90 52.63 52.63 48.27 74.29 52.38 50.00 52.63 70.27
Moderate Precision (%) 50.00 75.00 45.00 55.00 50.00 85.00 80.00 35.00 40.00 45.00 80.00
Recall (%) 43.48 51.72 50.00 50.00 71.43 50.00 61.54 46.67 53.33 50.00 66.67
F-measure (%) 46.51 61.22 47.37 52.38 58.82 62.96 69.57 40.00 45.71 47.37 72.73
High Precision (%) 53.33 73.33 53.33 66.67 73.33 66.67 60.00 66.67 66.67 60.00 60.00
Recall (%) 80.00 73.33 66.67 76.92 57.89 71.43 81.82 76.92 71.43 56.25 90.00
F-measure (%) 64.00 73.33 59.26 71.43 64.70 68.97 69.23 71.43 68.97 58.06 72.00
© ASCE 04016003-12 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
Demšar (2006) that the Friedman test be followed by the corre-
sponding post hoc Nemenyi test. According to the null hypothesis,
which states that all algorithms are equivalent and so their ranks Rj
should be equal, the Friedman statistic is given by (Demšar 2006;
Luengo et al. 2009)
x2f ¼ 12N
λðλþ 1Þ
�Xλ
j¼1
R2
j − λðλþ 1Þ2
4
�
ð6Þ
where λðλ > 2Þ = number of classifiers; N = number of data sets;
and Rj (Rj ¼
P
λ
j¼1 R
j
i=N) = average rank of classifier j on all
data sets.
However, the Friedman test has been shown to be unnecessarily
restrictive; therefore, Iman and Davenport (1980) tested FID, de-
rived from the Friedman test, which is less conservative than the
Friedman statistic (Luengo et al. 2009):
FID ¼ ðN − 1Þx2f
Nðλ − 1Þ − x2f
ð7Þ
FID is distributed according to the F-distribution with λ − 1 and
ðλ − 1ÞðN − 1Þ degrees of freedom (Sheskin 2006; Zar 2010).
Once computed, FID can be checked against critical values of
the F-distribution, and the null hypothesis with the critical level
α can accepted if FID ≥ Fα½λ − 1; ðλ − 1ÞðN − 1Þ� or rejected
if FID 1.927, the null hypoth-
esis was rejected for stating that there is no difference in the per-
formance of these 11 models over the 11 data sets.
The numbers in the scale represent the average rank; the higher
the rank, the worse the performance of a classifier (Sheskin 2006).
From Table 9, the average ranking obtained by the Friedman test
for accuracy shows that GBM was the best-performing algorithm
for TS, whereas QDA was the worst-performing. Although it has
traditionally been claimed that ANN and SVM provide a good per-
formance for rockburst problems, in this study they were found to
be significantly worse than the RF and GBM classifiers.
Nemenyi’s post hoc test compares the performance of classi-
fiers for TS, using the accuracy values from the 11 classifiers, via
10-fold CV. With 11 classifiers, the critical value qα was 3.219
(α ¼ 0.05), so CD ¼ 3.219×
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi½11× ð11þ 1Þ�=ð6× 11Þp ¼ 4.55.
Although model performance was compared using multiple experi-
ments from a single data set, the validity of this statistical inference
procedure did not diminish. Fig.6(a) plots the outcomes from the
Nemenyi test for TS according to rank. The learning systems con-
nected with a line are those that performed equivalently to the best-
ranked learning method according to Nemenyi’s test at α ¼ 0.05.
This means that all algorithms above this line performed significantly
worse than the best model. From Fig. 6(a), it can be concluded that
the average rank of GBM was 1.27, offering superior performance
among the 11 models, although with a 95% confidence interval
(CI) its performance could not be distinguished from the perfor-
mance of models developed using GBM, ANN, RF, CT, and
SVMR. When the difference between the average ranks of two
models was smaller than the value of CD, the difference in their
performance was not significant, as indicated by the bold straight
lines in Fig. 6(a). Other comparisons indicated statistically signifi-
cant differences in model performance.
The Friedman and Nemenyi tests were also applied for PS. The
average ranks of SL classifiers for PS are given in Table 9. The
Friedman test checks the null hypothesis: x2f ≈ 89.054, FID ¼
½ð11−1Þ×89.054�=½11ð11−1Þ−89.054�≈42.516> 1:927. Thus,
the null hypothesis was also rejected. Fig. 6(b) shows the outcome
of the Nemenyi post hoc tests for the test data. It can be seen that
RF and GBM achieved the lowest average rank (1.55), followed
by SVMR (4.36), as shown in Table 9. Unexpectedly, the PLSDA
classifier seemed to be the worst-performing algorithm. RF/GBM
performance could not be distinguished from models developed
using SVMR, NB, and KNN using a using 95% CI. Obviously, the
differences in performance among SVMR, NB, KNN, ANN, and
CT with SVML, as well as NB, KNN, ANN, CT, and SVML with
QDA, were statistically indistinguishable.
Results for the Full Data Set Using Empirical Criteria
Methods
Some empirical criteria [including Russenes criterion (1974), the
stress concentration factor (Wang et al. 1998), depth (Zhang et al.
2010), UCS (Zhang et al. 2010), the rock brittleness coefficient
(Wang et al. 1998; Zhang et al. 2010), the strain energy storage
index (Kidybinski 1981), GB50487-2008 (Ministry of Water Re-
sources of the People’s Republic of China 2008), and rockburst
measurable value S (Zhang et al. 2013)] for rockburst classification
were applied in evaluating the rockburst cases (246 cases). They are
shown in Table 10 along with their predictive performances. The
predictive accuracy of these methods varied between 21.14 [rock
brittleness coefficient B1 (Zhang et al. 2012c)] and 52.85% [meas-
urable value of rockburst S (Zhang et al. 2013)] of the original data
and less than 60% of the filtered data. It was clear, judging from the
predictive accuracy, that the empirical methods could not generate
satisfactory predictions for these cases. They were proposed based
on engineering experience, which probably is why their use re-
quires an engineering background. Whereas the rockburst cases
collected in the current study had a wide range of engineering types
and locations, the empirical methods did not work properly in all of
them. As can be seen from Table 10, no single criterion (strain en-
ergy storage index, stress concentration factor, rock brittleness in-
dex, measurable value of rockburst S) satisfied the need for accurate
estimation because of the influence of numerous conditions on
rockburst and the complexity of the variables. However, the prob-
lem of rockbursting is strongly site-specific, depending on many
factors such as the magnitude and direction of in situ stresses, the
strength of the rock mass, and the geometry of the tunnel, as well as
relative positions and excavation methods. Until now, apart from a
few methods describing the stress state in the rock mass, it has been
difficult to endorse universal and practical rockburst criteria.
Relative Importance of Variables
The generic function varImp () in the R caret package can be
used to characterize the general effects of predictors on a model
(Kuhn and Johnson 2013). It also works with objects produced
© ASCE 04016003-13 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
francisco.wong
Resaltado
Table 9. Strengths and Weaknesses of the 10 SL Methods
Method
Classification
accuracy
Computational
cost
Implementation
difficulty Strengths Weaknesses
LDA Low Low Low Simple, strong theoretical base, linear, easy
to interpret
Only performs well when all classes are
strictly homogeneous; cannot be used if the
number of variables is higher than the total
number of samples
QDA Low Low Low Simple, strong theoretical base, easy to
interpret
Only performs well when all classes are
strictly homogeneous and requires the
number of variables to be lower than the
number of objects in the smallest class
PLSDA Low Low Low Simple, strong theoretical base, linear Difficulty in interpreting loadings of the
independent latent variables because the
distributional properties of estimates are
not known
CT Low Low-medium Low Self-explanatory, easy to interpret, works
with categorical and continuous data,
makes no distribution assumptions
Dependent variable is restricted to
categorical data; may not perform well in
the presence of many complex
interactions; overfitting may lead to
instability
RF High Medium Medium Works well with high-dimensional small
sample sizes, robust to noise, fast
computation
Difficult to interpret and prone to
overfitting in certain data sets
SVM Medium Medium Medium Easy to classify complex nonlinear data,
avoids overfitting, robust to noise
Black box, computational scalability, lack
of transparency, restricted to pairwise
classification
MLPNN Medium High High Nonlinear adaptability, no assumptions
required for probability density and
distribution
Risk of overfitting, black box, difficult to
design an optimal architecture
KNN Low Low-medium Low Effective, simple, nonparametric and easy
to implement, intuitive, robust to predictor
outliers
Susceptible to irrelevant features and
correlated input; unable to handle mixed
data types
NB Low Low High Insensitive to irrelevant features, high
speed when applied to large databases
Assumes features are independent;
performs badly when dependency arises
GBM High Medium-high High Theoretical properties, outlier
identification
Black box, high computational cost, not
interpretable
Fig. 6. Comparison of 11 classifiers for accuracy using Nemenyi’s test with 95% CI; methods that are not significantly different are connected:
(a) training set; (b) test set
© ASCE 04016003-14 J. Comput. Civ. Eng.
 J. Comput. Civ. Eng., 04016003 
D
ow
nl
oa
de
d 
fr
om
 a
sc
el
ib
ra
ry
.o
rg
 b
y 
M
cG
ill
 U
ni
ve
rs
ity
 o
n 
01
/1
1/
16
. C
op
yr
ig
ht
 A
SC
E
. F
or
 p
er
so
na
l u
se
 o
nl
y;
 a
ll 
ri
gh
ts
 r
es
er
ve
d.
by strain, but it is a simple wrapper for the specific models inves-
tigated here. In this work, the RF and GBM methods determined
the relative importance of discriminating features. For most of the
classification models, each predictor had a separate variable impor-
tance for each class. The default variable importance metric con-
sidered the area under the curve (AUC) derived from a receiver
operating characteristic (ROC) analysis with regard to each predic-
tor and was model independent. In addition, all measures of impor-
tance were scaled to have a maximum value of 100.
Variables were sorted by average importance across the
classes. From Fig. 7(a), the importance of the variables was in-
vestigated for each rockburst class with RF Model A.Wet was the
most sensitive indicator, followed by MTS, SCF, D, UCS, B1,
UTS, and B2. A plotting method for varImp was included that
produces a “needle plot” of the importance values where the