Baixe o app para aproveitar ainda mais
Prévia do material em texto
J Clin Periodontol. 2020;47:303–319. wileyonlinelibrary.com/journal/jcpe | 303© 2020 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd 1 | INTRODUC TION Periodontitis is a chronic inflammatory disease that leads to the destruction of the periodontal attachment apparatus and pos- sible tooth loss (Page & Schroeder, 1976; Savage, Eaton, Moles, & Needleman, 2009). Disease progression induces an apical migration of the attachment apparatus with periodontal pocket formation (Bosshardt & Schroeder, 1988). The main goal of periodontal therapy is to arrest the destructive outcome of this disease. It is broadly ac- cepted that probing pocket depth (PPD) reduction via various treat- ment approaches may improve the prognosis of the individual teeth (Nunn et al., 2012). In order to maintain periodontal health, treat- ment aims are to reduce PPD, maintain or improve clinical attach- ment level (CAL), and resolve periodontal inflammation. In view of the previous objectives, treatment of periodontitis in- cludes a wide range of interventions: education of the patient for self-plaque control (Jonsson, Baker, Lindberg, Oscarson, & Ohrn, 2012; Jonsson, Ohrn, Lindberg, & Oscarson, 2010), smoking ces- sation (Chaffee, Couch, & Ryder, 2016; Ramseier & Suvan, 2015), Received: 14 August 2019 | Revised: 17 December 2019 | Accepted: 17 December 2019 DOI: 10.1111/jcpe.13246 S Y S T E M A T I C R E V I E W The efficacy of pocket elimination/reduction compared to access flap surgery: A systematic review and meta-analysis David Polak1 | Asaf Wilensky1 | Georgios N. Antonoglou2 | Lior Shapira1 | Moshe Goldstein1 | Conchita Martin3 Polak and Wilensky contributed equally to the article. 1Faculty of Dental Medicine (Periodontology), The Hebrew University – Hadassah, Jerusalem, Israel 2Universidad Complutense de Madrid (ETEP), Madrid, Spain 3Universidad Complutense de Madrid (BIOCRAN), Madrid, Spain Correspondence Lior Shapira, Department of Periodontology, The Hebrew University-Hadassah Medical Center, P.O. Box 12272, Jerusalem 91120, Israel. Email: lior.shapira@ekmd.huji.ac.il Abstract Aim: To assess the efficacy and adverse effects of resective surgery compared to ac- cess flap in patients with periodontitis. Methods: Randomized controlled trials with a follow-up ≥6 months were identified in ten databases. Screening, data extraction, and quality assessment were conducted by two reviewers. The primary outcome was probing pocket depth, and the main secondary outcome was clinical attachment level. Data on adverse events were col- lected. Meta-analysis was used to synthesize the findings of trials. Results: A total of 880 publications were identified. Fourteen publications from nine clinical trials met the inclusion criteria and were included for analysis. Meta-analysis was carried out using all available results. The results indicated superior pocket depth reduction following resective surgery compared to access flap after 6–12 months of follow-up (weighted mean difference 0.47 mm; confidence interval 0.7–0.24; p = .010). After 36–60 months of follow-up, no differences were found between the two treatments in pocket depth and attachment level. The prevalence of adverse ef- fects was not different between the groups. Post-operative recession tended to be more severe for the resective approaches. Conclusion: Resective surgical approach was superior to access flap in reducing pocket depth 6–12 months post-surgery, while no differences between the two mo- dalities were found at 36–60 months of follow-up. K E Y W O R D S periodontal pocket, periodontal surgery, periodontitis, pocket reduction, systematic review www.wileyonlinelibrary.com/journal/jcpe https://orcid.org/0000-0003-3211-1330 https://orcid.org/0000-0002-8254-5471 mailto: https://orcid.org/0000-0001-9145-5155 https://orcid.org/0000-0003-3997-6900 mailto:lior.shapira@ekmd.huji.ac.il http://crossmark.crossref.org/dialog/?doi=10.1111%2Fjcpe.13246&domain=pdf&date_stamp=2020-07-27 304 | POLAK et AL. and clinical interventions such as supra and subgingival mechani- cal instrumentation to remove plaque and calculus (Krishna & De Stefano, 2016), local and systemic medication (Chambrone et al., 2016; Jepsen & Jepsen, 2016; Slots, 2012), and surgical interven- tions (Carnevale, 2007; Cortellini & Tonetti, 2011; Kaldahl, Kalkwarf, Patil, Molvar, & Dyer, 1996a; Kirkland & Storer, 1931; Ramfjord & Nissle, 1974). The treatment of periodontitis is initiated by eliminating the in- fection and controlling its risk factors. This is mostly achieved by a non-surgical approach, which aims to diminish inflammation in the gingiva and to reduce probing depth. Following treatment, the ideal result is shallow periodontal pockets, which enable maintenance of periodontal health by a supportive periodontal therapy (SPT) pro- gramme (Axelsson, Nystrom, & Lindhe, 2004; Graziani, Karapetsa, Alonso, & Herrera, 2017). At times, persistent inflammation and residual deep periodontal pockets are present at the end of the anti-infective stage. This out- come is insufficient for successful supportive therapy, and in such cases, surgical intervention is advised. Patients presenting residual pockets with PPD ≥ 5 mm exhibit a higher risk for disease progression and eventually incidence of tooth loss (Matuliene et al., 2008,) and teeth with probing depth of 7 mm or more are at a significantly higher risk for tooth loss compared with shallow PPD (McGuire & Nunn, 1996). As such, residual pockets of 5 mm or more have been suggested as a clear indication for surgi- cal intervention (Claffey & Egelberg, 1995; Claffey, Nylund, Kiger, Garrett, & Egelberg, 1990; Lindhe, Socransky, Nyman, Haffajee, & Westfelt, 1982; Matuliene et al., 2008; Tonetti, Muller-Campanile, & Lang, 1998). The main objective of periodontal surgery is to access difficult to reach sites by direct vision and effective debridement, and to enable efficient self-performed plaque control and acceptable professional supportive therapy (Graziani et al., 2017; Graziani, Karapetsa, Mardas, Leow, & Donos, 2018; Heitz-Mayfield & Lang, 2013; Heitz-Mayfield, Trombelli, Heitz, Needleman, & Moles, 2002; Isidor & Karring, 1986). Ideally, periodontal surgery should result in shallow pockets (<5 mm) (Serino, Rosling, Ramberg, Socransky, & Lindhe, 2001). However, oc- casionally the surgical intervention only reduces deep PPD but not completely eliminates it (i.e., residual pockets exceeding 5 mm). While those cases are not the desired outcome, they still improve SPT effi- cacy compared with the pre-surgical state. A variety of surgical tech- niques exists and their value has been tested over the years, but the choice of a surgical approach still relies mainly on the decision-making process of the surgeon. The availability of evidence-based information for the clinicians as a part of his/her decision-making process should optimize the treatment outcome. Periodontal surgery includes access flap procedures, resective procedures, and regenerative procedures. Excluding regeneration procedures, the two remaining objectives comprise of two routine surgical approaches: 1. Access flap procedures that aim to facilitate visual access with minimal reduction in the shape and anatomy of the soft or bone tissues (Ammons & Smith, 1976; Cortellini & Tonetti, 2011; Froum et al., 1982; Heitz-Mayfield et al., 2002; Ramfjord, 1977; Takei, Han, Carranza, Kenney, & Lekovic, 1985; Wang & Greenwell, 2001; Yukna, 1976). The authors included under this heading the modified Widman flap (MWF) procedure, since its original design was set as access flap surgery, despite the fact that the procedure involves minimal soft tissue resection to allow healing by primary intention. 2. Pocket reduction/elimination procedures that, in addition to debridement with a direct vision, also aim to change the shape and outline of the soft and, occasionally hard tissues, in orderto achieve controlled PPD reduction (Kaldahl et al., 1996a; Kirkland & Storer, 1931; Ochsenbein, 1986; Olsen, Ammons, & Belle, 1985; Prichard, 1957; Schluger, 1949; Wang & Greenwell, 2001). The surgical treatment is expected to predictably improve the clinical parameters for long-term maintenance of the affected teeth. Several reviews of surgical periodontal procedures have been published in the past (Barrington, 1981; 1982; Wirthlin, 1981; Becker et al., 2001; Nowzari, 2001), and the outcome of the surgical approaches has been reported (Becker et al., 2001, 1987; Kaldahl, Kalkwarf, Patil, Dyer, & Bates, 1988; Kaldahl, Kalkwarf, Patil, Molvar, & Dyer, 1996b; Knowles, Burgett, Morison, Nisle, & Ramfjord, 1980; Ramfjord et al., 1987). Nevertheless, a system- atic comparative analysis of pocket elimination/reduction versus access flap procedures and their long-term effectiveness is not available. Clinical Relevance Scientific rationale for the study: The selection of a specific surgical approach should rely on success rate and side ef- fects as demonstrated in randomized clinical trials. The present systematic review analysed the clinical outcomes of resective surgery versus access flap procedures in sub- jects with periodontitis stages II-III (previously termed moderate to advanced periodontitis), in order to support the development of evidence-based guidelines for peri- odontal therapy. Principal findings: The results demonstrated a short-term (6–12 months post-treatment) superior pocket depth re- duction for resective surgery and equivalent outcome for the long term (36–60 months post-treatment) compared to access flap procedures. The side effects in both modalities were similar for all follow-up durations, except for recessions, which tended to be more severe for respective surgery. Practical implications: Since the present review found a significant difference between pocket reduction/elimi- nation versus access flap surgery for the short term only (6–12 months), the choice of treatment for each case should be depended on patient-specific considerations. | 305POLAK et AL. The aim of the present systematic review was to compare the effi- cacy of pocket reduction/elimination versus access flap surgical proce- dures in patients diagnosed with periodontitis. The analysis was limited to randomized controlled trials that included an initial step of supra-/ subgingival biofilm control and residual PPD of 5 mm or more. 2 | MATERIAL S AND METHODS 2.1 | Protocol development The protocol for this systematic review was developed a priori based on the corresponding Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) extension30 and registered a priori in PROSPERO (ID: CRD42019122805). This review was conducted and re- ported per the Cochrane Handbook (Higgins et al., 2019) and the PRISMA statement (Moher, Liberati, Tetzlaff, & Altman, 2009), respectively. The following focused PICOS question was proposed: “In adult patients with periodontitis after an initial step of supra-/subgingival biofilm control and residual PPD of 5 mm or more, what is the effi- cacy of pocket elimination/reduction surgery in comparison with ac- cess flap surgery in terms of PPD reduction?” In regard to the details of the PICOS question, the population, intervention, comparator, outcome, and study design were defined as follows. 2.1.1 | Population Individuals aged above 18 years with periodontitis, otherwise healthy, after the initial step of supra-/subgingival biofilm control with residual pockets of 5 mm or more. 2.1.2 | Interventions Pocket elimination or reduction surgery with or without osteoplasty/ ostectomy (including apically positioned flap and all the modified/per- tinent techniques that aim to reduce pocket depth by apically position- ing the soft tissue flap margins and occasionally resect hard tissues). 2.1.3 | Comparator Access flap surgery, including open flap curettage, open flap de- bridement, access flap, modified Widman flap (see introduction), and all the modified/pertinent techniques that do not include resection of soft and hard tissue. 2.1.4 | Outcomes The primary outcome was PPD reduction measured as mean differ- ence of pocket depth in millimetres. Secondary outcomes included CAL gain as mean difference of clinical attachment level in millime- tres, bleeding on probing (BOP) reduction; mean difference in the percentage of full-mouth bleeding on probing, percentage of pocket closure (%); and mean difference in the percentage of pocket clo- sure, tooth loss. Adverse effects were set as loss of attachment and PPD changes in initially shallow pockets and gingival recession. Patient-centred/related outcomes included the following: patient discomfort/pain, root-surface hypersensitivity, and oral health-re- lated quality of life. 2.1.5 | Study design Prospective clinical trials with 6 months or more of follow-up duration. 2.2 | Information sources and search strategy A comprehensive literature search was performed in duplicate (Table S1) from inception of each database up to January 24 2019 in seven electronic databases (MEDLINE, Scopus, Cochrane CENTRAL, Cochrane DARE, Web of Science, SciELO, LILACS, BBO, and DECS) without any limitations to publication year, language, type, or sta- tus (supplementary Table S1). In addition, several trial registers (Clini calTr ials.gov, UK clinical trials Gateway) were manually searched for any missed studies. The following search strategy was developed: (pocket elimina- tion surgery OR pocket reduction surgery OR periodontal flap sur- gery OR periodontal pocket surgery OR apically displaced flap OR apically repositioned flap OR apically positioned flap OR Neumann flap OR ostectomy OR osteoplasty OR osseous surgery OR gingi- vectomy) AND (access flap OR modified Widman OR open flap curettage OR open flap debridement OR Kirkland flap OR papilla preservation flap OR un-displaced flap) AND (longitudinal study OR randomised controlled study OR comparative study OR clinical trial) NOT (systematic review*[tiab] OR systematic literature[tiab] OR meta-analysis[tiab] OR meta-analyses[tiab] OR in vitro[tiab] OR animal*[tiab]). Selected databases considered as grey literature databases (Mahood, Eerd, & Irvin, 2014) did not allow for the recording of the reference lists in separate files and were therefore searched manu- ally. Reference lists of retrieved papers and of previously published systematic reviews were also hand-searched. The search queries on the previous were modified in order to allow the yielding of the max- imum number of hits. 2.3 | Study eligibility criteria This systematic review included only randomized controlled tri- als (RCTs, parallel, and split-mouth) as they are considered the most robust study for medical intervention assessment (Altman & Schulz, http://ClinicalTrials.gov http://ClinicalTrials.gov 306 | POLAK et AL. 2001). Non-randomized prospective controlled studies were planned to be included in order to provide as much as possible of the available published evidence, but none were identified. 2.3.1 | Inclusion criteria Prospective controlled clinical trials with more than 10 human sub- jects per group, patients >18 years old with periodontitis, all patients received full-mouth clinical examination (all teeth examined, with ≥4 sites per tooth), treated teeth had PPD of ≥5 mm, control groups un- derwent subgingival debridement with access flap or sister technique, and reports included PPD and CAL, 6 months or more of follow-up. 2.3.2 | Exclusion criteria Use of biomaterials/additional procedures for the same surgery in- cluding regenerative procedures, endodontic-periodontal lesions, periodontal treatment aiming to treat specifically furcation involve- ments (since this issue is specifically addressed in another systematic review in the current workshop), and studies published as editorials, letters, comments, case series,and case report. 2.4 | Record screening, selection, and data extraction process 2.4.1 | Screening and selection Two reviewers (PD & WA) performed the search and selected eligi- ble studies by reviewing the list of titles and abstracts independently. Studies were selected when appearing to meet the selection cri- teria, or when there were insufficient data to make a clear decision. Discrepancies between these reviewers pertaining to the selection and inclusion of any specific paper were solved by a third reviewer (ANG). In the second stage, the full text of the selected articles was assessed by two reviewers (SL & WA). All articles that did not fulfil all the selec- tion criteria were eliminated, and reasons for exclusion were recorded. All reports excluded at this stage were formally recorded, as well as the reason/s for their exclusion. Due to the limited number of identified eli- gible trials, the authors included two trials in which periodontal surgery was also performed prior to initial scaling and root planing (Kaldahl et al., 1988, 1996b; Westfelt et al., 1985). Inter-observer agreement value for the screening of complete articles was assessed via kappa score. Disagreements were resolved by consulting a third reviewer (ANG), and the kappa score was calculated. Special attention was paid to duplicate publications to avoid an effect on the agreement assessment. 2.5 | Data extraction Based on the Cochrane recommendations, a standardized, pre- piloted data extraction form was designed and used. Data were extracted from eligible studies and recorded by two reviewers (PD & ANG). Data extraction included study design (year of publication, study type, randomization process, setting, funding), patient charac- teristics (number, sex, age, disease severity, smoking), surgical char- acteristics (surgery type, brief description of surgery), maintenance protocol, and clinical measurements of the outcomes of interest and adverse effects. A third reviewer solved disagreements and cross- checked the accuracy and validity of all the data obtained from the studies (MC). Authors of studies or published protocols were con- tacted for further information when data were incomplete or miss- ing. When the results of a study were published more than once or if the results were presented in different publications, the most com- plete data set was included only once. 2.6 | Risk of bias of individual studies and reporting quality The quality assessment of the included studies was undertaken by two reviewers (MC & ANG) following the recommendations by Cochrane for assessing the risk of bias (RoB) (Higgins et al., 2011) and evaluating six criteria. RevMan software (Review Manager, version 5.2., Cochrane Collaboration) was used to obtain the RoB figures. Studies were defined as low RoB if these six criteria were clearly met in the study: random sequence allocation (selection bias), allocation concealment (selection bias), blinding of participants and personnel (performance bias), bias of outcome assessment (detection bias), incomplete outcome data (at- trition bias), selective reporting (reporting bias), and other bias. When missing one of these criteria, the study was classified as a “moderate potential risk of bias.” Missing two or more of these criteria resulted in a high potential risk of bias (Ten Heggeler, Slot, & Weijden, 2011). Other aspects regarding the quality of the studies were recorded in the re- porting quality assessment table based on some items extracted from the CONSORT guidelines (Schulz, Altman, & Moher, 2010). For interventional, non-randomized controlled trials, the Cochrane tool—ROBINS-I— was planned to be used (Sterne et al., 2016). 2.7 | Data analyses A quantitative synthesis of aggregate data was planned to be per- formed. To compare the selected studies, data on the primary (PPD reduction) and secondary outcomes (CAL gain) were pooled and analysed using weighted mean differences (WMDs) and 95% coef- ficient interval (CI). When the differences between (∆) baseline-end were not re- ported, they were calculated using baseline and final values. The variance of ∆Var was estimated with the formula: SVar2 = SVar12 + SVar22 − (2*r*SVar1*SVar2), where SVar2 is the variance of the difference, SVar12 is the variance of the mean baseline value, and Svar22 is the variance of the mean end value. A correlation r of 0.5 was assumed. If treatment effects (mean or standard deviation) were not reported in publications, authors were contacted. | 307POLAK et AL. The statistical heterogeneity among studies was assessed using the Q test according to DerSimonian and Laird (DerSimonian & Laird, 1986). As a complement to the Q test, the I2 index (Higgins, Thompson, Deeks, & Altman, 2003) was calculated in order to know the percentage of variation in the global estimate that was attribut- able to heterogeneity (I2 = 25%: low; I2 = 50%: moderate; I2 = 75%: high heterogeneity). In case of heterogeneity, in addition to the sum- mary estimate (WMD) and CI, prediction intervals will be reported to allow more informative inferences and illustrate which range of true effects can be expected in future settings, presenting the heteroge- neity in the same metric as the original effect size measure (IntHout, Ioannidis, Rovers, & Goeman, 2016). In addition, sources of heterogeneity were planned to be ana- lysed through pre-specified mixed-effects subgroup analyses and random-effects meta-regression using the following factors: (a) % of smokers (or different subgroups if provided, e.g. smokers vs. non-smokers) and (b) subsets of increasing periodontitis severity (4–6 mm PPD & >7 mm PPD). The study-specific estimates were pooled using both the fixed effect model (Mantel–Haenszel–Peto test) and the random effect model (DerSimonian–Laird test). If sig- nificant heterogeneity was found, the random effect model results were presented. Subgroup analysis (using the factors mentioned above), meta-re- gression, or network meta-analysis was considered in case the avail- able data allowed for these types of analysis. Forest plots were created to illustrate the effects in the me- ta-analysis of the global estimation and the different sub-analysis. STATA® 14 (StataCorp LP, Lakeway Drive) intercooled software was used to perform all analyses. Statistical significance was set at p ≤ .05. A summary of the key findings from the systematic review was presented using the GRADE approach in the form of a Summary of Findings (SoF) for grading the quality of evidence and the strength of our recommendations (Guyatt et al., 2011). 2.8 | Methods for addressing reporting biases Publication bias was addressed in different ways: (a) in case of a sufficient number of included studies (at least 10 studies; with fewer studies, the power of the tests is too low to distinguish chance from real asymmetry), funnel plots were planned to be used, although an asymmetrical funnel plot should not be equated with publication bias, (b) Egger's test was planned to be used to F I G U R E 1 Flow chart 308 | POLAK et AL. assess publication bias, and (c) results of tests for funnel plot asym- metry were planned to be interpreted in the light of visual inspec- tion of the funnel plot. The contribution made to the totality of the evidence by studies with statistically non-significant results compared to that from studies with statistically significant results will be taken into account. 3 | RESULTS 3.1 | Study selection Figure 1 depicts the study flow chart. The electronic search yielded 878 titles. The manual search added 2 additional studies, with a total of 880 studies. After the screening of titles and abstracts, 851 stud- ies were discarded (agreement = 97.44%; kappa = 0.705; p < .001) resulting in 29 studies. Following the review of the full-text articles, 15 studies were excluded and 14 were included in the qualitative synthesis (agreement = 93.10%; kappa= 0.877; p < .001), represent- ing 9 different studies. Excluded studies and reasons for exclusion are shown in Table S2. Five publications, corresponding to three tri- als, were considered for the quantitative synthesis. 3.2 | Study characteristics The methodological characteristics of the selected studies are pre- sented in Table 1. Some studies are presented in more than one publication, and these papers were united into one unit of analysis in the current report (Table 1). Most studies were based on a split- mouth design, and only three studies (Kyriazis et al., 2012; Nyman, Lindhe, & Rosling, 1977; Rosling, Nyman, Lindhe, & Jern, 1976) were based on a parallel-group design. Furthermore, the major- ity of the studies were conducted and published in the 1970s and 1980s, and only two articles were published after the year 2000 (Becker et al., 2001; Kyriazis et al., 2012), and of those only one study was performed after the year 2000 (Kyriazis et al., 2012). Table 2 summarizes the outcome variables of interest, surgical techniques, maintenance protocol, and adverse effects in the in- cluded trials. 3.3 | Risk of bias in individual studies Table S3 shows the risk of bias assessment as recommended by Cochrane. None of the 14 included studies could be qualified as having a low risk of bias since the information reported did not provide enough data to evaluate whether a criterion was met and therefore rated as unclear in many cases. The number of studies that had low risk of bias for specific criteria varied according to the assessed item: random sequence generation (n = 5), alloca- tion concealment (n = 1), blinding of participants (n = 7), blinding of outcome assessment (n = 5), incomplete outcome data (n = 5), TA B LE 1 M et ho do lo gi ca l c ha ra ct er is tic s of in cl ud ed tr ia ls a nd s tu di es re po rt in g th e sa m e tr ia l Tr ia l c od e (s tu di es re po rt in g sa m e tr ia l) Fi rs t a ut ho r, ye ar St ud y de si gn N um be r o f p at ie nt s (m /f ; a ge ; s m ok in g) N um be r o f tr ea tm en t g ro up s In iti al p oc ke t d ep th c at eg or ie s & si ng le -/ m ul ti- ca te go rie s Fo llo w -u p/ s i n m on th s Se tt in g/ fu nd in g 1 (2 ) Ra m fjo rd , 1 97 5 K no w le s, 1 98 0 Sp lit -m ou th 79 (N R; 3 9. 6; N R) 3 Ye s & N o 12 , 2 4, 3 6, 4 8, 60 , 9 6 U ni ve rs ity /N R 2 (1 ) Sm ith , 1 98 0 Sp lit -m ou th 12 (N R; N R; N R) 2 N o & N o 6 U ni ve rs ity /N R 3 (2 ) H ill , 1 98 1 Ra m fjo rd , 1 98 7 Sp lit -m ou th 90 (N R; N R; N R) 4 Ye s & N o 12 , 2 4, 3 6, 4 8, 60 U ni ve rs ity /N R 4 (2 ) Is id or , 1 98 4 Is id or , 1 98 6 Sp lit -m ou th 17 (N R; 4 0; N R) 3 N o & N o 3, 6 , 1 2, 1 8, 2 4, 36 , 4 8, 6 0 U ni ve rs ity /N R 5 (2 ) Be ck er , 1 98 7 Be ck er , 2 00 1 Sp lit -m ou th 16 (N R; 4 2; N R) 3 Ye s & N o 6, 1 2, 3 6, 4 8, 6 0 Pr iv at e pr ac tic e & U ni ve rs ity / N R 6 (2 ) K al da hl , 1 98 8 K al da hl , 1 99 6 Sp lit -m ou th 82 (N R; 4 3. 5; N R) 4 Ye s & N o 6, 1 2, 2 4, 3 6, 48 , 6 0, 7 2 U ni ve rs ity /N R 7 (1 ) W es tf al t, 19 85 Sp lit -m ou th 16 (N R; 3 5– 65 ; N R) 6 N o & N o 6 U ni ve rs ity /G ov er nm en t g ra nt 8 (1 ) Ky rg ia zi s, 2 01 2 Pa ra lle l 30 (1 6/ 14 ; 4 3. 5; N R) 2 N o & N o 6 U ni ve rs ity /N on e 9 (1 ) Ro sl in g, 1 97 6 Pa ra lle l 30 (N R; 4 6. 3; N R) 5 N o & N o 6, 1 2, 2 4 U ni ve rs ity /G ov er nm en t g ra nt A bb re vi at io ns m /f , m al e/ fe m al e; N R, N ot re po rt ed | 309POLAK et AL. TA B LE 2 Su m m ar y th e ou tc om e va ria bl es o f i nt er es t, su rg ic al te ch ni qu es , m ai nt en an ce p ro to co l, an d ad ve rs e ef fe ct s in th e in cl ud ed tr ia ls Tr ia l c od e A ut ho r, ye ar O ut co m es (o ut co m e m ea su re ) Su rg ic al tr ea tm en t p er g ro up M ai nt en an ce p ro to co l ( Ye s/ N o) & to ot h lo ss (n ) Re po rt ed a dv er se e ff ec ts (n o = 0; s ha llo w po ck et s = 1 ; r ec es si on = 2 ) 1 Ra m fjo rd , 1 97 5 K no w le s, 1 98 0 PP D , C A L (m ea n pa tie nt ch an ge ) M W F A PF w ith o st eo pl as ty Ye s: 3 m (N R) 1, 2 2 Sm ith , 1 98 0 PP D , C A L, G in gi va l I nd ex , Pl aq ue In de x, T oo th m ob ili ty (m ea n pa tie nt ch an ge ) O FC A PF w ith o st eo pl as ty N o (N R) 0 3 H ill , 1 98 1 Ra m fjo rd , 1 98 7 PP D , C A L (m ea n pa tie nt ch an ge ) M W F A PF w ith o st eo pl as ty Ye s (N R) 1, 2 4 Is id or , 1 98 4 Is id or , 1 98 6 PP D , C A L, P la qu e In de x, G in gi va l I nd ex (m ea n pa tie nt c ha ng e) M W F A PF w ith ou t o st eo pl as ty Ye s: 2 w /3 m /6 m (N R) 0 5 Be ck er , 1 98 7 Be ck er , 2 00 1 PP D , C A L, G in gi va l I nd ex , Pl aq ue In de x (m ea n pa tie nt c ha ng e) M W F O S Ye s: 3 m (N R) 1 6 K al da hl , 1 98 8 K al da hl , 1 99 6 PP D , C A L, R ec M W F O S Ye s: 6 m (9 2) 1, 2 7 W es tf al t, 19 85 C PD , P la qu e sc or es , Bl ee di ng s co re s (m ea n pa tie nt c ha ng e) M W F A PF w ith /w ith ou t os te op la st y Ye s− 2 w o ve r 6 m (N R) 0 8 Ky rg ia zi s, 2 01 2 PP D , C A L. P la qu e in de x, G in gi va l I nd ex , b on e re so rp tio n (m ea n pa tie nt ch an ge ) M W F A PF w ith ou t o st eo pl as ty N o (N R) 0 9 Ro sl in g, 1 97 6 PP D , C A L, G in gi va l I nd ex , Pl aq ue In de x (m ea n pa tie nt c ha ng e) M W F + os te op la st y M W F + bo ne cu re tt ag e G in gi ve ct om y A PF + o st ec to pl as ta nd A PF + b on e cu re tt ag e Ye s: 2 w o ve r 2 4 m (N R) 0 A bb re vi at io ns A PF , a pi ca lly p os iti on ed fl ap ; C A L, c lin ic al a tt ac hm en t l ev el ; C PD , c rit ic al p ro bi ng d ep th ; M , m on th ; M W F, m od ifi ed W id m an fl ap ; N R, n ot re po rt ed ; O FC , o pe n fla p cu re tt ag e; O S, o ss eo us su rg er y; P PD , p oc ke t p ro bi ng d ep th ; R ec , r ec es si on ; w , w ee ks . 310 | POLAK et AL. and selective reporting (n = 11). Regarding the independence of the study, in terms of funding, half of the studies (n = 8) did not report this aspect and the rest were either public or self-funded. All authors were independent researchers, which was considered as a low risk of bias. Figures S1 and S2 summarize this information graphically. 3.4 | Quality of design and reporting in individual studies Table S4 shows the items related to registration according to the information provided in the 14 included studies: ethical commit- tee approval (n = 4), informed consent (n = 8), reporting adherence to the CONSORT guidelines (n = 1), and registration of the RCT (n = 1). In the same table, aspects related to the statistical analyses and outcome assessment are summarized, showing the number of manuscripts reporting sample size and/or power analysis calculation (n = 1), the primary statistical test (n = 14), an intent-to-treat analysis (n = 0), calibration (n = 10), the number of examiners (n = 9), and their skills/experience (n = 9). 3.5 | Synthesis of the results Of the included publications, five presented the results in graphi- cal illustrations and not as values (Isidor, Karring, & Attström,1984; Kaldahl et al., 1996a; Nyman et al., 1977; Rosling, Nyman, & Lindhe, 1976), excluding them from the quantitative synthesis. The results of those studies and all other included publications are addressed in a narrative analysis that presents the summation of all publications. The follow-up duration of the included publications was up to seven years. PPD reduction following pocket reduction surgery showed diversity between groups, but overall showed superior re- duction compared to access flap during the first year post-surgery. However, in the long term (>3 years), no statistical differences in PPD reduction were found between the groups. Studies based on parallel study design (Kyriazis et al., 2012; Nyman et al., 1977; Rosling et al., 1976) show a similar pattern of re- sults as studies that used a split-mouth design. Kyriazis et al. found improved pocket reduction and CAL gain using apically positioned flap compared to MWF, with statistically significant differences between groups (Kyriazis et al., 2012). However, Rosling et al. (1976). demon- strated superior CAL gain for MWF compared to apically positioned F I G U R E 2 Forest plots showing comparisons of PPD (Probing Pocket Depth) changes (in mm) between groups. Subgroup analysis according severity of initial PPD: 1 = pockets 4–6 mm and 5–6 mm; 2 = pockets ≥7 mm. (a) 6-month follow-up results; (b) 12-month follow- up results; (c) 36-month follow-up results, and (d) 60-month follow-up results. Data presented in values are also available at Table S5. WMD, weighted mean difference; 95% CI, 95% confidence interval [Colour figure can be viewed at wileyonlinelibrary.com] www.wileyonlinelibrary.com | 311POLAK et AL. flap (Rosling et al., 1976). Nyman et al. did not show any difference be- tween groups regarding changes in PPD and CAL (Nyman et al., 1977). In accordance with the outcome variables analysed in the current review, adverse effect was set as loss of attachment in initially shal- low pockets (as primary outcome) and PPD changes (as secondary outcome). Analysis according to initial pockets (Becker et al., 2001, 1987; Hill et al., 1981; Kaldahl et al., 1988; Ramfjord et al., 1987) showed primary CAL loss in initial shallow pockets in both treatment modalities. Data regarding patient discomfort/pain, root-surface hy- persensitivity, and oral health-related quality of life were not avail- able in most of the included studies and thus were not included in the adverse effect analysis. For a similar reason, BOP, percentage of pocket closure, and tooth loss as secondary outcome variables were not analysed. Post-operative recessions may also be considered as an adverse effect. Most studies did not report post-surgical recession, and only four publications from two clinical trials (Becker et al., 2001, 1987; Kaldahl et al., 1988, 1996a, 1996b) presented recession val- ues. In shallow pockets (1–3 mm), both trials showed that recession severity was higher following resective surgery than after access flap approach. Becker et al. showed post-operative recession mean of 0.95 ± 0.58 mm for resective surgery and 0.66 ± 0.4 mm for access flap at one year, and 0.85 ± 0.52 mm and 0.57 ± 0.48 mm, respectively, at five years. Similarly, Kaldahl et al., 1988, found 0.89 ± 0.1 mm recession post-resective surgery and 0.38 ± 0.06 mm post-access flap after 2 years of follow-up. Moderately deep to deep initial pockets exhibited more re- cession than shallow initial pockets. Both studies showed higher F I G U R E 3 Meta-regressions showing comparisons of PPD (probing pocket depth) changes (in mm) between groups with follow-up (in months) as a covariate. (a) Intermediate pockets (4–6 mm and 5–6 mm); (b) Deep pockets (≥7 mm). Data presented in values are also available at Table S6. WMD, weighted mean difference [Colour figure can be viewed at wileyonlinelibrary.com] www.wileyonlinelibrary.com 312 | POLAK et AL. recession values following resective surgery compared to access flap at 1 year post-surgery (moderately deep pockets: 1.25 ± 0.59 mm vs. 1.05 ± 0.59 mm for Becker et al., and 1.87 ± 0.09 mm vs. 0.65 ± 0.07 mm for Kaldahl et al.; deep pockets: 2.42 ± 0.91mm vs. 1.83 ± 0.84 mm for Becker et al., and 2.49 ± 0.12 mm vs. 1.02 ± 0.11 mm for Kaldahl et al.). However, at the five years of fol- low-up, Becker et al. did not find a difference in recession values be- tween the two modalities (moderately deep pockets: 1.03 ± 0.46 mm vs. 0.94 ± 0.74 mm; deep pockets: 1.7 ± 1.23mm vs. 2.18 ± 1.86 mm), while Kaldahl et al. showed higher recession values following resec- tive surgery compared to access flap (moderately deep pockets: 1.2 ± 0.12 mm vs. 0.48 ± 0.07 mm; deep pockets: 1.54 ± 0.14 mm vs. 0.71 ± 0.11 mm). 3.6 | Quantitative analysis Probing pocket depth analysis according to follow-up duration and initial pocket depth showed, at 6 and 12 months, better results for pocket reduction surgery compared to access flap procedures, with a mean difference of 0.59 mm (95% CI: −1.12, −0.06; 95% prediction interval ranged from −3.05 to 1.86) and 0.47 mm (95% CI: −0.70, −0.24; 95% prediction interval ranged from −1.17 to 0.23) for 6 and 12 months, respectively (Figure 2; Table S5). In the 12 months of fol- low-up interval, moderately deep pockets (4–6 mm) showed modest differences between the two tested modalities (mean difference of 0.34 mm; 95% CI: −0.48, −0.19; 95% prediction interval ranged from −1.55 to 0.88), while deep pockets (>7 mm; 95% CI: −1.17, −0.35) showed a greater difference between the groups (0.76 mm, 95% pre- diction interval ranged from −4.68 to 3.16). At 36 and 60 months of follow-up, the differences between the groups were markedly reduced with mean PPD reduction difference of 0.05 mm (95%CI: −0.39, 0.28; 95% prediction interval ranged from −1.27 to 1.17) for 36 months and 0.1 mm (95%CI: −0.31, 0.12; 95% prediction interval ranged from −0.57 to 0.38) for 60 months. At those follow-up inter- vals, no differences were found between moderately deep and deep initial pockets (Figure 2). Meta-regression models exhibit a reduction in the differences in PPD between the two treatment modalities, with a similar pattern for moderately deep and deep initial pockets (Figure 3; Table S6). Clinical attachment level analysis shows similar results as the PPD, with greater CAL gain for pocket reduction procedure at 6 months (mean differences of 0.28 mm; 95% CI: −0.12, 0.68; 95% prediction interval ranged from −1.50 to 2.05), and limited differences at all the F I G U R E 4 Forest plots showing comparisons of CAL (clinical attachment level) changes (in mm) between groups. Subgroup analysis according severity of initial PPD (probing pocket depth): 1 = pockets 4–6 mm and 5–6 mm; 2 = pockets ≥7 mm. (a) 6-month follow-up results; (b) 12-month follow-up results; (c) 36-month follow-up results; and d, 60-month follow-up results. Data presented in values are also available at Table S5. WMD, weighted mean difference; 95% CI, 95% confidence interval [Colour figure can be viewed at wileyonlinelibrary.com] www.wileyonlinelibrary.com | 313POLAK et AL. other tested follow-up intervals, ranging between 0.13 mm (95% CI: −0.11, 0.38; 95% prediction interval ranged from −0.54 to 0.81) and 0.17 mm (95% CI: −0.14, 0.48; 95% prediction interval ranged from −0.85 to 1.19) (Figure 4; Table S5). Furthermore, initially deep pock- ets showed more CAL gain than initially moderately deep pockets at 6 months of follow-up (Figure 3). Meta-regression analysis of initial moderately deep and deep pockets shows an initial superior CAL gain for pocket reduction procedures compared to access flap, with no difference between the procedures in the long-term follow-up (Figure 5; Table S6). Clinical attachment level loss and increased PPD for initially shal- low pockets were considered as adverse effects. The results showed an increase in CAL of 0.25 mm (95%CI: 0.16, 0.35;95% prediction in- terval ranged from −0.08 to 0.59) for pocket reduction surgery com- pared to access flap (Figure 6; Table S7). This difference tended to reduce over time, as can be seen from the meta-regression analysis (for follow-up= −0.0038; 95% CI: −0.01 to 1, Figure 7). PPD changes showed a modest difference between the 2 surgical modalities of 0.03 mm (95%CI: −0.04, 0.1; 95% prediction interval ranged from −0.24 to 0.30), with stability of the results at all tested intervals. Meta-regression results were similar to PPD changes (coefficient for follow-up= −0.0035; 95% CI: −0.01 to 1, Figure S5). There were no remarkable differences in PPD between the 2 treatment modal- ities in shallow pockets, and their effect varied between 0.01 and 0.33 mm (Table S7). A summary of the key findings (SoF) from this systematic review is presented in Table S8 using the GRADE method. The following out- come variables were included and assessed in the SoF table: PPD re- duction and CAL changes for moderately deep and deep pockets at 12 months of follow-up, and CAL changes at 60 months of follow-up. F I G U R E 5 Meta-regressions showing comparisons of CAL (clinical attachment level) changes (in mm) between groups with follow-up (in months) as covariate. (a) Intermediate pocket (4-6 mm) analysis; (b) deep pocket (≥7 mm) analysis. Data presented in values are also available at Table S6. WMD, weighted mean difference [Colour figure can be viewed at wileyonlinelibrary.com] www.wileyonlinelibrary.com 314 | POLAK et AL. 4 | DISCUSSION While individual studies describe and indicate the advantages of sur- gical procedures and provide valuable clinical information, system- atic analysis of controlled trials is required as part of the decisions making process in clinical situations. The current systematic review illustrates the limited amount and heterogeneous available studies focusing on access flap and resective surgery. The results of the current review demonstrate a short-term su- perior outcome in PPD reduction for resective surgery compared F I G U R E 6 Forest plots for adverse effects. (a) CAL (clinical attachment level) changes (in mm) and (b) PPD (probing pocket depth) changes (in mm). Subgroup analysis according to follow-up (in months): 6, 12, 24, 36, 48, and 60 months. Data presented in values are also available at Table S7. ARF, apically repositioned flap; WMD, weighted mean difference; 95% CI: 95% confidence interval [Colour figure can be viewed at wileyonlinelibrary. com] www.wileyonlinelibrary.com www.wileyonlinelibrary.com | 315POLAK et AL. to access flap procedures, alongside a "cost" of attachment level loss. This was more evident for initially deep pockets (>7 mm) than moderately deep pockets (4–6 mm). However, in the long-term fol- low-up, the superiority of resective surgery diminished, and the two approaches demonstrate an equivalent outcome. Similarly, CAL changes show comparable long-term outcomes for resective and access flap procedures. Taken together, resective surgery ap- proach provides transient superior PPD reduction and long-term equivalent effectiveness compared with access flap approach. The side effect of both approaches was PPD and CAL deterioration in sites with initial shallow pockets, but the differences between the two approaches were minimal (0.25 mm of ΔPPD and 0.03 mm of ΔCAL), with minimal clinical significance. It should be stated that the meta-analyses in the current review are based on 2–3 studies, and not all of the studies provided data for the long-term analyses (36 and 60 months). Unfortunately, there is the lack of publications in the last two decades related to access flap or pockets reduction surgical proce- dures, with scarce description of surgical techniques or reviews of techniques (Carnevale, 2007; Cortellini & Tonetti, 2011; Graziani et al., 2012; Nowzari, 2001) and only one new RCT (Kyriazis et al., 2012). Considering the limited outcome of non-surgical ther- apy in deep pockets (Heitz-Mayfield & Lang, 2013) and the lim- ited anatomical conditions which allow predictable regenerative procedures (Avila-Ortiz, Buitrago, & Reddy, 2015; Kao, Nares, & Reynolds, 2015), non-regenerative surgical approaches are an essential and common part of periodontal therapy. The lack of evidence derived from the lack of research with contemporary methodology, absence of studies using newly acceptable end- points (such as “closed pockets”), and the lack of evidence from studies implementing modern surgical approaches (i.e., micro- surgery, use of magnification systems, etc.) preclude definitive F I G U R E 7 Meta-regression for adverse effects. (a) CAL (clinical attachment level) changes (in mm) between groups in shallow pockets with follow-up (in months) as a covariate. (b) PPD (probing pocket depth) changes (in mm) between groups in shallow pockets with follow-up as (in months) covariate [Colour figure can be viewed at wileyonlinelibrary.com] www.wileyonlinelibrary.com 316 | POLAK et AL. conclusions. On the other hand, the strength of the included RCTs lies in the long-term follow-up. Most of the included studies used a split-mouth design, which has the weakness of carry-across effect (Lesaffre, Philstrom, Needleman, & Worthington, 2009). This is based on the microbial aetiology of periodontitis that postulates potential changes of the oral microbiota when one niche of a quadrant is treated, it may affect the other quadrants in the same subject. However, this design pro- vides the advantage of subject variable response to therapy. Of interest is the possible impact of SPT protocols of the tri- als on the results. Some of the included studies included very fre- quent SPT at two-week intervals (Isidor & Karring, 1986; Nyman et al., 1977; Rosling et al., 1976; Westfelt et al., 1985), while in other studies SPT was performed every 3 months (Becker et al., 2001, 1987; Hill et al., 1981; Kaldahl et al., 1988, 1996a, 1996b; Ramfjord et al., 1987). Since the results of all studies are relatively uniform, it seems that SPT every 3 months may be sufficient. Moreover, since the majority of studies were carried out decades ago, current RCT guidelines did not apply (such as non-split-mouth design). When con- sidering only the parallel design studies (three studies) (Kyriazis et al., 2012; Nyman et al., 1977; Rosling et al., 1976), their results are conflicting and prevent a concrete and uniform clinical conclusions. Nevertheless, the fact is that SPT has a substantial impact on the recurrence of periodontitis (Costa et al., 2015). Therefore, it is rea- sonable to conclude that a meticulous SPT programme is mandatory to maximize any surgical outcome. Based on the risk of bias of individual studies and on the risk of bias assessment using the RoB tool, most factors that would introduce bias were undetectable or absent. Nonetheless, loses to follow-up and inadequate blinding were frequently encountered in the included RCTs. The overall quality of the individual studies was considered poor although this needs to be evaluated in the specific historical context. Publication bias tests were not performed due to the limited number of studies that could be used for this purpose. In addition, the quality of evidence across studies assessing se- lected outcomes was deemed “LOW” in relation to all outcomes due to the possibility of serious risk of bias and imprecision of the clin- ical measurements. This can be explained by the fact that all three studies that dropped data considered in the Summary of Findings table (Table S8) were rated with 2 or more domains of unclear risk of bias using the RoB tool. Also, the inherent imprecision of clinical measurement using periodontal probes has a considerable effect on assessing effects lower than 2 mm. The former may reduce the confidence of our conclusions because the observed effects can be substantiallydifferent from the true effects. The present systematic review focused on PPD reduction and CAL changes despite the fact that these clinical parameters are weak predictors of long-term disease progression or as risk markers for tooth loss. There is a great need to single out more reliable indices for risk assessment, and include them in future RCTs that compare periodontal surgical approaches. In addition, the difference between the surgical approaches was small, and it would be beneficial to carry out an additional assessment of the effect size (%) for the overall changes in different PPDs. However, such analysis was not possible due to the presentation of the results in the RCTs by pocket depth groups only. In conclusion, when the aim of the periodontal treatment is to reduce the depth of the residual periodontal pockets: 1. Pocket reduction/elimination techniques were found to be su- perior to access flap approaches 6–12 months post-surgery, particularly in sites with initial PPD of >6 mm. 2. Longer-term follow-up (36–60 months) was not able to find sig- nificant differences between the two surgical approaches. Clinically, since the present review found a significant difference between pocket reduction/elimination versus access flap surgery for the short term (6–12 months) only, the authors feel that the choice of treatment for each case may rely on other factors, such as the sur- geon's experience as well as individual patient-related preferences, aesthetic considerations, and post-surgical restorative aspects. CONFLIC T OF INTERE S T Dr. Shapira reports grants from MIS Implants; personal fees from MIS Implants Technologies, Dexcel Pharma, and BoneSci; and other from Datum Dental, outside the submitted work. Dr. Goldstein re- ports personal fees from MIS Implants Technologies outside and with no possible conflict to the submitted work. Dr. Polak reports personal fees from Dexcel Pharma, outside the submitted work. Dr. Antonoglou, Dr. Martin, and Dr. Wilensky have nothing to disclose. ORCID David Polak https://orcid.org/0000-0003-3211-1330 Georgios N. Antonoglou https://orcid. org/0000-0002-8254-5471 Lior Shapira https://orcid.org/0000-0001-9145-5155 Conchita Martin https://orcid.org/0000-0003-3997-6900 R E FE R E N C E S Altman, D. G., & Schulz, K. F. (2001). Statistics Fnotes: Concealing treat- ment allocation in randomised trials. British Medical Journal, 323, 446–447. https ://doi.org/10.1136/bmj.323.7310.446 Ammons, W. F. Jr, & Smith, D. H. (1976). Flap curettage: Rationale, technique, and expectations. Dental Clinics of North America, 20, 215–226. Avila-Ortiz, G., De Buitrago, J. G., & Reddy, M. S. (2015). Periodontal re- generation - furcation defects: A systematic review from the AAP Regeneration Workshop. Journal of Periodontology, 86, S108–S130. https ://doi.org/10.1902/jop.2015.130677 Axelsson, P., Nystrom, B., & Lindhe, J. (2004). The long-term ef- fect of a plaque control program on tooth mortality, caries and periodontal disease in adults. Results after 30 years of mainte- nance. Journal of Clinical Periodontology, 31, 749–757. https ://doi. org/10.1111/j.1600-051X.2004.00563.x Barrington, E. P. (1981). An overview of periodontal surgical procedures. Journal of Periodontology, 52, 518–528. https ://doi.org/10.1902/ jop.1981.52.9.518 Becker, W., Becker, B. E., Caffesse, R., Kerry, G., Ochsenbein, C., Morrison, E., & Prichard, J. (2001). A longitudinal study comparing scaling, osseous surgery, and modified widman procedures: Results https://orcid.org/0000-0003-3211-1330 https://orcid.org/0000-0003-3211-1330 https://orcid.org/0000-0002-8254-5471 https://orcid.org/0000-0002-8254-5471 https://orcid.org/0000-0002-8254-5471 https://orcid.org/0000-0001-9145-5155 https://orcid.org/0000-0001-9145-5155 https://orcid.org/0000-0003-3997-6900 https://orcid.org/0000-0003-3997-6900 https://doi.org/10.1136/bmj.323.7310.446 https://doi.org/10.1902/jop.2015.130677 https://doi.org/10.1111/j.1600-051X.2004.00563.x https://doi.org/10.1111/j.1600-051X.2004.00563.x https://doi.org/10.1902/jop.1981.52.9.518 https://doi.org/10.1902/jop.1981.52.9.518 | 317POLAK et AL. after 5 years. Journal of Periodontology, 72, 1675–1684. https ://doi. org/10.1902/jop.2001.72.12.1675 Becker, W., Becker, B. E., Prichard, J. F., Caffesse, R., Rosenberg, E., & Gian-Grasso, J. (1987). Root isolation for new attachment pro- cedures. A surgical and suturing method: Three case reports. Journal of Periodontology, 58, 819–826. https ://doi.org/10.1902/ jop.1987.58.12.819 Bosshardt, D., & Schroeder, H. E. (1988). Histopathologic features of spontaneous periodontitis in adult Macaca fascicularis mon- keys. Journal of Periodontal Research, 23, 352–362. https ://doi. org/10.1111/j.1600-0765.1988.tb016 13.x Carnevale, G. (2007). Fibre retention osseous resective sur- gery: A novel conservative approach for pocket elimination. Journal of Clinical Periodontology, 34, 182–187. https ://doi. org/10.1111/j.1600-051X.2006.01027.x Chaffee, B. W., Couch, E. T., & Ryder, M. I. (2016). The tobacco-using periodontal patient: Role of the dental practitioner in tobacco cessa- tion and periodontal disease management. Periodontology 2000, 71, 52–64. https ://doi.org/10.1111/prd.12120 Chambrone, L., Vargas, M., Arboleda, S., Serna, M., Guerrero, M., de Sousa, J., & Lafaurie, G. I. (2016). Efficacy of Local and Systemic Antimicrobials in the Non-Surgical Treatment of Smokers With Chronic Periodontitis: A Systematic Review. Journal of Periodontology, 87, 1320–1332. https ://doi.org/10.1902/ jop.2016.160268 Claffey, N., & Egelberg, J. (1995). Clinical indicators of probing attach- ment loss following initial periodontal treatment in advanced peri- odontitis patients. Journal of Clinical Periodontology, 22, 690–696. https ://doi.org/10.1111/j.1600-051x.1995.tb008 28.x Claffey, N., Nylund, K., Kiger, R., Garrett, S., & Egelberg, J. (1990). Diagnostic predictability of scores of plaque, bleeding, suppura- tion and probing depth for probing attachment loss. 3 1/2 years of observation following initial periodontal therapy. Journal of Clinical Periodontology, 17, 108–114. https ://doi.org/10.1111/j.1600- 051x.1990.tb010 71.x Cortellini, P., & Tonetti, M. S. (2011). Clinical and radiographic out- comes of the modified minimally invasive surgical technique with and without regenerative materials: A randomized-controlled trial in intra-bony defects. Journal of Clinical Periodontology, 38, 365–373. https ://doi.org/10.1111/j.1600-051X.2011.01705.x Costa, F. O., Cota, L. O., Cortelli, J. R., Cortelli, S. C., Cyrino, R. M., Lages, E. J., & Oliveira, A. P. (2015). Surgical and non-surgical procedures as- sociated with recurrence of periodontitis in periodontal maintenance therapy: 5-year prospective study. PLoS ONE, 10, e0140847. https :// doi.org/10.1371/journ al.pone.0140847 DerSimonian, R., & Laird, N. (1986). Meta-analysis in clini- cal trials. Controlled Clinical Trials, 7, 177–188. https ://doi. org/10.1016/0197-2456(86)90046-2 Froum, S. J., Coran, M., Thaller, B., Kushner, L., Scopp, I. W., & Stahl, S. S. (1982). Periodontal healing following open debridement flap proce- dures. I. Clinical assessment of soft tissue and osseous repair. Journal of Periodontology, 53, 8–14. https ://doi.org/10.1902/jop.1982.53.1.8 Graziani, F., Karapetsa, D., Alonso, B., & Herrera, D. (2017). Nonsurgical and surgical treatment of periodontitis: How many options for one disease? Periodontology 2000, 75, 152–188. https ://doi.org/10.1111/ prd.12201 Graziani, F., Karapetsa, D., Mardas, N., Leow, N., & Donos, N. (2018). Surgical treatment of the residual periodontal pocket. Periodontology 2000, 76, 150–163. https ://doi.org/10.1111/prd.12156 Graziani, F., Vescovi, P., Campisi, G., Favia, G., Gabriele, M., Gaeta, G. M., … Colella, G. (2012). Resective surgical approach shows a high performance in the management of advanced cases of bisphospho- nate-related osteonecrosis of thejaws: A retrospective survey of 347 cases. Journal of Oral & Maxillofacial Surgery, 70, 2501–2507. https :// doi.org/10.1016/j.joms.2012.05.019 Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., … Schünemann, H. J. (2011). GRADE guidelines: 1. Introduction- GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology, 64, 383–394. https ://doi.org/10.1016/j.jclin epi.2010.04.026 Heitz-Mayfield, L. J., & Lang, N. P. (2013). Surgical and nonsurgical peri- odontal therapy. Learned and unlearned concepts. Periodontology 2000, 62, 218–231. https ://doi.org/10.1111/prd.12008 Heitz-Mayfield, L. J., Trombelli, L., Heitz, F., Needleman, I., & Moles, D. (2002). A systematic review of the effect of surgical debridement vs non-surgical debridement for the treatment of chronic periodonti- tis. Journal of Clinical Periodontology, 29(Suppl 3), 92–102. https ://doi. org/10.1034/j.1600-051X.29.s3.5.x Higgins, J. P., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., … Sterne, J. A. C. (2011). The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. British Medical Journal, 343, d5928. https ://doi.org/10.1136/bmj.d5928 Higgins, J. P., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.) (2019). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019), 2nd ed. Chichester, UK: John Wiley & Sons. Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 557–560. https ://doi.org/10.1136/bmj.327.7414.557 Hill, R. W., Ramfjord, S. P., Morrison, E. C., Appleberry, E. A., Caffesse, R. G., Kerry, G. J., & Nissle, R. R. (1981). Four types of periodontal treatment compared over two years. Journal of Periodontology, 52, 655–662. https ://doi.org/10.1902/jop.1981.52.11.655 IntHout, J., Ioannidis, J. P., Rovers, M. M., & Goeman, J. J. (2016). Plea for routinely presenting prediction intervals in meta-analysis. British Medical Journal Open., 6, e010247. https ://doi.org/10.1136/bmjop en-2015-010247 Isidor, F., & Karring, T. (1986). Long-term effect of surgical and non-surgical periodontal treatment. A 5-year clinical study. Journal of Periodontal Research, 21, 462–472. https ://doi.org/10.1111/j.1600-0765.1986. tb014 82.x Isidor, F., Karring, T., & Attström, R. (1984). The effect of root plan- ing as compared to that of surgical treatment. Journal of Clinical Periodontology, 11, 669–681. https ://doi.org/10.1111/j.1600- 051X.1984.tb013 15.x Jepsen, K., & Jepsen, S. (2016). Antibiotics/antimicrobials: Systemic and local administration in the therapy of mild to moderately ad- vanced periodontitis. Periodontology 2000, 71, 82–112. https ://doi. org/10.1111/prd.12121 Jonsson, B., Baker, S. R., Lindberg, P., Oscarson, N., & Ohrn, K. (2012). Factors influencing oral hygiene behaviour and gingi- val outcomes 3 and 12 months after initial periodontal treat- ment: An exploratory test of an extended Theory of Reasoned Action. Journal of Clinical Periodontology, 39, 138–144. https ://doi. org/10.1111/j.1600-051X.2011.01822.x Jonsson, B., Ohrn, K., Lindberg, P., & Oscarson, N. (2010). Evaluation of an individually tailored oral health educational programme on peri- odontal health. Journal of Clinical Periodontology, 37, 912–919. https ://doi.org/10.1111/j.1600-051X.2010.01590.x Kaldahl, W. B., Kalkwarf, K. L., Patil, K. D., Dyer, J. K., & Bates, R. E. (1988). Evaluation of four modalities of periodontal therapy. Mean probing depth, probing attachment level and recession changes. Journal of Periodontology, 59, 783–793. https ://doi.org/10.1902/ jop.1988.59.12.783 Kaldahl, W. B., Kalkwarf, K. L., Patil, K. D., Molvar, M. P., & Dyer, J. K. (1996a). Long-term evaluation of periodontal therapy: I. Response to 4 therapeutic modalities. Journal of Periodontology, 67, 93–102. https ://doi.org/10.1902/jop.1996.67.2.93 Kaldahl, W. B., Kalkwarf, K. L., Patil, K. D., Molvar, M. P., & Dyer, J. K. (1996b). Long-term evaluation of periodontal therapy: II. Incidence https://doi.org/10.1902/jop.2001.72.12.1675 https://doi.org/10.1902/jop.2001.72.12.1675 https://doi.org/10.1902/jop.1987.58.12.819 https://doi.org/10.1902/jop.1987.58.12.819 https://doi.org/10.1111/j.1600-0765.1988.tb01613.x https://doi.org/10.1111/j.1600-0765.1988.tb01613.x https://doi.org/10.1111/j.1600-051X.2006.01027.x https://doi.org/10.1111/j.1600-051X.2006.01027.x https://doi.org/10.1111/prd.12120 https://doi.org/10.1902/jop.2016.160268 https://doi.org/10.1902/jop.2016.160268 https://doi.org/10.1111/j.1600-051x.1995.tb00828.x https://doi.org/10.1111/j.1600-051x.1990.tb01071.x https://doi.org/10.1111/j.1600-051x.1990.tb01071.x https://doi.org/10.1111/j.1600-051X.2011.01705.x https://doi.org/10.1371/journal.pone.0140847 https://doi.org/10.1371/journal.pone.0140847 https://doi.org/10.1016/0197-2456(86)90046-2 https://doi.org/10.1016/0197-2456(86)90046-2 https://doi.org/10.1902/jop.1982.53.1.8 https://doi.org/10.1111/prd.12201 https://doi.org/10.1111/prd.12201 https://doi.org/10.1111/prd.12156 https://doi.org/10.1016/j.joms.2012.05.019 https://doi.org/10.1016/j.joms.2012.05.019 https://doi.org/10.1016/j.jclinepi.2010.04.026 https://doi.org/10.1016/j.jclinepi.2010.04.026 https://doi.org/10.1111/prd.12008 https://doi.org/10.1034/j.1600-051X.29.s3.5.x https://doi.org/10.1034/j.1600-051X.29.s3.5.x https://doi.org/10.1136/bmj.d5928 https://doi.org/10.1136/bmj.327.7414.557 https://doi.org/10.1902/jop.1981.52.11.655 https://doi.org/10.1136/bmjopen-2015-010247 https://doi.org/10.1136/bmjopen-2015-010247 https://doi.org/10.1111/j.1600-0765.1986.tb01482.x https://doi.org/10.1111/j.1600-0765.1986.tb01482.x https://doi.org/10.1111/j.1600-051X.1984.tb01315.x https://doi.org/10.1111/j.1600-051X.1984.tb01315.x https://doi.org/10.1111/prd.12121 https://doi.org/10.1111/prd.12121 https://doi.org/10.1111/j.1600-051X.2011.01822.x https://doi.org/10.1111/j.1600-051X.2011.01822.x https://doi.org/10.1111/j.1600-051X.2010.01590.x https://doi.org/10.1111/j.1600-051X.2010.01590.x https://doi.org/10.1902/jop.1988.59.12.783 https://doi.org/10.1902/jop.1988.59.12.783 https://doi.org/10.1902/jop.1996.67.2.93 https://doi.org/10.1902/jop.1996.67.2.93 318 | POLAK et AL. of sites breaking down. Journal of Periodontology, 67, 103–108. https ://doi.org/10.1902/jop.1996.67.2.103 Kao, R. T., Nares, S., & Reynolds, M. A. (2015). Periodontal regeneration - intrabony defects: A systematic review from the AAP Regeneration Workshop. Journal of Periodontology, 86, S77–104. https ://doi. org/10.1902/jop.2015.130685 Kirkland, H., & Storer, R. V. (1931). Gonococcal rhinitis in an infant. British Medical Journal, 1, 263–264. https ://doi.org/10.1136/ bmj.1.3658.263 Knowles, J., Burgett, F., Morison, E., Nisle, R., & Ramfjord, S. (1980). Comparison of results following three modalities of periodontal ther- apy related to tooth type and initial pocket depth. Journal of Clinical Periodontology, 7, 32–47. https ://doi.org/10.1111/j.1600-051X.1980. tb019 47.x Krishna, R., & De Stefano, J. A. (2016). Ultrasonic vs. hand instrumenta- tion in periodontal therapy: clinical outcomes. Periodontology 2000, 71(1), 113–127. https ://doi.org/10.1111/prd.12119 Kyriazis, T., Gkrizioti, S., Mikrogeorgis, G., Tsalikis, L., Sakellari, D., Lyroudia, K., & Konstantinides, A. (2012). Crestal bone resorption after the application of two periodontal surgical techniques. A ran- domized, controlled clinical trial. Journal of Clinical Periodontology, 39, 971–978. https ://doi.org/10.1111/j.1600-051X.2012.01928.x Lesaffre, E., Philstrom, B., Needleman, I., & Worthington, H. (2009). The design and analysis of split-mouth studies: What statisticians and cli- nicians should know. Statistics in Medicine, 28, 3470–3482. https :// doi.org/10.1002/sim.3634 Lindhe, J., Socransky, S. S., Nyman, S., Haffajee, A., & Westfelt, E. (1982). "Critical probing depths"in periodontal therapy. Journal of Clinical Periodontology, 9, 323–336. https ://doi.org/10.1111/j.1600- 051X.1982.tb020 99.x Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: Challenges and benefits. Research Synthesis Methods, 5, 221–234. https ://doi.org/10.1002/jrsm.1106 Matuliene, G., Pjetursson, B. E., Salvi, G. E., Schmidlin, K., Bragger, U., Zwahlen, M., & Lang, N. P. (2008). Influence of residual pockets on progression of periodontitis and tooth loss: Results after 11 years of maintenance. Journal of Clinical Periodontology, 35, 685–695. https :// doi.org/10.1111/j.1600-051X.2008.01245.x McGuire, M. K., & Nunn, M. E. (1996). Prognosis versus actual outcome. III. The effectiveness of clinical parameters in accurately predicting tooth survival. Journal of Periodontology, 67, 666–674. https ://doi. org/10.1902/jop.1996.67.7.666 Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group (2009). Preferred reporting items for systematic reviews and me- ta-analyses: The PRISMA statement. Journal of Clinical Epidemiology, 62, 1006–1012. https ://doi.org/10.1016/j.jclin epi.2009.06.005 Nowzari, H. (2001). Aesthetic osseous surgery in the treatment of peri- odontitis. Periodontology 2000, 27, 8–28. https ://doi.org/10.103 4/j.1600-0757.2001.02700 1008.x Nunn, M. E., Fan, J., Su, X., Levine, R. A., Lee, H. J., & McGuire, M. K. (2012). Development of prognostic indicators using classification and regression trees for survival. Periodontology 2000, 58, 134–142. https ://doi.org/10.1111/j.1600-0757.2011.00421.x Nyman, S., Lindhe, J., & Rosling, B. (1977). Periodontal surgery in plaque-infected dentitions. Journal of Clinical Periodontology, 4, 240– 249. https ://doi.org/10.1111/j.1600-051x.1977.tb018 96.x Ochsenbein, C. (1986). A primer for osseous surgery. International Journal of Periodontics & Restorative Dentistry, 6, 8–47. Olsen, C. T., Ammons, W. F., & van Belle, G. (1985). A longitudinal study com- paring apically repositioned flaps, with and without osseous surgery. International Journal of Periodontics & Restorative Dentistry, 5, 10–33. Page, R. C., & Schroeder, H. E. (1976). Pathogenesis of inflamma- tory periodontal disease. A summary of current work. Laboratory Investigation; A Journal of Technical Methods and Pathology, 34, 235–249. Prichard, J. (1957). Regeneration of bone following periodontal therapy; report of cases. Oral Surgery Oral Medicine Oral Pathology, 10, 247– 252. https ://doi.org/10.1016/0030-4220(57)90089-0 Ramfjord, S. P. (1977). Present status of the modified Widman flap procedure. Journal of Periodontology, 48, 558–565. https ://doi. org/10.1902/jop.1977.48.9.558 Ramfjord, S. P., Caffesse, R. G., Morrison, E. C., Hill, R. W., Kerry, G. J., Appleberry, E. A., … Stults, D. L. (1987). 4 modalities of periodontal treatment compared over 5 years. Journal of Clinical Periodontology, 14, 445–452. https ://doi.org/10.1111/j.1600-051x.1987.tb022 49.x Ramfjord, S. P., & Nissle, R. R. (1974). The modified widman flap. Journal of Periodontology, 45, 601–607. https ://doi.org/10.1902/ jop.1974.45.8.2.601 Ramseier, C. A., & Suvan, J. E. (2015). Behaviour change counselling for tobacco use cessation and promotion of healthy lifestyles: A system- atic review. Journal of Clinical Periodontology, 42(Suppl 16), S47–S58. https ://doi.org/10.1111/jcpe.12351 Rosling, B., Nyman, S., & Lindhe, J. (1976). The effect of systematic plaque control on bone regeneration in infrabony pockets. Journal of Clinical Periodontology, 3, 38–53. https ://doi.org/10.1111/j.1600- 051X.1976.tb018 49.x Rosling, B., Nyman, S., Lindhe, J., & Jern, B. (1976). The healing potential of the periodontal tissues following different techniques of periodontal surgery in plaque-free dentitions. A 2-year clinical study. Journal of Clinical Periodontology, 3, 233–250. https ://doi.org/10.1111/j.1600- 051x.1976.tb000 42.x Savage, A., Eaton, K. A., Moles, D. R., & Needleman, I. (2009). A sys- tematic review of definitions of periodontitis and methods that have been used to identify this disease. Journal of Clinical Periodontology, 36, 458–467. https ://doi.org/10.1111/j.1600-051X.2009.01408.x Schluger, S. (1949). Osseous resection; a basic principle in periodontal surgery. Oral Surgery Oral Medicine Oral Pathology, 2, 316–325. https ://doi.org/10.1016/0030-4220(49)90363-1 Schulz, K. F., Altman, D. G., Moher, D., & Consort Group (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group ran- domised trials. Journal of Clinical Epidemiology, 63, 834–840. https :// doi.org/10.1016/j.jclin epi.2010.02.005 Serino, G., Rosling, B., Ramberg, P., Socransky, S., & Lindhe, J. (2001). Initial outcome and long-term effect of surgical and non-surgi- cal treatment of advanced periodontal disease. Journal of Clinical Periodontology, 28, 910–916. https ://doi.org/10.1034/j.1600- 051x.2001.02801 0910.x Slots, J. (2012). Low-cost periodontal therapy. Periodontology 2000, 60(1), 110–137. https ://doi.org/10.1111/j.1600-0757.2011.00429.x Sterne, J. A., Hernan, M. A., Reeves, B. C., Savovic, J., Berkman, N. D., Viswanathan, M., … Higgins, J. P. (2016). ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. British Medical Journal, 355, i4919. https ://doi.org/10.1136/bmj.i4919 Takei, H. H., Han, T. J., Carranza, F. A. Jr, Kenney, E. B., & Lekovic, V. (1985). Flap technique for periodontal bone implants. Papilla preser- vation technique. Journal of Periodontology, 56, 204–210. https ://doi. org/10.1902/jop.1985.56.4.204 Ten Heggeler, J. M., Slot, D. E., & Van der Weijden, G. A. (2011). Effect of socket preservation therapies following tooth extraction in non-molar regions in humans: A systematic review. Clinical Oral Implants Research, 22, 779–788. https ://doi.org/10.1111/j.1600-0501.2010.02064.x Tonetti, M. S., Muller-Campanile, V., & Lang, N. P. (1998). Changes in the prevalence of residual pockets and tooth loss in treated periodon- tal patients during a supportive maintenance care program. Journal of Clinical Periodontology, 25, 1008–1016. https ://doi.org/10.1111/ j.1600-051X.1998.tb024 06.x Wang, H. L., & Greenwell, H. (2001). Surgical periodontal ther- apy. Periodontology 2000, 25(1), 89–99. https ://doi.org/10.103 4/j.1600-0757.2001.22250 107.x https://doi.org/10.1902/jop.1996.67.2.103 https://doi.org/10.1902/jop.1996.67.2.103 https://doi.org/10.1902/jop.2015.130685 https://doi.org/10.1902/jop.2015.130685 https://doi.org/10.1136/bmj.1.3658.263 https://doi.org/10.1136/bmj.1.3658.263 https://doi.org/10.1111/j.1600-051X.1980.tb01947.x https://doi.org/10.1111/j.1600-051X.1980.tb01947.x https://doi.org/10.1111/prd.12119 https://doi.org/10.1111/j.1600-051X.2012.01928.x https://doi.org/10.1002/sim.3634 https://doi.org/10.1002/sim.3634 https://doi.org/10.1111/j.1600-051X.1982.tb02099.x https://doi.org/10.1111/j.1600-051X.1982.tb02099.x https://doi.org/10.1002/jrsm.1106 https://doi.org/10.1111/j.1600-051X.2008.01245.x https://doi.org/10.1111/j.1600-051X.2008.01245.x https://doi.org/10.1902/jop.1996.67.7.666 https://doi.org/10.1902/jop.1996.67.7.666 https://doi.org/10.1016/j.jclinepi.2009.06.005 https://doi.org/10.1034/j.1600-0757.2001.027001008.x https://doi.org/10.1034/j.1600-0757.2001.027001008.x https://doi.org/10.1111/j.1600-0757.2011.00421.x https://doi.org/10.1111/j.1600-051x.1977.tb01896.x https://doi.org/10.1016/0030-4220(57)90089-0 https://doi.org/10.1902/jop.1977.48.9.558 https://doi.org/10.1902/jop.1977.48.9.558 https://doi.org/10.1111/j.1600-051x.1987.tb02249.x https://doi.org/10.1111/j.1600-051x.1987.tb02249.x https://doi.org/10.1902/jop.1974.45.8.2.601 https://doi.org/10.1902/jop.1974.45.8.2.601 https://doi.org/10.1111/jcpe.12351 https://doi.org/10.1111/j.1600-051X.1976.tb01849.x https://doi.org/10.1111/j.1600-051X.1976.tb01849.x https://doi.org/10.1111/j.1600-051x.1976.tb00042.x https://doi.org/10.1111/j.1600-051x.1976.tb00042.x https://doi.org/10.1111/j.1600-051X.2009.01408.xhttps://doi.org/10.1016/0030-4220(49)90363-1 https://doi.org/10.1016/0030-4220(49)90363-1 https://doi.org/10.1016/j.jclinepi.2010.02.005 https://doi.org/10.1016/j.jclinepi.2010.02.005 https://doi.org/10.1034/j.1600-051x.2001.028010910.x https://doi.org/10.1034/j.1600-051x.2001.028010910.x https://doi.org/10.1111/j.1600-0757.2011.00429.x https://doi.org/10.1136/bmj.i4919 https://doi.org/10.1902/jop.1985.56.4.204 https://doi.org/10.1902/jop.1985.56.4.204 https://doi.org/10.1111/j.1600-0501.2010.02064.x https://doi.org/10.1111/j.1600-051X.1998.tb02406.x https://doi.org/10.1111/j.1600-051X.1998.tb02406.x https://doi.org/10.1034/j.1600-0757.2001.22250107.x https://doi.org/10.1034/j.1600-0757.2001.22250107.x | 319POLAK et AL. Westfelt, E., Bragd, L., Socransky, S. S., Haffajee, A. D., Nyman, S., & Lindhe, J. (1985). Improved periodontal conditions following ther- apy. Journal of Clinical Periodontology, 12, 283–293. https ://doi. org/10.1111/j.1600-051x.1985.tb022 94.x Wirthlin, M. R. (1981). The current status of new attachment therapy. Journal of Periodontology, 52, 529–544. https ://doi.org/10.1902/ jop.1981.52.9.529 (1982). Proceedings from the State of the Art Workshop on Surgical Therapy for Periodontitis. Sponsored by National Institute of Dental Research, National Institutes of Health May 13–14, 1981 Workshop background paper. Journal of Periodontology, 53, 475–501. https :// doi.org/10.1902/jop.1982.53.8.475 Yukna, R. A. (1976). A clinical and histologic study of healing follow- ing the excisional new attachment procedure in rhesus monkeys. Journal of Periodontology, 47, 701–709. https ://doi.org/10.1902/ jop.1976.47.12.701 SUPPORTING INFORMATION Additional supporting information may be found online in the Supporting Information section. How to cite this article: Polak D, Wilensky A, Antonoglou GN, Shapira L, Goldstein M, Martin C. The efficacy of pocket elimination/reduction compared to access flap surgery: A systematic review and meta-analysis. J Clin Periodontol. 2020;47:303–319. https ://doi.org/10.1111/jcpe.13246 https://doi.org/10.1111/j.1600-051x.1985.tb02294.x https://doi.org/10.1111/j.1600-051x.1985.tb02294.x https://doi.org/10.1902/jop.1981.52.9.529 https://doi.org/10.1902/jop.1981.52.9.529 https://doi.org/10.1902/jop.1982.53.8.475 https://doi.org/10.1902/jop.1982.53.8.475 https://doi.org/10.1902/jop.1976.47.12.701 https://doi.org/10.1902/jop.1976.47.12.701 https://doi.org/10.1111/jcpe.13246
Compartilhar