Showing 1–17 of 17 results for author: Han, H

Reproducibility Assessment of Magnetic Resonance Spectroscopy of Pregenual Anterior Cingulate Cortex across Sessions and Vendors via the Cloud Computing Platform CloudBrain-MRS

Authors: Runhan Chen, Meijin Lin, Jianshu Chen, Liangjie Lin, Jiazheng Wang, Xiaoqing Li, Jianhua Wang, Xu Huang, Ling Qian, Shaoxing Liu, Yuan Long, Di Guo, Xiaobo Qu, Haiwei Han

Abstract: Given the need to elucidate the mechanisms underlying illnesses and their treatment, as well as the lack of harmonization of acquisition and post-processing protocols among different magnetic resonance system vendors, this work is to determine if metabolite concentrations obtained from different sessions, machine models and even different vendors of 3 T scanners can be highly reproducible and be p… ▽ More Given the need to elucidate the mechanisms underlying illnesses and their treatment, as well as the lack of harmonization of acquisition and post-processing protocols among different magnetic resonance system vendors, this work is to determine if metabolite concentrations obtained from different sessions, machine models and even different vendors of 3 T scanners can be highly reproducible and be pooled for diagnostic analysis, which is very valuable for the research of rare diseases. Participants underwent magnetic resonance imaging (MRI) scanning once on two separate days within one week (one session per day, each session including two proton magnetic resonance spectroscopy (1H-MRS) scans with no more than a 5-minute interval between scans (no off-bed activity)) on each machine. were analyzed for reliability of within- and between- sessions using the coefficient of variation (CV) and intraclass correlation coefficient (ICC), and for reproducibility of across the machines using correlation coefficient. As for within- and between- session, all CV values for a group of all the first or second scans of a session, or for a session were almost below 20%, and most of the ICCs for metabolites range from moderate (0.4-0.59) to excellent (0.75-1), indicating high data reliability. When it comes to the reproducibility across the three scanners, all Pearson correlation coefficients across the three machines approached 1 with most around 0.9, and majority demonstrated statistical significance (P<0.01). Additionally, the intra-vendor reproducibility was greater than the inter-vendor ones. △ Less

Submitted 6 March, 2025; originally announced March 2025.

RegMixMatch: Optimizing Mixup Utilization in Semi-Supervised Learning

Authors: Haorong Han, Jidong Yuan, Chixuan Wei, Zhongyang Yu

Abstract: Consistency regularization and pseudo-labeling have significantly advanced semi-supervised learning (SSL). Prior works have effectively employed Mixup for consistency regularization in SSL. However, our findings indicate that applying Mixup for consistency regularization may degrade SSL performance by compromising the purity of artificial labels. Moreover, most pseudo-labeling based methods utiliz… ▽ More Consistency regularization and pseudo-labeling have significantly advanced semi-supervised learning (SSL). Prior works have effectively employed Mixup for consistency regularization in SSL. However, our findings indicate that applying Mixup for consistency regularization may degrade SSL performance by compromising the purity of artificial labels. Moreover, most pseudo-labeling based methods utilize thresholding strategy to exclude low-confidence data, aiming to mitigate confirmation bias; however, this approach limits the utility of unlabeled samples. To address these challenges, we propose RegMixMatch, a novel framework that optimizes the use of Mixup with both high- and low-confidence samples in SSL. First, we introduce semi-supervised RegMixup, which effectively addresses reduced artificial labels purity by using both mixed samples and clean samples for training. Second, we develop a class-aware Mixup technique that integrates information from the top-2 predicted classes into low-confidence samples and their artificial labels, reducing the confirmation bias associated with these samples and enhancing their effective utilization. Experimental results demonstrate that RegMixMatch achieves state-of-the-art performance across various SSL benchmarks. △ Less

Submitted 17 April, 2025; v1 submitted 14 December, 2024; originally announced December 2024.

Comments: Accepted in AAAI Conference on Artificial Intelligence (AAAI-25)

Mitigating Spurious Correlations via Disagreement Probability

Authors: Hyeonggeun Han, Sehwan Kim, Hyungjun Joo, Sangwoo Hong, Jungwoo Lee

Abstract: Models trained with empirical risk minimization (ERM) are prone to be biased towards spurious correlations between target labels and bias attributes, which leads to poor performance on data groups lacking spurious correlations. It is particularly challenging to address this problem when access to bias labels is not permitted. To mitigate the effect of spurious correlations without bias labels, we… ▽ More Models trained with empirical risk minimization (ERM) are prone to be biased towards spurious correlations between target labels and bias attributes, which leads to poor performance on data groups lacking spurious correlations. It is particularly challenging to address this problem when access to bias labels is not permitted. To mitigate the effect of spurious correlations without bias labels, we first introduce a novel training objective designed to robustly enhance model performance across all data samples, irrespective of the presence of spurious correlations. From this objective, we then derive a debiasing method, Disagreement Probability based Resampling for debiasing (DPR), which does not require bias labels. DPR leverages the disagreement between the target label and the prediction of a biased model to identify bias-conflicting samples-those without spurious correlations-and upsamples them according to the disagreement probability. Empirical evaluations on multiple benchmarks demonstrate that DPR achieves state-of-the-art performance over existing baselines that do not use bias labels. Furthermore, we provide a theoretical analysis that details how DPR reduces dependency on spurious correlations. △ Less

Submitted 20 December, 2024; v1 submitted 3 November, 2024; originally announced November 2024.

From Urban Clusters to Megaregions: Mapping Australia's Evolving Urban Regions

Authors: M. K. M Ng, Z. Shabrina, S. Sarkar, H. Han, C. Pettit

Abstract: This study employs percolation theory to investigate the hierarchical organisation of Australian urban centres through the connectivity of their road networks. The analysis demonstrates how discrete urban clusters have developed into integrated regional entities, delineating the pivotal distance thresholds that regulate these urban transitions. The study reveals the interconnections between dispar… ▽ More This study employs percolation theory to investigate the hierarchical organisation of Australian urban centres through the connectivity of their road networks. The analysis demonstrates how discrete urban clusters have developed into integrated regional entities, delineating the pivotal distance thresholds that regulate these urban transitions. The study reveals the interconnections between disparate urban clusters, shaped by their functional differentiation and historical development. Furthermore, the study identifies a dichotomy of urban agglomeration forces and a persistent spatial disconnection between Australia's wider urban landscape. This highlights the interplay between urban densification and peripheral growth. It suggests the need for new thinking on potential integrated governance structures that bridge urban development with broader social and economic policies across regional and national scales. Additionally, the study emphasises the growing importance of national coordination in Australian urban development planning to ensure regional consistency, equity, and productivity. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Uncertainty-enabled machine learning for emulation of regional sea-level change caused by the Antarctic Ice Sheet

Authors: Myungsoo Yoo, Giri Gopalan, Matthew J. Hoffman, Sophie Coulson, Holly Kyeore Han, Christopher K. Wikle, Trevor Hillebrand

Abstract: Projecting sea-level change in various climate-change scenarios typically involves running forward simulations of the Earth's gravitational, rotational and deformational (GRD) response to ice mass change, which requires high computational cost and time. Here we build neural-network emulators of sea-level change at 27 coastal locations, due to the GRD effects associated with future Antarctic Ice Sh… ▽ More Projecting sea-level change in various climate-change scenarios typically involves running forward simulations of the Earth's gravitational, rotational and deformational (GRD) response to ice mass change, which requires high computational cost and time. Here we build neural-network emulators of sea-level change at 27 coastal locations, due to the GRD effects associated with future Antarctic Ice Sheet mass change over the 21st century. The emulators are based on datasets produced using a numerical solver for the static sea-level equation and published ISMIP6-2100 ice-sheet model simulations referenced in the IPCC AR6 report. We show that the neural-network emulators have an accuracy that is competitive with baseline machine learning emulators. In order to quantify uncertainty, we derive well-calibrated prediction intervals for simulated sea-level change via a linear regression postprocessing technique that uses (nonlinear) machine learning model outputs, a technique that has previously been applied to numerical climate models. We also demonstrate substantial gains in computational efficiency: a feedforward neural-network emulator exhibits on the order of 100 times speedup in comparison to the numerical sea-level equation solver that is used for training. △ Less

Submitted 21 June, 2024; originally announced June 2024.

Optimal subsampling algorithm for the marginal model with large longitudinal data

Authors: Haohui Han, Liya Fu

Abstract: Big data is ubiquitous in practices, and it has also led to heavy computation burden. To reduce the calculation cost and ensure the effectiveness of parameter estimators, an optimal subset sampling method is proposed to estimate the parameters in marginal models with massive longitudinal data. The optimal subsampling probabilities are derived, and the corresponding asymptotic properties are establ… ▽ More Big data is ubiquitous in practices, and it has also led to heavy computation burden. To reduce the calculation cost and ensure the effectiveness of parameter estimators, an optimal subset sampling method is proposed to estimate the parameters in marginal models with massive longitudinal data. The optimal subsampling probabilities are derived, and the corresponding asymptotic properties are established to ensure the consistency and asymptotic normality of the estimator. Extensive simulation studies are carried out to evaluate the performance of the proposed method for continuous, binary and count data and with four different working correlation matrices. A depression data is used to illustrate the proposed method. △ Less

Submitted 15 November, 2023; originally announced November 2023.

IROS 2019 Lifelong Robotic Vision Challenge -- Lifelong Object Recognition Report

Authors: Qi She, Fan Feng, Qi Liu, Rosa H. M. Chan, Xinyue Hao, Chuanlin Lan, Qihan Yang, Vincenzo Lomonaco, German I. Parisi, Heechul Bae, Eoin Brophy, Baoquan Chen, Gabriele Graffieti, Vidit Goel, Hyonyoung Han, Sathursan Kanagarajah, Somesh Kumar, Siew-Kei Lam, Tin Lun Lam, Liang Ma, Davide Maltoni, Lorenzo Pellegrini, Duvindu Piyasena, Shiliang Pu, Debdoot Sheet , et al. (11 additional authors not shown)

Abstract: This report summarizes IROS 2019-Lifelong Robotic Vision Competition (Lifelong Object Recognition Challenge) with methods and results from the top $8$ finalists (out of over~$150$ teams). The competition dataset (L)ifel(O)ng (R)obotic V(IS)ion (OpenLORIS) - Object Recognition (OpenLORIS-object) is designed for driving lifelong/continual learning research and application in robotic vision domain, w… ▽ More This report summarizes IROS 2019-Lifelong Robotic Vision Competition (Lifelong Object Recognition Challenge) with methods and results from the top $8$ finalists (out of over~$150$ teams). The competition dataset (L)ifel(O)ng (R)obotic V(IS)ion (OpenLORIS) - Object Recognition (OpenLORIS-object) is designed for driving lifelong/continual learning research and application in robotic vision domain, with everyday objects in home, office, campus, and mall scenarios. The dataset explicitly quantifies the variants of illumination, object occlusion, object size, camera-object distance/angles, and clutter information. Rules are designed to quantify the learning capability of the robotic vision system when faced with the objects appearing in the dynamic environments in the contest. Individual reports, dataset information, rules, and released source code can be found at the project homepage: "https://lifelong-robotic-vision.github.io/competition/". △ Less

Submitted 26 April, 2020; originally announced April 2020.

Comments: 9 pages, 11 figures, 3 tables, accepted into IEEE Robotics and Automation Magazine. arXiv admin note: text overlap with arXiv:1911.06487

Neural Network Solutions to Differential Equations in Non-Convex Domains: Solving the Electric Field in the Slit-Well Microfluidic Device

Authors: Martin Magill, Andrew M. Nagel, Hendrick W. de Haan

Abstract: The neural network method of solving differential equations is used to approximate the electric potential and corresponding electric field in the slit-well microfluidic device. The device's geometry is non-convex, making this a challenging problem to solve using the neural network method. To validate the method, the neural network solutions are compared to a reference solution obtained using the f… ▽ More The neural network method of solving differential equations is used to approximate the electric potential and corresponding electric field in the slit-well microfluidic device. The device's geometry is non-convex, making this a challenging problem to solve using the neural network method. To validate the method, the neural network solutions are compared to a reference solution obtained using the finite element method. Additional metrics are presented that measure how well the neural networks recover important physical invariants that are not explicitly enforced during training: spatial symmetries and conservation of electric flux. Finally, as an application-specific test of validity, neural network electric fields are incorporated into particle simulations. Conveniently, the same loss functional used to train the neural networks also seems to provide a reliable estimator of the networks' true errors, as measured by any of the metrics considered here. In all metrics, deep neural networks significantly outperform shallow neural networks, even when normalized by computational cost. Altogether, the results suggest that the neural network method can reliably produce solutions of acceptable accuracy for use in subsequent physical computations, such as particle simulations. △ Less

Submitted 25 April, 2020; originally announced April 2020.

Comments: 16 pages, 12 figures, 1 table

Journal ref: Phys. Rev. Research 2, 033110 (2020)

EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction

Authors: Youru Li, Zhenfeng Zhu, Deqiang Kong, Hua Han, Yao Zhao

Abstract: Time series prediction with deep learning methods, especially long short-term memory neural networks (LSTMs), have scored significant achievements in recent years. Despite the fact that the LSTMs can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within multiple time-steps is insufficient. To address this issue, an evolutionary attent… ▽ More Time series prediction with deep learning methods, especially long short-term memory neural networks (LSTMs), have scored significant achievements in recent years. Despite the fact that the LSTMs can help to capture long-term dependencies, its ability to pay different degree of attention on sub-window feature within multiple time-steps is insufficient. To address this issue, an evolutionary attention-based LSTM training with competitive random search is proposed for multivariate time series prediction. By transferring shared parameters, an evolutionary attention learning approach is introduced to the LSTMs model. Thus, like that for biological evolution, the pattern for importance-based attention sampling can be confirmed during temporal relationship mining. To refrain from being trapped into partial optimization like traditional gradient-based methods, an evolutionary computation inspired competitive random search method is proposed, which can well configure the parameters in the attention layer. Experimental results have illustrated that the proposed model can achieve competetive prediction performance compared with other baseline methods. △ Less

Submitted 8 November, 2018; originally announced November 2018.

Neural Networks Trained to Solve Differential Equations Learn General Representations

Authors: Martin Magill, Faisal Qureshi, Hendrick W. de Haan

Abstract: We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden… ▽ More We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layer is general, and that deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we visualize the general representations of the first layers, and interpret them as generalized coordinates over the input domain. △ Less

Submitted 29 June, 2018; originally announced July 2018.

Comments: 14 pages, 9 figures. Submitted to NIPS 2018

Simulating outcomes of interventions using a multipurpose simulation program based on the Evolutionary Causal Matrices and Markov Chain

Authors: Hyemin Han, Kangwook Lee, Firat Soylu

Abstract: Predicting long-term outcomes of interventions is necessary for educational and social policy-making processes that might widely influence our society for the long-term. However, performing such predictions based on data from large-scale experiments might be challenging due to the lack of time and resources. In order to address this issue, computer simulations based on Evolutionary Causal Matrices… ▽ More Predicting long-term outcomes of interventions is necessary for educational and social policy-making processes that might widely influence our society for the long-term. However, performing such predictions based on data from large-scale experiments might be challenging due to the lack of time and resources. In order to address this issue, computer simulations based on Evolutionary Causal Matrices and Markov Chain can be used to predict long-term outcomes with relatively small-scale lab data. In this paper, we introduce Python classes implementing a computer simulation model and presented some pilots implementations demonstrating how the model can be utilized for predicting outcomes of diverse interventions. We also introduce the class-structured simulation module both with real experimental data and with hypothetical data formulated based on social psychological theories. Classes developed and tested in the present study provide researchers and practitioners with a feasible and practical method to simulate intervention outcomes prospectively. △ Less

Submitted 26 November, 2017; originally announced November 2017.

Intraoperative margin assessment of human breast tissue in optical coherence tomography images using deep neural networks

Authors: Amal Rannen Triki, Matthew B. Blaschko, Yoon Mo Jung, Seungri Song, Hyun Ju Han, Seung Il Kim, Chulmin Joo

Abstract: Objective: In this work, we perform margin assessment of human breast tissue from optical coherence tomography (OCT) images using deep neural networks (DNNs). This work simulates an intraoperative setting for breast cancer lumpectomy. Methods: To train the DNNs, we use both the state-of-the-art methods (Weight Decay and DropOut) and a newly introduced regularization method based on function norms.… ▽ More Objective: In this work, we perform margin assessment of human breast tissue from optical coherence tomography (OCT) images using deep neural networks (DNNs). This work simulates an intraoperative setting for breast cancer lumpectomy. Methods: To train the DNNs, we use both the state-of-the-art methods (Weight Decay and DropOut) and a newly introduced regularization method based on function norms. Commonly used methods can fail when only a small database is available. The use of a function norm introduces a direct control over the complexity of the function with the aim of diminishing the risk of overfitting. Results: As neither the code nor the data of previous results are publicly available, the obtained results are compared with reported results in the literature for a conservative comparison. Moreover, our method is applied to locally collected data on several data configurations. The reported results are the average over the different trials. Conclusion: The experimental results show that the use of DNNs yields significantly better results than other techniques when evaluated in terms of sensitivity, specificity, F1 score, G-mean and Matthews correlation coefficient. Function norm regularization yielded higher and more robust results than competing methods. Significance: We have demonstrated a system that shows high promise for (partially) automated margin assessment of human breast tissue, Equal error rate (EER) is reduced from approximately 12\% (the lowest reported in the literature) to 5\%\,--\,a 58\% reduction. The method is computationally feasible for intraoperative application (less than 2 seconds per image). △ Less

Submitted 31 March, 2017; originally announced March 2017.

Comments: 16 pages, 9 figures

Predicting Long-term Outcomes of Educational Interventions Using the Evolutionary Causal Matrices and Markov Chain Based on Educational Neuroscience

Authors: Hyemin Han, Kangwook Lee, Firat Soylu

Abstract: We developed a prediction model based on the evolutionary causal matrices (ECM) and the Markov Chain to predict long-term influences of educational interventions on adolescents development. Particularly, we created a computational model predicting longitudinal influences of different types of stories of moral exemplars on adolescents voluntary service participation. We tested whether the developed… ▽ More We developed a prediction model based on the evolutionary causal matrices (ECM) and the Markov Chain to predict long-term influences of educational interventions on adolescents development. Particularly, we created a computational model predicting longitudinal influences of different types of stories of moral exemplars on adolescents voluntary service participation. We tested whether the developed prediction model can properly predict a long-term longitudinal trend of change in voluntary service participation rate by comparing prediction results and surveyed data. Furthermore, we examined which type of intervention would most effectively promote service engagement and what is the minimum required frequency of intervention to produce a large effect. We discussed the implications of the developed prediction model in educational interventions based on educational neuroscience. △ Less

Submitted 30 November, 2016; originally announced December 2016.

Journal ref: Trends.Neurosci.Educ. 5 (2016) 157-165

Quantile Dependence between Stock Markets and its Application in Volatility Forecasting

Authors: Heejoon Han

Abstract: This paper examines quantile dependence between international stock markets and evaluates its use for improving volatility forecasting. First, we analyze quantile dependence and directional predictability between the US stock market and stock markets in the UK, Germany, France and Japan. We use the cross-quantilogram, which is a correlation statistic of quantile hit processes. The detailed depende… ▽ More This paper examines quantile dependence between international stock markets and evaluates its use for improving volatility forecasting. First, we analyze quantile dependence and directional predictability between the US stock market and stock markets in the UK, Germany, France and Japan. We use the cross-quantilogram, which is a correlation statistic of quantile hit processes. The detailed dependence between stock markets depends on specific quantile ranges and this dependence is generally asymmetric; the negative spillover effect is stronger than the positive spillover effect and there exists strong directional predictability from the US market to the UK, Germany, France and Japan markets. Second, we consider a simple quantile-augmented volatility model that accommodates the quantile dependence and directional predictability between the US market and these other markets. The quantile-augmented volatility model provides superior in-sample and out-of-sample volatility forecasts. △ Less

Submitted 25 August, 2016; originally announced August 2016.

Galton's Family Heights Data Revisited

Authors: Hao Han, Yeming Ma, Wei Zhu

Abstract: Galton's family heights data has been a preeminent historical dataset in regression analysis, on which the original model and basic results have survived the close scrutiny of statisticians for 125 years. However by revisiting Galton's family data, we challenge whether Galton's classic model and his regression towards mean interpretation are proper. Using Galton's data as a benchmark for different… ▽ More Galton's family heights data has been a preeminent historical dataset in regression analysis, on which the original model and basic results have survived the close scrutiny of statisticians for 125 years. However by revisiting Galton's family data, we challenge whether Galton's classic model and his regression towards mean interpretation are proper. Using Galton's data as a benchmark for different regression methods, such as least squares, orthogonal regression, geometric mean regression, and least sine squares regression - a newly developed nonparametric robust regression approach, we elucidate that his regression model has fundamental drawbacks not only in variable and model selection by "transmuting" women into men thus the simple linear model, but also a strong bias in least squares regression leading to otherwise alternative conclusions on the true relationships between the heights of the child and his or her parents. △ Less

Submitted 12 August, 2015; originally announced August 2015.

RCR: Robust Compound Regression for Robust Estimation of Errors-in-Variables Model

Authors: Hao Han, Wei Zhu

Abstract: The errors-in-variables (EIV) regression model, being more realistic by accounting for measurement errors in both the dependent and the independent variables, is widely adopted in applied sciences. The traditional EIV model estimators, however, can be highly biased by outliers and other departures from the underlying assumptions. In this paper, we develop a novel nonparametric regression approach… ▽ More The errors-in-variables (EIV) regression model, being more realistic by accounting for measurement errors in both the dependent and the independent variables, is widely adopted in applied sciences. The traditional EIV model estimators, however, can be highly biased by outliers and other departures from the underlying assumptions. In this paper, we develop a novel nonparametric regression approach - the robust compound regression (RCR) analysis method for the robust estimation of EIV models. We first introduce a robust and efficient estimator called least sine squares (LSS). Taking full advantage of both the new LSS method and the compound regression analysis method developed in our own group, we subsequently propose the RCR approach as a generalization of those two, which provides a robust counterpart of the entire class of the maximum likelihood estimation (MLE) solutions of the EIV model, in a 1-1 mapping. Technically, our approach gives users the flexibility to select from a class of RCR estimates the optimal one with a predefined regression efficiency criterion satisfied. Simulation studies and real-life examples are provided to illustrate the effectiveness of the RCR approach. △ Less

Submitted 12 August, 2015; originally announced August 2015.

The Cross-Quantilogram: Measuring Quantile Dependence and Testing Directional Predictability between Time Series

Authors: Heejoon Han, Oliver Linton, Tatsushi Oka, Yoon-Jae Whang

Abstract: This paper proposes the cross-quantilogram to measure the quantile dependence between two time series. We apply it to test the hypothesis that one time series has no directional predictability to another time series. We establish the asymptotic distribution of the cross quantilogram and the corresponding test statistic. The limiting distributions depend on nuisance parameters. To construct consist… ▽ More This paper proposes the cross-quantilogram to measure the quantile dependence between two time series. We apply it to test the hypothesis that one time series has no directional predictability to another time series. We establish the asymptotic distribution of the cross quantilogram and the corresponding test statistic. The limiting distributions depend on nuisance parameters. To construct consistent confidence intervals we employ the stationary bootstrap procedure; we show the consistency of this bootstrap. Also, we consider the self-normalized approach, which is shown to be asymptotically pivotal under the null hypothesis of no predictability. We provide simulation studies and two empirical applications. First, we use the cross-quantilogram to detect predictability from stock variance to excess stock return. Compared to existing tools used in the literature of stock return predictability, our method provides a more complete relationship between a predictor and stock return. Second, we investigate the systemic risk of individual financial institutions, such as JP Morgan Chase, Goldman Sachs and AIG. This article has supplementary materials online. △ Less

Submitted 20 January, 2018; v1 submitted 9 February, 2014; originally announced February 2014.

Journal ref: Journal of Econometrics, 2016, 193, 251-270