Cybernetics And Systems Analysis logo
Editorial Board Announcements Abstracts Authors Archive
Cybernetics And Systems Analysis
International Theoretical Science Journal
Volume 59 >>> № 3 MAY — JUNE 2023
-->

UDC 004.891.3
O.A. Zhukovska1, L.S. Fainzilberg2


1 National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”, Kyiv, Ukraine

 zhukovskaya71@gmail.com

2 International Scientific and Training Center of Information Technologies and Systems, National Academy of Sciences of Ukraine and Ministry of Education and Science of Ukraine, Kyiv, Ukraine; National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”, Kyiv, Ukraine

 fainzilberg@gmail.com

EVALUATION OF THE USEFULNESS OF BINARY CLASSIFIER
BASED ON ENHANCED ROC-ANALYSIS

Abstract. The definition of the usefulness of the binary classifier from the point of view of reducing the a priori risk of false classification is formulated. Sufficient conditions are proposed to guarantee the utility of a diagnostic test according to this definition. The obtained conditions improved the traditional ROC analysis by limiting the corresponding region of the ROC curve. The line limiting the region of the guaranteed useful test is shown to coincide with the known iso-performance line corresponding to the a priori risk level. Permissible limits of the ratio of losses from target misses and false alarms were determined, according to which a test with appropriate operational characteristics remains useful for screening a disease with a known prevalence. Based on the obtained results, the authors substantiated the effectiveness of the new method of the analysis and interpretation of electrocardiograms, which is based on determining the original diagnostic feature in the phase space and enables detecting persons with a high risk of coronary heart disease in the early stages of the disease.

Keywords: binary classifier, ROC curve, diagnostic feature, analysis and interpretation of ECG.


full text

REFERENCES

  1. Maxim D., Niebo R., Utel M.J. Screening tests: A review with examples. Inhalation Toxicology. 2014. Vol. 26. Iss. 13. P. 811–828. http://doi.org/10.3109/08958378.2014.955932.

  2. Zhukovska O. Decision-making model on potential borrower lending for in-dependent experts group. Proc. IEEE 3rd Intern. Conf. on System Analysis & Intelligent Computing (SAIC) (4–7 Oct. 2022, Kyiv, Ukraine). P. 118–121. http://doi.org/10.1109/SAIC57818.2022.9923015 .

  3. Dendek C., Mandziuk J. Improving performance of a binary classifier by training set selection. Proc. of the 18th International Conference on Artificial Neural Networks (September 3–6, 2008, Prague Czech Republic). Part I. 2008. P. 1–8. http://doi.org/10.1007/978-3-540-87536-9_14 .

  4. Metz C.E. Fundamental ROC analysis. In: Handbook of Medical Imaging. R.L. Van Metter; Beutel J., Kundel H.L. (Eds). Vol. 1. Physics and Psychophysics. Bellingham: SPIE Press, 2000. P. 754–769. https://doi.org/10.1117/3.832716.ch15 .

  5. Fawcett T. Using rule sets to maximize ROC performance. Proc. of the IEEE International Conference on Data Mining (ICDM-2001). Los Alamitos, CA: IEEE Computer Society, 2001. P. 131–138. https://doi.org/10.1109/ICDM.2001.989510.

  6. Flach P., Wu S. Repairing concavities in ROC curves. Proc. of the 2003 UK Workshop on Computational Intelligence. University of Bristol, 2003. P. 38–44.

  7. Sonego P., Kocsor A., Pongor S. ROC analysis: Applications to the classification of biological sequences and 3D structures. Briefings in Bioinformatics. 2008. Vol. 9, Iss. 3. P. 198–209. https://doi.org/10.1093/bib/bbm064.

  8. Feng K., Hong H., Nang K., Wang J. Decision making with machine learning and ROC curves. arXiv:1905.02810v1 [stat.ME] 5 May 2019. https://doi.org/10.48550/arXiv.1905.02810.

  9. Davis J., Goadrich M. The relationship between Precision-Recall and ROC curves. Proc. of the 23rd International Conference on Machine Learning (ICML’06). 2006. P. 233–240. https://doi.org/10.1145/1143844.1143874 .

  10. Van den Hout W. The area under an ROC curve with limited information. Medical Decision Making. 2003. Vol. 23, Iss. 2. P. 160–166. https://doi.org/10.1177/0272989X03251246.

  11. Pepe M.S, Longton G., Janes H. Estimation and comparison of receiver operating characteristic curves. The Stata Journal. 2009. Vol. 9, N 1. P. 1–16. PMID: 20161343.

  12. Alonzo T.A., Pepe M.S. Distribution-free ROC analysis using binary regression techniques. Biostatistics. 2002. N 3. P. 421–432. https://doi.org/10.1093/biostatistics/3.3.421 .

  13. Provost F., Fawcett T. Robust classification for imprecise environments. Machine Learning. 2001. Vol. 42, N 3. P. 203–231. https://doi.org/10.1023/A:1007601015854.

  14. Majnik M., Bosni Z. ROC analysis of classifiers in machine learning: A survey. Intelligent Data Analysis. 2013. Vol. 17, N 3. P. 531–558. https://doi.org/10.3233/IDA-130592 .

  15. Hu N. Using receiver operating characteristic (ROC) analysis to evaluate information-based decision-making. In: Advanced Methodologies and Technologies in Business Operations and Management. 2019. P. 764–776. https://doi.org/10.4018/978-1-5225-7362-3.ch057.

  16. Hand D. J., Till R. J. A simple generalization of the area under the ROC curve to multiple class classification problems. Machine Learning. 2001. Vol. 45, N 2. P. 171–186. https://doi.org/10.1023/A:1010920819831.

  17. Ferri C., Hernїndez-Orallo J., Salido M.A. Volume under the ROC surface for multi-class problems. In: Machine Learning: ECML 2003. ECML 2003. Lavrac N., Gamberger D., Blockeel H., Todorovski L. (Eds.). Lecture Notes in Computer Science. Vol. 2837. Berlin; Heidelberg: Springer, 2003. P. 108–120. https://doi.org/10.1007/978-3-540-39857-8_12.

  18. Edwards D.C, Metz C.E, Kupinski M.A. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans. Med. Imaging. 2004. Vol. 23, N 7. P. 891–895. https://doi.org/10.1109/TMI.2004.828358 .

  19. Edwards D.C, Metz C.E, Nishikawa R.M. The hypervolume under the ROC hypersurface of “Near-Guessing” and “Near-Perfect” observers in N-class classification tasks. IEEE Trans. Med. Imaging. 2005. Vol. 24, N 3. P. 293–299. https://doi.org/10.1109/tmi.2004.841227.

  20. He X, Fry E.C. An optimal three-class linear observer derived from decision theory. IEEE Trans. Med. Imaging. 2007. Vol. 26, N 1. P. 77–83. https://doi.org/10.1109/TMI.2006.885335.

  21. Sahiner B, Chan H-P, Hadjiiski L.M. Performance analysis of 3-class classifiers: Properties of the 3D ROC surface and the normalized volume under the surface. IEEE Trans. Med. Imaging. 2008. Vol. 27, Iss. 2. P. 215–227. https://doi.org/10.1109/TMI.2007.905822.

  22. Fawcett T. ROC graphs with instance-varying costs. Pattern Recognition Letters. 2006. Vol. 27, Iss. 8. P. 882–891. .

  23. Meekins R., Adams S, Beling P.A., Farinholt K., Hipwell N., Chaudhry A., Polter S., Dong Q. Cost-sensitive classifier selection when there is additional cost information. Proceedings of Machine Learning Research. 2018. Vol. 88. P. 17–30. PMLR 88:17-30.

  24. Holte R.C., Drummond C. Cost-sensitive classifier evaluation using cost curves. In: Advances in Knowledge Discovery and Data Mining. PAKDD 2008. Lecture Notes in Computer Science. Berlin; Heidelberg: Springer, 2008. Vol. 5012. P. 26–29. https://doi.org/10.1007/978-3-540-68125-0_4.

  25. Fainzilberg L.S. Plausible but groundless premises when constructing diagnostic models. Journal of Automation and Information Sciences. 2020. Vol. 52, Iss. 5. P. 38–50. https://doi.org/10.1615/JAutomatInfScien.v52.i5.40 .

  26. Fainzilberg L.S. Conditions of utility of diagnostic tests from the point of view of the statistical theory of decision making. Journal of Automation and Information Sciences. 2003. Vol. 35, Iss. 4. P. 63–73. https://doi.org/10.1615/JAutomatInfScien.v35.i4.30.

  27. Fainzilberg L.S. New opportunities of phasegraphy in medical practice. Science and Innovation. 2017. Vol. 1, Iss. 3. P. 37–50. https://doi.org/10.15407/scine13.03.037.

  28. Fainzilberg L.S. New approaches to the analysis and interpretation of the shape of cyclic signals. Cybernetics and Systems Analysis. 2020. Vol. 56, N 4. P. 665–674. https://doi.org/10.1007/s10559-020-00283-0.




Editorial Board Announcements Abstracts Authors Archive
about scope office editorial board instructions for authors subscriptions contacts
© 2023 Kibernetika.org. All rights reserved.