Cinnamon-GUI: Convolutional Neural Network And Multimodal Learning with Graphic User Interface for Digital Pathology

lunanfoldomics · September 9, 2024, 5:02am

Authors: Luca Zammataro

Abstract: CINNAMON-GUI is an advanced digital pathology tool that leverages the power of convolutional neural networks (CNN) and multimodal learning to provide an intuitive graphical user interface for research professionals. This tool is designed to enhance the analysis and interpretation of pathological samples, making it easier to study various diseases. Cinnamon-GUI has evolved from its initial implementation with ipywidgets for Jupyter Lab and Notebook to a fully-fledged Shiny app in Python. This transition brings several advantages, notably an enhanced user interface and improved interactivity. The Shiny framework allows for dynamic and responsive web applications, making it more straightforward for users to interact with machine learning models and visualize results in real time. This update ensures that Cinnamon-GUI remains at the forefront of digital pathology tools, providing a more robust and user-friendly experience for researchers and clinicians. The user interface, built with Shiny components, provides a responsive and interactive experience. Images and plots are generated dynamically using base64 encoding, ensuring efficient memory usage and performance.

Full Description: ## Key Features

Advanced CNN Models: Utilize state-of-the-art convolutional neural networks optimized for digital pathology.
Multimodal Learning Capabilities: Cinnamon-Gui is predispoded for integrating multiple types of data to improve analysis accuracy (only for research purpose).
User-Friendly Interface with Dynamic UI Components: Simplify the workflow with an intuitive GUI, designed for professionals without technical expertise.
Image and Plot Display: The application dynamically generates and displays images and plots using base64 encoding, ensuring efficient memory usage and performance.
Scalable Solutions: Suitable for individual researchers and large organizations.

Detailed Workflow

This schematic illustrates two primary operational pathways within the Cinnamon-GUI platform: 1) Data Processing and Model Training Workflow, where users can load pickle files from datasets to train models and generate reports post-testing, and 2) Biospecimen Annotation and Analysis Workflow, where users can load annotated Pap smear images, with annotations applied using the integrated Labelme software. This workflow supports loading the corresponding JSON annotation files and converting them and the images into a pickle dataset format for subsequent cell type prediction and classification. Additionally, this pathway facilitates the creation and expansion of datasets, enhancing the utility of Cinnamon-GUI for diverse research applications.

Initialization and Logging

Log Initialization: The application initializes a log file (log.txt) at startup to record all activities and messages throughout the session.
Notification System: Provides real-time notifications to users about the status and progress of various operations.

Dataset Loading and Preprocessing

Dataset Upload: Users can upload a dataset in the form of a pickle file. The dataset is then loaded into memory, normalized, and reshaped for processing.
Class Definitions: Users can upload a classes file (TSV format) to define the classes used in the dataset. These classes are used for labeling and classification during model training and evaluation.

Model Loading and Summary

Model Upload: Users can upload a pre-trained model file.
Model Summary: The loaded model's architecture is summarized and displayed in a tabular format, providing detailed information about each layer and its parameters.

Annotation Processing and Dataset Creation

Labelme Integration: The application can open Labelme for defining regions of interest (ROIs) and labeling cells in specimen images.
Annotation Processing: Users can upload an image file and its corresponding annotation file. The application processes these annotations to extract cells, classify them, and create a new dataset.
In-Memory Image Handling: All images are handled in memory using BytesIO and base64 encoding.

Model Training

Training Configuration: Users can configure various parameters for training the model, including the seed, epochs, batch size, learning rate, and data augmentation settings.
Training Process: The application trains the model using the specified configuration. Progress is displayed in real-time, including a progress bar and status messages.
Early Stopping and Learning Rate Reduction: The training process includes callbacks for early stopping and reducing the learning rate based on validation loss.
Model and Log Saving: The trained model and log file are automatically saved to disk upon completion of training.

CNN Architectures

Through the Training Tab, users can configure and implement various Convolutional Neural Network (CNN) architectures tailored to their specific needs. The interface allows customization of multiple hyperparameters, including the number of convolutional layers, the number of filters, kernel sizes, activation functions, and pooling layers. Additionally, users can add fully connected layers with customizable activation functions, dropout rates, and regularization techniques such as L1 and L2 regularization. This flexibility enables the creation of both simple and complex CNN models, making it suitable for a wide range of applications in digital pathology, from basic image classification to more advanced feature extraction and pattern recognition tasks.

Users can also choose from a variety of optimization algorithms to train their models, including:

Stochastic Gradient Descent (SGD): A straightforward and widely-used optimizer that updates model parameters iteratively based on each training example.
Adam: Combines the benefits of two other extensions of stochastic gradient descent, Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Propagation (RMSProp), making it efficient for large datasets and high-dimensional parameter spaces.
RMSprop: An adaptive learning rate method designed to perform well in the online and non-stationary settings.
Adagrad: Adapts the learning rate to the parameters, performing larger updates for infrequent and smaller updates for frequent parameters.
Adadelta: An extension of Adagrad that seeks to reduce its aggressive, monotonically decreasing learning rate.
Adamax: A variant of Adam based on the infinity norm.
Nadam: Combines Adam and Nesterov Accelerated Gradient (NAG) to provide better convergence in some scenarios.

These options allow users to fine-tune the learning process to achieve optimal performance for their specific datasets and tasks.

Image Augmentation

The Training Tab includes robust Image Augmentation capabilities that enhance the diversity of the training dataset and improve the generalization of the model. Image Augmentation involves applying various transformations to the training images, which helps the model become invariant to these transformations and thus perform better on unseen data. Users can customize the following augmentation parameters:

Rotation Range: Randomly rotates images within a specified degree range, allowing the model to become invariant to rotational changes.
Width Shift Range (W-shift): Randomly shifts images horizontally, enabling the model to handle horizontal translations of objects within the images.
Height Shift Range (H-shift): Randomly shifts images vertically, helping the model cope with vertical translations of objects.
Shear Range: Applies shearing transformations to the images, effectively tilting them to simulate changes in the camera angle or object orientation.
Zoom Range: Randomly zooms into images, allowing the model to be robust to variations in object size within the images.
Horizontal Flip: Randomly flips images horizontally, enhancing the model's ability to recognize objects irrespective of their left-right orientation.
Vertical Flip: Randomly flips images vertically, useful for datasets where objects can appear upside down.

These augmentation techniques are applied in real-time during training, generating new variations of the images in each epoch. This process helps prevent overfitting and makes the model more resilient to variations in real-world data.

Evaluation and Analysis

Image Display and Feature Mapping: Users can select an image index to display the image along with its predicted class. Optionally, feature mapping plots can be generated to visualize the activations of different layers in the model.
Performance Testing: The application generates tests on the testing dataset to evaluate the model's performance. It compares the predicted labels against the actual labels to provide counts of predictions versus actual values. The application generates a detailed report of the predicted and actual class counts, along with a summary table and plot. All reports are displayed in the application without saving to disk.
Interactive Plot: An interactive plot feature enhances user interaction. This plot allows for the immediate identification of images classified by the model through a mouse-over function.
This addition aims to improve the usability and functionality of the app, making the analytical process more intuitive and accessible.
External Specimen Screening: When used to screen an external specimen with unknown characteristics, the application provides a report to identify specific cells, such as cancerous cells in a Pap smear. This allows for precise identification and classification in real-world diagnostic scenarios.

Dataset

As example of implementation we uses the Sipakmed Dataset.

The SIPaKMeD database is publicly available and it can be used for experimental purposes with the request to cite the following paper:

Marina E. Plissiti, Panagiotis Dimitrakopoulos, Giorgos Sfikas, Christophoros Nikou, Olga Krikoni, Antonia Charchanti, SIPAKMED: A new dataset for feature and image based classification of normal and pathological cervical cells in Pap smear images, IEEE International Conference on Image Processing (ICIP) 2018, Athens, Greece, 7-10 October 2018.

Also, you can download a pickle version of Sipakmed from here:

and pre-trained models from here:

Model-B:

Other models:

The classes.tsv file

The file "classes.tsv" contains pathological classes associated with the labels. It is pivotal for displaying the results.
The classes.tsv file for SIPAKMED is the follow:

Class	Label
Dyskeratocytes	0
Koilocytes	1
Metaplastic Cells	2
Parabasal Cells	3
Superficial-Intermediate Cells	4

The classes.tsv file associates numerical labels with more intuitive literal characters.
There are no predefined rules for generating class names, but using only a short class name is a good practice. Using letters with at most two characters to define a label is better. Remember to start with the label with a zero value when defining the first class, as we are in a Python system, and we know that Python always starts from zero and never from one!
The classes have been abbreviated to letters to make the various output more readable. They correspond respectively to the five classes 'Diskeratotic,' 'Koilocytes,' 'Metaplastic,' 'Parabasal,' and 'Superficial-Intermediate,' as reported in the Sipakmed dataset.
Generating your dataset.pickle and classes. tsv files is a straightforward process. Always ensure that a classes.tsv file accompanies your dataset and is located in the same directory as the pickle file. Additionally, make sure that both files share the same name. For example, if you decide to name your dataset.pickle 256X256.pickle, ensure that its accompanying classes file is named 256X256.classes.tsv

CINNAMON-GUI includes internal functions for image normalization and a suite of functions for randomly splitting the dataset into training and testing sets for CNN learning. Users can select from a wide range of seeds for random splitting via the scikit-learn package using a dedicated sliding bar in the GUI's Training Tab.
The Table illustrates the architecture implemented for classifying the Sipakmed.

Layer (type)	Output Shape	Param #
conv2d (Conv2D)	(None, 254, 254, 32)	896
max_pooling2d (MaxPooling2D)	(None, 127, 127, 32)	0
conv2d_1 (Conv2D)	(None, 125, 125, 64)	18496
max_pooling2d_1 (MaxPooling2D)	(None, 62, 62, 64)	0
conv2d_2 (Conv2D)	(None, 60, 60, 128)	73856
max_pooling2d_2 (MaxPooling2D)	(None, 30, 30, 128)	0
conv2d_3 (Conv2D)	(None, 28, 28, 256)	295168
max_pooling2d_3 (MaxPooling2D)	(None, 14, 14, 256)	0
flatten (Flatten)	(None, 50176)	0
dense (Dense)	(None, 256)	12845312
dense_1 (Dense)	(None, 5)	1285

Other parameters used or this architecture:

seed: 8791092
total epochs: 100
test size: 0.2
batch size: 32
rotation_range: 20
width_shift_range: 0.2
height_shift_range: 0.2
shear_range: 0.2
zoom_range: 0.2
horizontal_flip: True
vertical_flip: False
regularization params: (0.001, 0.001)
dropout_value: 0.5
optimizer: Adam
loss: categorical_crossentropy
learning_rate: 0.0001

With this architecture, the CNN for the Sipakmed achieved a training accuracy of 95% on the validation test after 100 epochs of learning, with both regularization parameters set to 0.001.

License

SPDX-License-Identifier: AGPL-3.0-only

This project is licensed under the terms of the GNU Affero General Public License version 3. See the LICENSE file in the root directory of the repository for details.

Disclaimer: the software use is intended ONLY for experimental purposes, not for clinical.

For more information, visit our website: Lunan Foldomics LLC

Contact

For support or inquiries, please contact us via email at lucazammataro@lunanfoldomicsllc.com or at info@lunanfoldomicsllc.com or visit the contact page on our website.

How To Cite

Zammataro L. CINNAMON-GUI: Revolutionizing Pap Smear Analysis with CNN-Based Digital Pathology Image Classification [version 1; peer review: awaiting peer review]. F1000Research 2024, 13:897 (CINNAMON-GUI: Revolutionizing Pap Smear Analysis... | F1000Research)

Shiny app: Posit Connect Cloud
Repo: GitHub - lunanfoldomics/Cinnamon-GUI: Convolutional Neural Network And Multimodal Learning with Graphic User Interface for Digital Pathology

Thumbnail:

Full image:

Cinnamon-GUI: Convolutional Neural Network And Multimodal Learning with Graphic User Interface for Digital Pathology - Shiny Contest Submission