Attention 3D U-NET for dose distribution prediction of high-dose-rate brachytherapy of cervical cancer: Direction modulated brachytherapy tandem applicator

Background: Direction Modulated Brachytherapy (DMBT) enables conformal dose distributions. However, clinicians may face challenges in creating viable treatment plans within a fast-paced clinical setting, especially for a novel technology like DMBT, where cumulative clinical experience is limited. Deep learning-based dose prediction methods have emerged as effective tools for enhancing efficiency. Purpose: To develop a voxel-wise dose prediction model using an attention-gating mechanism and a 3D UNET for cervical cancer high-dose-rate (HDR) brachytherapy treatment planning with DMBT six-groove tandems with ovoids or ring applicators. Methods: A multi-institutional cohort of 122 retrospective clinical HDR brachytherapy plans treated to a prescription dose in the range of 4.8–7.0 Gy/fraction was used. A DMBT tandem model was constructed and incorporated onto a research version of BrachyVision Treatment Planning System (BV-TPS) as a 3D solid model applicator and retrospectively re-planned all cases by seasoned experts. Those plans were randomly divided into 64:16:20 as training, validating, and testing cohorts, respectively. Data augmentation was applied to the training and validation sets to increase the size by a factor of 4. An attention-gated 3D UNET architecture model was developed to predict full 3D dose distributions based on high-risk clinical target volume (CTV_HR) and organs at risk (OARs) contour information. The model was trained using the mean absolute error loss function, Adam optimization algorithm, a learning rate of 0.001, 250 epochs, and a batch size of eight. In addition, a baseline UNET model was trained similarly for comparison. The model performance was evaluated on the testing dataset by analyzing the outcomes in terms of mean dose values and derived dose-volume-histogram indices from 3D dose distributions and comparing the generated dose distributions against the ground-truth dose distributions using dose statistics and clinically meaningful dosimetric indices. Results: The proposed attention-gated 3D UNET model showed competitive accuracy in predicting 3D dose distributions that closely resemble the ground-truth dose distributions. The average values of the mean absolute errors were 1.82 ± 29.09 Gy (vs. 6.41 ± 20.16 Gy for a baseline UNET) in CTV_HR, 0.89 ± 1.25 Gy (vs. 0.94 ± 3.96 Gy for a baseline UNET) in the bladder, 0.33 ± 0.67 Gy (vs. 0.53 ± 1.66 Gy for a baseline UNET) in the rectum, and 0.55 ± 1.57 Gy (vs. 0.76 ± 2.89 Gy for a baseline UNET) in the sigmoid. The results showed that the mean absolute error (MAE) for the bladder, rectum, and sigmoid were 0.22 ± 1.22 Gy (3.62%) (p = 0.015), 0.21 ± 1.06 Gy (2.20%) (p = 0.172), and -0.03 ± 0.54 Gy (1.13%) (p = 0.774), respectively. The MAE for D₉₀, V_100%, and V_150% of the CTV_HR were 0.46 ± 2.44 Gy (8.14%) (p = 0.018), 0.57 ± 11.25% (5.23%) (p = 0.283), and -0.43 ± 19.36% (4.62%) (p = 0.190), respectively. The proposed model needs less than 5 s to predict a full 3D dose distribution of 64 × 64 × 64 voxels for any new patient plan, thus making it sufficient for near real-time applications and aiding with decision-making in the clinic. Conclusions: Attention gated 3D-UNET model demonstrated a capability in predicting voxel-wise dose prediction, in comparison to 3D UNET, for DMBT intracavitary brachytherapy planning. The proposed model could be used to obtain dose distributions for near real-time decision-making before DMBT planning and quality assurance. This will guide future automated planning, making the workflow more efficient and clinically viable.