Extending the Segment Anything Model (SAM) for Marine Debris Segmentation in Sentinel-2 Imagery

By Joost van Dalen

Abstract
Marine debris is a growing environmental concern, with optical satellite data being vital in monitoring efforts. However, manual visual interpretation using spectral indices and the development of deep segmentation models are limited by the heterogeneous nature of debris and its surroundings in medium-resolution satellite images.

This thesis addresses these challenges through two primary approaches. First, it introduces SAMSelect, an automated method that uses the Segment Anything Model (SAM) to optimize spectral band combinations from Sentinel-2 data for improved marine debris visualization, thereby improving interpretation for environmental scientists. Second, it develops scene-specific models for coastal regions in Accra, Ghana, and Durban, South Africa, utilizing few-shot learning with an adapted PerSAM-F model. These models are designed to overcome limitations related to data scarcity and environmental variability.

SAMSelect identified new spectral indices that significantly outperformed existing methods, achieving mean intersection over union (mIoU) scores 8% to 29% higher. The best visualizations achieved segmentation scores of 45.8% and 42.0% for Accra and Durban, respectively. Despite these advancements, the scene-specific model faced challenges with prompt generation, resulting in lower accuracy compared to domain-specific models such as the Marine Debris Detector and MariNeXt.

Future work should focus on applying SAMSelect to larger datasets and validating the identified indices with in-situ targets. Additionally, exploring alternative deep feature extractors could improve prompt generation for PerSAM-F, further advancing marine debris monitoring and making it more accessible for environmental scientists.