Fine-tuning Segment Anything Model (SAM) for flood inundation mapping

  • Data Science
  • Deep Learning and pixel classification

Introduction

Flooding is one of the most frequent and costly forms of natural disasters. They often strike without warning and can occur when large volumes of water fall in a short time, causing flash floods. Flood mapping is typically performed using aerial observations and ground surveys. However, when flooding is widespread, these methods become prohibitively expensive and time consuming. Furthermore, aerial observation and optical imagery can often prove difficult, if not impossible, due to obstructive weather conditions. During flooding conditions, clouds can prevent the use of optical satellite imagery for visualization and analysis. In these instances, synthetic-aperture radar (SAR) allows us to penetrate through clouds and hazy atmospheric conditions to continuously observe and map flooding.

North India faced severe rainfall in the first two weeks of July 2023 due to the interaction between Western Disturbances (a low pressure system originating from the Mediterranean region) and Monsoon trough (a low pressure zone located in East west direction from Northwest Rajasthan towards the Bay of Bengal). From July 9th - 13th, a large volume of water was released from Hathnikund Barrage on the Yamuna river. This raised the water level downstream to the highest level since floods in 1978, causing flooding in Delhi and the adjoining areas.

In such situations, mapping of flood affected areas can help in quick and efficient disaster management and relief efforts. Here, we have used the SamLora deep learning model that finetunes the foundational pixel classification model by Meta's Segment Anything Model (SAM). SamLoRA allows the use of the underlying learnings of SAM and quickly finetunes on a smaller dataset to work for a particular use case.

Necessary imports

import os
from datetime import datetime
from pathlib import Path

from arcgis.gis import GIS
from arcgis.learn import prepare_data, SamLoRA
from arcgis.raster import Raster, convert_raster_to_feature
from arcgis.features.manage_data import overlay_layers
from arcgis.features.analysis import dissolve_boundaries

Connect to your GIS

from arcgis import GIS
gis = GIS('home')

Export training data

We have used the Sentinel-1 SAR imagery as the input raster for the training dataset. A composite of VV, VH & VV/VH band has been created, which works best for floods, according to the literature.

input_raster = gis.content.get("3b55a6f703234c278e76704e6ff50666")
input_raster
yamuna_flood_input_raster

Tiled Imagery Layer by api_data_owner
Last Modified: July 11, 2024
0 comments, 12 views

The feature layer contains a single class for water bodies. The feature layer will be used as the Input Feature Class in the Export Training Data For Deep Learning tool.

label_raster = gis.content.get("cb0e42c0ae714c5e90b09aeb2add2baa")
label_raster
yamuna flood_labels

Feature Layer Collection by api_data_owner
Last Modified: July 10, 2024
0 comments, 16 views

The polygon feature class will be used as Input Mask Polygons in the Export Training Data For Deep Learning tool to delineate the area where image chips will be created.

aoi = gis.content.get("8fcf0ba8de7947c3b696778a1fdc88ea")
aoi
yamuna_flood_aoi_mask
Area of Interest Mask for Yamuna floods labels
Feature Layer Collection by api_data_owner
Last Modified: July 11, 2024
0 comments, 12 views

The Export Training Data For Deep Learning tool is used to prepare training data for training a deep learning model. The tool is available in both ArcGIS Pro and ArcGIS Enterprise.

Next, we will utilize Jupyter Notebooks. Documentation on how to install and setup the necessary environment is available here.

Model training

Get training data

We have already exported the data, and it can be directly downloaded using the following steps:

training_data = gis.content.get('5dff2d39729c44949b3024c1f1ead403')
training_data
finetuning_sam_for_flood_inundation_mapping

Image Collection by api_data_owner
Last Modified: July 11, 2024
0 comments, 1 views
filepath = training_data.download(file_name=training_data.name)
import zipfile
with zipfile.ZipFile(filepath, 'r') as zip_ref:
    zip_ref.extractall(Path(filepath).parent)
data_path = Path(os.path.join(os.path.splitext(filepath)[0]))

Prepare data

We will specify the path to our training data and a few hyperparameters.

  • path: path of the folder/list of folders containing training data.
  • batch_size: Number of images your model will train on each step inside an epoch. Depends on the memory of your graphic card.
  • chip_size: The same as the tile size used while exporting the dataset to ensure the entire chip is used at once and not cropped.
data = prepare_data(data_path, batch_size=4, chip_size=400)

Visualize training data

To get a sense of what the training data looks like, the show_batch() method will randomly select a few training chips and visualize them.

data.show_batch(alpha=0.8)
<Figure size 1500x1500 with 9 Axes>

Load model architecture

The code below will create a SamLoRA model. It is based on the Segment Anything Model (SAM) by MetaAI. It uses a Low Rank Adaptation to finetune the model for a specific task. This model can be used for 'pixel classification'. By default, this model will be loaded with a vit_b backbone. vit_h or vit_l backbone can be chosen by providing the backbone parameter. More details about SamLoRA can be found here.

model = SamLoRA(data)

Train the model

Next, we will train the model for a few epochs.

model.fit(epochs=10, lr=0.001)
epochtrain_lossvalid_lossaccuracydicetime
00.7554350.6529480.7878040.28423300:21
10.7241420.6951300.3509400.38839500:10
20.7031780.3243600.8570220.72312500:10
30.5986240.2473130.9215130.80209800:10
40.5159510.2044780.9235920.82037100:10
50.4607030.2342490.9247200.79936600:10
60.4136910.2016800.9299740.82463300:10
70.3771300.1947760.9305210.83108700:10
80.3562350.1863280.9306450.83774500:10
90.3347900.1871670.9308610.83697800:10
model.plot_losses()
<Figure size 640x480 with 1 Axes>

Visualize results in validation set

It is a good practice to see the results of the model viz-a-viz ground truth. The code below picks random samples and shows us ground truth and model predictions, side by side. This enables us to preview the results of the model we trained. In the following code the alpha parameter sets the transparency of the results.

model.show_results(alpha=0.8)
<Figure size 800x1600 with 8 Axes>

Evaluate model performance

model.accuracy()
0.9306446313858032

arcgis.learn provides the mIOU() method that computes the mean Intersection Over Union (IOU) of the results provided by the model on the validation set for each class.

model.mIOU()
100.00% [2/2 00:00<00:00]
{'0': 0.9152871021073077, '1': 0.7232920825582878}

To calculate the precision, recall, and f1 score for each class, use the per_class_metrics function.

model.per_class_metrics()
NoData1
precision0.9493300.860626
recall0.9622980.819254
f10.9557700.839431

Save the model

We will save the model that we trained as a 'Deep Learning Package' ('.dlpk' format). A Deep Learning Package is the standard format used to deploy deep learning models on the ArcGIS platform.

We will use the save() method to save the trained model. By default, it will be saved to the 'models' sub-folder within the training data folder.

model.save('samlora_yamuna_flood_model', publish=True, overwrite=True)
Published DLPK Item Id: b25d811c5caa4f71b30cd82356630894
WindowsPath('C:/temp/data/samlora_yamuna_flood_model')

Model inferencing

Using ArcGIS Pro, we can use the trained model on a test image/area to classify flood inundated areas in the SAR satellite image.

After training the SamLoRA model and saving the weights for classifying images, we can use the Classify Pixels Using Deep Learning tool available in ArcGIS Pro and ArcGIS Enterprise for inferencing.

flood_model = gis.content.get('b25d811c5caa4f71b30cd82356630894')
flood_model
samlora_yamuna_flood_model

Deep Learning Package by api_data_owner
Last Modified: July 12, 2024
0 comments, 0 views
raster_for_inferencing = gis.content.get('5faf61fdf96845a28f105d473c2c61a3')
raster_for_inferencing
raster_for_inferecing_samlora

Tiled Imagery Layer by api_data_owner
Last Modified: July 15, 2024
0 comments, 4 views

Results visualization

The classified output raster is generated using ArcGIS Pro. The output raster is published on the portal for visualization.

sar_ras2 = gis.content.get('5faf61fdf96845a28f105d473c2c61a3')
inf_ras2 = gis.content.get('0237c0324a284463a03aecb4fddc417d')
from arcgis.raster import colormap
inf_cmap2 = colormap(inf_ras2.layers[0], colormap=[[1, 0, 204, 204]])

Create map widgets

Three map widgets are created showing flood affected areas in Delhi and the adjoining region.

map1 = gis.map('Mathura, India', 8)
map1.basemap='satellite'
map2 = gis.map('Mathura, India', 8)
map2.add_layer(sar_ras2)
map3 = gis.map()
map3.add_layer(sar_ras2)
map3.add_layer(inf_cmap2)

Set the map layout

from ipywidgets import HBox, VBox, Label, Layout
map1.sync_navigation(map2)
map2.sync_navigation(map3)

Hbox and Vbox were used to set the layout of map widgets.

hbox_layout = Layout()
hbox_layout.justify_content = 'space-around'

hb1,hb2=HBox([Label('True Colour Image'),Label('Sentinel-1 Imagery'),Label('Predictions'),]),\
                HBox([Label('True Colour Image'),Label('Sentinel-1 Imagery'),Label('Predictions')])
hb1.layout,hb2.layout=hbox_layout,hbox_layout

Flood inundation mapping

The resulting predictions are provided as a map for better visualization. The results show the spatial distribution of flood water in the Delhi region during the 2023 floods. Sentinel-1 imagery of July 2023 are used for the analysis.

VBox([hb1,HBox([map1,map2,map3])])

Three map widgets were created. The left widget displays natural color high resolution satellite imagery prior to flooding, the middle widget displays the Sentinel-1 imagery during the flood event, and the right map widget displays the predictions of the trained SamLoRA model. In the maps, Delhi and other cities of the National Capital Region of India can be seen on both sides of the Yamuna river. The model is able to identify all the flooded regions. The True Color Imagery can be used for visual interpretation for model accuracy.

Conclusion

In this notebook, we have demonstrated how to use SamLoRA to finetune SAM model with ArcGIS API for Python and extract flood area with a small dataset and short training. We covered how Sentinel-1 SAR data can be used for flood inundation mapping and monitoring. This process involved steps to prepare the input data, train a pixel-based classification model, visualize the results, generate accuracy metrics, and predict on a test raster.

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.