ncsa/criticalmaas-pipeline repository overview

To get started with this pipeline you will need to clone the repository and and install requirements.txt⁠. We recommend using python venv here to keep the working environment clean.

# If you are on hydro you will need to load the python and cuda module.
# module load python/3.9.13 cuda/11.7.0 

git clone [email protected]:criticalmaas/pipeline.git
cd pipeline_inference
python3 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt

This repository also makes use of submodules which will need to be initialized.

git submodule init
git submodule update

To perform inference with our pipeline, only one data input is required and that is the map that you want to perform inference on. There are other data inputs we can use to speed up and perform optional steps with. Each of these optional inputs needs to be structured so that the name is consistent with the input map. E.g. if you have CA_Sage.tif the legend will need be named CA_Sage.json

This is visualization of what that structure looks like.

data
├── Map_1.tif
├── Map_2.tif
├── ...
└── Map_N.tif

legends # Optional
├── Map_1.json
├── Map_2.json
├── ...
└── Map_N.json

layouts # Optional
├── Map_1.json
├── Map_2.json
├── ...
└── Map_N.json

validation # Optional
├── Map_1_lgd_1_poly.tif
├── Map_1_lgd_2_poly.tif
├── ...
├── Map_1_lgd_N_poly.tif
├── ...
├── Map_N_lgd_1_poly.tif
├── Map_N_lgd_2_poly.tif
├── ...
└── MapN_lgdN_poly.tif

It is also important to note that if you specify --legends and there is no corresponding legend for a map file, that is completely fine. Pipeline will just fallback to generating a legend for that specfic map. The same is true for --layouts and --validation.

To perform inference with one of our models, we will need to run pipeline.py. Pipeline.py has 3 core required arguments to run:

• --model : The model to use for inference.
• --data : Directory containing data to perform inference on.
• --output : Directory to save the output data of the pipeline to.

The list of available models can be found below⁠ with the release-tag being what you want to use for the argument.

Note* You must have a GPU available to run pipeline.py

# Example call to pipeline.py
python pipeline.py --model "golden_muscat" --data mydata/images/ --output mydata/output/

Running this will have "golden-muscat" run inference on every .tif file in the directory specifed by --data. The output rasters of this inference will then be saved as .tifs to the directory specifed by --output along with a geopackage file for each map. The geopackage file contains vector data for each legend item in the map. Output is saved as the pipeline runs so even if the pipeline were to crash in the middle of running, all maps that ran before the crash will have been saved.

By default the pipeline will save logging information to logs/Latest.log this can be useful if you have any problems or want to see a detailed view of what the pipeline is doing. You can also change the log file location with --log.

For the further documentation on all the pipeline options see below⁠.

For running the pipeline on hydro there are two options. You can manually run the pipeline with an interactive srun session or we can submit an automatic job using sbatch. You can learn how to manually run with srun in the hydro docs⁠. You will need to make sure to srun with --partition=a100 flag as these are the only nodes with GPUs on hydro.

For running with sbatch we have two scripts submit.sh and start_pipeline.sh. When we run submit.sh that script will automatically start start_pipeline.sh on an a100 node.

First, we will want to set the parameters for pipeline.py in start_pipeline.sh. Then, once we are ready to run, all we have to do is call

sbatch submit.sh

and that will start the job. We can view our pipelines progess by looking at logs/job_%yourjobid%.log. The slurm logs can also be found at logs/slurm/%yourjobid%.e if you have any errors.

*Hint tail -f logs/job_%yourjobid%.log can be very useful for viewing these logs. You can also use nvitop when on the node that is running the job to view GPU statistics in real-time.

*Please note that our job script assumes that you are using venv to setup your environment. If you are using another python environment manager, E.g. Conda or virtualenvwrapper, you will need to adapt the start_pipeline.sh script to your setup.

Pipeline can produce quite a few output files so it can be important to understand what each is. The key arguments here ar --output and --feedback as they control what files the pipeline will output and where. --output controls where the results of inference will get saved; A Raster tif for each legend and a geopackage for each map containing the vectorized legend data for every legend. --feedback controls whether the pipeline will output files that are intended for debugging. When feedback is enabled, the pipeline will save any legend data that was generated by the pipeline, save the image of the legend label, create a visualization image for each legend analyzed in the validation step, and save the validation score csv for each individual map. This results in the following output structure.

output
├── full_dataset_scores.csv # If validation was enabled and feedback was not
├── Map1_lgd1.tif
├── ...
├── Map1_lgdN.tif
├── Map1.gpkg
├── ...
├── MapN_lgd1.tif
├── ...
├── MapN_lgdN.tif
└── MapN.gpkg

feedback
├── full_dataset_scores.csv # If validation was enabled
├── Map1
│   ├── Map1.json # If a map legend was generated by pipeline
│   ├── Map1_Scores.csv   # If validation was enabled
│   ├── lgd_map1_lgd1.tif # Legend label image
│   ├── ...               # ''
│   ├── lgd_map1_lgdN.tif # ''
│   ├── val_map1_lgd1.tif # Legend validation image
│   ├── ...               # ''
│   └── val_map1_lgdN.tif # ''
├── ...
└── MapN
    ├── MapN.json # If a map legend was generated by pipeline
    ├── MapN_Scores.csv   # If validation was enabled
    ├── lgd_mapN_lgd1.tif # Legend label image
    ├── ...               # ''
    ├── lgd_mapN_lgdN.tif # ''
    ├── val_mapN_lgd1.tif # Legend validation image
    ├── ...               # ''
    └── val_mapN_lgdN.tif # ''

Note that if feedback is not turned on and validation is, pipeline will still save all the scores in the output directory to #%data%_results.csv

Git Repository : https://github.com/Dongjiahua/DARPA_torch⁠
Lead Developer : Dong Jiahua⁠
Description :

Release Tags :

• flat_iceberg

Git Repository : https://git.ncsa.illinois.edu/nj7/darpa_proj⁠
Lead Developer : Nattapon Jaroenchai (Nathan)⁠
Description : Attention U-net model

Release Tags :

• primordial_positron_3
• primordial_positron_4

Git Repository : https://git.ncsa.illinois.edu/nj7/darpa_proj⁠
Lead Developer : Nattapon Jaroenchai⁠
Description : Attention U-net model

Release Tags :

• customer_backpack

Git Repository : https://github.com/xiyuez2/Darpa_Unet_Release⁠
Lead Developer : xiyuez2⁠
Description : U-net model

Release Tags :

• golden_muscat

Git Repository : https://github.com/xiyuez2/Darpa_Unet_Release⁠
Lead Developer : xiyuez2⁠
Description : U-net model

Release Tags :

• rigid_wasabi

Git Repository : https://github.com/xiyuez2/Darpa_Unet_Release⁠
Lead Developer : ziruiwang409⁠
Description : Superpixel U-net model

Release Tags :

• blaring_foundry