Optical calibration, bathymetry, water column correction and bottom typing of shallow marine areas, using passive remote sensing imageries
Busy? 4SM in 10 lines
from raw data ..........>>>4SM flow chart.........>>> to final bottom typing

Same day operation
using only OpenEV, 4SM and a few other opensource executables


4SM modeling performance
 


 
1 - NO NEED for field data, nor for atmospheric correction
2 - this is demonstrated in this website, using a variety of hyper/multi spectral data
 
Requirements are
1 - homogeneous water body and atmosphere
2 - some coverage of optically deep water
3 - some coverage of dry land
 
Problems are
1 - the precision on estimated depth is found wanting, because the noise-equivalent change in radiance  of accessible data is too high for shallow water column correction work 
2 - radiance data should be preprocessed by the provider at level 1 in order to improve S/N ratio
3 - exponential decay: the deeper/darker the bottom, the poorer the performances
 
So
I keep digging
until suitable data
become available
 
   Your comments are invited                                                              See you on 4SM blog   
Where is the competition?

4SM procedure 2017

see 4SM demonstration
see also HYPHOON




Busy? 4SM in 10 lines
FLOWCHART
Procedure for operating 4SM.7.00
 
this file is displayed each time you run -MakePIX
Raw image data=============================
  •    may be formated in a variety of formats
  •    4SM uses PCIDSK format for its working database.pix
  •    TIF    format is OK, but must first be exported to PCIDSK
  •    OpenEV uses GDAL for handling very many image formats
Seatruth data==================================
  •    DTM: best is a gridded digital terrain model overlay, co-registered
  •    XYZ: depth sounding profiles as ASCII text file are also OK
Georeferencing==================================
  •    UTM please
Metadata==================================
  •    for OLI data, textfile L8..._MTL.txt must be available
  •    for WV2 data, textfile ..........IMD must be available
I use OpenEV to export raw data to PCIDSK image format
  •    specify a subwindow   if required
  •    resample the PAN band as required
 
Format the command line script as required
  •    use an existing command line
  •    update/modify as required
  •    pay special attention to georeferencing details and byte_swap
Run -MakePIX===================================
  •    to write database header
  •    to write database.pix structure, FILE interleaving
  •    and stop
Run -Import...=================================
  •    to import raw 16U data into database
  •    and stop
Run -Prepare_Scale==============================
  •    to write 8 bits scaled from 16 bits raw data
  •    and stop
Display 8 bits scaled image in OpenEV
  •    beware of byte_swap
  •    investigate using various enhancement schemes
  •    get a feeling of the image radiometric quality
  •    get a feeling of illumination conditions
  •    get a feeling of atmospheric conditions
  •    get a feeling of homogeneity of water bodies
  •    look for negative bottom areas
  •    which areas of this image are worthy of shallow water work?
Prepare shapefiles: Lsw.shp and dLsw.shp============
  • Lsw should be as low as possible: deep water radiance
  • cloud shadows or the area with lowest radiance in the NIR/SWIR band
  • dLsw.shp to locate deep water radiance at an optimized location
     
Prepare shapefiles: glint.shp====================
  •    must absolutely sample optically deep areas in the blue/green range
  •    to the extent possible, glint.shp should avoid clouds and whitecaps
  •    polygons and PolyLines may be mixed
Prepare shapefiles: LsM.shp and mSOIL.shp=======
  •    LsM.shp should sample bright genuine dryland
  •    LsM is one point to represent maximum brightness of shallow bottoms
  •    either with Points or Polygons/PolyLines.   Beware of wetlands
Prepare shapefiles: mSOIL.shp====================
  •    mSOIL.shp may include less bright dryland.  Beware of wetlands
  •    mSOIL shall give a sense of the Soil Line through a scatter of select pixels
  •    either with Points or Polygons/PolyLines.   Beware of wetlands
Prepare  mask_1.shp shapefile==================
  •    these unsafe marine areas shall be skipped while selecting calibration data
  •    like polluted areas, wave breakers, marinas, ships, alien floating things, etc
Prepare    land.shp shapefile==================
  •    these areas shall be forced to land
Prepare    vegetation.shp shapefile==================
  •    this shall help estimate water volume reflectance in the Blue range
Prepare shallow.shp shapefile==================
  •    these areas shall be forced to marine
Run AutoCalibrator: does the following:========
  •    reads  raw radiances from 16 bits raw data channels in database.pix
  •    scales raw radiances from 16 bits range to 8 bits range
  •    scans the image for nodata pixels
  •    extracts and format the glint_regressions.gli textfile
  •    formats glint_regressions.sh script
  •    Lsw: computes provisional Lsw deep water radiance
  •    Lw : computes provisional Lw water volume reflectance
  •    La : computes provisional La path radiance
  •    prepares the special effect mask
  •    BPL and DPL: extracts the calibration data for BPL and DPL
  •    BDH:         extracts the calibration data for bidimensional histograms
  •    runs the AutoCalibrator
  •    displays the provisional AutoCalibration diagram
  •    formats and displays the 4SM image-specific command line
  •    and stops
  •    this provisional calibration diagram may now be inspected
Are you satisfied with the mask mSE=====
  •    if not, see what needs to be modified, and start again
Run glint_regressions.sh=======================
  •    this displays the various glint regressions, pair wise
  •    investigate the glint regressions
  •    if not satisfied, modify the glint.shp shapefile and start again
Display the glint text file====================
  •    investigate the glint_regressions_text_file.gli
Run ABProfiler=================================
  •    in order to investigate deglinting performance, and Lsw and Lw
  •    -ProfileAB/LVW/profile_red/chAB_6_7
  •    see whether dLsw should be specified to adjust Lsw
  •    adjust as required, until satisfied
  •    ensure Lw is consistent with the physics of underwater  light field
  •    ensure La is consistent with the physics of atmospheric light field
Deglint the image==============================
  •    -Deglint/vRbad/q/GlintM20.0
  •    and adjust as required until satisfied
Run Extractor=================
  •    if mot satisfied with provisional calibration dataset
  •    specify your conditions for BPL: deglinted or not
  •    specify your conditions for BPL: mask mSE condition
  •    specify your conditions for BPL: NIRmax threshold
  •    specify your conditions for BDH: deglinted or not
  •    specify your conditions for BDH: whole image or marine only
  •    specify your conditions for extraction: whole image or select window
  •     -Extract/v/rawBDH/fullBDH/NIRband7/NIRmax5/mBPL2/mask_3
Run Calibrator=================
  •    then only, engage in the calibration process
  •     -Calibrate/V/BDSNpzg/plot_BPL/BC_6_5_3_2/BDh_12_13_34
  •    establish/adjust LsM
  •    establish Ki/Kj for all pairs, as precisely as possible
  •    estimate Lw for all bands: very sensitive for dark botoms
  •    consolidate your choices for LsM, Lsw and Lw
Run Modeler
  •    then only you may start modeling
  •    deglinting range may be tuned
  •    smoothing parameters may be tuned
  •    water column correction may be specified TOA or BOA
  •    Bmin is specified in order to alleviate very dark bottom artefacts
  •    mask_deep may be used to mask out deep water areas
  •    and various other operational tricks
Run ABProfiler to investigate your results
  •     -ProfileAB/ZrvwB/profile_blue/dTM/chAB_1_2_3
  •    adjust command line as required
  •    until satisfied
Start again
  •    until satisfied
Run ZZRegressor for sea truth evaluation
 

     
Et voila!
  •    you are now ready for bottom typing
When finished: run the -Archiver
It takes me ~5 hours
to process a new Landsat 8 OLIP scene,
from extracting the image downloaded from USGS,
to running seatruth tests and reporting to the web



more details below
WARNING: 
Due to progress achieved since january 2009,
many aspects of the HELP and HOWTO services are outdated
 
  • Use OpenEV to prepare your raw image

  • Get hold of ancillary infos:
    • nbRow, nbLin, nb of BGRNir channels
    • sensorType and sensorMode
    • georeferencing infos
    • pixel size
    • Export your raw image from your petty format to PCIDSK format
    • imaging conditions
    • wavelengths...
  • Use 4SM to create a PCIDSK formated database

 
  • If 16 bits data, use OpenEV to sample some bright bareland
    • this will be used by 4SM to scale the data from 16 bits to 8 bits (floating)
 
  • For 16 bits data, a -SCL... commndline argument must be specified
  • Specify the -Import... commandline argument and run your script

  • Do some optional digitizing for house keeping

    • NoData.shp
    • baddata.shp
    • land.shp
    • mSOIL.shp for bright bare soil
    • shallow.shp
    • badwaters.shp
    • clouds.shp
    • waves.shp
    • deep.shp for optically deep water
  • Do some compulsory digitizing of training sites

    • Lsw.shp           for deep water radiance

    • LsM.shp           for bright bare soil

    • glint.shp           for glint/haze regression

    • vegetation.shp  for healthy vegtation

  • Add a -AutoCAL... argumentand run your script

    • this shall create the database structure with all required channel
    • then run a few statistics
    • then work out the glint regressions and write a database.gli textfile, and display the regressions
    • then automtically write the special effect mask mSE
    • then extract the deglinted calibration data textfile
    • then attempt to run a calibration
    • then offer a fully developped commandline script

If all gos well, 4SM offers
a  fully developped commandline script

 

 which now needs to be verified
before use

Add a -Deglint... commandline argument
and run your script: this shall

  • work out a glint regression
  • then write a deglinted image
  • disable   -deglint... commandline argument  
  • Add an -Extract... commandline argument
  • and run your script
  • this shal  write a datbase.cal calibration textfile
  • Use 4SM to operate the simplified shallow water radiative transfer optical model on all shallow water pixels

 
  • Use OpenEV to digitize training sites for bottom typing:

    • digitize small and homogeneous training sites for retreiving bottom type signature

  • Use 4SM to perform bottom typing
  • add a -Classify... commandline argument
  • run 4SM to collect bottyom type signatures
  • run 4SM to clasify the shallow water areas
  •  
  • this writes a U8 channel to map the bottom typing
  •  
  • then if desired run 4SM to apply a modal filter
  •  
  • this writes a U8 channel to map a cleaner bottom classification
  • Use 4SM to investigate Sea Truth

  • Add a -RegressZZ... commandline argument

  • and run your script: this shall

    • write a datbase_RegressZZ.txt text file

    • write andd then run a datbase_RegressZZ.sh script
    • and finally display a datbase_RegressZZ.eps
  • Use 4SM to plot sea truth profiles

  • Use 4SM to plot profiles