to make special effect mask mSE automatically
Automatic Mode of 4SM: 
from raw data to first calibration and modeling

back to Help

1 - NO NEED for field data, nor for atmospheric correction
2 - this is demonstrated in this website, using a variety of hyper/multi spectral data
Requirements are
1 - homogeneous water body and atmosphere
2 - some coverage of optically deep water
3 - some coverage of dry land
Problems are
1 - the precision on estimated depth is found wanting, because the noise-equivalent change in radiance  of accessible data is too high for shallow water column correction work 
2 - radiance data should be preprocessed by the provider at level 1 in order to improve S/N ratio
3 - exponential decay: the deeper/darker the bottom, the poorer the performances
I keep digging
until suitable data
become available

About segmentation of the image
Automatic Mode of 4SM
4SM Defaults
The -AutoCAL sequence of events 
HowTo MakeLand Automatic
How To Make Mask mSE
4SM FlowChart
How to Develop A GUI for 4SM

About the segmentation of the image


-AutoCAL shall use all default values

-AutoCAL/.......... shall use all default values except those specified here (in random order)
AutoCAL values currently available

/C(c)clouds MaskClouds   default=0     to mask clouds
/G(g)lint to deglint 
/G(g)lintM_25 GlintM=25.........................deglints in the range Lsnir=Lswnir to Lsnir=25
/GlintM_50........................deglints in the range Lsnir=Lswnir to Lsnir=50
/GlintM_255......................deglints in the range Lsnir=Lswnir to Lsnir=255
/GlintM_0..........................prevents any deglinting!!!!!                    
/P(p)rint_bi_bj         printbi    default printbi=0   to print details of KIKJ regression for the pair of bands bi vs bj
/M(m)apBPL mapBPL   to map BPL pixels in database.pix             default mapBPL=0

to collect BPL_data for mSE range  mBPL_to_mBPLmax      
                  default  mBPL=2 and mBPLmax=20
/mBPL_1   to collect BPL data from mSE=1  up to mSE=20
/mBPL_3   to collect BPL data from mSE=3  up to mSE=20
/MBPL_3   to collect BPL data from mSE=3  up to mSE=3
/M(m)akeNewMask default=M      makeNewMask would keep existing mask SE unchanged
/L(l)and_5_1.7_7 default    offsetLand=5 slopeLand=1.18 chLand=bands  
specifies the segmentation of land/non-land pixels
/G(g)T25 WaterMax    specifies the segmentation of land/non-land pixels:
pixels with Ls[bands]>=WaterMax shall be mapped to Land
/N(n)oSmooth default=n      NoSmooth would prevent any smoothing
/NIRmax_25 default dLsNirBPL=5  
collects BPL pixels in the range Lsnir=Lswnir to Lsnir=Lswnir+dLsNirBPL
/NIRmax_25......collects BPL pixels in the range Lsnir=Lswnir to Lsnir=Lswnir+25
/NIRmax_255....collects BPL pixels in the range Lsnir=Lswnir to Lsnir=255
G(g)etGLINT getGLINT  default getGLINT=1   shall extract glint regressions   Deglint=Make_GlintDatabase=getGLINT=1
/L(l)astGlint_3 default is NIR band number
/LastGlint_3........excludes the NIR band in a BGRN image
/L(l)sM_248_7569_5 XLsM XLsM WinRadLsM         defaults XLsM=YLsM=WinRadLsM=0
/L(l)sw_248_7569_5 XLsw XLsw WinRadLsw         defaults XLsw=YLsw=WinRadLsw=0
/E(e)xt_1_nbRow_1_nbLin default Cext_1_nbRow_1_nbLin extraction window for calibration data


Image segmentation
  • 4SM has an AutoCAL module to segment
    • all land areas into mSE=248
    • all marine areas to mSE=2
  • Specific marine areas specified by relevant shapefiles recoded to the mask_NewVal value,
    • like to map at mSE=3 under mask_3.shp
  • How does that work: below is an illustration in a bi-dimensional histogram of Lsred vs Lsnir
    • Land    below (or right of) a certain straight line
    • Marine above (or left  of) a certain straight line

Natural radiances, Green vs Red

Natural radiances, Red vs NIR

View mSE mask in channel_4 as created by the AutoCAL process

  • The Automatic 4SM process is an exciting development
  • which followed time intensive processing of a series of 12 Landsat ETM images in Tanzania, and many more, early in 2009
  • after 5 years, it now seems to work to satisfaction

The AutoCAL feature paves the way for the preparation of a graphic user interface
Several aspects had to be addressed


  • Specify default values for most of the command line arguments
  • Automatic creation of desired database structure
  • Automatic import of raw data channels from original image
  • Automatic preparation of special effect mask mSE
  • Automatic specification of glint regression, if applicable
  • Automatic extraction of calibration data
  • Automatic calibration, (and, if allowed, deglinting and modeling)
  • Wavelength specification
  • Preparation of a detailed command line, complete with a choice of supplementary tools

The practioner only needs to prepare a few shapefiles
Lsw.shp.................for an area where to measure provisional spectral deep water value
LsM.shp.................for an area where to measure provisional spectral LsM values
glint.shp.................for an area where to estimate glint correlations (if necessary)
vegetation.shp........for an area of healthy vegetation
mask_1.shp, mask_2.shp, required

Automatic Mode of 4SM

the next step after importing the raw data:
from raw data to first calibration (and deglinting/modeling)

nice -20  ./4SM.8.02 -Process/13_NOV_2017  -Origin/FUGRO-NPA @@
-DB/tmnov/31_3s_0s_0s/4_1/1265_1560/208.350_2398.770 @@
-Mis/Bahamas/Caicos/Landsat/TM/FUGRO-NPA/bOA/UTM_19_008/0.030_0.030/22_NOV_1990 @@
-MakePIX/Import @@
-Import/v/dTM/data*TM_009_045_1990-11-22/dbnc_6_0S_0s_0s/R4341_L4462/Origin_129.510_2448.060_0_0/chIn1,4/chOut1,4/import @@
R2KKmin is specified here at 0.95
In the AutoCalibration process:
KIKJRegression R2
  • First an attempt is made to derive the ratio Kblue/Kgreen of attenuation coefficients 
    • automatically through a linear regression of BPL pixels for the Blue/Green pair of bands
    • this regression derives a R2 coefficient of determination
    • this also process includes the optimization the -LsM... argument, in order to achieve a tight fit of the BPL model
  • if R2>=R2KKmin: success
  • else: fall back to default below
  • If the coefficient of determination R2 of this regression is less than the value specified r2KKmin
    • then a default value of KIKJ=0.5 is enforced by 4SM in the AutoCalibration process
  • A value of .../R2_1.00/... would cause the regression to be always rejected,
    • and the ratio Kblue/Kgreen=0.50 to be enforced

The AutoCAL process won't work unless

  • LsM.shp shapefile: at least a few non-saturated pixels of bright land may be sampled using the LsM.shp shapefile
  • Lsw.shp shapefile: an area of optically deep water pixels may be sampled using a Lsw.shp shapefile
    • optically deep is to mean : optically deep in all wavebands
  • glint.shp shapefile: if conspicuous clouds or glint is observed over optically deep water in the image, an area of glinted pixels may be sampled using a glint.shp shapefile
  • bright and homogeneous bottoms are expected to exist at various depths
  • suitable default values are available and applicable

4SM defaults
  • Apart from image-specific metadata which need to be specified in -DB..., -Mis... and -Import... arguments,
    • many commandline argument variables are given a default value in the 4SM code
    • any such default value may be overridden in the command line itself
  • Upon the first execution of 4SM in the current directory,
    • 4SM writes a 4sm.def editable textfile to contain such default values
    • default values are written or read at the end of decoding the -Mis... argument
    • default values are then eventually overridden upon reading the rest of the commandline
The -AutoCAL sequence of events
- to produce a special effect mask mSE in channel bands+1
- and perform a provisional optical calibration
  • Erase existing mask mSE in channel bands+1, and write a new one
  • Recode
    • NoData........recode to 239 in mask mSE all pixels which read 0 in any of the raw data bands
    • baddata.......recode to 239 in mask mSE all pixels which are included in shapefile baddata.shp
    • cloud...........recode to 234 in mask mSE all pixels which are included in shapefile cloud.shp
    • wave...........recode to 233 in mask mSE all pixels which are included in shapefile wave.shp
  • Lsw and LsM
    • Lsw.............if a -Lsw... argument is not provided in the commandline,
    • 4SM reads shapefile Lsw.shp to compute and specify provisional spectral Lsw
  • LsM.............if a -LsM... argument is not provided in the commandline,
    • 4SM reads shapefile LsM.shp to compute and specify provisional spectral LsM
    • Glint regression
      • if a shapefile named glint.shp is available, 4SM reads it
      • to specify the glint correlations
      • to specify the -Lsw... argument
  • Make Land, Marine, Wave  and Cloud....
    • A provisional Soil Line is defined using provisional LsM and Lsw values in the Red and Nir bands
    • The image is then segmented as explained elsewhere
      • recode land pixels       to   248
      • recode shallow pixels  to   2
      • recode cloud pixels     to   234 (if a cloud.shp shapefile is not available)
      • recode wave breakers to   233 (if a wave.shp shapefile is available)
  • Recoder
    • Clean the land area
      • land.shp: recode to 248 pixels which are included in shapefile land.shp
    • Clean the marine area
      • shallow.shp............recode to 2 all pixels which are included in shapefile shallow.shp
      • badwaters.shp..................recode to 249 all pixels which are included in shapefile badwaters.shp
      • deep.shp..........................recode to 249 all pixels which are included in shapefile deep.shp
      • mask_1.shp......................recode to 1 all pixels which are included in shapefile mask_1.shp
      • mask_3.shp......................recode to 3 all pixels which are included in shapefile mask_3.shp
      • mask_4.shp......................recode to 4 all pixels which are included in shapefile mask_4.shp
      • mask_5.shp......................recode to 5 all pixels which are included in shapefile mask_5.shp
      • etc
      • mask_20.shp....................recode to 20 all pixels which are included in shapefile mask_20.shp
      • negative.shp.....................recode to 244 areas where negative bottom contrast is allowed
  • Extractor:
    • if a suitable calibration textfile is not available, 4SM runs the extraction
  • KIKJ Regressor:
    • estimates the ratio Kblue/Kgreen by regression on the linearized BPL pixels
      • and use the linearized BPL, the optical calibration, and the provisional Soil Line to specify final LsM
    • if the coefficient of determination is too low (R2<R2KKmin),
      • KIKJ=Kblue/Kgreen=0.50 in enforced
  • Calibrator
    • All parameters are now specified
    • 4SM proceeds to compute and display the calibration plot
  • autoCAL.txt: 4SM writes and displays a textfile named autoCAL.txt
    • which contains, among other things, a complete detailed commandline
    • which may in turn be used for the practioner to take full control of all aspects of the 4SM process