Optical calibration, bathymetry, water column correction and bottom typing of shallow marine areas, using passive remote sensing imageries


 
1 - NO NEED for field data, nor for atmospheric correction
2 - this is demonstrated in this website, using a variety of hyper/multi spectral data
 
Requirements are
1 - homogeneous water body and atmosphere
2 - some coverage of optically deep water
3 - some coverage of dry land
 
Problems are
1 - the precision on estimated depth is found wanting, because the noise-equivalent change in radiance  of accessible data is too high for shallow water column correction work 
2 - radiance data should be preprocessed by the provider at level 1 in order to improve S/N ratio
3 - exponential decay: the deeper/darker the bottom, the poorer the performances
 
So
I keep digging
until suitable data
become available
 


Your comments are invited                            See you on 4SM blog

intel_x86-64    core I7    64 bits           bash   


 

 
What is OpenEV?
I entirely depend on FWTool/OpenEV. 
FWTools is a 12 years old 32 bits executable.
Over the last few years it has become more and more difficult to run OpenEV,
because of unmet dependencies.
As of june 2017, it has become impossible.
The OpenEV problem as of june 2017
This is the end unless
a solution is found using 
Geomatica or ENVI or GRASS 
The 4SM problem as of june 2017

Install your 4SM workstation:
a tutorial by Fabio Favoretto, 2017

 
  • My workstation: from 1999 to 2014: I used Toshiba Satellite laptops.
    • then I moved to MSI GS70 64 bits Intel x86_64 portable PC: Intel Core I7 2.13 GHz, 16 GB RAM, 17" screen 1920*1080 (16:9, full HD)
  • My operating system: Ubuntu Linux_64 bits/Gnome/bash. Ubuntu Linux is "for human beings", it is free, it installs like a dream in "dual boot" on a Windows 8 machine.
  • 4SM code is proprietary ; it is written and compiled C.
  • 4SM system calls: 4SM calls some executables or Linux commands
    • GMT: gmtset , psbasemap, pstext , psxy, and psscale: 4SM writes some scripts which call a few modules of GMT from U. of Hawaii (please note that GMT may be installed using Ubuntu's Synaptic Package Manager). These executables are included in the 4sm.calls.zip zipfile
    • shp2text: using this executable, 4SM converts shapefiles to textfile SHP.txt, for use with the
    • wget, gv and cpuid: install them using Ubuntu Software Center
      • gv: default PostScript viewer in 4SM. If you use another viewer, you have to place a -Calls/epsviewer_yourviewer argument in your commandline
    • firefox: default browser in 4SM. If you use another browser, you have to place a 
    • gedit: default text editor in 4SM. If you use another texteditor, you have to place a 
    • chmod, rm, mkdir, dir, cp, zip, bzip2, whoami come with the Linux distribution
    • cpuid: if you want to become a superuser of 4SM
    • wget: if you want to download/update a mirror copy of 4SM website on your computer/network
  • Compiler: 4SM is compiled under Linux/Gnome/bash: " gcc -lm -w -m32 -g 4SM.c -lm -o 4SM"
  • Image format: image.pix from PCIgeomatica, band-interleaved, preferably file-interleaved.
  • Image preparation and display:
    • I formerly used ImageWorks and EASI/PACE from PCI .
    • Since 2005, I only use OpenEV FWTools for import/export, image display and vector activities.
  • GUI: I would like to develop a GUI with Qt Designer or GLADE in order to manage the 4SM command line.


 

Prepare your work directory

 
 
  • This 1440*900 screen display shows
    • ?the tmnov tutorial command line script,
    • the OPENEV display of the tmnov.pix image created through 4SM's_MakePIX, complete with georeferencing
    • and some of my XTERM terminal where tmnov.pix image is being processed.
  • This demonstrates that the OPENEV solution is workable for all, and FREE!
    • Accelerated X graphics is advisable though, as OPENEV tends to hang: see "OpenEV Performance Tuning: Accelerated OpenGL Support" in OPENEV's Help
 

Linux flavor

 

For workstation installation,
refer to workstation installation
 
 

Screen resolution:
FULL HD 1920*1080 (16:9) please

 
  • Mine is 17" 1920 * 1080: best is that you have similar FULL HD resolution
  • I tend to use it liberally
  • shrinking my 4SM displays so that they fit a smaller screen resolution might take a lot of time.
  • You can reduce the font size of your GNOME terminal:
  • mine is Monospace size 10
  • mine is Monospace size 10
 
Path and aliases
some of my favorite /etc/bash.bashrc (I don't bother with .bashrc in my home directory)

PATH="$PATH:~/software/FWTools-1.1.0/bin_safe"
export PATH MANPATH

alias aliasetc="gedit /etc/bash.bashrc &"
alias sourcetc="source /etc/bash.bashrc "

alias te="gedit "
alias mv="mv -i "
alias cp="cp -i "
alias rm=" rm -i "
alias rmforce="rm -f "
alias getridof="rm -f -r "
alias cpr="bzip2 "
alias ucp="bunzip2 "
alias dir="ls -l -k --color "
alias dirw="ls -l -k --color; pwd "
alias dira="ls -l -a -k --color "
alias dirs="ls -a -k --color "
alias cmall="chmod a-x *; chmod a+x *sh*; chmod a+x 4SM*; chmod a+x .*; chmod a+rw *; dira"
alias df="df -m "
alias showdu="du -k | sort -nr > du.txt; te du.txt &"
alias ym="cd /home/ymorel; dira ; df; pwd "
alias home="cd /home/ymorel; dira ; df; pwd "
alias openev="openev -h &"



 
The OpenEV problem as of 2017
I have used Ubuntu and OpenEV/FWTools binaries since 2005: most convenient and sufficient. 
FWTools is 32 bits and orphaned since 2005.
Everything these days has to be 64 bits: I am in trouble, the problem has become too much.
Although there might be a way to revive it: see below.
Gamers have that sort of problem (Steam), they know ways around!!!
I tried using 32 bits Ubuntu 16.0.4 LTS ubuntu-16.04.2-desktop-i386.iso: see below.
This is beyond me, and it means the end of my 4SM project.
I need help!!

Never tried using Windows version of
OpenEV/FWTools through WINE
maybe this is the solution for now!!!!!!!!

 
Now I need to use GRASS GIS behemoth, see if it can take over from FWTools
ymorel@u16-i386-GS70:/usr/lib/i386-linux-gnu/dri$ LIBGL_DEBUG=verbose openev
[1] 10877
ymorel@u16-i386-GS70:/usr/lib/i386-linux-gnu/dri$ Setup for user installed hardware acceleration
 
Gtk-WARNING **: Failed to load module "libgail.so": libgail.so: cannot open shared object file: No such file or directory
 
Gtk-WARNING **: Failed to load module "libatk-bridge.so": libatk-bridge.so: cannot open shared object file: No such file or directory
 
Gtk-WARNING **: Failed to load module "libunity-gtk-module.so": libunity-gtk-module.so: cannot open shared object file: No such file or directory
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/gvrastertools.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/mil_symbols.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/isodata.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/calculator.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/Tool_autopan.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/compose.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/wmstool.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/imgproctemplate.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/Tool_Export.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/Tool_ShapesGrid.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/open_subarea.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/rendertest.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/fft.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/open_raw.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/reproject.py
Loading tools from /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/bin/../tools/Tool_DriverList.py
libGL: Can't open configuration file /home/ymorel/.drirc: No such file or directory.
libGL: pci id for fd 3: 8086:0416, driver i965
libGL: OpenDriver: trying /usr/lib/i386-linux-gnu/dri/tls/i965_dri.so
libGL: OpenDriver: trying /usr/lib/i386-linux-gnu/dri/i965_dri.so
libGL: dlopen /usr/lib/i386-linux-gnu/dri/i965_dri.so failed (/home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/lib/fallback/libgcc_s.so.1: version `GCC_3.4' not found (required by /usr/lib/i386-linux-gnu/dri/i965_dri.so))
libGL: OpenDriver: trying ${ORIGIN}/dri/tls/i965_dri.so
libGL: OpenDriver: trying ${ORIGIN}/dri/i965_dri.so
libGL: dlopen ${ORIGIN}/dri/i965_dri.so failed (${ORIGIN}/dri/i965_dri.so: cannot open shared object file: No such file or directory)
libGL: OpenDriver: trying /usr/lib/dri/tls/i965_dri.so
libGL: OpenDriver: trying /usr/lib/dri/i965_dri.so
libGL: dlopen /usr/lib/dri/i965_dri.so failed (/usr/lib/dri/i965_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: i965_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: i965
libGL: pci id for fd 3: 8086:0416, driver i965
libGL: OpenDriver: trying /usr/lib/i386-linux-gnu/dri/tls/i965_dri.so
libGL: OpenDriver: trying /usr/lib/i386-linux-gnu/dri/i965_dri.so
libGL: dlopen /usr/lib/i386-linux-gnu/dri/i965_dri.so failed (/home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/lib/fallback/libgcc_s.so.1: version `GCC_3.4' not found (required by /usr/lib/i386-linux-gnu/dri/i965_dri.so))
libGL: OpenDriver: trying ${ORIGIN}/dri/tls/i965_dri.so
libGL: OpenDriver: trying ${ORIGIN}/dri/i965_dri.so
libGL: dlopen ${ORIGIN}/dri/i965_dri.so failed (${ORIGIN}/dri/i965_dri.so: cannot open shared object file: No such file or directory)
libGL: OpenDriver: trying /usr/lib/dri/tls/i965_dri.so
libGL: OpenDriver: trying /usr/lib/dri/i965_dri.so
libGL: dlopen /usr/lib/dri/i965_dri.so failed (/usr/lib/dri/i965_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: i965_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: i965
libGL: OpenDriver: trying /usr/lib/i386-linux-gnu/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib/i386-linux-gnu/dri/swrast_dri.so
libGL: dlopen /usr/lib/i386-linux-gnu/dri/swrast_dri.so failed (/home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/lib/fallback/libgcc_s.so.1: version `GCC_3.4' not found (required by /usr/lib/i386-linux-gnu/dri/swrast_dri.so))
libGL: OpenDriver: trying ${ORIGIN}/dri/tls/swrast_dri.so
libGL: OpenDriver: trying ${ORIGIN}/dri/swrast_dri.so
libGL: dlopen ${ORIGIN}/dri/swrast_dri.so failed (${ORIGIN}/dri/swrast_dri.so: cannot open shared object file: No such file or directory)
libGL: OpenDriver: trying /usr/lib/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib/dri/swrast_dri.so
libGL: dlopen /usr/lib/dri/swrast_dri.so failed (/usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory)
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
Gdk-ERROR **: BadValue (integer parameter out of range for operation)
  serial 858 error_code 2 request_code 155 minor_code 3
Here OpenEV goes into frenzy spinning, CPU overheats, then goes to throttle
Killed
 
[1]+  Exit 137                LIBGL_DEBUG=verbose ~/Software/Linux_software/OpenEV/FWTools-2.0.6/bin_safe/openev -h
 
 
 
ymorel@u16-i386-GS70:~/Software/Linux_software/OpenEV/FWTools-2.0.6/lib/fallback$ locate libgcc_s.so.1
/lib/i386-linux-gnu/libgcc_s.so.1
cd /home/ymorel/Software/Linux_software/OpenEV/FWTools-2.0.6/lib/fallback/
mkdir old
ln -s /lib/i386-linux-gnu/libgcc_s.so.1 libgcc_s.so


 





 
The 4SM problem as of 2017

please compile and run 4SM 
  under Ubuntu 16 LTS 64 bits 
 gcc -lm -w -m32 -g 4SM.7.10.ghost.c -lm -o 4SM.7.10.ghost
 
  • Write 4SM in python?
    • "system programming languages (like C) are well suited to building components where the complexity is in the data structures and algorithms"
    • "scripting languages (like python) are well suited for gluing applications where the complexity is in the connections"
      • "my conclusion: keep 4SM in C, which is "a system programming language"
    • please read http://webcem01.cem.itesm.mx:8005/s201513/tc2026/scripting_xxi.pdf
      • Does the application implement complex algorithms or data structures?
        • I think probably YES
        • "stratified" waters over the visible range makes for a fairly complex description of diffuse attenuation properties
        • 4SM makes good use of dynamic memory allocation, and of structure data type
      • Does the application manipulate large datasets (e.g. all the pixels in an image) so that execution speed is critical?
      • Are the application's functions well-defined and changing slowly?
        • I think YES: took me 23 years!!
      • Is the application's main task to connect together pre-existing components?
        • I think NO, certainly NOT the main task
        • system calls to pre-existing gv, shp2text, or OpenEV are very simple and effective in C
      • Will the application manipulate a variety of different kinds of things?
        • I think NO
      • Does the application include a graphical user interface?
        • NO: entirely controlled by a powerfull and well structured command line
        • GUI not needed actually;
        • although much desired/demanded by lazy would-be practitioners: can wait!
      • Does the application do a lot of string manipulation?
        • NO, just a few for building output file names, and for deciphering command line
      • Will the application's functions evolve rapidly over time?
        • I think NO
        • after 23 years, all main features have been identified and implemented
        • catters for multi/hyperspectral imageries
        • catters for various pixel sizes
        • catters for very large images
      • Does the application need to be extensible?
        • I think NO, for the years to come
        • already does a lot
        • covers all image formats and ground sample distances
        • does both the optical calibration and the raster processing
        • does the deglinting, pan sharpening, smart smoothing, model inversion, and bottom typing
        • runs and displays spatial and spectral profiles