User Tools

Site Tools


courses:msc:msc-phygeo-remote-sensing:worksheets:rs-ws-05-1

W05-1: Scaling

Over the last sessions, you have compiled quite a variety of data sets. Before we jump into a classification approach it is time to consolidate the data and compile a comprehensive data set which will finally be made available through a geo-database.

After completing this worksheet you should be able to prepare consolidate your project folder structure and prepare automatic scripts for the computation of the final baseline data sets.

Things you need for this worksheet

  • GIS — most commercial or open source GIS systems will be fine. For open source software running on Ubuntu Linux, the Ubuntu GIS package is a good choice. For Windows, it will be a good choice installing everything via the OSGeo4W environment and not individually. Actually both repositories make available a comprehensive bundle of GI -tools and software packages. We will focus on GRASS, QGIS and SAGA. On the commercial side, ArcGIS is one of the powerful options.

  • R — the interpreter can be installed on any operation system. For Linux, you should use the r-cran packages supplied for your Linux distribution. If you use Ubuntu, this is one of many starting points. If you use Windows, you could install R from the official CRAN web page.

  • R Studio — we recommend to use R Studio for (interactive) programming with R. You can download R Studio from the official web page.

  • Git environment for your operating system. For Windows users with little experience on the command line we recommend GitHub Windows.

  • your finally cleaned aerial files, derived indices and filters which have been compiled as part of worksheets W02-1, W02-2, W03-1, W04-1, and W04-2.

Prepare a "server quality" script for compiling a final comprehensive data set

:-\ Please co-ordinate with the other groups in order to compile a joint comprehensive data set which contains each kind of product only once. Prepare a script which compiles the data automatically starting with the results from session 3 (i.e. no white stripes any more, eastern files are cut to the LiDAR extent).

Once your script is ready, we will run it on one of our servers in order to speed up the process.

This is also the time to finally synchronize all our data structures.

setPathGlobal.R
# rs-ws-05-1
#' @description  MOC - Advanced GIS (T. Nauss, C. Reudenbach)
#' setPathGlobal
#'@return 
#' defines and creates (if necessary) all folders variables
#' set the SAGA path variables and other system variables
#' exports all variables to the global environment
#'
#'@param filepath_git  project github root directory (your github name)
#'@param csess= current session "01",
#'@param ccourse current course options are "gi", "rs", "da"
#'@param moc=TRUE creates a folder structure according to the needs of the MOC courses, FALSE creates a simple project structure
#'\preformatted{
#'   If moc=TRUE the following folderstructure is exported. If folders do not exist thesy will be created.
#'.
#'├── data
#'│   ├── data_analysis
#'│   │   ├── csv
#'│   │   └── raw
#'│   ├── gis
#'│   │   ├── input
#'│   │   ├── output
#'│   │   ├── RData
#'│   │   ├── run
#'│   │   └── temp
#'│   └── remote_sensing
#'│       ├── aerial
#'│       ├── aerial_croped
#'│       ├── aerial_merged
#'│       ├── input
#'│       ├── RData
#'│       ├── run
#'│       └── temp
#'└── MOC
#'    ├── data_analysis
#'    │   └── da-ws-01
#'    │       └── rmds
#'    │       └── scripts
#'    ├── fun
#'    ├── gis
#'    │   └── gi-ws-01
#'    │       └── rmds
#'    │       └── scripts
#'    └── remote_sensing
#'        └── rs-ws-01
#'    │       └── rmds
#'            └── scripts
#'    proj_root_data<-c(path.expand(paste0(dirname(filepath_git),"/data/")))
#' 
#' ############
#' 
#' if moc=FALSE
#' .
#' └── project1
#'     ├── control
#'     │   └── log
#'     ├── data
#'     │   ├── input
#'     │   └── output
#'     ├── run
#'     └── src
#'         └── fun
#'   } 
#'
#'@author Thomas Nauss, Chris Reudenbach
#'
#'@return  setPathGlobal< creates if necessary the directories and export the corresponding pathes as global variables\cr
 
setPathGlobal<- function(filepath_git,csess=15,ccourse="gi", moc=TRUE) {
 
  # switch backslash to slash and expand path to full path
  filepath_git<-gsub("\\\\", "/", path.expand(filepath_git))  
 
  # check  tailing / and if not existing append
  if (substr(filepath_git,nchar(filepath_git)-1,nchar(filepath_git)) != "/") {
    filepath_git<-paste0(filepath_git,"/")
  }
 
  ### moc = FALSE feel free to adapt 
  default_folders<- c(paste0(filepath_git,"src/"),
                      paste0(filepath_git,"src/fun/"),
                      paste0(filepath_git,"data/input/"),
                      paste0(filepath_git,"data/output/"),
                      paste0(    proj_root_data<-c(path.expand(paste0(dirname(filepath_git),"/data/")))filepath_git,"control/log/"),
                      paste0(filepath_git,"run/"))
 
  ### moc=TRUE
  # script and function folder for each course session can be adapted 
  session_working_folder<-c("/scripts/", "/rmds/")
  # currently implemented data folders can be adapted 
  # list1 is linked to "remote_sensing, list2 is linked to "gis" list3 is linked to "data_analysis"
  data_working_folder<-list(list("aerial/","aerial_merged/","aerial_croped/","RData/","temp/","run/","input/"),
                            list("RData/","temp/","run/","input/","output/"),
                            list("csv/","raw/"))  
      proj_root_data<-c(path.expand(paste0(dirname(filepath_git),"/data/")))
  if (moc) {
    # static course structure - better keep the below folders
    proj_root_git<-c(path.expand(filepath_git))
    proj_root_data<-paste0(substr(proj_root_git,1,gregexpr(pattern ='/',proj_root_git)[[1]][as.numeric(lengths(gregexpr(pattern ='/',proj_root_git))[[1]]-2)]),"data/")
    sub_root<-c("remote_sensing/","gis/","data_analysis/")
    session_ID<-c("rs-ws-","gi-ws-","da-ws-")
 
    # create sessionstring
    ns<-1:csess
    session_number<- sapply(ns, function(ns){
                     if (ns<10) {ns<-paste0("0",ns)}
                     return(ns)
                   })
        proj_root_data<-c(path.expand(paste0(dirname(filepath_git),"/data/")))
    # create folder and varibales 
    # function folder for all courses
    name<-paste0("pgit_fun")
    value<-paste0(filepath_git,"/fun/")
    exV(name, value)
    # and the rest
    if (!file.exists(file.path(paste0(filepath_git,"/fun/")))) {
      dir.create(file.path(paste0(filepath_git,"/fun/")), recursive = TRUE)
    }  
    for (i in 1:length(proj_root_git)) {
      for (j in 1:length(sub_root)) {
        for (k in 1:length(session_ID)) {
          for (l in 1:length(session_number)) {
            for (m in 1:length(session_working_folder)) {
              name<-paste0("pg_", substr(session_ID[j],1,2),"_",as.character(gsub("/", "", session_number[l])),"_",as.character(gsub("/", "",session_working_folder[m])))
              value<- paste0(proj_root_git[i],sub_root[j],session_ID[j],session_number[l],session_working_folder[m])
              exV(name, value)
              if (!file.exists(file.path(paste0(proj_root_git[i],sub_root[j],session_ID[j],session_number[l],session_working_folder[m])))) {
                dir.create(file.path(paste0(proj_root_git[i],sub_root[j],session_ID[j],session_number[l],session_working_folder[m])), recursive = TRUE)
              }
            }
          }
        }
      }
    }
 
    # data structure NOTE it is outside the proj_root_git folder
    for (i in 1:length(proj_root_data)){
      for (j in 1:length(sub_root)) {
        for (k in 1:length(data_working_folder[[j]])) {
          name<-paste0("pd_",substr(session_ID[j],1,2),"_",as.character(gsub("/", "",data_working_folder[[j]][k])))
          value<- paste0(proj_root_data[i],sub_root[j],data_working_folder[[j]][k])
          exV(name, value)
          if (ccourse==substr(session_ID[j],1,2) && data_working_folder[[j]][k]=="run/"){
            path_temp<- value
          }
          if (!file.exists(file.path(paste0(proj_root_data[i],sub_root[j],data_working_folder[[j]][k])))) {
            dir.create(file.path(paste0(proj_root_data[i],sub_root[j],data_working_folder[[j]][k])), recursive = TRUE)
 
          }
        }
      }
    }
  } # end of moc=TRUE
  # create a default project structure
  else {
    # create directories if needed
    path_temp<-paste0(filepath_git,"run/")
    for(folder in default_folders){
      name<-paste0("p",as.character(gsub("/", "_",substr(folder,nchar(filepath_git),nchar(folder)-1))))
      exV(name, folder)
      if (!file.exists(file.path(folder))) {
        dir.create(file.path(folder), recursive = TRUE)
      }
    }
  }
 
  # set rasteroptions to working directory
  raster::rasterOptions(tmpdir = path.expand(path_temp))
 
  # setting R environ temp folder to the current working directory
  Sys.setenv(TMPDIR = file.path(path_temp))
 
  # set R working directory
  setwd(file.path(path_temp))
 
  # (R) set pathes  of SAGA modules and binaries depending on OS  
  exist<-FALSE
  if(Sys.info()["sysname"] == "Windows"){
    exV("saga_bin", "C:/OSGeo4W/apps/saga")
    exV("saga_mod",  "C:/OSGeo4W/apps/saga/modules/")
    # check if already in system path
    p<- Sys.getenv("PATH")
    if(substr(p, 1,nchar(saga_bin)) == saga_bin){
      exist<-TRUE
    }
    # append SAGA path to systempath
    if (!exist){
      Sys.setenv(PATH=paste0(saga_bin,";",saga_mod,";",Sys.getenv("PATH")))}
  }else {
    exV("saga_bin",  "/usr/local/bin")
    exV("saga_mod","/usr/local/lib/saga")
    # check if already in path
    p<- Sys.getenv("PATH")
    if(substr(p, 1,nchar(saga_bin)) == saga_bin){
      exist<-TRUE
    }
    # append SAGA path to systempath
    if (!exist){
      Sys.setenv(PATH=paste0(saga_mod,":",saga_mod,":",Sys.getenv("PATH")))}
  }
 
}
 
# assigns a variable in .GlobalEnv 
exV <- function(name,value) {
  if(!exists(name, envir = .GlobalEnv)) {
    assign(name, value, envir = .GlobalEnv, inherits = TRUE)
  } 
}
courses/msc/msc-phygeo-remote-sensing/worksheets/rs-ws-05-1.txt · Last modified: 2016/11/21 15:03 by creudenbach