User Tools

Site Tools


courses:msc:msc-phygeo-remote-sensing:worksheets:rs-ws-06-2

W06-2: Accuracy assessment

This worksheet covers the quantitative accuracy assessment of your first land-cover classification. After completing this worksheet you should be able to compute and interpret the Kappa index of agreement.

Things you need for this worksheet

  • QGIS and OTB — For remote sensing related tasks, we recommend to use the capabilities of QGIS including the Orfeo ToolBox (OTB) plugin. Regarding the installation, the unstable repository of the Ubuntu GIS package is a good choice for Linux while we strongly recommend the OSGeo4W environment as installation source for Windows users. Please have a look at the actual OTB download options prior your installation of QGIs. If you experience troubles with the OTB plugins in QGis, you can also install Monteverdi2 which is a stand alone GUI for OTB.

  • R — the interpreter can be installed on any operation system. For Linux, you should use the r-cran packages supplied for your Linux distribution. If you use Ubuntu, this is one of many starting points. If you use Windows, you could install R from the official CRAN web page.

  • R Studio — we recommend to use R Studio for (interactive) programming with R. You can download R Studio from the official web page.

  • your deliveries from W05-2: Training sites
  • your deliveries from W06-1: Land-cover classification
  • [Kuhnert2005] or a quick Wikipedia search if you do not know how to calculate Kappa

Cohen's Kappa

Eyeball verification of map-type figures generally gives a very good (first) impression. For a quantitative description however, a more “objective” index might be requested by others. Regarding classification accuracies, Cohen's Kappa index of agreement might likely be the most commonly used index (see e.g. [Kuhnert2005] for a short overview and calculation information). Although Kappa is certainly not a one fits all purposes index, we will start with this one for today.

Learning log assignments

8-) As always, please add these entries to your today's learning log in the beginning of your Rmd file you will use to push to your GitHub classroom.

  • Favorite aspect of the session (if any)
  • Superfluous aspect of the session (if any)
  • Eureka effect (if any)
  • Links to what I've learned so far (if any)
  • Questions (if any)

As today's special, please complete the following assignment:

In W06-1: Land-cover classification, you have computed a land-cover classification and already performed some qualitative analysis. Let's explore a more quantitative way using Cohen's Kappa index of agreement in a leave-many-out setting.

:-? Please repeat the computation of your land-cover prediction 6 times in a leave-many-out-setup. This means, that you exclude a complete sample of all land covers from the model training and use it as independent test sample. For the accuracy assessment you can use the function provided below which will compute the Kappa index. The function requires a contingency table as input which can be generated using OTB's Compute Confusion Matrix function in QGIS. Please implement the computation of Kappa in a Rmd file and include a print statement in the end which provides the different Kappa values.

:-\ Knitr your Rmd file, update (i.e. commit) it and publish (i.e. push) it to the GitHub classroom. Make sure that the created html file is also part of your git repository.

fun_compKappa.R
# rs-ws-03-2
# MOC - Remote Sensing (T. Nauss, C. Reudenbach)
# 
#' Compute Kappa based on contingency table
#' 
#' @description
#' Compute Kappa index.
#'
#' @param ctable Contingency table
#'
#' @return Kappa index.
#'
compKappa <- function(ctable){
  ctable <- ctable/sum(ctable)
  categories <- nrow(ctable)
 
  # Fraction of agreement
  pagrm <- 0
  for(i in seq(categories)){
    pagrm <- pagrm + ctable[i,i]
  }
 
  # Expected fraction of agreement subject to the observed distribution
  pexpct <- 0
  for(i in seq(categories)){
    pexpct <- pexpct + sum(ctable[i,]) * sum(ctable[,i])
  }
 
  # Kappa index
  kappa <- (pagrm - pexpct)/(1 - pexpct)
 
  return(kappa)
}
courses/msc/msc-phygeo-remote-sensing/worksheets/rs-ws-06-2.txt · Last modified: 2016/12/08 16:38 by tnauss