Tools for visual scene recognition

Abstract

Scene recognition is an important step towards a full understanding of an image. This

thesis presents novel ideas related to semantic-spatial content capture and local-global

feature fusion techniques and applies them for scene recognition. It shows how the proper

use of these approaches, without trying to recognize objects in the scene images, can lead

to improvement in recognition accuracy for scene classi cation.

First, we propose a method to build a semantic visual vocabulary. The features are

extracted from the image patches and the initial vocabulary is constructed by performing

k-means clustering on the extracted features and choosing the cluster centers as the visual

words. The feature vectors are quantized based on the initial vocabulary to form a wordimage

matrix that describes the occurrence of words in the images. The codebooks

are then embedded into the concept space by latent semantic models. We demonstrate

this embedding using Latent Semantic Analysis (LSA) as well as Probabilistic Latent

Semantic Analysis (pLSA). In the proposed space, the distances between words represent

the semantic distances, which are used to construct a discriminative and semantically

meaningful vocabulary. The main contributions of the rst chapter are as follows: 1.

Using semantic word space to co-cluster similar words together to form a semantic visual

vocabulary. This will improve the results compared to other methods that use document

space directly after pLSA embedding. 2. Investigating changes in the number of latent

variables. 3. Using LSA embedding when all other vision systems to date only use pLSA.

This method has shown promising results on 15-Scene categories when the proposed

model extracts one type of visual feature.

Second, since fusing local and global features is bene cial for achieving a promising

performance for scene categorization systems [7], we propose a novel Local-Global Feature

Fusion (LGFF) method with the capability to fuse latent semantic patches adaptively. The image local feature space is embedded into the latent semantic space by employing

pLSA modeling; afterwards, this semantic space and the global contextual feature space

are mapped into a kernel-space. To perform this embedding, the global features and latent

variables are relatively weighted for each scene category. The following is a summary

of the main contributions of the second chapter: 1. Weighting latent semantic topics

based on their discriminative power 2. De ning a novel exemplar-based distance learning

3. De ning a category-dependent map function; the experimental evaluations indicate

radical improvements in the 15-Scene and 67-Indoor Scenes datasets.

Third, every scene image contains a high value of spatial information. Capturing the

spatial positions of visual features plays an essential role in scene categorization; however,

the methods proposed in the previous chapters disregard this position information.

Inspired by methods that construct pyramid levels over image primitives [8, 9], pyramid

matching kernels are employed to measure the dissimilarity scores of image pyramids.

We improved the semantic vocabulary framework by considering the position of semantic

local patches and their surrounding neighborhood properties using either the global or

region-based spatial pyramid method. In the global method, after projecting the image

into the concept space, it is divided into ner sub-scenes and then the pyramid match

kernels are employed over the proposed space for co-clustering the semantic visual words.

The region-based method initially divides the image into sub-scenes and then projects

each sub-scene into the concept space. In the Local-Global Feature Fusion framework,

by using the global features gist and CENsus TRansform hISTogram (CENTRIST), the

global spatial layout of the scene is already captured and, to further improve the results,

the image is divided into sub-regions at di erent levels of resolution. The pyramid

matching kernels are then applied over these sub-regions. The representation of these

sub-regions is obtained either by applying CENTRIST or a bag of Scale-Invariant Feature

Transform (SIFT) visual features. The experimental results outperform most of the

best published results for both the 15-Scene and 67-Indoor Scenes datasets.

This thesis concludes with a discussion of future directions for extending the proposed

works on scene recognition.