The spatial analysis of remotely sensed images evolves in three stages: physical classification, semantic extraction, and information recognition. The traditional approaches to image processing use only the first stage. In this paper, a hierarchical multi-resolution structure was developed by the use of segmentation to integrate the three stages into one common platform. Following a spatial segmentation, hierarchical multi-resolution layers were formed to handle the various information needs inherent in the various ground features identified within the image. The relationship between image objects in one layer, or several layers, characterized the segmental objects in spectral, spatial, and hierarchical perspectives, generating new additional layers of information for image analysis. Consequently when the classification was finished, the semantic hierarchy of the ground features and information hidden within the image pixels were extracted. Finally this classification is achieved through image processing by initially decomposing the image into the most basic physical objects and the recomposing into semantic objects. The superiority of the method is obvious: (i) semantic extraction and information recognition can be synthetically performed with the classification procedure in the image processing; (ii) the way that per-pixel handling is promoted to per-object analysis makes image processing more natural; and (iii) the mechanism analysing the relationship in a multi-resolution hierarchy yields a better understanding of the image objects, both semantically and physically.