Reconstruction is commonly achieved using points and even lines which can be grouped in planes. Thus, the problem could be reformulated at a higher level, in a more effective way for recognition. However, conventional plane sweep approaches introduce two important limitations related with the high multiplicity of planes and the reconstruction of non-planar shapes. Both limitations can be tackled using two different strategies:
Plane arrangements introduce a combinational problem more complicated than line arrangements. They can be managed using discrete groups of symmetries which spreads the information by replication with respect to the spatial symmetries linked to the planes. This information can be represented using the set of normals of the planes.
Non-planar shape of the reconstructed objects can be treated using some kind of manifold learning. More specifically, LLE (Locally Linear Embedding) which can be efficiently implemented using sparse matrix algorithms. LLE begins by finding a set of the nearest neighbors of each point. Then, it computes a set of weights for each point that best describe the point as a linear combination of its neighbors. Finally, it uses an eigenvector-based optimization technique to find the low-dimensional embedding of points, such that each point is still described with the same linear combination of its neighbors. This approach has been commonly used to solve recognition problems but it has never been used in reconstruction since 3D objects are not described in terms of plane hulls.
As the main conclusion we can connect the reconstruction and recognition processes using plane hulls of 3D objects controlled by LLE. In such situation, vectors of features detected over an image would be collections of normals linked to configurations of $(k+1)$ planes in a global space. Each matrix representing a plane is an element of a grassmannian $Grass(k+1, N+1)$ which is the universal space to factorize applications defined over a manifold $M$ in normal meaning context.