Waldhausen's algebraic $K$-theory of spaces provides a critical link in the classification of manifolds and their diffeomorphisms. We will explain this connection and discuss work in progress on the analogous story for manifolds with group action.