In the study of high dimensional knot concordance groups, Levine's classification hinges on the primary decomposition along self-dual factors of Alexander polynomials. We propose a notion of general primary decompositions for the study of low dimensional cases. Especially, for the smooth concordance group of topology slice knots, we show that primary decomposition reveals new rich structures which are invisible to modern smooth invariants. To detect this, we combine amenable $L^2$-signatures with information from Heegaard Floer homology. We also address primary decomposition of the associated grades of the bipolar filtration of topologically slice knots.