We will discuss cohomological obstructions to extending group actions on the boundary $\partial M$ of a $3$-manifold to a $C^0$-action on $M$ when $\partial M$ is diffeomorphic to a torus or a sphere. In particular, we show that for a $3$-manifold $M$ with torus boundary which is not diffeomorphic to a solid torus, the torus action on the boundary does not extend to a $C^0$-action on $M$.