Constraints on Galactic intermediate mass black holes