An understanding of the balance of interspecific competition and the physical environment in structuring organismal communities is crucial because those communities structured primarily by their physical environment typically exhibit greater sensitivity to environmental change than those structured predominantly by competitive interactions. Here, using detailed phylogenetic and functional information, we investigate this question in macrofaunal assemblages from Northwest Atlantic Ocean continental slopes, a high seas region projected to experience substantial environmental change through the current century. We demonstrate assemblages to be both phylogenetically and functionally under-dispersed and thus conclude that the physical environment, not competition, may dominate in structuring deep-ocean communities. Further, we find temperature and bottom trawling intensity to be amongst the environmental factors significantly related to assemblage diversity. These results hint that deep-ocean communities are highly sensitive to their physical environment and vulnerable to environmental perturbation, including by direct disturbance through fishing, and indirectly through the changes brought about by climate change.