Based on the theories of the multimodal pedagogy, this article focuses on the evaluation of the teaching effectiveness of multimodal college English listening teaching and the configuration principle of its multimodal system. Considering the lack of empirical research on multimodal college English listening teaching in China, the research applies empirical study to compare the differences between the experimental group and the control group to find out the effectiveness of the multimodal college English listening teaching. After the one-year experiment, the subjects of experimental group and control group, who were basically similar in their English achievement and English listening ability before the experiment, manifest significant differences in both English test and English listening test. Superior characteristics are reflected by the experimental group received multimodal college English listening teaching in two test points compared with the control group, which proves the feasibility and effectiveness of the multimodal college English listening teaching system. Besides, the following interviews on the subject of the experimental group, combined with the multimodal pedagogy theories, provide the possible configuration to ensure an effective, dynamic and three-dimensional multimodal college English listening teaching system.