Evaluation of students’ ability in constructing scientific explanations on scientific phenomena is essential as an effort to obtain information and feedback for innovation in learning process and curriculum development. Unfortunately, this issue is still left unexplored by researchers in chemistry education. Such is presumed to occur due to validated instruments, measurements, analysis techniques, and diverse epistemological values that leave much space to be investigated. Employing a Rasch model, we intended to validate test of ability in constructing scientific explanations on chemical phenomena, examine data fit with the Rasch model, evaluate difference in the students’ ability in constructing scientific explanations, investigate items with different functions, and diagnose causes for difference in item difficulty level. The respondents were 550 students from seven senior high schools in three regencies/cities and 153 university students in Sulawesi, Indonesia. Data were collected by 30 test items; each item consisted of three questions measuring students’ ability in proposing their knowledge (Q1), evidence (Q2), and reasoning (Q3). Their responses were assessed on criteria and analyzed using the Rasch partial credit model. This model applies an individual-centered statistical approach allowing researchers to measure up to item and individual level. Results suggested that data fit the Rasch model measurement. Also, students’ ability in constructing scientific explanations varied significantly. We found no items with different functions, signifying that sex and hometown do not influence students’ ability. However, based on item logit value grouping, it was discovered that item difficulty level also varied among students. This was particularly due to students’ lack of chemistry concepts mastery that lowered their ability and accuracy in constructing scientific explanation. This shows lack of epistemological engagement of students in learning process. In conclusion, this study provides valuable insights into students’ ability to construct scientific explanations and sheds light on factors that influence their performance in this area. Findings highlight need for targeted interventions that address students’ conceptual understanding and engagement with chemistry concepts, as well as promote critical thinking and scientific reasoning skills. This has important implications for science education and can inform curriculum development and evaluation policies.