Aiming to address food safety issues, this research analyzes the influence of "moral deficiency" and "inadequate ability" on consumers' willingness to punish food companies when they are in behaviors of Corporate Social Irresponsibility (CSI). Based on the attribution theory and expectation theory, this paper analyzes 342 valid questionnaires and finds that the two kinds of CSI attribution both have significant positive influences on consumers' punishment intention, yet they vary in terms of the degrees of influence. Compared to the "insufficient capacity" of companies, "moral deficiency" is more likely to incur punishments from consumers. Consumer perception plays a mediating role between CSI behavior attribution of responsibility and consumers' intention to punish. The CSI perception of consumers is also affected by the size of companies: consumers tend to have a stronger CSI behavior perception of large companies than ordinary size. Therefore, food companies should stick to the moral bottom line, strengthen their management, and improve technology, while large companies should play a leading role in the industry.