I am filing MA (Resident) and CA (non-resident) state tax return. In MA tax return, it asks for "5% Income on Which You Paid Tax". What does 5% mean here?

This refers to the Total Taxable 5.0% Income reported on line 21 of your Massachusetts Form 1, and the Tax On 5.0% Income reported on line 22.

According to the instructions,

If line 21 is less than $24,000, find the proper tax by using the tax tables found in the back of this booklet. If line 21 is greater than $24,000 multiply by 0.05 and enter the result in line 22.

Note: Personal income tax forms must provide an election to voluntarily pay tax at a rate of 5.85% on taxable income which would otherwise be taxed at a rate of 5.0%.