"Total of all your payments and withholding" means "what did you pay to that taxing agency (state or local) in 2016" (or the year of the refund), including what was withheld and any other additional payments made in 2016. DO NOT include any estimated tax you paid for tax year 2016 but you paid after Dec. 31, 2016. If the 1099-G is for a tax refund, the program is trying to determine if it is taxable.
If the refund is from 2016, and you used the "Standard Deduction" on your 2016 return (you did not itemize), the refund is not taxable and does not need to be reported.
If you did itemize in 2016 but used Sales Tax Paid, rather than State/Local Income Tax Paid, it is not taxable and does not need to be reported.
If you did itemize in 2016 and used the State and/or Local Income Tax Paid last year, the refund may be taxable this year depending on 1) when you paid the tax, 2) how it affects your itemized deductions versus the available standard deduction this year and/or last.
You may need to look on your 2016 return to answer this particular question.
You can also check the tax return that matches the refund. For example, if the refund is from a 2016 Michigan return, look at the 2016 MI-1040 that you filed and see what is reported as "Payments and withheld".
