The penalty is related to what you owe after subtracting tax withheld and estimated payments made well as any credits.
