You won't need to pay a gift tax on the additional $5000 and you may not have to file a gift tax return (Form 709).
If you are married, 1/2 of the $20,000 could be considered as coming from your spouse - in that case, each of you would be treated as making a $10,000 gift to your father, which is below the annual $15,000 gift exclusion limit.
Even if you are single, you're also entitled to a lifetime exemption from gift and estate tax. In 2020, that exemption amount jumps to $11.58 million. In this case, you'd have to file a gift tax return so the IRS has a record of your lifetime giving.
A 709 is only needed for the year of a gift over the annual exclusion amount (currently $15k) . If there is no future gift, no future 709 need be filed. Be sure to keep the 709. Also keep the address it is sent to. (That is asked for on a 709 if it is filed).
