Penalties? Only if it is in a IRA account and you don't replace it within 60 days. For a regular account you might pay tax on it even if you put it back.
