If you are ever audited and they review your banking transactions and you don't have some kind of records of where this money came from then the IRS can consider it all unreported income and it would be your responsibility to prove them wrong. Income audits is how they catch people who work under the table or have illegal income.
