Your situation will be dependent on the relationship of the two people that own the joint account.
If married, the annual gift exclusion doesn't apply as one may gift to their spouse without limit.
If the two owners are not married, recordkeeping is never a bad thing. Tracking the deposits and withdrawals as you referenced would allow you to know exactly who owns the funds going in and out of the account. If one owner does use the other owner's funds for their personal purposes with no intent of repayment, it definitely could be construed as a gift.
Additionally, be aware that the gift exclusion for 2023 is $17,000.
If you find this information helpful, please indicate by selecting the thumbs up button below.
Elizabeth W, EA