Office is in CT but live in NY. We have been working from home since March 2020. Do I owe taxes to CT or NY for the work from home period in this case?

Nothing has changed ... you will file both a non resident CT return and a resident NY return in the program IN THAT ORDER  so that you get a credit on the NY return for the taxes owed to CT.

https://www.daypitney.com/insights/publications/2021/01/11-update-increased-telecommuting-raises-state-tax#:~:text=Connecticut%2C%20unlike%20other%20states%20with,convenience%20of%20the%20employer%20rule.

I’m aware I would still be filing in both states. The question is whether the work from home income is considered NY or CT income? Does it go by where physically you worked or where your office is?

Both ... it is still CT sourced income and NY taxes everything. 

I suggest you talk to someone locally to see what considerations have been made between NY & CT if there are any.