Do I need to put Amazon orders in on the "Untaxed Out-Of-State Purchases" for CT taxes. Because Amazon orders to CT are apparently taxed already, does it count?

You are referring to a "Use Tax", which is a tax assessed by states on out-of-state purchases that were not charged taxes by either your state of residence or the state where the state was made. 18 or more states have enacted these taxes to recoup local sales taxes that were lost because items were bought on the Internet.

If, however, your Amazon receipt shows sales taxes were collected, then you don't need to include these purchases with other items for the Use Tax.