Supply chain personnel are not owned by Amazon. In Most cases and region, a third party vendor takes up the delivery work.
So you would have to thoroughly vet the delivery guys to make this happen because you are basically giving the key to your house to a stranger.
What happens if the home has pets?
What happens when there is an actual robbery but cops are needlessly wasting their time checking if someone misused Amazon key.
What if you have kidnapped someone and she escaped because you have a delivery?