It’s entirely possible! I remember listening to a podcast on AI, where they mentioned someone once asked the questions “which mammal lays the largest eggs” to which the ai responded with elephants, and proceeded to argue with the user that it was right and he was wrong.
It has become a lot easier as I’ve learned how to kind of coach it in the direction I want, pointing out obvious errors and showing it what I’m really looking to do.
Ai is a great tool, when it works. As the technology improves I’m sure it will rapidly get better.
When you end up resorting to saying things like “wow, this is wonderful, but… It breaks my code into a million tiny pieces” Or “for the love of God do you have any idea what you’re actually doing?” It’s a sign that perhaps stack overflow is still your best (and only) ally