webplodder said:

OK, but assuming we're talking about a system capable of emulating or even exceeding human reasoning,

Such a machine does not exist yet, and the possibility of creating one is as probable as bringing back to life body parts sewn together, à la Frankenstein:

https://www.sciencenewstoday.org/can-a-machine-ever-truly-think-like-a-human said:

The Path Toward Human-Like Minds​

Could future machines evolve beyond clever mimicry? Many scientists believe the answer may lie in artificial general intelligence (AGI)—a hypothetical AI capable of understanding, learning, and reasoning across the full range of human cognitive tasks.

AGI would not simply be good at chess or conversation but could tackle any intellectual challenge a human can, adapting flexibly to new situations. It would integrate vision, language, reasoning, planning, and perhaps even emotion.

Achieving AGI demands breakthroughs in multiple areas:

• World models: Human thought relies on rich mental models of the world. We simulate scenarios, predict consequences, and imagine possibilities. Machines would need similar models to think beyond surface patterns.
• Common sense: Humans possess vast background knowledge that informs our judgments. Machines need to acquire comparable common sense to navigate everyday situations.
• Embodied cognition: Many scientists argue that true understanding requires a body interacting with the physical world. Robots, not just chatbots, might be essential to creating human-like intelligence.
• Consciousness and emotion: If thinking requires subjective experience, as some philosophers argue, then machines might need new architectures to support inner awareness.

Yet whether AGI will ever be conscious—or merely produce an impeccable imitation of human behavior—remains one of science’s deepest mysteries.

webplodder said:

is it be possible for complex chatbots to develop behaviours that can’t be predicted by simply analyzing the sum of their parts?

I'm a big fan of the "embodied cognition" requirement for AGI; so, in my opinion, no, a simple chatbot would never achieve consciousness or the like.

PeroK said:

The argument that they are just dumb machines is, however, a dangerous fallacy.

They are dumb machines. Thinking they are intelligent and relying on their decisions without further verification is what is dangerous.