Deeply disagree that transformer based architectures can be scaled to AGI by adding multi-modal features. Check the part-1 of this article which talks about what AGI needs mathematically.

https://medium.com/autonomous-agents/why-llms-will-never-lead-to-agi-aa7bcff9805d