Pioneering Sovereign, Human-Centric AI

AI models typically rely on a limited set of subwords to process text. This rigid tokenization limits adaptability and efficiency. Our Hierarchical Autoregressive Transformer (HAT) changes the game. By combining character-level flexibility with word-level efficiency, it eliminates fixed vocabularies. This approach increases efficiency making AI more resilient to typos, and more adaptable to new languages or domains.

Training AI models at scale is challenging. Hyperparameters that work for small models often fail at larger sizes, and low-precision training introduces instability. u-µP solves both problems. By combining Maximal Update Parametrization (µP) with Unit Scaling, it enables models to train efficiently at any size while keeping hyperparameters stable and transferable. This approach simplifies tuning, accelerates training, and allows models to work seamlessly in FP8 without complex rescaling. With u-µP, AI development becomes faster, more predictable, and more scalable.

Tokenizers are essential for encoding text in LLMs, but they face significant limitations, including computational overhead, inefficient vocabulary use, and bias toward reference corpora. Trigram addresses these issues by embedding words through sparse activation patterns over character triplets (“trigrams”), eliminating the need for a reference corpus. This approach exploits morphological similarities and allows for strong compression of embedding layers, achieving competitive downstream performance with over 85% parameter reduction.

Every result or decision of an AI system is based on patterns learned during training and knowledge accessed during inference. AtMan makes these patterns visible and usable. We can show the developer or user how evidence has caused AI outputs, which relationship needs to be added to improve AI systems, and where potentially contradicting observations may hint at a complex context. This transparency of the multidimensional context is crucial to optimize prompts and systems, guide UX and workflows, and for humans to take responsibility.

LLM’s growing size presents deployment challenges. Our innovative Divergent Token Metrics (DTMs) offer a solution by providing a nuanced evaluation of compressed LLMs, overcoming the limitations of traditional metrics. DTMs focus on token divergences, offering insights into the intricacies of model compression, especially when assessing individual components. This innovation ensures efficient deployment while maintaining high performance, making DTMs an essential tool for quantizing LLMs in practical applications.

Text-to-image AI models are powerful but often limited – stuck in a single language and reliant on only text inputs. MultiFusion changes that. By seamlessly integrating pre-trained models, it enables AI to understand multilingual and multimodal prompts, combining text and images in any order. MultiFusion transfers capabilities from existing individual modules, making image generation more expressive, intuitive, and accessible. With this approach, users can create rich, detailed visuals using any mix of words and images, across multiple languages, for a truly global and flexible creative experience.

Vision-language modeling has traditionally relied on tailored pretraining objectives and labeled data, limiting its scalability and flexibility. MAGMA introduces a new approach by augmenting generative language models with visual inputs through adapter-based finetuning. This method keeps the language model’s weights unchanged, preserving its encyclopedic knowledge and in-context learning abilities. By leveraging a simple next-token prediction objective, MAGMA achieves state-of-the-art results on the OKVQA benchmark and performs competitively across various VL tasks, all while using significantly less data than existing methods.