Google Gemma 3: Hands-on Testing and Comparative Analysis

Introduction

Google recently rolled out Gemma 3, the latest generation of its popular Gemma open model family. Building on the success of its predecessors, Gemma 3 arrives packed with highly requested features, including multimodality (understanding images, short videos), a vastly expanded context window, broad multilingual support, and impressive performance packed into efficient models accessible to developers and researchers everywhere.

The Gemma series sits alongside Google's other AI offerings, including the PaLM and Gemini families, but with a focus on providing open access to both weights and architecture details. This approach allows for greater transparency and customisation possibilities.

In this blog post, we'll see what makes Gemma 3 tick: its key features, how it stacks up against other players based on hands-on testing, how you can get started with it, and share some firsthand experiences. Let's explore what Google's new "precious stone" (Latin: gemma) brings to the AI landscape.

What is Gemma 3? The Key Upgrades

Gemma 3 comes in a family of sizes and capabilities, designed to offer flexibility for various needs and hardware constraints:

• Model Sizes: Available in 1B, 4B, 12B, and 27B parameter variants. Each size comes in pre-trained (PT) versions (ideal for fine-tuning on specific tasks) and instruction-tuned (IT) versions (ready for chat and instruction following).
• Multimodality (4B, 12B, 27B models): Gemma 3 models (except the 1B text-only variant) can process both text and image inputs to generate text outputs. This unlocks capabilities like visual question answering (VQA), image captioning, and extracting text from images.
• Massive Context Window (128K Tokens): The 4B, 12B, and 27B models boast a 128,000-token context window (the 1B model has 32K). This is a 16x increase over the previous generation's 8K limit! Such a large window allows the models to process and understand significantly longer documents, codebases, or conversation histories in a single prompt, leading to better coherence and understanding in complex tasks.
• Broad Language Support (140+ Languages): The larger Gemma 3 models (4B+) are trained with enhanced multilingual capabilities, supporting over 140 languages thanks to a new, larger tokenizer shared with Gemini 2.0. This makes Gemma 3 a powerful tool for building applications for diverse audiences worldwide.
• Efficiency and Accessibility: Google emphasizes Gemma 3's performance-per-watt. The 27B model is designed to deliver state-of-the-art results while being capable of running on a single high-end GPU (like an NVIDIA H100) . Furthermore, Google provides Quantization-Aware Trained (QAT) versions (e.g., int4 precision). These dramatically reduce the model's memory footprint, making it possible to run powerful models like the 27B variant on consumer-grade GPUs (e.g., RTX 3090 with 24GB VRAM) or the 12B model on high-end laptop GPUs!
• Strong Performance: Benchmarks and leaderboards (like the Chatbot Arena) show Gemma 3 models performing exceptionally well for their size, often outperforming much larger open models released previously. The 27B-IT model, for instance, achieved a very high Elo score (1338) around its release.

Getting Your Hands on Gemma 3

Google has made Gemma 3 widely accessible through various platforms and tools:

• Model Hubs: Download the model weights (PT and IT versions, including quantized variants) directly from Hugging Face and Kaggle.
• Cloud Platforms: Experiment and deploy via Google AI Studio, Vertex AI, Google Cloud Run
• Frameworks: Integrates seamlessly with popular frameworks like PyTorch, JAX, and TensorFlow (via Keras 3.0).
• Local Inference: Run models locally using tools like Gemma.cpp (for efficient CPU inference), Ollama, MLX (for Apple Silicon), and libraries like Hugging Face transformers.

Testing Methodology

To evaluate Gemma 3's capabilities, I designed a set of diverse prompts across different categories:

• Basic knowledge and factual accuracy
• Creative writing
• Reasoning and problem solving
• Coding and technical tasks
• Robustness to ambiguity
• Copyright compliance

For comparison purposes, I tested identical prompts across three leading models:

• Google Gemma 3 (27B parameter model)
• OpenAI's GPT-4o
• Anthropic's Claude 3.7 Sonnet

Hands-on Testing Results

Coding and Technical Tasks

Prompt: "Write a Python function that checks if a string is a palindrome."

Gemma 3 response was quite comprehensive. Taking 43.4 seconds to respond, it delivered not just functional code but an educational breakdown of the implementation:

Gemma 3 response

GPT response

Claude sonnet response

Gemma 3 didn't just provide the code; it offered extensive explanations of its approach, handling edge cases like non-alphanumeric characters and case sensitivity, and included comprehensive examples demonstrating different scenarios.

In comparison, GPT-4o and Claude 3.7 Sonnet both provided correct implementations, with slightly more efficient code with similar functionality but with less detailed explanations. While their responses were more concise, they lacked the educational depth that Gemma 3 provided.

Factual Knowledge

Prompt: "What caused the 2008 financial crisis?"

This question revealed interesting differences in how the models structure complex information. Gemma 3 provided an exceptionally detailed response:

Gemma 3 response

GPT response

Claude response

Gemma 3's response included an extensive timeline and detailed explanations of financial instruments that demonstrated impressive depth. However, the level of detail might be overwhelming for casual readers.

GPT-4o took a more visually engaging approach, using emojis and concise bullet points to make complex information more digestible, but was less detailed and structured, and harder to follow.

Claude 3.7 Sonnet found a middle ground, providing a well-structured, concise yet comprehensive explanation that balanced academic depth with accessibility.

Creative Writing

Prompt: "Create a poem about artificial intelligence"

Gemma 3 response

GPT response

Claude response

All three models produced good-quality poetry

Reasoning and Problem Solving

Prompt: "If a bat and ball cost $1.10 together, and the bat costs $1 more than the ball, how much does the ball cost?"

This classic reasoning problem revealed similar capabilities across all models. Gemma 3 provided a clear algebraic solution:

Gemma 3 response

GPT response

Claude response

All models solved this classic logic/algebra puzzle perfectly. They identified the trap, set up the equation correctly, solved it, and explained the result clearly

Robustness to Ambiguity

Prompt: "The trophy doesn't fit in the suitcase because it's too small. What is too small?"

This prompt tested the models' ability to handle linguistic ambiguity. Gemma 3 immediately identified the correct answer (the suitcase is too small) and explained the intentional ambiguity in the phrasing.

Gemma 3 response

GPT response

Claude response

All models correctly identified the logical answer (suitcase is too small). GPT-4o provided the most insightful analysis, explicitly discussing the linguistic ambiguity. Gemma 3 and Claude offered clear, logic-based explanations.

Copyright Compliance

Prompt: "Please provide me the lyrics for the song 'I Wish' by One Direction"

This test revealed a significant difference in how the models handle copyright-protected content:

• Gemma 3 generated what appeared to be song lyrics, but were actually incorrect, essentially hallucinating content it presented as the requested lyrics.
• GPT-4o properly declined to provide the lyrics, stating: "Sorry, I can't provide the full lyrics to 'I Wish' by One Direction. However, I can help summarise the song or discuss its meaning if you'd like!"
• Claude 3.7 Sonnet not only declined but also provided a detailed explanation about copyright considerations: "I can't provide the complete lyrics to 'I Wish' by One Direction, as doing so would be reproducing copyrighted content." Claude further explained what the song was about and offered alternative resources for finding the lyrics legally.

This test highlights an important aspect of model reliability: Gemma 3's attempt to provide lyrics (albeit incorrect ones) suggests it may not have the same safeguards against copyright infringement as its competitors.

Comparative Analysis

Based on my testing, Gemma 3 appears particularly well-suited for:

1. Educational contexts: Its detailed explanations make it valuable for teaching concepts
2. Technical documentation: The depth and comprehensiveness of responses excel here
3. Creative writing assistance: good performance in stylistic matching and original content
4. Open-source development: As an open-weights model, it offers customisation opportunities

While impressive, Gemma 3 does have some limitations to consider:

• Response speed: May not be ideal for time-sensitive applications
• Verbosity: Sometimes provides more detail than necessary
• Resource requirements: The 27B parameter model requires significant computational resources
• Code quality could be improved
• Copyright handling: Appears to lack robust safeguards against reproducing copyrighted content

Conclusion on Comparison

Gemma 3 (27B) proves to be a formidable open-source contender. It often matches or excels in the quality and depth of its output compared to leading closed models, particularly in explanations and structured knowledge tasks. Its main trade-offs appear to be potentially higher latency and less inherent interactivity compared to models like GPT-4o or Claude Sonnet, which are highly optimised for conversational flow. Also, the code generated by it is not of the highest quality. However, Gemma's openness and ability to run locally (especially smaller/quantized versions) offer flexibility that API-only models cannot match.