Google Gemini new functions and features

Okay, let’s check it out!

AI Overviews coming now

Google kicked off the I/O 2024 event with a major announcement: the rollout of its Search Generative Experience (SGE) labs feature to US users, scheduled within the week.

AI Overviews will automatically answer specific searches in the US, offering concise explanations at the top of search results pages before the traditional list of links. Over the next few days, hundreds of millions of users in the US will experience AI overviews, with plans to expand to over a billion users worldwide by the end of the year.

AI Overviews will help with increasingly complex questions. Rather than breaking your question into multiple searches, you can ask your most complex questions, with all the nuances and caveats you have in mind, all in one go.

For example, maybe you’re looking for a new yoga or pilates studio, and you want one that’s popular with locals, conveniently located for your commute, and also offers a discount for new members. Soon, with just one search, you’ll be able to ask something like “find the best yoga or pilates studios in Boston and show me details on their intro offers, and walking time from Beacon Hill.”

Beyond finding the right answer or information for a complex question, Search will also be able to plan with you.

With planning capabilities directly in Search, you can get help create plans for whatever you need, starting with meals and vacations. Search for something like “create a 3-day meal plan for a group that’s easy to prepare,” and you’ll get a starting point with a wide range of recipes from across the web.

With advancements in video understanding, you can now search using videos. For instance, if you bought a record player at a thrift shop and the needle arm is drifting unexpectedly, you can simply search with a video of the issue. This saves you from trying to describe the problem in words and provides an AI Overview with troubleshooting steps and resources.Video search will soon be available for Search Labs users in the U.S. in English, with plans to expand to more regions over time.

Talk to your gallery with Ask Photos

Gemini is making its way further into the Photos app, which soon will be able to complete tasks you tell it to do.

In the upcoming months, Google Photos will introduce context-aware voice and text prompts to help users search for specific images or details within images. The Ask Photos feature goes beyond conventional image searches by utilizing Gemini to recognize image content. For example, it can detect a car license plate and prompt users to inquire about a specific plate number on a particular car model, providing accurate identification.

The rollout of Ask Photos is expected to begin in the coming months, with a tentative release timeframe set for summer.

“Double the tokens, please!”

Pichai also revealed that Gemini 1.5 Pro, the newest iteration of its AI model, will now be accessible to all users through the Gemini Advanced app. The public version comes with a context window of 1 million tokens. Additionally, Google has upgraded Gemini 1.5 Pro to handle 2 million tokens, but this feature will be limited to developers in a private preview.

In AI, a token is like a building block or a piece of a puzzle. It’s a small unit of information that represents something meaningful, like a word or a part of a sentence. Tokens help AI understand and process language by breaking it down into manageable pieces, making it easier for computers to analyze and generate text.

• Summarize emails: With this feature, Gemini can analyze email threads and provide a summarized view directly in the Gmail app. Simply tap the summarize button at the top of your email thread to get the highlights. This will be available to Workspace Labs users this month, and to all Gemini for Workspace customers and Google One AI Premium subscribers next month.
• Contextual Smart Reply: Soon, Gemini in Gmail will offer even more detailed and nuanced suggested replies based on context from your email thread. With Contextual Smart Reply, you can edit or simply send as-is. This will be available to Workspace Labs users on mobile and web starting in July.
• Gmail Q&A: Soon when you click the new Gemini icon in the mobile app, Gemini in Gmail will offer helpful options, like “summarize this email,” “list the next steps” or “suggest a reply.” And similar to the side panel on desktop, you can use the open prompt box when you have more specific requests. For instance, you could ask Gemini to “find the bid from the roofing contractor” that’s buried somewhere in your inbox. Gmail Q&A will be available to Workspace Labs users on mobile and web starting in July.

Google is improving NotebookLM, its AI tool for understanding documents, by adding “audio overviews” that create a podcast-style conversation between two speakers.

This upgrade is great for people who prefer learning by listening rather than reading. In a demo, NotebookLM was given some physics lessons to work with. It then made a conversation between two speakers, explaining how basketball relates to the physics topic, like force, when asked by Google’s Josh Woodward.

Google is introducing a new model called Gemini 1.5 Flash, designed to be fast and efficient.

Gemini 1.5 Flash is “great at summarizing, chatting, captioning images and videos, extracting data from long documents and tables, and more,” wrote Demis Hassabis, CEO of Google DeepMind, in a blog post. Hassabis explained that Google made Gemini 1.5 Flash because developers wanted a model that was lighter and cheaper than the Pro version announced in February. 

Gemini 1.5 Flash is in between Gemini 1.5 Pro and Gemini 1.5 Nano, Google’s smallest model that runs directly on devices. Even though it’s lighter than Gemini Pro, it’s still powerful.

Also, Google announced two new AI tools for media creation: Veo, which can create high-quality 1080p videos, and Imagen 3, the latest version of its text-to-image framework.

Google says Veo understands natural language and visual concepts to generate the video you want. These AI-generated videos can be over a minute long and include advanced cinematic techniques like timelapses.

Imagen 3 is described as Google’s highest-quality text-to-image model, producing highly detailed and photorealistic images with fewer errors. Google claims Imagen 3 is better at understanding and managing detailed prompts and handles text more effectively than previous versions.

Trillium offers a 4.7 times increase in performance per chip compared to the previous TPU v5e, with double the memory and bandwidth. It includes a third-generation SparseCore accelerator for processing large data sets in ranking and recommendation tasks.

Google claims Trillium can train AI models faster with lower latency and cost, and it’s their most energy-efficient TPU yet, using 67% less energy than the previous version.

Full Multimodal Capabilities Coming to Gemini Nano

Android is set to become the first mobile operating system to feature a built-in, on-device foundation model with the introduction of Gemini Nano. This innovation aims to deliver fast and secure experiences while keeping user information private. Starting with Pixel devices later this year, the latest model, Gemini Nano with multimodality, will be launched. This upgrade will enable phones to process not only text input but also understand contextual information such as sights, sounds, and spoken language.

Picture this scenario: You tell Gemini you’re heading to Miami for Labor Day with your family. Your son loves art, and your husband craves fresh seafood. Can Gemini pull flight and hotel details from your Gmail and assist in planning the weekend?

Gemini does more than just provide generic suggestions. It considers your flight schedule, dining preferences, and local attractions. By accessing your Gmail for flight information, tapping Google Maps for nearby restaurant and museum suggestions, and utilizing Search for additional activities, Gemini creates a personalized itinerary. Whether it’s a walking tour of the Design District or beach time, Gemini ensures your day is filled with activities that match your interests. Plus, the itinerary updates automatically if you make changes or add more details.

This dynamic planning experience will be available on Gemini Advanced in the coming months.

Personalized Gems and Live for Gemini Advanced

Gemini Advanced subscribers will soon have the option to create Gems for an even more personalized experience. Gems are customized versions of Gemini tailored to your preferences. Whether you need a gym buddy, sous chef, coding partner, or creative writing guide, Gems can be designed to suit your needs.

Creating a Gem is straightforward. You simply describe what you want your Gem to do and how you want it to respond. For example, you could request a running coach to provide daily plans with a positive and motivating attitude. Gemini will then take your instructions and, with a single click, create a Gem that fulfills your specific requirements.

Also, Google is introducing new ways to interact with Gemini more naturally, whether you’re texting or speaking. With Gemini in Google Messages, you can now chat with it within the same app you use to message your friends.

In the upcoming months, the tech giant will be launching Live for Gemini Advanced subscribers, offering a new mobile conversational experience. This feature utilizes cutting-edge speech technology to make conversing with Gemini more intuitive. With Gemini Live, you can engage in a conversation with Gemini and choose from various natural-sounding voices for its responses. You can also speak at your own pace or interrupt with clarifying questions, mimicking a real conversation.

For instance, if you’re preparing for a job interview, you can go Live and ask Gemini to assist you. It can help you rehearse and even suggest skills to emphasize during the interview. Later this year, you’ll also be able to use your camera during Live sessions, enabling discussions about your surroundings.

Since its debut at Samsung Unpacked in January 2024, Circle to Search has been enhanced with new features such as full-screen translation, and its availability has been extended to more Pixel and Samsung devices.

As of today, Circle to Search can assist students with their homework, providing them with a deeper understanding rather than just delivering answers, directly from their phones and tablets. When students encounter a problem they’re stuck on, circling the prompt prompts Circle to Search to offer step-by-step instructions for solving a range of physics and math word problems, all without leaving their digital materials. Later this year, Circle to Search will expand its capabilities to solve even more complex problems involving symbolic formulas, diagrams, graphs, and beyond.

Currently available on over 100 million devices, Circle to Search aims to double its reach by the end of the year, with plans to extend the experience to more devices.