How to Build an Android XR App for Smart Glasses From Scratch

Google has tried wearable AR before. Glass Enterprise Edition carved out a small niche in warehouses and operating rooms, but it never became a consumer platform. Android XR is a completely different animal. Announced with Samsung, Qualcomm, and a roster of OEM partners, Android XR brings the full weight of the Android ecosystem to smart glasses and headsets. That means billions of existing Google Play users, familiar developer tooling, and an open hardware spec that lets multiple manufacturers ship competing devices.

The first wave of Android XR glasses from Samsung, launched in mid-2031, sold over 3 million units in the first quarter. Qualcomm's XR2+ Gen 3 chip gives these glasses enough compute for real-time spatial mapping, on-device ML inference, and smooth 3D overlay rendering at 90fps. Unlike Apple's closed ecosystem or Meta's vertically integrated approach, Android XR lets any OEM build glasses that run the same apps. That fragmentation is a double-edged sword for developers, but it also means the addressable market is enormous.

We have been building Android XR apps at Kanopy since the developer preview dropped, and this guide distills everything we have learned into a practical, step-by-step walkthrough. If you have already shipped apps for Meta Orion or Apple Vision Pro, some spatial computing concepts will transfer, but Android XR has its own SDK, its own interaction model, and its own set of hardware constraints you need to respect.

Before you write a line of code, you need to understand the hardware your app will run on. Android XR glasses are not phones with a head strap. They are power-constrained, thermally limited, optically complex devices with unique input methods. Ignoring these constraints leads to apps that overheat the device, drain the battery in an hour, or render text so small it is illegible through a waveguide display.

Display Technology

Samsung's Galaxy Glass XR uses a micro-LED waveguide with a 60-degree diagonal field of view and an effective resolution of about 1440x1080 per eye. That is sharp enough for UI elements, text overlays, and lightweight 3D objects, but it will not compete with a phone screen for media consumption. Colors look excellent indoors. In bright sunlight, you lose contrast fast, so your app needs an adaptive brightness mode that boosts opacity and switches to high-contrast color palettes when ambient light exceeds a threshold. The SDK provides a light sensor API that makes this straightforward.

Compute and Thermal Envelope

The Qualcomm XR2+ Gen 3 delivers roughly the GPU performance of a Snapdragon 8 Gen 2 phone, which is solid for 2D overlays and moderate 3D rendering. The catch is thermal throttling. A phone can dissipate heat through its metal chassis and your hand absorbs some of that warmth. Glasses sit on your face. There is almost no thermal mass. Sustained heavy GPU usage triggers throttling within 8 to 12 minutes, dropping frame rates from 90fps to 45fps or lower. Design your rendering budget around 60% of peak GPU capacity to avoid thermal cliffs during extended sessions.

Input Methods

Android XR glasses support four input channels: voice commands via Google Assistant integration, a companion controller (sold separately), hand tracking via outward-facing cameras, and a touchpad built into the right temple of the glasses. Most users will rely on voice and the temple touchpad for daily interactions. Hand tracking works well in controlled environments but struggles outdoors when hands are partially occluded or when the user is carrying something. Design your primary navigation around voice and touchpad, then layer in hand tracking as a secondary input for spatial interactions like grabbing and placing virtual objects.

Battery Life

Expect 3 to 5 hours of mixed usage depending on display brightness and GPU load. A lightweight notification overlay app can stretch to 6 hours. A constantly-on navigation app with full scene understanding will drain the battery in under 2.5 hours. The SDK provides a power budget API that tells you in real time how many milliwatts your app is consuming. Use it aggressively during development to catch power regressions early.

Google built the Android XR SDK as an extension of the standard Android development stack. If your team already builds Android apps, the ramp-up is significantly faster than switching to an entirely new platform like visionOS or the Meta Horizon SDK. Here is exactly what you need to install and configure.

Android Studio with XR Plugin

Download Android Studio Ladybug (2031.3) or later and install the Android XR plugin from the SDK Manager. The plugin adds an XR emulator, spatial scene editor, and XR-specific Lint rules that flag common mistakes like blocking the main thread during anchor resolution or exceeding the recommended polygon budget. The XR emulator alone requires at least 16GB of RAM and a dedicated GPU with Vulkan support. On an M-series Mac or a modern Windows workstation with an RTX 4060 or better, the emulator runs smoothly. On anything less, expect stuttering that makes testing spatial interactions painful.

Jetpack XR Libraries

Google ships the core XR functionality as a set of Jetpack libraries, which means you pull them in via Gradle just like any other Android dependency. The key libraries are: androidx.xr.scenecore for spatial anchors, scene understanding, and 3D rendering; androidx.xr.compose for building spatial UIs with Jetpack Compose in 3D space; androidx.xr.input for gesture recognition, gaze tracking, and controller events; and androidx.xr.arcore for the updated ARCore pipeline that handles SLAM, plane detection, and depth mapping on glasses hardware. Pin your dependency versions explicitly. The XR libraries are still shipping breaking changes between minor versions, and auto-updating will burn you.

Kotlin and Compose for Spatial UI

The recommended language is Kotlin, and the recommended UI framework is Jetpack Compose with the XR Spatial extensions. If you have built Compose UIs for phones or tablets, the mental model is similar: you declare composable functions that describe your UI, and the framework handles rendering. The difference is that your composables now exist as panels floating in 3D space rather than filling a flat screen. You position panels using spatial anchors, and users interact with them via gaze, touch, or voice. The learning curve from standard Compose to Spatial Compose is about two to three weeks for an experienced Android developer.

Physical Device for Testing

The XR emulator covers about 65% of testing needs. For the remaining 35%, you need real hardware. Samsung sells the Galaxy Glass XR Developer Edition for $1,299, which includes the glasses, a companion controller, a charging case, and early access to beta firmware. Google also offers a loaner program for registered XR developers, though the waitlist is currently 6 to 8 weeks. Do not attempt to ship a polished XR app without testing on a real device. We have seen rendering bugs, gesture recognition failures, and spatial drift issues that only appear on physical hardware because the emulator's sensor simulation is too clean.

Every Android XR glasses app relies on three foundational SDK systems. Getting these right early prevents the kind of architectural rewrites that push timelines back by weeks. Let us walk through each one with practical implementation details.

Spatial Anchors and Persistence

A spatial anchor pins a virtual object to a specific point in the physical world. When a user places a virtual checklist on their refrigerator, that checklist's 3D position, orientation, and scale are stored as a spatial anchor. Android XR supports two types: session anchors (ephemeral, lost when the app closes) and cloud anchors (persisted via Google Cloud and shareable across devices). Session anchors are free and unlimited. Cloud anchors cost nothing for the first 10,000 resolutions per month, then $0.01 per resolution after that. For most apps, session anchors are sufficient. Use cloud anchors only when you need multi-user shared experiences or cross-device persistence.

Anchor accuracy depends heavily on the environment. In a well-lit room with textured surfaces, anchors hold position to within 1 to 2 centimeters. In dim or featureless spaces, drift can exceed 10 centimeters. The SDK provides an anchor quality callback that reports confidence scores in real time. If confidence drops below 0.6, show the user a re-localization prompt rather than letting virtual objects float away from their intended positions. We learned this the hard way on a retail shelf-labeling project where anchors drifted in aisles with uniform white shelving.

Scene Understanding Pipeline

Android XR's scene understanding system identifies planes (floors, walls, tables, ceilings), meshes (arbitrary 3D surfaces), and semantic labels (door, chair, window, screen). The pipeline runs on a dedicated DSP core, which means it does not compete with your app's GPU budget, but it does consume significant battery. You should enable scene understanding only when your app needs it and disable it during idle states.

The plane detection API returns plane boundaries as convex polygons with surface normal vectors. This is what you use to place UI panels on tables, mount navigation arrows on walls, or detect the floor for ground-plane interactions. Mesh data is denser and more detailed, suitable for occlusion rendering where virtual objects need to appear behind real-world surfaces. The semantic labeling system identifies about 30 object categories with roughly 85% accuracy in indoor environments.

Jetpack Compose Spatial Panels

This is where Android XR diverges most from other spatial platforms. Instead of building UIs with Unity or a custom rendering engine, you build them with Jetpack Compose. A spatial panel is essentially a Compose layout that floats in 3D space. You define its content using standard Compose components (Text, Button, Image, LazyColumn) and position it using a SpatialAnchor or relative to the user's head position. Panels can be transparent, semi-transparent, or opaque. They respond to gaze focus, touchpad taps, and voice commands just like on-screen elements respond to touch on a phone.

The performance characteristics are excellent for 2D content. A panel with a scrollable list of 200 items renders at a locked 90fps with no perceptible jank. 3D content embedded in panels (using the SceneView composable) is more demanding, but for most information-display use cases, 2D panels are the right choice. Save 3D rendering for spatial objects that genuinely need to exist in the physical world, like navigation arrows or product visualizations.

Theory is useful, but you learn a platform by building on it. Let us walk through creating a practical app: a contextual note-taking tool that lets users pin voice memos to physical locations. When they return to that location, the notes reappear. This covers spatial anchors, voice input, Compose panels, and persistence, which are the building blocks of most XR glasses apps.

Step 1: Project Setup and Manifest Configuration

Create a new Android project in Android Studio targeting API level 35 (Android XR). In your app manifest, declare the XR-specific permissions: SCENE_UNDERSTANDING for spatial mapping, SPATIAL_ANCHOR for placing persistent objects, MICROPHONE for voice input, and CAMERA for environment tracking. You also need to declare the XR feature requirement so the Play Store only shows your app to XR-capable devices. Add the Jetpack XR dependencies to your build.gradle file, making sure to use the same version across all XR libraries to avoid binary compatibility issues.

Step 2: Initialize the XR Session

In your main Activity, create an XrSession object and request the capabilities your app needs. The session is the gateway to all spatial APIs. Initialize it in onCreate, passing a configuration that specifies which subsystems to activate: spatial anchors, scene understanding, and input. The session creation is asynchronous and can fail if the hardware does not support a requested capability, so handle the failure callback by falling back to a reduced feature set or showing an informative error.

Step 3: Detect Surfaces and Offer Anchor Placement

Enable plane detection and listen for plane updates. When the system detects a vertical surface (like a wall or whiteboard), highlight it with a subtle outline rendered via a SpatialPanel with a translucent border. When the user taps the temple touchpad while looking at a detected surface, create a spatial anchor at the gaze intersection point. Store the anchor's UUID and the associated note data in a local Room database so you can restore everything on the next app launch.

Step 4: Capture and Display Voice Notes

When an anchor is created, immediately start recording audio via the MediaRecorder API. Show a small Compose panel near the anchor with a recording indicator and a timer. When the user taps the touchpad again, stop recording and transcribe the audio using Google's on-device speech recognition, which runs locally without a network round-trip and delivers transcripts in under 500ms for clips under 30 seconds. Display the transcript in a floating panel attached to the anchor, with options to play back the original audio or edit the text.

Step 5: Restore Notes on Return

When the app launches, query the Room database for all saved anchors and attempt to resolve each one against the current environment. The ARCore pipeline compares saved visual features against the live camera feed to determine if the user is near any previously created anchors. Resolved anchors trigger a callback with the anchor's 3D pose, at which point you instantiate the corresponding Compose panel with the saved note content. Anchors that fail to resolve (because the user is in a different location) are silently skipped and retried periodically in the background.

Step 6: Polish and Edge Cases

Real apps need to handle interruptions gracefully. What happens when the user takes the glasses off mid-recording? Save a draft automatically. What if two anchors are so close together that their panels overlap? Implement a collision avoidance system that nudges panels apart or stacks them vertically. What about low-light environments where anchor resolution fails? Show a toast suggesting the user move to a better-lit area. These edge cases separate a demo from a shippable product, and they typically consume 30 to 40% of total development time.

An XR glasses app that stutters, overheats the device, or drains the battery in an hour will be uninstalled immediately. Performance is not a nice-to-have on this platform. It is the difference between a usable app and an expensive tech demo. Here are the specific techniques we use at Kanopy to keep Android XR apps running smoothly.

Frame Budget Management

At 90fps, you have 11.1 milliseconds per frame to handle input, run logic, and render the scene. In practice, you should target 8ms to leave headroom for system processes and garbage collection. Use Android Studio's XR Profiler to measure per-frame timings broken down by CPU and GPU. The most common performance killer is overdraw: rendering transparent panels on top of each other forces the GPU to blend pixels multiple times. Keep your panel count under 10 simultaneously visible panels, and avoid stacking transparent panels whenever possible.

Thermal Throttling Prevention

Monitor the device's thermal state using the PowerManager API. When the thermal status reaches THERMAL_STATUS_MODERATE, reduce your rendering complexity: drop shadow quality, disable anti-aliasing, reduce scene understanding update frequency from 30Hz to 10Hz. At THERMAL_STATUS_SEVERE, consider pausing non-essential rendering entirely and showing a minimal "cooling down" UI. We build thermal state monitoring into every XR project from day one because retroactively adding thermal management to a codebase is painful and error-prone.

Memory Management

Android XR glasses typically have 8 to 12GB of RAM, but the system reserves a significant portion for its own spatial computing processes. Your app's usable memory ceiling is around 2 to 3GB. The biggest memory traps are 3D model loading (always use compressed formats like glTF with Draco compression), texture atlasing (combine small textures into larger atlases to reduce draw calls), and scene understanding mesh data (prune mesh triangles beyond a 5-meter radius to avoid unbounded memory growth). Use LeakCanary during development to catch memory leaks early.

Testing on Multiple Devices

The Android XR ecosystem already includes glasses from Samsung, XREAL, and Lenovo, each with different display characteristics, thermal profiles, and sensor configurations. If you are targeting the Play Store, test on at least two different devices. Samsung's Galaxy Glass XR and XREAL's Air 3 XR are the two highest-volume devices as of late 2031. The differences in field of view, display brightness, and gesture recognition sensitivity are significant enough to cause layout issues and interaction bugs that only appear on one device. Google's XR compatibility testing service (part of Firebase Test Lab) lets you run automated UI tests on real devices in the cloud, which helps but does not replace manual testing for spatial interactions.

Building an Android XR glasses app is not cheap, but it is more accessible than developing for Apple Vision Pro or building a fully custom AR hardware solution. Here is a realistic breakdown of what to expect based on the projects we have delivered.

Development Costs

A simple XR overlay app (notifications, weather, calendar HUD) takes 6 to 10 weeks with a two-person team and costs between $40,000 and $70,000. A mid-complexity app with spatial anchors, scene understanding, and custom UI panels takes 12 to 18 weeks with a three-person team, landing between $80,000 and $150,000. A complex spatial app with multi-user shared experiences, cloud anchor synchronization, custom ML models, and polished 3D content can run $200,000 to $400,000 over 6 to 9 months. For a deeper breakdown of pricing across AR and VR platforms, check out our guide to AR/VR development costs.

Team Composition

You need at least one senior Android developer with Kotlin and Compose expertise, one 3D/spatial designer who understands the constraints of waveguide displays, and a QA engineer with access to physical XR hardware. For apps with custom ML features, add a machine learning engineer. For apps with cloud anchor or multi-user features, add a backend developer. The spatial designer role is the hardest to hire for because the discipline is new. Look for designers with experience in AR (ARKit, ARCore), VR (Unity, Unreal), or industrial HMI design.

Play Store Submission for XR

Google requires XR apps to pass a set of platform-specific quality checks before listing on the Play Store's XR section. Your app must maintain 72fps or higher for 95% of a 10-minute test session (measured by their automated testing), respond to the system "pause" event within 500ms, properly release camera and sensor resources when backgrounded, and display content within the safe rendering zone (the central 80% of the field of view). The review process takes 5 to 10 business days, which is longer than a standard Android app review. Submit early and expect at least one round of rejection feedback.

Go-to-Market Strategy

The Android XR app ecosystem is still young, with fewer than 2,000 apps specifically built for glasses. That means organic discovery is high if your app is genuinely useful. Focus on one specific use case rather than building a Swiss Army knife. A navigation app that does indoor wayfinding exceptionally well will outperform a "do everything" AR app that does nothing particularly well. Partner with Samsung's XR developer relations team for potential featuring in the Galaxy Store's XR section, which drives significant download volume for early apps.

If you are ready to build an Android XR smart glasses app and want a team that has already shipped on this platform, book a free strategy call with us at Kanopy. We will help you scope the project, estimate costs, and figure out whether Android XR, Meta Orion, or Apple Vision Pro is the right platform for your specific use case.