PokeGoalsHelper/UI_MODERNIZATION_TASKS.md

UI Modernization Tasks - Capture Mode

Overview

This document outlines the planned modernization of the capture mode UI, transforming it from a basic floating button system to a modern, feature-rich interface.

Implementation Journey & Current Status

✅ What We Successfully Implemented

• Decoupled Architecture: FloatingUIActivity + ScreenCaptureService with Binder communication
• Material 3 Compose FAB: Beautiful, animated FAB with proper Material Design
• Expandable Menu: Smooth animations, haptic feedback, proper Material 3 theming
• Service Communication: Clean Binder-based API for detection commands

❌ Critical Issues Encountered

1. Activity-Based Approach Issues

• Problem: FloatingUIActivity not truly "floating" - positioned like regular activity
• Symptom: Couldn't drag freely, brought up navigation/status bars
• User Feedback: "It's not floating, I can drag it to the left a little but that's it"

2. ComposeView in Overlay Context

• Problem: ViewTreeLifecycleOwner not found when using ComposeView in WindowManager overlay
• Root Cause: ComposeView requires LifecycleOwner, but overlay context has none
• Attempts Made:
  • Created custom OverlayLifecycleOwner implementing LifecycleOwner + SavedStateRegistryOwner
  • Tried ViewTreeLifecycleOwner.set() - API not available
  • Tried View.setTag() with lifecycle IDs - didn't work
  • Simplified approach without explicit lifecycle - still crashed

3. Transparency & System UI Issues

• Problem: Activity showed system bars, wasn't truly transparent
• Symptoms: Navigation bar visible, black status bar, couldn't see underlying apps
• Attempts: Immersive mode, SYSTEM_UI_FLAG_*, WindowInsetsController - partially worked

🔄 Current Architecture State

MainActivity → ScreenCaptureService → FloatingComposeOverlay (BROKEN)
                                   ↓
                              ComposeView + WindowManager (Crashes on lifecycle)

Files Modified:

• FloatingUIActivity.kt - Working Compose UI (but not truly floating)
• FloatingComposeOverlay.kt - Crashes due to lifecycle issues
• ScreenCaptureService.kt - Clean service communication
• AndroidManifest.xml - Activity registration with transparent theme

Phase 1: Core Modernization ⚡

1.1 Material 3 Floating Action Button

Priority: HIGH
Status: PENDING

Convert the current basic floating orb to a proper Material 3 FAB:

• Use Material 3 FAB component with proper elevation and shadows
• Smooth expand/collapse animations using AnimatedVisibility
• Material 3 color theming and dynamic colors
• Proper ripple effects and touch feedback
• Size variants (mini FAB for menu items)

Technical Details:

• Replace Button with FloatingActionButton in Jetpack Compose
• Implement AnimatedVisibility for menu expansion
• Use Material3 theme colors and elevation tokens
• Add haptic feedback for interactions

1.2 Visual Design Modernization

Priority: MEDIUM
Status: PENDING

• Glass-morphism Effects: Semi-transparent backgrounds with blur
• Smooth Animations: Choreographed transitions between states
• Modern Icons: Vector icons with proper scaling
• Color Schemes: Material You dynamic theming
• Typography: Material 3 typography scale

Technical Details:

• Implement blur effects using Modifier.blur()
• Custom animation specs for fluid motion
• Vector Drawable icons with animated state changes
• Dynamic color extraction from wallpaper (Android 12+)

Phase 2: Enhanced Functionality 📊

2.1 Detection Results Overlay

Priority: MEDIUM
Status: PENDING

Real-time overlay showing detection results:

• Bounding boxes with confidence scores
• Class labels with color coding
• Performance metrics (FPS, inference time)
• Detection count by category

Technical Details:

• Custom Canvas drawing for overlays
• WindowManager overlay with proper Z-ordering
• Real-time data binding from detection controller
• Optimized rendering to avoid performance impact

2.2 Live Preview & Status

Priority: MEDIUM
Status: PENDING

• Mini preview window showing current screen region
• Real-time detection status indicators
• Processing queue visualization
• Error state handling with retry options

2.3 Settings Panel Integration

Priority: MEDIUM
Status: PENDING

Built-in settings accessible from floating UI:

• Detection sensitivity sliders
• Class filter toggles with visual preview
• Coordinate transformation mode selection
• Debug options panel

Phase 3: Advanced UX Patterns 🎯

3.1 Gesture-Based Interactions

Priority: MEDIUM
Status: PENDING

• Long Press: Context menu with advanced options
• Swipe Gestures: Quick filter switching
• Drag: Repositioning FAB location
• Double Tap: Quick detection trigger

Technical Details:

• Custom gesture detection using Modifier.pointerInput()
• Haptic feedback patterns for different gestures
• Visual feedback during gesture recognition
• Gesture customization settings

3.2 Contextual Intelligence

Priority: MEDIUM
Status: PENDING

Smart behavior based on current context:

• Auto-hide during active detection
• Context-aware menu options
• Smart positioning to avoid UI occlusion
• Adaptive timeout based on usage patterns

3.3 Accessibility Enhancements

Priority: HIGH
Status: PENDING

• Screen reader compatibility
• Voice commands integration
• High contrast mode support
• Large text scaling support
• Keyboard navigation

Phase 4: Advanced Features 🚀

4.1 Detection Analytics Dashboard

Priority: LOW
Status: PENDING

• Mini-map showing detection regions
• Historical detection data
• Performance trend graphs
• Export detection logs

4.2 Smart Automation

Priority: LOW
Status: PENDING

• Auto-detection based on screen content changes
• Smart filtering based on user behavior
• Predictive UI adaptation
• Background processing optimization

4.3 Integration Features

Priority: LOW
Status: PENDING

• Screenshot annotation and sharing
• Detection result export (JSON, CSV)
• Cloud sync for settings
• Integration with external Pokemon databases

Technical Architecture

Component Structure

FloatingActionButtonUI (Compose)
├── FABCore (Material 3 FAB)
├── ExpandableMenu (AnimatedVisibility)
├── DetectionOverlay (Canvas)
├── SettingsPanel (BottomSheet/Drawer)
└── StatusIndicators (Badges/Chips)

Performance Considerations

• Lazy composition for menu items
• Efficient recomposition boundaries
• Background thread processing
• Memory management for overlays
• Battery optimization strategies

Implementation Timeline

Sprint 1 (Week 1-2): Material 3 FAB conversion Sprint 2 (Week 3-4): Visual design modernization
Sprint 3 (Week 5-6): Detection results overlay Sprint 4 (Week 7-8): Enhanced UX patterns Sprint 5+ (Future): Advanced features as needed

🎯 Critical User Requirements (UPDATED)

Core Floating Behavior

• ✅ "Floating Action Button" - Not basic Button, actual FAB with Material 3 design
• ✅ "Truly floating" - Like CalcIV, draggable anywhere on screen
• ✅ "Not bringing up navigation bar" - System UI completely hidden
• ✅ "Show underlying apps" - Transparent background, no window blocking content
• ❌ "Blocky static developer UI" - Must be polished, smooth, professional

🆕 NEW Requirements (Critical for Implementation)

• 🎯 "Soft dock to screen edges" - When dragged within ~50px of left/right edge, animate to snap against edge
• 🎯 "Touch-through for underlying apps" - User must be able to interact with Pokemon GO UI:
  • Scroll through Pokemon lists
  • Tap buttons, menus, items
  • Pinch to zoom, swipe gestures
  • Only FAB itself should intercept touches, everywhere else passes through

Technical Implications of New Requirements

Edge Docking Implementation

// Pseudo-code for edge snapping:
onDragEnd { finalPosition ->
    val screenWidth = getScreenWidth()
    val snapThreshold = 50.dp.toPx()
    
    val targetX = when {
        finalPosition.x < snapThreshold -> 0f // Snap to left edge
        finalPosition.x > screenWidth - snapThreshold -> screenWidth - fabWidth // Snap to right edge  
        else -> finalPosition.x // Stay where dropped
    }
    
    animateToPosition(targetX, finalPosition.y)
}

Touch-Through Requirements

This is the most complex requirement and changes our approach significantly:

Option A Issues: Activity approach likely won't work - Activities capture all touches within their bounds

WindowManager Requirements:

// Window must have precise touch handling:
WindowManager.LayoutParams(
    WRAP_CONTENT, WRAP_CONTENT, // Only FAB size, not fullscreen
    TYPE_APPLICATION_OVERLAY,
    FLAG_NOT_TOUCH_MODAL or  // Don't capture outside touches
    FLAG_LAYOUT_NO_LIMITS,   // Allow positioning anywhere
    PixelFormat.TRANSLUCENT
)

// FAB must be exactly positioned with minimal window size
// Everything outside FAB boundaries must pass through to underlying app

Revised Option Analysis

Option A: Activity Approach - ⚠️ PROBLEMATIC

• Major Issue: Activities have window boundaries that may interfere with touch-through
• Edge Docking: Possible with window positioning
• Touch-Through: Very difficult - Activity windows tend to capture touches

Option B: WindowManager + Views - ⭐ BEST FOR TOUCH-THROUGH

• Touch-Through: Excellent - Window can be sized exactly to FAB dimensions
• Edge Docking: Perfect control with WindowManager.updateViewLayout()
• Polish: Need to ensure Material 3 styling with Views

Option C: Advanced Compose - ❓ RESEARCH NEEDED

• Touch-Through: Unknown if ComposeView can be made touch-through
• Edge Docking: Would work if lifecycle issues solved
• Complexity: High due to lifecycle issues

RECOMMENDED APPROACH: Option B Enhanced

Given the new requirements, WindowManager + Views is now the clear winner:

1. Precise Touch Handling: Window sized exactly to FAB, everything else passes through
2. Edge Snapping: Full control over window positioning and animations
3. No Lifecycle Issues: Views work perfectly in overlay context
4. Material 3 Polish: Use com.google.android.material.floatingactionbutton.FloatingActionButton

// Implementation approach:
class EnhancedFloatingFAB {
    // Material 3 FloatingActionButton with proper theming
    // ObjectAnimator for smooth edge snapping
    // WindowManager for precise positioning
    // Touch handling only within FAB bounds
}

Success Metrics

• Touch Behavior: FAB interactive, underlying app fully usable
• Edge Snapping: Smooth 300ms animation to screen edges
• Visual Polish: Indistinguishable from Compose Material 3 FAB
• Performance: 60fps animations, no interference with underlying app

---

UPDATED STATUS: Option B (WindowManager + Material Views) now recommended due to touch-through requirements. Ready to implement enhanced version with edge snapping and precise touch handling.