Unity Mixed Reality Process Stepper Overview Prototype

We are seeking qualified contractor or individual developer to submit proposals for the development of a single-user Unity-based Mixed Reality (MR) prototype targeting the PICO 4 Ultra Enterprise device. The prototype will deliver a hands-free, gaze-controlled, and voice-activated interface to guide users through procedural process-flows using checklists stored in external YAML files.

This solution is designed as an early demonstration application to showcase immersive productivity and training capabilities within enterprise and industrial workflows.

2. Objectives

The objective of this contract is to work with us to design, co-develop, and deliver a working Mixed Reality prototype application on the PICO 4 Enterprise using Unity, that includes the following:

• Hands-free user interaction

• Gaze, gesture, and voice-activated controls

• Modular protocol loading via external YAML

• Real-time performance duration tracking

• Audio feedback and user guidance

3. Functional Scope & Workflow

The contractor will be required to implement the following user workflow within Unity:

3.1 Onboarding & Setup

• The user launches the application on the PICO headset.

• The app loads a set of available procedural workflows from a locally stored YAML files.

• The user selects one protocol via gaze, hand interaction or voice.

3.2 Protocol Start

• The selected protocol displays a title screen with a brief overview and a visible Start Button.
  (Protocol Title and Title Description parsed from the YAML file)

• The protocol can be initiated in three ways:

  • By using a hand gesture to press the "Start" button

  • By maintaining gaze on the button for 3 seconds

  • By saying the phrase "Start the Procedure"

3.3 Protocol Execution

• Upon start:

  • A voice (using Unity Sentis – Piper TTS) plays the protocol introduction, safety instructions, and interaction guidance.
    (Protocol PreRequisites , Safety Instructions and Primary Contact parsed from the YAML file)

  • The application starts screen recording of the session (for review purposes).

• The procedure UI highlights the first 6 steps, with:

  • Full protocol UI positioned right, outside the field of view to avoid clutter.

  • Active step fixed to the bottom of the user's field of view.

  • Navigation methods:

    • Gaze at the step’s checkmark for 3 seconds

    • Voice command: "Step {n} completed"

  • System tracks time spent per step (in milliseconds) and record it in a {ISOdate}-sessionperformance.log file

  • System 

3.4 Dynamic Navigation Rules

• in the right UI Always display:

  • The last 2 completed steps

  • The next 3 steps

• Completed steps are greyed out with a light green tint and a green check mark.

• The active step is highlighted.

• Future steps are greyed out.

3.5 Completion

• After all steps are completed:

  • Show a visual and voice-based summary screen.

  • Return to the protocol selection screen.

4. Technical Requirements & Integrations

The solution must be implemented using Unity (LTS version recommended) and must integrate the following components:

• Unity Base Template provided by client

• PICO Unity Integration SDK

• PICO Unity Live Preview Plugin

• Unity Sentis

  • Whisper Tiny (Speech Recognition)

  • Piper TTS (Text-to-Speech)

• MRTK3 Volumetric UI framework for immersive interaction

• Local file loader and YAML parser for loading protocols

YAML Schema Example:

Protocol Title: "Sterile Compounding Room Startup"


Protocol Title Description: "Validated protocol for initiating sterile manufacturing operations in Grade A/B Cleanroom."


Protocol Introduction: >
  This protocol outlines the validated steps required to initiate operations in a sterile production suite. 
  All steps must be completed in sequence and signed off digitally or via voice/gaze control.


Protocol PreRequisites: >
  Ensure personnel are properly gowned according to SOP-001-A.
  Confirm environmental monitoring system is operational.
  Confirm HVAC and HEPA filtration systems have been running continuously for a minimum of 30 minutes prior to start.


Protocol Safety Instructions: >
  All operators must adhere strictly to aseptic techniques.
  No unauthorized materials may enter the Grade A zone.
  Report any anomalies to the QA Supervisor immediately.


Protocol Primary Contact: "Dr. Lisa Meinhart, QA Supervisor, Ext. 2205"


Protocol Steps:
  - step: 1
    title: "Enter Gowning Room"
    instructions: "Follow full-body sterile gowning procedure according to SOP-001-A. Use mirror to verify full coverage."
  
  - step: 2
    title: "Perform Initial Cleanroom Disinfection"
    instructions: "Using sterile wipes saturated with sporicidal agent, disinfect all high-touch surfaces in the Grade B area."


  - step: 3
    title: "Activate Cleanroom Monitoring Systems"
    instructions: "Check and activate particle counter, differential pressure monitors, and temperature/humidity sensors. Log values."


  - step: 4
    title: "Prepare and Stage Materials"
    instructions: "Transfer pre-approved sterile materials into the pass-through chamber. Wipe down each item with 70% IPA."


  - step: 5
    title: "Perform Airflow Smoke Test (Visual Confirmation)"
    instructions: "Use sterile smoke wand to visualize unidirectional airflow across work surfaces. Confirm laminar flow integrity."


  - step: 6
    title: "Verify HEPA Filter Functionality"
    instructions: "Check filter status indicators or refer to Building Management System (BMS) for real-time performance data."


  - step: 7
    title: "Confirm Line Clearance"
    instructions: "Verify that the production suite is free from any previous batch remnants, with QA sign-off completed."


  - step: 8
    title: "Document Environmental Parameters"
    instructions: "Log temperature, RH%, differential pressure readings in the batch record or eBR."


  - step: 9
    title: "Sanitize Gloves and Enter Grade A Zone"
    instructions: "Sanitize sterile gloves with 70% IPA spray before entering the aseptic processing area."


  - step: 10
    title: "Begin Batch Execution"
    instructions: "Initiate batch record by scanning lot ID and confirming material and equipment readiness."

5. Deliverables

The appointed contractor will deliver the following:

• ✅ Fully functional Unity prototype as described

• ✅ Integration with YAML parser

• ✅ Working interaction models (voice, gaze, gesture)

• ✅ Time tracking per step

• ✅ Codebase with documented architecture regularly pushed to Github for version control and collaboration,

• ✅ Unity project files and build packages

• ✅ Technical handover and walkthrough session

6. Timeline

We expect a delivery window of 10 days, starting from contract signature.