Robotics Teleoperation Operating System
Developing a user-friendly robot interface (RUI) aiming to simplify robot adoption and transform operations in the shipping, security, and journalism industries.
Role / UX Designer
Methodology / Research, User interviews, Discovery, Storyboarding, Wireframes, Interaction Design, Mockups
Concentrations / Remote Operation, Real Time 3D, Computer Vision, Digital Twins, Machine Learning, Robotics, 5G Network
Overview
Our client sought to address operational inefficiencies in large industries by leveraging robotics for remote operations. They wanted to create a user-friendly robot interface (RUI) that would simplify robot adoption and revolutionize business operations. To achieve this, three proofs of concept were developed to address common pain points in different industries.
The Opportunity
The existing interfaces for robots were technical, challenging to use, and lacked cross-compatibility, making it difficult for companies to transition to robotic adoption. The main goal was to design an intuitive RUI that would enable businesses to easily connect, control, and monitor robots from anywhere, thereby transforming the way they operate and grow.
Our RUI's mission is to accelerate the world's transition to robotic automation.
How might we decrease the barrier to leveraging robots to help solve operation inefficiencies?
Areas of Interest
The areas of interest for our RUI were defined by our client and their client partners. The client's focus areas for the RUI were security, freight & logistics, and journalism.
Security
Security firms are finding it more difficult to find and compensate security personnel in an industry with different levels of human risk.
Freight & Logistics
Ports around the world lack efficiency in their operations. There is a clear need to shift to smart ports that utilize robotic automation.
Journalism
Robotic integration could aid in removing journalists from dangerous situations and provide content creation for more immersive media.
Requirements & Constraints
Designing for remote robot control on 5G cellular networks presented several requirements and constraints to consider in our design.
Requirements
-
Must be intuitive and easy to use
-
Need to visualize operational safety
-
Compatibility with different types of robots
-
Controls must be intuitive and familiar
-
Must communicate depth
Contstraints
-
Feed fidelity of feed for users
-
Latency over cellular
-
Cameras/sensors/payload limitations
-
Types of actions that can be reproduced by robots
Research
To validate observations and assumptions made by our client, I along with another designer investigated the industries’ and RUI landscape. Alongside secondary research, we conducted interviews with a specialist in the different areas to supplement our understanding before partnering with a UX researcher to conduct more in depth interviews to gain insight into possible solution approaches.
Participant Title
Security Specialists
Longshoreman, Port of Los Angeles
XR Specialist
Number of Participants
3
1
1
Participant breakdown for user interviews.
Example exploration flow for understanding the logistics of the shipping industry. This flow provided visual aid to determine the areas where robotics could be incorporated.
In addition to gaining expertise on the different working components of each scenario, we were able to validate that with the effect of the pandemic, companies and industries were finding themselves more exposed to inefficiencies, with their previous challenges being amplified. We also found that our client was far from the first in the market to tackle the problem, but current solutions were far from having an option that could enable robotic automation to be widely adopted due to various barriers of entry.
BARRIERS TO ENTRY
Receding Labor Pool
Many companies are struggling to garner interest in a competitive labor market. This is a result of receding labor forces, and an increasing demand for purposeful work with better benefits and work culture.
Cost
Start up and initial operations cost is steep for those who would be interested in robotic automation.
Stigma
Implementation is opposed by the workforce as they do not want to lose their jobs and be replaced by a robot
Technology Limitations
With the state of current cameras, sensors, and payloads, it is difficult for remote users to have the same understanding, comprehension, and awareness of their surroundings as someone in person. Aspects such as depth are difficult to communicate at this time.
Learning Curve
Robot user interfaces are currently not the most intuitive with each robot often requiring special controls to operate. Complicated UI makes it difficult to adopt as additional training and different human skills and resources are required.
Insights
The research identified two key themes for the solution's key selling points: ease of use and universality.
Ease of use
Currently, robotics have a very high barrier of entry. Only those with technical skills can set up and control a robot. Our plan is to reduce this barrier by prioritizing an intuitive and friendly user experience for the RUI platform.
WHAT DOES THIS MEAN?
Connect to any robot from anywhere.
The RUI should allow any user, regardless of technical experience, to easily connect and set up robotic systems for their business operations.
RUI will be available on the cloud to support global access from anywhere.
Control your robots with ease.
RUI will be designed to make operating a robot intuitive to pick up and user friendly.
Providing real time feedback, visual aids, and a digital twin with low latency will be prioritized to ensure our technology doesn’t get in the way of user’s productivity.
Universal
Our aim is to support any robot, with any input control setup. Current RUI require proprietary control input devices that can sometimes be replaced with game controllers or tablets, our goal is to empower users to flexibly integrate robots into their operations.
WHAT DOES THIS MEAN?
Leverage technology to evolve user’s business.
Businesses can optimize their process and operations through RUI’s powerful ML/AI algorithms to evolve their robots’ value.
This will increase the robots’ autonomy to complete tasks and missions without human aid, increasing productivity and reducing costs.
Solve & adapt to unique problems.
RUI will be a flexible platform that can adapt to multiple industries by introducing an app store that will support the creation and distribution of robot applications.
This will provide other developers with the only tools they need to accelerate the robotic market and reduce the barrier to entry to robotic automation.
To define a clear product vision, whiteboarding sessions were held with the client. The solution was divided into four main components:
-
Control Hub (Space): A dynamic digital twin of the work environment, including any robots that work there.
-
Mission Planner: This feature will allow operators to send robots out into designated areas to accomplish specific tasks at specific moments of the day.
-
Teleoperation: Users can remotely connect to a robot and manually operate it.
-
Skill Library: The platform aimed to compile a list of all the skills of a robot using hard code and ML/AI.
Although existing applications could accomplish one or two of the features, we would aim to enhance the RUI experience by combining the above features with Unity’s real time 3D digital twin representations and be aware of user considerations to make the application more accessible to different users
Design
I collaborated extensively to create documentation around defining the scope of each scenario and what each proof of concept would entail. For each demo, I worked with a creative director and another designer on creating flows and maps to work out how each application would fit into our greater RUI solution, then visualize what would be required for automated robotic operations, skill library/ machine learning training operations, mission based operations, and finally the MVP, manual remote operations, for the teleoperated demo.
Example high level flow for the target product. By mapping out on the macro level, the team was able to focus on what areas we would focus on for the demo MVP.
Throughout the process, I participated in creating visual summaries from whiteboarding and envisioning exercises. To communicate the collective vision for the RUI, I provided storyboards for the overall experience. This helped align the Unity and client teams and highlight areas for more clarification.
Storyboards for the journalistic application scenario for the Boston Dynamics Spot Agile Mobile Robot
As the design needed to give users the best understanding of their robot’s surroundings from a remote location, it was imperative for design to work alongside 3D artists, robotics experts, developers, and product. There were many considerations such as how to control cameras and robotic movement in addition to communicating depth, movement, and representing items in the dynamic digital twin in real time to achieve accuracy and reliability. To demonstrate a proof of concept for what the experience might look like, I and another designer worked alongside developers to create an environment where the client and their prospective users could try controlling the robot remotely.
Design proposal for visual aids for communicating statuses and nuances about the relationship of URe5 robotic arm to its surroundings and objects.
Design proposal for visual aids for communicating statuses and nuances about the relationship of URe5 robotic arm to its surroundings and objects.
Example notations and schema for controlling a Boston Dynamics Spot Agile Mobile Robot with joystick and keyboard.
Retrospective
The project aimed to develop a universal solution with ambitious goals, but the lack of a clear vision posed challenges. While the team successfully created demonstrations to secure additional funding, the client’s over-optimism resulted in a lack of clarity and alignment. Much of the project time was spent deciding on the best course of action, as generalized ideas needed to be refined into defined scenarios. The team struggled to prioritize among the three identified scenarios, leading to divided focus and slowed progress due to frequent pivoting.
Additionally, design would have benefited from earlier and more extensive investment in general and user research. Initially, I partnered with another designer to conduct all the research and interviews for each scenario, however, the inability to consult experts and the simultaneous work on multiple scenarios hindered design’s progress. In contrast, collaboration with a dedicated researcher later facilitated the acquisition of more insight to inform the design in each scenario.
Despite the challenges faced, the project proved to be rewarding, pushing myself to collaborate closely with diverse disciplines and gain in-depth knowledge in areas such as digital twins, computer vision, robotics, and machine learning. Although the technical nature of the subject matter initially seemed daunting, it ultimately provided a greater appreciation for designing new and complex experiences.