Augmented reality is a promising field of study which aims to enhance our daily experiences by overlaying digital information into our view of the world, thus “augmenting” our perception of reality in useful ways. A key component of AR is computer vision which enables a computer to map and interpret the physical world. My proposed experiment will examine the effectiveness of several computer vision algorithms in balancing a pendulum attached to a movable cart. A camera, suspended by the pendulum, will collect observational data and relay this information to a microcontroller. Then I will utilize Hough Transform and Canny Edge Detection algorithms to recognize stable lines/edges in the environment, so a reference frame can be constructed in which to balance the pendulum. As the pendulum tips and the camera begins to fall, the system will compare the input frame to the reference frame and adjust the cart accordingly via servo motors. My research will compare several methods of video tracking and object recognition as well as SLAM (Simultaneous Localization and Mapping) as viable options to balance and orient systems with the real world. By designing a system that dynamically responds to its environment using visual cues, I aim to contribute to a critical component in the realization of emerging technologies such as autonomous vehicles, domestic robots and augmented reality systems.

This work seeks to address some problems faced by deaf and hard of hearing persons who work in confined or hazardous spaces. The issue they face that I am focusing on is the timely access of alerts and alarm, which are commonly provided solely through sound. Many approaches used in more general work conditions, such as flashing wall lights or another person notifying them, are heavily impeded by a confined work area. Additionally, a hazardous environment requires strict safety features regarding electronics. Any tool used must not have exposed electrical connections that could cause a spark or that it could break in such a way that could provide a source of ignition to flammable gases. Currently available bluetooth headsets and pagers do not meet these safety specifications and would be an additional piece of equipment for workers to carry. We have worked to develop a device that can clip onto existing safety equipment, conform to hazardous environment safety standards, and pair with the user’s current cell phone and radio to notify them through vibration and light patterns. We are working with people who would benefit from the device as the design improves to its signals are clear in an industrial environment and that it suits their need. Future developments include allowing the device to relay more complex messages from multiple sources.

Obstructive sleep apnea (OSA) is a condition estimated to affect 5% to 15% of the population, in which individuals stop breathing for extended periods while asleep. Treatment is usually successful when provided; however, the vast majority of OSA patients go undiagnosed, in part because the superficial symptoms of OSA in a wakeful state are varied and nonspecific, including drowsiness, hypertension, heart disease, diabetes, and depression. The current diagnostic gold standard is an overnight sleep study, which is expensive and requires access to a specialized sleep medicine facility. This motivates a need for a convenient and accurate technology to screen for sleep apnea in homes and clinics. A ubiquitous screening solution must be both sensitive to slight variations, and specific to apnea in spite of many interfering physiological factors; these challenges are further complicated by the constraints of low-cost sensing devices. Our approach is to detect persistent changes in the sympathetic nervous system that are caused by frequent apnea events; although these changes cannot be measured directly, they manifest in discernible changes to heart signals. To test our system we record cardiac and respiratory signals while participants execute a series of breathing maneuvers, such as breath holding and breath rate control. We record in parallel with (1) a smartphone running a custom application and (2) a commercially available wireless biomedical recorder. Using digital signal processing, we extract informative features from the locations and amplitudes of peaks and troughs in the PPG, SCG, and ECG signals. Equipped with these extracted features and the ground truth diagnoses provided for each patient by the Harborview Sleep Medicine Center, we use existing machine learning libraries to train a predictive model for apnea. We expect results to demonstrate a correlation between cardiac regulation and sleep apnea severity.

Augmented reality (AR), which overlays virtual content on top of the user's perception of the real world, is now beginning to enter the consumer market. Besides smartphone platforms, early-stage head-mounted displays such as the Microsoft HoloLens are being actively developed. Many of the most compelling uses of these AR technologies are multi-user: for instance, in-person collaborative tools, multiplayer gaming, and telepresence. Although AR technologies enable new forms of interaction, new security and privacy challenges will also arise when users can augment each other's reality, and it is imperative that these challenges be addressed while the technology is still new and highly malleable. In this work, I explore these emerging challenges in secure and private content sharing for multi-user AR. I systematize design goals for security and functionality that an AR content sharing framework should support, and I design and prototype a framework for the HoloLens that meets these goals. By evaluating my framework against representative application case studies, I show that it meets desired security and functionality goals flexibly across a range of use cases. Preliminary investigations into developer effort suggest that applications' content sharing needs can be achieved in relatively few lines of code. I plan to convert my research prototype into an open-source toolkit so developers can address these challenges in practice. This work opens up directions for future research in how these underlying paradigms should manifest to users in the form of an application's user interface. By building foundations for secure multi-user AR content sharing, my work takes steps toward allowing AR to securely reach its full potential.

Though many teleoperation interfaces have been made for mobile manipulation tasks, the majority of these interfaces are inaccessible to people with severe physical disabilities, and the interfaces that are accessible are extremely slow, requiring around three minutes to complete a simple manipulation task, such as picking up and moving a soda can a few feet. People with severe physical disabilities are those who could benefit the most from these teleoperation interfaces, because through teleoperation, they can complete tasks that they can not perform on their own, such as opening the door, fetching an object, or making the bed. For a teleoperation interface to be accessible to people with severe physical disabilities, it must rely on cursor movement and clicks, or speech commands. Because the current accessible interfaces are slow, my research goal is to design new interfaces that are both accessible and brisk. I have designed three of these interfaces so far and am in the process of finishing implementation. The interfaces use two orthogonal views instead of the standard one view, thus allowing the user to grasp a complete picture of the robot and the world without needing to change the viewing angle, a slow and frustrating process. The three interfaces which allow for control of the end effector consist of two types of voice commands, and one button interface. Soon I will be conducting user studies with the new interfaces to quantify their improvement over the current standard.

The purpose of this research is to find data patterns of DC motor failure through radio frequency emission. To collect data for classification, three sets of five DC motors were modified to exhibit three common motor fault issues: armature winding resistance variation, brush contact/air gap, and bearing friction. A set of control data from unmodified motors was also collected. The motors were run for a set period of time at the manufacturer recommended voltage and rpm band with no load. The data was collected via a RF receiver and processed digitally. Using a Python-based program, known-good and known-bad data was run through a machine learning program. The program was then tested for accuracy and precision of diagnosis using a wider spectrum of faulted motors, including variation of fault, load, and size of motor. The hypothesis of this research is that radio frequency emissions generated from a defective DC motor show detectable differences from a normally functioning motor, the differences of which may show signature patterns that can be isolated using classification  through machine learning programming and used to predict onset of failure.

Over the past few years, the presence of smart homes has increased rapidly. Previous research has observed that people make errors in smart home programming with their mental models differing from their actual programming implementation which can cause serious security and safety concerns. The most common errors include missing half-rules and the incorrect use of conjunctions in triggers. For example, if users desire to program a rule for turning on the light when they are at home, they might program a rule such as “when I come home, turn on the lights”, forgetting to program the rule for “when I leave home, turn off the lights”. Our research focuses on reducing user errors in end-user programming by using programming by demonstration. Our hypothesis is that when users physically demonstrate a trigger-action rule, they are less likely to commit the trigger-action programming errors explained above. We compare the number and types of errors made within the IFTTT (if-this-then-that) interface that does not implement programming by demonstration and two interfaces that we built that implement programming by demonstration: a web interface and an augmented reality based interface within the Microsoft HoloLens. If our hypothesis holds true, we might observe that programming by demonstration significantly reduces programming errors in end-user programming in the case of both missing half rules and incorrect trigger conjunctions. We postulate that there might not be a difference in the case of error reduction between both the web and augmented reality interface; however, the augmented reality interface might provide for a broader range of programming by demonstration interfaces such as those related to temperature, humidity etc. Moreover, if our hypothesis holds, we might further suggest that smart home providers incorporate programming by demonstration models in their programming processes for end users within smart homes.

The first-year population, with students transitioning from high school to college, is full of novel experiences. These experiences, be they academic, social, etc., influence the well-being of the student. The purpose of the study is to understand the first-year experience in a data-driven manner to create a cornerstone for institutional change. Although replicating a previously piloted study at Carnegie Mellon University in Spring 2017, this study extends focus to investigating stressful life events of first-years. Over the 2018 Winter and Spring quarters, a participant sample size of around 200 UW first-year undergraduate students (around 50% pre-/engineers) contribute data to the study. Data consists of both active and passive collection methods: participants actively respond to Qualtrics surveys, with questions investigating stressors, and passively via our phone software and FitBits. The two quarters of this study will yield tangible data that can be analyzed to advance the science of behavioral health and trauma. This data is unique in providing a picture of student behavioral changes after stressful experiences. Additionally, this advancement in science can become a basis for institutional change for student wellness. As the first-year population is only a slice of the overall student experience, we hope to expand this study further by encompassing the graduate student experience and the full four-year college experience in addition to a larger sample size.

Pregnant women often get their bellies touched without their permission by people ranging from family members, to coworkers, to strangers. While usually well-intentioned, people who touch pregnant bellies are unaware that their actions may be unwanted and invasive. Previous works, such as Asta Roseway’s Printing Dress, have shown that wearable computing can bring attention to social issues by provoking discourse. Those works lead us to our wearable design that highlights the issue of personal space for pregnant women. We implemented a maternity shirt that reacts to inappropriate touches upon contact in various ways, such as displaying messages, vibrating, and setting off an alarm. We plan to deploy this shirt on experimental participants and gather qualitative data in addition to the quantitative data collected through the shirt about when and where touches occur. Moreover, findings from the experiment will provide insight into pregnant women’s experiences, and how to design wearables that benefit to pregnant women. Our future goal is to have the shirt deter the toucher as they approach the belly. Through this interactive design piece we hope to raise awareness and provoke conversation around themes of consent, inappropriate touching, and women's bodies.

There has been a research effort in exploring RFID (Radio-frequency identification) interfaces. RFID is a form of wireless communication that uses electromagnetic coupling in the radio frequencies to uniquely identify an object, person or animal. By expanding on previous work in paper and RFID technologies, we are able to create new interactive capabilities from paper and RFID. During my time with the Ubiquitous Computing Lab, my goal is to find a technique using capacitive sensing, conductive ink, and RFID tags in order to create an interactive touch surface. An RFID antenna can read an RFID tag and give information pertaining to that tag, for example, tag ID, time, a tag’s doppler shift, tag’s signal strength and more. My research has been exploring the relationship among these variables and creating a quantitative model that can map for continuous interactions. With such a model, one can make that surface customizable for a variety of interactions. 2D Sense strives to enable 2-dimensional touch interactions on paper by applying signal processing techniques towards existing RFID technology.

In scientific research labs, in general, experiments are generally treated as a black box: a prepared sample goes in, something happens, and one gets results that are then obtained via elaborate characterization steps. Several important dependent or correlated parameters are either discarded or ignored because of a lack of coherent dependency analyses that require critical thinking, linking, and pattern recognition. In this research we are working to stop treating experiments and computational simulations as black boxes, and create a cohesive platform where materials used, processes and parameters utilized and results achieved can be brought together as separate but related sets of databases. In the next step, the relationships between all the different parameters can then be connected, analyzed and visualized. Machine learning and AI techniques can then be used to predict results using these databases, thereby reducing experiment time, and taking away the traditional ‘trial and error’ method of experimentation. The research involves creation of a software interface, with numerous image and signal processing tools and applications running on libraries made customizable to research fields, types of experiments, etc. Assorted variety of services such as parallelization, compression, data analysis, and visualization, caching (among others) are also provided. We are improving the accuracy of time series data analysis and using fingerprinting to depict all parameters for improved predictability, flexibility and accuracy. When fully developed, we anticipate that the program will enable experimental and computational researchers to extensively use, customize and apply data analytics, machine learning and AI even in niche research in the hard sciences at the intersection of biology and genetics, materials science (physics, chemistry) and engineering, and computational modeling and informatics, enabling faster and accurate cross disciplinary innovation in technology and medicine. The research is supported by NSF-DMREF (DMR-1629071) program at GEMSEC-MSE, as part of National Materials Genome Initiative.

BrainNet is a system designed to be the first multi-person direct brain-to-brain interface in human brains. This interface combines electroencephalography (EEG) to record brain signals and Transcranial Magnetic Stimulation (TMS) to deliver information to the brain. Through this interface, three subjects will collaborate to complete a Tetris-like block game using direct brain-to-brain communication. Two of the three subjects are designated as "senders", whose brain signals are decoded by real time EEG data analysis. The decoding process extracts information about their decisions to rotate a Tetris block before it is dropped to fill the line. Then, this information is transmitted via TCP and directly delivered through magnetic stimulation of the occipital cortex to the brain of the "receiver", who cannot see the game screen. The receiver integrates the information s/he acquired from the senders and conveys this information back through EEG to the computer, which completes the intended action of either turning the block or keeping it in the same position. The performance of this brain network is evaluated in terms of the accuracies attained in (1) decoding decisions through EEG analysis, (2) delivering information through TMS, and (3) making correct group-level decisions during the block game. This interface enables for the very first time simple transmission of information directly among three brains without any verbal or physical action. Our preliminary result has shown proof of successful communication between the senders and the receiver with 14 out of 15 trails correct. BrainNet enables direct brain-to-brain communication of simple information between multiple humans, and is a step forward towards ultimate brain-to-brain communication. 

The field of robotics attempts to replicate human intelligence by harnessing vast amounts of information, and will heavily influence next generation technologies. Of course, in order to operate in an intelligent manner, our robots must be equipped with sensors that can provide informative data. By endowing our robots which such sensors, we can enable them to assist humans that are dependent on others for completing every day tasks, such as opening doors, picking up fallen objects, and pulling wheelchairs. Up until recently, many robots required their sensors to be wired to their main computers, potentially impeding their movement and/or limiting the types of sensors that they can employ. Our work, in particular, focuses on pre-touch sensors, which are sensors mounted to the robot's fingers that allow it to sense an object prior to making contact. This project redeveloped the pre-touch sensing system such that data from up to five sensors is wirelessly transferred to the robot's main computer. There are five micro USB ports connecting to the five fingertip sensors, all of which are controlled by the main USB hub. Vocore, a single-board computer, is incorporated, and this Linux based mini-computer is an essential component for sending the sensor data via Wi-Fi to any selected computer. Additionally, the elimination of wires makes it much easier to reconfigure the form factor of the pre-touch sensing system. As a result, these versatile sensors could be applied to any robot in the world.

Mobile and worn sensors have been increasingly used in order to gather data in long term behavioral studies, especially as the cost has decreased for such sensors. These kinds of studies provide increased information that can be used in order to understand behaviors and to elicit changes in them. Such studies have been conducted at CMU, MIT, and Dartmouth. UW is conducting a similar study that uses mobile phone sensors via a custom app, surveys via Qualtrics, and FitBit sensors in order to collect relevant behavioral data that may be related to academic performance in college. Because such studies rely on participants following the protocol and actively participating in the study, it is important to create and modify tools that allow for analysis of participant compliance, such as the rate of survey completion and amount of sensor data. These tools allow researchers to keep track of participants and work on collecting more high quality data for later analysis. There is little information about the design of compliance tools for similar studies. In this work, the focus is on modifying an existing dashboard from a similar study at CMU. Development will be focused on adding additional functionality based on the needs of the research staff. In particular, work will begin by developing filtering functionality for non-compliant participants and creating useful views for researchers in terms of contacting participants around possible broken sensors or missing survey data. Usability of the tool will be analyzed for future improvement. The hope is that this tool can be used to gather better quality data in this UW study and can shape future dashboards for similar experiments that gather behavioral information.

Color is often a very intentional and comprehensive choice as a visual element; artists, filmmakers, designers, and animators recognize color as a tool that signficantly shifts a viewer's perception of their work. While there are universal associations of solid colors- blue for sadness, and red for love, for example, the vastness and complexity of the color wheel leaves much room for nuance between the relationship of humans and color. Color in animation makes use of this nuance, but with a more varied stylistic choice and a greater control over very specific palettes. Through the conceptualization and production of a series of short animation clips, I aim to use color as a deliberate element to evoke subtle shifts in mood, as well as further explore the relationship between animation and color psychology. I've taken inspiration from color scripting in films through the use of a specific color palette to achieve visual balance and strong story support. My work, through tradittional and digital animation methods, aims to bring color in harmony with motion and composition to create a unique visual experience. I hope for my project to contribute to the growing understanding of color as an artistic, technical, and psychological asset in the viewer's engagement in animated films.

Deep learning has pushed the boundaries of natural language processing, and neural network architectures have achieved state of the art results in tasks such as automatic machine translation, question answering, and language modeling, among others. The backbone of many of these models is the recurrent neural network (RNN), which serves as a general-purpose module for encoding sequences of text as a vector of real numbers. RNNs are ubiquitous in recent natural language processing research; in fact, it is often difficult to find work that doesn't use them! Despite their popularity, RNNs are often treated as black box text encoders, and little is understood about why they are so effective and how to improve them. Motivated by synthetic data experiments and observations from real-world natural language tasks, we introduce several new variants of RNNs and measure their performance on several tasks. Improving RNNs would subsequently improve the quality of modern language technologies as a whole.

Digital mapping and routing has completely changed the way people interact with maps. However, a large percent of the population is, in general, not served well by most current popular digital mapping services due to two primary reasons. First, most maps do not focus services for pedestrian access. Second, unlike other areas of digital information delivery, mapping and routing has not gone through customization and personalization strategies to improve people's experience of and access to digital maps. There are many such human factors, ability characteristics, cognitive or emotional parameters, that are difficult for users to articulate numerically, but significantly impact the way in which urban environments are experienced and enjoyed by pedestrians, or the manner in which digital mapping information can be accessed. AccessMap is a map for those with disabilities, providing routes and information to wheelchair users, cane users, and others who may have difficulties getting around. Currently, users don't have too much freedom to control their preferences when they use AccessMap and they didn't have a profile to save the preferences they chose. Therefore, we tried to understand users' needs and modified existing interfaces to include more features. In our research, we are trying to design a better interface to address the diverse spectrum of user needs in a comprehensive and human-understandable way, as well as provide a platform to save and refine those preferences. We focus more on the user experience side of this project, we run varied research including scholar research and survey. We consolidated findings into personas and ideated scenarios and user flows to guide our design and envision how users would interact with the product. Our future goal is to run more usability tests with potential users for the purpose of improving application’s user experiences and evaluation our design.

The ultimate goal of robotics is to be able to interact with robots as one would with another person; hand in hand with this goal lies the ability for robots to understand natural language and carry out tasks in practical environments. Let us take for example, if a human were to command a robot to “pick up a can of coke inside a cardboard box on a table”. The robot should be able to extract relational information such as a can of coke is inside a cardboard box and a cardboard box is on a table. If “coke” is unknown, the robot should be able to ask clarifying questions to ascertain the qualitative aspects that define it. So, robots should be able to reason in the presence of uncertainties during object manipulation in the real world. We use a Fuzzy Markov Logic Network (Fuzzy-MLN) to learn and predict object classifications given input from multiple sensory modalities. Inputs from vision (RBG-D), haptics, and audio are used along with a prior knowledge base to identify objects in the robot’s environment and annotate various attributes which are added to the MLN. Audio input is processed for natural language which can provide information to contextualize the robot’s environment or give instructions to/answer queries from the robot in order to learn new relations, update Fuzzy-MLN weights, and improve annotator accuracy.

At the terminus of every robotic arm is an end-effector. These are the hand-like devices that directly interact with the environment. Collecting data from the vantage point of an end-effector is highly desirable, though difficult to accomplish in practice. Wired data transmission solutions which use internal cables suffer from signal degradation due to proximity to power conduits. Moreover, the solutions which use external cables are subject to tangling and often impose limits on the robot’s kinematics. Finally, existing wireless data transmission solutions are rarely used due to the low transmission rate, high latency time and unstable connection. This research focuses on developing a general approach to stream visual (RGBD) and haptic (force) data wirelessly to other devices. The resulting system should consist of an architecture which interfaces with a suite of end-effector sensors and handles signal compression, transmission, and decompression efficiently. Additionally, this project involves a hardware design portion which aims to neatly package and mount this infrastructure at the robot end-effector while satisfying power and kinematic constraints for the robot. This research chose the Intel Joule, an embedded Linux board, as the infrastructure to handle wireless transmission. The embedded board uses a low-level Python networking package, which forms the connection to transmit data to other devices on the same network. This project implements packages to encode, transmit and decode the data stream with Python. The receiving device decodes the compressed data and makes the data available to other devices on the same network. This project allows sensors and the main computer of the robot wireless connected and future robot would operate movement more freely.