SmartCare Project
The SmartCare project is a multi-discipline health technologies project between the Nursing and Health Innovation and Computer Science and Engineering departments at the University of Texas at Arlington. SmartCare offers an unobtrusive and pervasive system that provides in-home health monitoring for the elderly. Using a range of embedded sensing technologies along with hardware, software, and communication infrastructure within one’s home allows continuous health monitoring and alleviates the burden on older individuals in providing reliable activity and health information to care providers and loved ones. The SmartCare system can provide early diagnosis support for medical professionals, continuous activity monitoring, deviation detection, and self-management assistance in the form of home automation and medication intake reminders.
SmartCare In The News:
SmartCare Apartment Infrastructure
The SmartCare apartment is located at Lakewood Village Retirement Community in Fort Worth, Texas. The interior has the appearance of a normal apartment with the latest high-end appliances and fixtures, but embedded under the floor, in the ceiling, and in the walls are various sensing and automation technologies to provide 24/7 health monitoring for the elderly. It is a live-in laboratory to run various experiments and gather short-term and long-term data. The apartment’s unveiling was in May 2015.
The apartment infrastructure was designed and constructed under the supervision of Dr. Záruba, Dr. Huber, and Dr. Daniel. Graduate student Nicholas Burns was instrumentally involved in all aspects, including cleaning, construction, product research, hardware and sensor installation, software design, and networking which made the smart apartment a reality.
From the recruitment efforts of Dr. Daniel, residents of the Lakewood Village Retirement Community have volunteered to live in our SmartCare apartment. From May 2017 to May 2019 various single and coupled residents have stayed in our apartment for various lengths of time; usually about one month. During these stays we have recorded 24/7 data from the smart floor, IR sensors, door sensors, water usage, electricity activity, etc. This semi long-form data will hopefully provide insight into building a model of a resident’s activities and learning health information. We also had a resident stay twice at our apartment with a significant time gap in between, hopefully this data can identify any differences this resident underwent over that period.
System Architecture and Embedded Technologies
The apartment’s technologies include a smart pressure-sensitive floor, Z-Wave sensors and actuators, home automation devices, high-resolution bed mats, water/electricity monitoring and control all fed to a computer system that runs custom-built software to manage the hundreds of sensors and data collection. The hardware, software, visualization, and infrastructure details are explained in our previous papers SmartCare - An Introduction and PESTO: Data Integration for Visualization and Device Control in the SmartCare Project. The visualization was through the hard work of Peter Sassaman.
The figure above shows an overview of the elements of the architecture of the system. At the bottom of this architecture is the sensor layer containing sensor and actuator components that provide actual data and perform assistive actuations. These components contain their own data accumulation and communication hardware and interact with the data layer of the system. The data layer forms the heart of the architecture. Each of the senor-layer modules interacts with the data layer through a client-server interface. Through this, acquired data is put in the database and commands are read from the database and sent to the sensor/actuator module. As a result, interaction between any two modules is strictly performed through the database, leading to a uniform interaction scheme as well as to an infrastructure that facilitates research by maintaining data and allowing re-processing of information independent of the running sensor modules. The analysis layer contains the processing modules. Again, these interact through client-server interfaces with the database, accessing the necessary information and returning processed analysis results to the database for other processes to access. Similarly, the interface layer contains user interface and visualization components.
Below is a picture of the SmartCare apartment's computer control room. All sensors, data acquisition boards, and computers feed locally into the databases located on the computers in this room.
Smart Floor
For in-depth details on the Smart Floor progress, please visit the Smart Floor Project Page.
The entire SmartCare apartment is equipped with a pressure sensitive smart floor with a resolution of about one pressure sensor per square foot. The choice for pressure sensors was the Tekscan FlexiForce pressure sensor. The floor is built on disc-like sensors with rubber padding deployed in a 1-sqft square matrix configuration. On top of these sensors we have a click-together ceramic tile rigid flooring structure that can bend along the tile lines. Thus, each single tile is floating over four Tekscan sensors in the sensor-matrix. These tiles are rigid enough to handle people and object weights usually encountered in a home by being supported only by their corners.
The floor is built in 4ft by 8ft sections with each having a custom designed acquisition board using a PIC24FJ64GA004 microcontroller recording the readings of 32 sensors at 25Hz. Data from the acquisition boards is relayed via BeagleBone Black computers (BBB) deployed in the walls of the apartment, which in turn preprocess and relay the data to the central server using Ethernet. The central SmartCare server is responsible for ensuring that all smart floor acquisition boards and their controlling BBBs are running correctly with the appropriate servers. There are a total of 712 floor sensors covering the entire apartment.
Z-Wave Home Automation
Many of the "smart" aspects of the technologies described below are Z-Wave enabled. Z-Wave is a wireless communications protocol designed for home automation purposes. Z-Wave sensing and actuating devices (lights, outlets, valves, door monitoring, etc.) communicate with the central hub (Z-Stick USB fob device). This device acts as a transmitter and receiver between the various Z-Wave devices throughout the apartment and the central computer's database software in the computer room.
Z-Wave allows data to be gathered and stored regarding energy usage, on/off states, activity, light usage, door states, and other measurable quantities of the SmartCare apartment. The main computer system has custom-built Z-Wave interface software for monitoring and controlling all of the Z-Wave devices (written by Nicholas Burns in C#).
Smart Outlets
Every 120VAC outlet in the SmartCare apartment is equipped with an AEON Labs smart energy switch (MSES: DSC26103-ZWUS) which is placed in the outlet box right behind the power outlet (a total of 44 MSESs). These z-wave devices can be remotely turned on and off; they provide instantaneous power and accumulated energy readings over to the main z-wave controller and the ZAPS software. ZAPS runs a server process on a Windows machine. A client process on magoo connects to ZAPS and retrieves the state, power, and energy usage of the MSES switches, which are then placed in the mongo database. Each BBB’s power supply is also controlled by one of these MSES switches, enabling an easy remote cycling of the power to the BBBs.
Smart Lighting
Every light and fan in SmartCare is also connected to a AEON Labs micro smart dimmer switch MSDS: DSC27103- ZWUS (a total of 13 MSDSs). The MSDS-s are controlled similarly to the MSESs, with the exceptions that MSDSs are also connected to regular light switches that can trigger them and that they have power control (dimming) capabilities.
Water Monitoring and Shutoff
We believe that water usage monitoring could provide value added input to activity recognition. Thus in SmartCare all water sources are measured. In addition we wanted to have the capability to shut off each water source in case any of them are accidentally left running. Thus after each water valve (two for the kitchen sink, one for the dishwasher, two for the tub, two for the bathroom sink, one for the toilet and one for all hot water) we have installed an additional ball valve with a servo motor controlled shutoff, and a water flow meter.
The water flow sensors are Uxcell’s hall effect sensors (YF-S201 and FS300A for ½” and ¾” pipes). These sensors need to be powered with 5V and output a conditioned digital signal with the frequency corresponding to the water flow. After level adjusting the signal with CMOS buffers (the BBBs cannot handle signals larger than 3.3V) they are connected to some of the digital general purpose inputs of the BBBs. One BBB can handle six water flow sensors and six shutoff valves. The shutoff valves are BoxLink BL-OC-VAL; Solenoid based shutoff valves could not have maintained the state of the valve in case of a power outage. The shutoffs we are using are mounted on regular ball valves and have powerful servos to turn the valves. They are controlled using H-bridges and MOSFET drivers from the GPIOs of the BBBs.
For the entire water infrastructure of the SmartCare apartment we use two BBBs, one controlling water in the kitchen (3) and one for the rest (6). Each BBB is set up similarly to the BBBs that control the SmartFloor, i.e., magoo has direct ssh access to them. They both have the same software running, that establishes a TCP server socket, and can then send measurement data (ticks/second for each sensor) to connected TCP clients as well as receive “open valve” or “close valve” commands from them. A script on magoo starts and maintains the server processes on the BBBs as well as starting client processes on magoo that control the valves and write data coming from them into the mongo database.
Additionally, the apartment has three z-wave based flood sensors (Everspring ST812-2) placed at strategic locations in the bathroom and kitchen. They send their data through ZAPS to magoo. A client process monitors data coming from these sensors and can instruct any of the active water sources to be shut off to prevent further damage.
Smart Curtains
In addition to the privacy film applied to each window, we also have curtains as window treatments, providing the option to darken the rooms. Each curtain is affixed to a BEME erod (ERODCN68-120) remote controllable curtain rod; they are adjustable in size. These curtain rods come with IR remote controls (open, close, stop) so humans can control them conveniently. They have an electronic head unit with a powerful (and heavily geared) motor. Unfortunately, like many “smart home” technologies, they do not provide an interface to advanced users.
To be able to control them, we use IR remote controls, where the “buttons” are triggered by the outputs of a BBB (through MOSFETs). However, we also wanted the inhabitants to be able to use the remote controls that come with these units. Thus in order to know when they have been triggered (and how far they were instructed to pull the curtain) we also need feedback coming from the motors. Originally (in order not to have to hack the electronics) we decided to use optical encoders on the motor shaft; this turned out to be unreliable. Our current design has a small current sensing resistor in series of the motor (inside the H bridge) connected to an H-bridge capable difference amplifier IC, of which the output is fed into an A/D converter pin of the BBB controlling the curtains. This way we have a relatively precise position feedback (after we learn the current drawing characteristics of each motor) on the position of the curtains. The four curtains thus are controlled by a BBB, using three outputs each to control the curtains and four analog inputs to measure any activation (and thus position of the current).
Smart Appliances
In general, there are very few off-the-self “smart home appliances” that provide an API for advanced users. In our experience, companies manufacturing “smart appliances” attempt to market these appliances and then due to low interest from customers, they cease to manufacture them. We have observed this vicious cycle for most of the past two decades.
Refrigerator and Range
The fridge we have selected is an LG smartThinQ LFX31995ST. The “smartness” of this refrigerator is in that it has an Android driven touchscreen from which the user can access a few configuration options, and can retrieve basic information from the Internet. In addition the fridge can talk wirelessly to a remote control application as well as to LG manufactured smart ranges (preheat them to the right temperature when a recipe is selected). Unfortunately, there is no easy to use API to access these options from the outside and thus controlling the refrigerator or receiving data from it from/to our own software remains future work.
Similarly to the refrigerator, the range we picked is a “smart appliance” manufactured by LG (smartThinQ LRE3027ST). This is the model that can actually indeed talk to the refrigerator. Interfacing the stove with our own software remains a task for the future.
Faucets and Toilet
Both the kitchen faucet and the bathroom sink faucet are Delta Touch2o variants; these faucets turn water on and off when a human touches the faucet body (capacitive sensing). Unfortunately, they do not have any external interfaces; however our water monitoring and shutoff setup can control water. The toilet is a KOHLER Touchless appliance, where flushing is done by waving one’s hand over a battery operated sensor that triggers a motor to pull the flushing flap. Like with the faucets there is no external interface provided to the toilet.
Bathtub and Water Heater
Neither of these units is “smart”, they do not provide any interfacing capabilities. However, the tub is a walk-in “geriatric” tub with a Jacuzzi feature (Meditub 3060wirbd) (we can measure or shut off water to it). The water heater is a Stiebel-Eltron (Tempra 20) tankless electric heater (unfortunately without a computer interface). This heater was added (having had to upgrade the main electrical cabling) to enable inhabitants to indeed enjoy a full tub of hot water.
Vaccum Cleaner
Our choice for the vacuum cleaner was a Neato Botvac D80. These vacuum cleaners do room mapping in order to be knowledgeable about obstacles and thus know where they cleaned already. They do not provide a native interface but researchers have hacked them previously so that they can be remote controlled. We will apply similar hacks in future work to be able to interface the Neato with our data collection and appliance control.
Smart Coffee Table
The coffee table in the living area is a lowered Samsung SUR40 Microsoft PixelSense table; essentially a large (40” – 101cm diagonal) optical-multi-touch tablet. This table will serve as a remote control and health visualization appliance as well as it will host geriatric serious gaming applications currently under development in our lab.
Recumbent Exercise Bike
Proper exercise can significantly contribute to an extension of independent living. It is important to be able to provide appropriate exercise appliances and measure their usage. One of the least stress-inducing exercise machines are recumbent bikes. Unfortunately, exercise bikes that provide an open interface are rare. For SmartCare we opted to purchase the Stamina Elite Total Body recumbent exercise bike. In order to be able to interface with it, we connected a BBB’s GPIO input with appropriate signal conditioning to the speed sensor of the bike. (We reused an available pin of a nearby SmartFloor control BBB.) The software architecture of reading from this “SmartBike” is similar to that of the window treatments.
Xbox
The inhabitants also have access to an Xbox/Kinect system. A preloaded yoga game (Your Shape: Fitness Evolved), can post data on a Facebook page, from which it could be downloaded into our database. This interfacing is future work.
Smart Bed
The queen size bed in the bedroom is 5-way position adjustable on both sides. In addition to this convenience feature, each side of the bed has Vista Medical pressure sensitive sensors under the bedsheet (BodiTrak BT3510). Each of these bedsheets provides pressure images 26 times a second with a resolution of 256x64 “pixels”. Both sensors are USB connected to an Intel NUC i5 (sandman) microcomputer running Windows 8, located under the bed. Our software on sandman establishes a TCP server, reads both the bedsheets, and provides a compressed stream of data to connected clients. Magoo, has a client running that connects to sandman and stores the incoming pressure data in the mongo database. A separate client connected to the mongo database can retrieve the data and estimate sleep parameters of the inhabitants.
Smart Mirror
The Smart Mirror is an experimental component under research in our lab and thus is not permanently in the SmartCare apartment. The Smart Mirror is planned to be located in the bathroom vanity cabinet. Interestingly, it is the only component (besides the Xbox) that contains a camera and that camera is indeed in the bathroom. The smart mirror is engineered so that no raw data can be extracted from it and so that any but meta data reading access to the computer requires physical access to the computer. The camera is not visible to the inhabitant as it is behind the mirror. The smart mirror detects anomalies in facial expressions and structures to detect signs of sickness, ailments, mood, and water retention. In addition it can detect hemoglobin and melanin levels and measure pulse.
Motion, Temperature, HVAC, and Registers
Each room is equipped with one to three z-wave based mutlisensors (AEON Labs DSB05-ZWUS) monitoring the room. These sensors measure temperature, humidity, and have a passive IR detector on them. As they are z-wave, they interface with magoo through ZAPS. The HVAC unit has been replaced for the apartment with a high efficiency unit (Texas heat). The thermostat is z-wave compatible and thus can be controlled through ZAPS as well. One of our future goals is to be able to set “micro-climates” for different rooms; thus we have z-wave controllable ceiling HVAC inlet registers. This way we can shut off or enable induced cold or hot air from the HVAC unit entering individual rooms.
Doors
Each door (entrance, patio door, room doors, even cabinet doors) and drawers have z-wave open/close door sensors attached to them. Data coming from these sensors is recorded by magoo through the ZAPS interface. This data, together with the SmartFloor data are going to be essential components for our activity recognition tasks.
The main entry door has a Samsung RFID/biometric safety lock (SHS-P718) that can be remote unlocked (by a remote controller). The door handles on this lock mechanism are “intuitive” as pushing it inwards opens the door from the outside while pulling it inward opens the door from the inside. The same door has high-end electronic actuator (DORMA ED100LE) that helps in opening and closing the door.
Our plans include the placement of smaller actuators on each door so a robot can easily traverse the entire area. For this we will also need to add solenoid enabled door strikes (so that doors can actually be unlocked before trying to open them). Although we possess the actuators and strikes, their installation and interfacing is future work.
