The Internet of Things (IoT) draws upon a long held view of ubiquitous computing: to extend computing's reach to everyday physical objects by embedding computational services in them. Generally, today's IoT envisions the adoption of these cutting-edge technologies will occur as homes replace their possessions with these new, internet-connected objects. Currently, commercially available IoT devices support families in upgrading their homes by switching to internet-connected versions of common household objects like light bulbs, audio speakers, and blinds. Even non-durable goods, such as paper towels, can be part of the commercially supported smart home through automated purchasing at the click of a brand-specific, physical button routed to online retailers, such as the Amazon Dash button.

This vision of how homes will adopt IoT overlooks the costs of replacing everyday objects with internet equivalents. Early depictions of the smart home involved contracting with specialists to modify home architecture with sensors and install operating systems for the home. This view makes IoT available only to those who own their own homes. Even when IoT shifts infrastructure to artifacts, which can be easily purchased and brought into the home by consumers, families confront substantial costs when domesticating these devices. For example, by replacing common household objects, adoption contributes to waste, and if electronic, can be especially damaging for the environment. There are also social costs. Discarding household possessions to make room for new ones means owners must let go of their attachments to these objects and eliminate the entrenched routines and practices sustained by them. Relatedly, new objects mean users need to acquire new skills, and so, renders their existing skills obsolete. Thus, adopting IoT by replacing current possessions carries many hidden costs for the home.

Instead, upcycling can extend computing to everyday objects by adapting computing capabilities to existing possessions. Rather than replace domestic possessions with internet-enabled equivalents, upcycling could support their renewal by retrofitting them with the latest computing capabilities. Within this vision, upgrading to a smart home consists of augmenting domestic possessions with an internet connection and related IoT services. Domestic possessions could then persist with an adaptable user interface capable of accommodating their existing forms. Since they are grounded in the home's existing relationships, values, and routines, these upcycled objects could offer families greater discretion and control over IoT upgrade costs. We lay the groundwork for an upcycled IoT in our work on the "IoT Codex"—a book of inexpensive, battery-free interactive devices and composition patterns—to support customizing everyday objects with software and web services using IoT stickers (see Figure 1).

Consider how simple and cheap IoT stickers could transform mundane household objects in the following scenario.

Realizing that the overweight cat frequently tries to trick Judy and her partner into feeding it twice, Judy uses the IoT Codex to create a sticker for tracking whether the cat has already been fed. She attaches a two-state toggle sticker to the cat food box in her home to track morning and evening feedings by logging the event. Later, when Judy arrives home at the end of the day and is greeted by her cat meowing loudly and directing Judy to the cat bowl to feed it. Judy can check the IoT sticker on the cat food. She sees that its hand-written label says "Morning" and realizes that her cat's plea for food is genuine. To use the sticker, Judy slides the sticker's switch to the state "Evening" when she feeds it. A nearby speaker chimes in feedback that the cat's feeding is logged to the household's data store. When the cat feeding is logged, a cat food digital object reaches its threshold count and adds cat food to a list that Judy shares with her partner so that they can replenish the food next time they make a purchase.

This scenario illustrates how stickers could support even temporary, situated items—like cat food—and leverages Judy's knowledge of the state of the world to support her tracking and managing household activities in coordination with other household members.

  Lightweight Modification Framework

Considering how to upcycle everyday objects centers the consideration of the costs of IoT adoption in design. By designing for the programming and customization of IoT as part of the adoption process, we can begin to address the costs of replacement such as minimizing waste, evolving family members' attachment to their possessions, and reconfiguring entrenched, household routines.

We ran an experience prototyping study with 10 diverse, American families to develop a framework of the dimensions of home life implicated when families tailor an IoT system to their domestic possessions [2]. When families do so, they suspend belief in the current home to consider how objects and rooms might be reconfigured. This process destabilizes family members' working model of home to include previously unthought of arrangements. This allows families to imagine how new computing techniques could augment home life and enable them to extend family norms to include computing mediated forms of coordination and communication. This pushes families to adopt differing community dynamics to coordinate family input and collaboration on how the new technology will work within the family's day-to-day life. The principal dimensions of our "Lightweight Modification Framework" (see Figure 2) is outlined as follows:

• Object relationships. When family members generate ideas for upcycling, their relationships to their objects are at the center of this process. Personal items—like those with a single owner—can be targeted for improvement to help realize their owner's ideals, while modifications of shared items need to be negotiated with other family members.
• Family relationships. Household members have distinct roles and responsibilities for maintaining and caring for the home and its objects, and this includes computing devices or potential upcycled artifacts. Families find it important to reach an agreement on the household's norms and a shared idea of how the family should live when modifying their possessions with computing capabilities.
• Societal relationships. In order to determine what modification ideas will be considered by the family and how those ideas will unfold in any particular home by embodying community norms, social relationships are considered. For example, public places help families imagine new ways to stretch their domestic role by suggesting alternative modes of care and maintenance for the home's objects and family relationships.
• Time use. The time investment an IoT system demands of family members is reflected in time use. Families already have established routines that they use to take time to reflect and plan for day-to-day activities. While time is precious, families are willing to use it to try out new ways of accomplishing subtasks for their already complex routines, or, for less ingrained routines, creating any routine at all.

While families value buy-in from the rest of the family on an object modification idea, as the level of analysis shifts to include all four of our framework's axes, we find household members begin to disagree about how a modification idea should evolve object, social, and societal relationships and be realized in the future, desired home. Time use constitutes the processes through which families negotiate and compromise on their planned changes.

---

The IoT Codex and stickers provide the enabling technology to support families with reusing their possessions as material for IoT devices.

---

Our work engages upcycling questions shared by the research community on whether upcycled objects can, indeed, offer greater value than before modification. Our findings and framework help lay the groundwork for thinking about these questions by revealing opportunities and needs within current household dynamics around upgrading the home to address experimentation with household roles and historical, social legacies (e.g., gender or age). We identify opportunities to destabilize traditional groupings, such as gender or age, during the upcycling process by experimenting with the home's malleable relationships.

  Everyday Objects as Computing Materials

A lightweight and customizable approach that makes use of the home's existing possessions could give owners control over how IoT is integrated into the home. In our work, we created a novel approach to authoring IoT experiences in a lightweight form: a codex of stickers. Our IoT Codex consists of a set of programmable stickers that can be drawn on, customized through variables and through composition, and attached to everyday objects [4]. IoT stickers give householders control over the style, form, and location of IoT services. These stickers use RFID to provide a set of battery-free and wireless interactive devices that integrate the process of IoT setup with the family's existing negotiation processes when arranging and modifying their home. Thus, the IoT Codex enables family members to use their existing possessions as a design material for IoT systems, and thereby upcycle their everyday objects and routines into IoT experiences.

In our IoT Codex, stickers are the interaction primitives. Each of the five sticker types contributes a user interface component similar to a widget (see Figure 3). Thus, stickers serve as the building blocks for creating an interface for a user's IoT application. Like links developed for paper-user interfaces, we use stickers to link everyday objects in the home to both pages within our codex and electronic programs and content.

We fabricated our stickers using a layered fabrication approach. This allows us to embed electronics in the sticker in a fabrication-friendly manner. Each sticker variously modifies the basic button layers: sticker paper, double-sided adhesive with a slot for an RFID tag, and a top layer for aesthetic customization like drawing, writing, or icons. We use RFID tags as a proof-of-concept example of cheap, battery-free, wireless electronics capable of embedding digital data in our sticker book without requiring a line of sight as needed for computer vision, optical sensors, or infrared-based approaches typical of barcodes or glyphs. This point is especially important for our application area—the home—where sensing systems that depend on cameras and microphones risk breaching home dweller's privacy in large part because the computer-recognized semantics are likewise human-recognizable semantics. In contrast, RFID sensing techniques often depend on metadata about tag reads, such as received signal strength and channel hopping, that do not carry immediately recognizable information such as who said what to whom. These properties, and the current commercial availability of RFID tags, have motivated their use in the current IoT Codex system implementation. However, it is also important to note many of the conceptual properties of our system, such as scaffolding of learning needed for end-users to advance to customization and composition of stickers, are independent of this particular technology. These concepts could be adapted to other identification technologies such as bar codes in a different implementation of the interactive book.

RFID systems consist of readers and tags. In our system, we use passive, ultra-high frequency (UHF) tags capable of wireless communication up to a range of 11 meters with an appropriate antenna setup. Prior work has demonstrated a reader and antenna could be embedded in form factors amenable to easy home deployment. For example, a light bulb equipped with a Wi-Fi RFID reader has been demonstrated [1], and in a mass-market setting these could be produced much more inexpensively than current RFID readers. In the future, such advanced form factors may be able to facilitate novice installation of a home-scale RFID system. Our research builds on this work by focusing on the tags and the interpretation of tag data as an extension of this kind of home-scale setup. Passive RFID tags, like ours, communicate with the reader when it interrogates the environment for present tags by emitting an RF signal in the 840-960MHz range. When the tag receives the signal, it communicates with the reader through modulated backscatter—changing how it absorbs or reflects the signal back to the reader in order to encode an ID number unique to the tag. When the tag is covered with conductive material, the reader is unable to communicate with the tag. Thus, individual tags can be represented as having a simple binary state: covered or uncovered. We use this binary state for our stickers.

When the IoT Codex is opened to the section for the sticker, page text provides initial guidance. It instructs the user to peel the sticker from the page and attach it to the object of their choice to create a sticker button. Our first sticker button provides a blank face so that anything can be written or drawn on it. We use it to introduce the user to the logging pattern since this can signal any user-specific meaning and can easily function even with minimal user intervention. This pattern provides immediate value and functionality with little to no mastery of the system's programming capabilities. When the user first peels the logging button sticker from the page, this triggers the setup process for the sticker within our system. The logging pattern triggers a brief audio message letting the user know that the sole function of the sticker is to log every time the person presses the button. Once the sticker is attached to the object of the user's choice, it can then be activated by covering the sticker fully with the hand. Once the user covers the sticker with their hand, nearby an audio speaker chimes in recognition of the log. This progressive disclosure provides encouragement and incentive to continue with the button setup process. The steps are also written in the sticker book to serve as a reference if the user has questions.

The IoT Codex's architecture supports a hardware abstraction layer that 1) produces events, 2) tracks the life of identified objects (in our case RFID tags), and 3) facilitates interactive manipulation of parameters through advanced parameter types and actions that enable thread-like control (see Figure 4). These features combine to provide advanced capabilities for the user in a concrete, physically instantiated form without requiring knowledge/mastery of the abstract programming concepts they embody. Taken together, these features support users with manipulating parameters through proprioceptive control overexposing/blocking tag reads to instantiate stickers and update their state, as well as directly supplying parameter values through interaction. This creates a ramp for users with no programming skills to develop competencies needed to tinker with a programming language. The IoT Codex extends the power of working with abstractions to everyday users by concretizing the architecture's central abstractions of an identifiable object, stickers, and actions in a tangible form.

The IoT Codex progressively introduces our system's five sticker types, as shown in Figure 3, through the book's form factor. Page-turning sequences provide an introduction to progressive levels of sticker complexity starting with the "button sticker" and progressing to a "wrapper sticker." The IoT Codex provides substantial capabilities through fixed form and function button stickers created for common household tasks. The IoT Codex then gradually introduces customizable button stickers and eventually progresses to the composition of sticker functionality using wrapper stickers. Each sticker concretizes the link between user interaction and action execution by explicitly embodying it in the physical sticker's kinetic mechanism.

The current IoT Codex supports four types of stickers that serve as widgets and a fifth sticker type with composition mechanics for a customizable IoT user interface. To provide guidance on how our stickers could be tailored to domestic IoT use cases that are situated and idiosyncratic, we implemented example stickers that we parameterized to suit highly specific needs. The IoT Codex uses kinetic mechanisms to gradually introduce more complex embedded computing capabilities by leveraging the sequential nature of the book form factor, and encouraging manipulation and tinkering with the stickers themselves. By concretizing the association between user actions in the physical world and computational actions by means of stickers, the IoT Codex exposes users to tinkering with low-level details without requiring mastery of low-level knowledge or programming.

  Scaffolding Creative Reuse

The IoT Codex and stickers provide the enabling technology to support families with reusing their possessions as material for IoT devices. Yet, an outstanding question for an upcycled IoT is how to help individuals conceive of their current possessions as potential IoT devices. We find that families already employ coordination patterns that could enable collaborative creativity by building on one another's ideas and experimenting with alternative roles. There is an opportunity for the embeddedness of IoT to scaffold object-oriented creativity through reshaping object relationships by suggesting experiments with new object roles and with the alternative social arrangements afforded through those objects. We examined families' socio-material practices to distill opportunities for lightweight modification of household objects as patterns, which could be easily recognizable to end-users.

---

Considering how to upcycle everyday objects centers the consideration of the costs of IoT adoption in design.

---

We found families use objects and spatial divisions of the home (for example, like a side of a room, or the room itself) to set and enforce social norms [3]. Families implicitly map their social relationships onto spatial jurisdictions. Further, they use their objects to negotiate or modify these jurisdictions as their relationships with one another evolve. For example, siblings, college roommates, and spouses split sides of a bedroom so that each has their own side. Similarly, family members cluster personally owned objects in shared spaces to lay claim to portions of a room like setting up one's laptop and papers at the kitchen table to study or work from home in the afternoon, then moving that cluster elsewhere when dinner time arrives. As a result, a dining room is temporarily converted into a home office that should provide a quiet workspace. This use of objects and rooms allows for family members to coordinate with one another about how the home should function at a specific time of day.

We are currently using the IoT Codex to explore whether we can enhance families' socio-material practices with end-user programming to scaffold the upcycling process. If successful, this would enable families to transform their everyday objects into IoT devices. Since IoT stickers can be attached to everyday objects, we can leverage the families' use of objects and spatial jurisdictions to facilitate context-sensitive programming. For example, the same sticker type could be parameterized differently based on user input when they set up a sticker for a specific object. We can use the IoT Codex to compose complex behavior to set policies for a group of stickers that are embedded in everyday objects. This would allow for a sticker type to be parameterized with the policies set for a group of stickered objects in a particular room.

  Conclusion

In summary, our research enables an upcycled IoT in two ways: 1. deep empirical engagement with socio-material practices implicated by augmenting everyday objects, and 2. building the technology to enable end-users to transform their existing possessions into IoT devices. By creating the IoT Codex, an interactive book of programmable and customizable stickers, we build on the advances made in paper-based computing to create a low-cost and lightweight smart home infrastructure. The book's IoT stickers can be written on and stuck to everyday objects, and thus, bring computational behavior to the idiosyncratic contexts of household objects and clutter. Further, these stickers integrate battery-free and wireless communication into the stickers using RFID (costing approximately $0.03 USD) but enhance these communication capabilities using kinetic mechanisms capable of revealing or blocking RFID reads in order to manage and track state. To handle communication with web services and collecting information supplied by the user, we extended the event-based architecture typical of interactive systems and the parameter gathering process typical of procedural languages to handle asynchronous and just-in-time values. This enables a variety of interaction techniques for householders to customize web services and software-supported behavior by manipulating the stickers. It also makes those capabilities available in situ through the sticker devices. Our work presents these stickers to the user in a book form factor to support the gradual introduction of more complex programming behavior—such as the composition of two stickers together—as well as cross-generational and collaborative programming of IoT behavior. Taken together, the IoT Codex and stickers support families with installing, customizing, and managing the introduction of new IoT capabilities using an upcycling approach.

  References

[1] Gummeson, J., Mccann, J., Yang, C., Ranasinghe, D., Hudson, S., and Sample, A. Rfid light bulb: Enabling ubiquitous deployment of interactive RFID systems. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1–16.

[2] Williams, K., Pulivarthy, R., Hudson, S. E., and Hammer, J. Understanding family collaboration around lightweight modification of everyday objects in the home. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–24.

[3] Williams, K., Pulivarthy, R., Hudson, S. E., and Hammer, J. The Upcycled Home: Removing barriers to lightweight modification of the home's everyday objects. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, 2020, 1–13.

[4] Williams, K., Hammer, J., and Hudson, S. E. The IoT Codex: A book of paper engineering techniques for authoring and composing embedded computing applications. 2020.

  Authors

Kristin Williams is a Ph.D. student in the HCII at Carnegie Mellon University. She holds an M.S. in HCI from the University of Maryland and a B.A. in philosophy from Reed College. She has received the NSF EAPSI fellowship and an AAUW Career Development grant.

Jessica Hammer is the Thomas and Lydia Moran Assistant Professor of Learning Science, jointly appointed between the HCI Institute and the Entertainment Technology Center at Carnegie Mellon University. She is also an award-winning game designer.

Scott Hudson is a Professor of HCI at Carnegie Mellon and previously held positions at the University of Arizona and Georgia Tech. He has received the ACM SIGCHI Lifetime Research and Service Awards, was elected to the CHI Academy, and received the Allen Newell Award for Research Excellence at CMU.

  Figures

Figure 1. The IoT Codex is a book of inexpensive, battery-free sensors and interaction patterns to support linking everyday objects to software and web services using stickers.

Figure 2. The four axes of the "Lightweight Modification Framework."

Figure 3. An image of each sticker type on its own page in the IoT Codex.

Figure 4. IoT Codex's event architecture to lower the floor for potential end-user programmers of paper user interfaces and IoT applications. IoT stickers concretize event emitters in a tangible form, and their kinetic mechanisms afford tinkering with parameters and manipulating control flow.

Figure 5. This image shows how IoT stickers can be given contextual and spatial meaning by attaching them to everyday objects in the home.

 

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.

 

To comment you must create or log in with your ACM account.