Getting Started with the SLN-TLHMI-IOT | NXP Semiconductors

Automotive

Industrial

Consumer

Technologies

Getting Started with the SLN-TLHMI-IOT

Last Modified: 2022-11-10 21:34:46

Contents of this document

  • 1

    Plug it In

  • 2

    Get Software

  • 3

    Build, Run

Contents of this document

1. Plug it In

Welcome to the SLN-TLHMI-IOT Getting Started guide. This guide walks you through the process of getting up and running with your SLN-TLHMI-IOT board. This guide takes you through the steps of unboxing your kit, running the out-of-box coffee machine demo application, as well as downloading, modifying and debugging the firmware source code for your kit.

Although the SLN-TLHMI-IOT supports multiple out of box applications (which can be selected by pressing and holding SW1, SW2 or SW3 on the back of the kit), this guide will focus on the Coffee Machine out of box demo.

The Coffee Machine application demonstrates not only traditional touch-based controls, but also the face recognition, multi-language far-field voice recognition and graphics capabilities of the kit.

Before we begin, make sure to check the box your kit came in for any marks or damage and should you find anything, be sure to report it to your local NXP representative.

1.1 Unbox

The kit your box arrived in with few different things, such as:

  • A packing list paper
  • Fully assembled TLHMI kit
  • USB-C cable

Figure 1. Items inside the box.

1.2 Power On

Before we begin, remove the protective film from the RGB and IR camera, as shown in Figure 2. This protective film is used to protect the lens of each camera during transport. However, failure to remove may cause the image capture not to work correctly.

Figure 2. Protective film covering the two cameras: RGB and IR.

To get started, follow the steps below:

  1. Take the provided USB-C cable and plug one end into your laptop/PC

    2. Figure 3. USB-C power port.

  2. Once successfully plugged in, the display shows a startup screen with the icon for the currently enabled application most prominently displayed. For the purposes of this guide, we will focus on the Coffee Machine application

    3. Figure 4. Startup Screen.

  3. Touch the screen or wait a few seconds and the Coffee Machine standby screen will be shown. This screen is used as a screensaver to indicate the kit is waiting for user interaction. The standby screen is enabled when the board is first turned on, once a coffee is finished brewing and after a timeout period is reached without any interaction with the kit

    4. Figure 5. Coffee Machine Standby Screen.

  4. Once the Standby Screen is enabled, touch the screen one more time to begin using the Coffee Machine demo. Touching the screen will enable the Home Screen

    5. Figure 6. Home Screen.

Note: If no voice commands and/or touch events are received, the board automatically reenters the standby screen after a 60 second timeout.

1.3 Saving a New Face

Let us get started with a demonstration of this application’s out-of-box features.

To demonstrate the SLN-TLHMI-IOT's ability to create tailored custom experiences for each saved user, we are using our face to save a custom coffee profile.

Before diving in, let us take a quick tour of the home screen layout.

The home screen is divided into two halves: the camera preview pane on the left, and the coffee selection menu on the right.

Additionally, there is a Start button in the lower-left corner of the screen and a language selector drop-down menu in the top-right corner of the screen.

Figure 7. Start Button and Language Dropdown.

On the camera preview pane, a bounding box is provided to help you properly align your face so that it may be identified by the kit.

The different colors of the bounding box are used to indicate face recognition status. By default, the box is colored blue to indicate that no face has been detected.

Figure 8. Blue indicates that No face has been detected.

The color red is used to indicate that an unrecognized face has been detected. Align your face with the camera until the bounding box turns red. This should take < 1s if done properly

Figure 9. Red indicates that an unrecognized face has been detected.

Once your face has been detected by the camera, it can later be used to associate with your preferred coffee selection.

Now that your face has been detected, you are ready to select your coffee preferences. The coffee configuration options can be seen on the right half of the home screen as shown in Figure 10. These options include coffee type, size and strength.

Figure 10. Coffee configuration options: coffee type, size and strength.

To choose your coffee preferences, follow the steps below:

  1. Select the coffee type by pressing the icon associated with your preference
  2. Next, click the checkbox selector next to your preferred size
  3. Finally, select the checkbox selector next to your preferred coffee strength
  4. The coffee selection has been customized. Before moving on, ensure that your face has been detected by the camera (indicated by a red bounding box) before pressing the Start button to begin brewing the coffee

    5. Note: The start button will be grayed out and cannot be pressed until a coffee type, size and strength have been selected.

    Figure 11. Selecting coffee preferences.

  5. After pressing the start button, a brewing animation is shown. The brewing screen displays a progress bar and a summary of the chosen selections on the left

    6. Figure 12. Brewing screen.

  6. Once the brewing animation is finished, the chosen coffee is shown on the final screen

    7. Figure 13. Final screen.

  7. After a brief delay, a voice prompt plays via the onboard speakers, asking “Save your coffee selection? Confirm or Cancel.” Say “Confirm” to save this order. Answering “Confirm” associates the coffee that was brewed with your face. Next time your face is recognized, the coffee machine automatically asks whether you would like to reuse your saved order

After your order is complete and the prompt has been responded to, the device will automatically reenter the standby screen that was shown at startup.

Figure 14. Standby screen.

1.4 Modifying a Saved Order

Now that we have demonstrated the core features using traditional touch-based controls, we will walk through the next generation hands-free voice control capabilities made possible by the processing power of the i.MX RT117H.

Note: The stock out-of-box software uses an evaluation version of the VoiceSeeker AFE or Cyberon ASR library which limits users to 25 hours of continuous operation and 100 detections, respectively. Simply power-cycling the board will resolve the issue and restart the timer/counter. This issue is not present in full/non-evaluation license versions. See the User Guide for more details.

First, wake the board out of Standby mode using the “Hey NXP” wake word.

Figure 15. Home screen.

Note: The wake word is used to exit Standby mode but is not necessary to be used prior to each voice command.

With the board active once more, align your face inside the bounding box until the color of the box turns green.

Figure 16. Green indicates that your face has been recognized.

A green bounding box indicates that you are a recognized user. The coffee machine plays an audio prompt asking if you would like to order “another Espresso? Confirm or Cancel?”

Responding with “Confirm” immediately takes you to the brewing screen again, while “Cancel” can be used to choose a different coffee selection.

Instead of reusing our saved order, respond to the prompt with the “Cancel” voice command, allowing us to choose something different. This time around, however, we will use hands-free voice commands instead of the touch controls we used previously.

Follow the steps below:

  1. Set the coffee type by using the “Cappuccino” voice command. Successfully recognized voice commands are indicated by a green LED and brief audio tone

    2. Note: Each of the supported voice commands matches the words shown on screen.

    Figure 17. “Cappuccino” voice command updates the coffee type selection.

  2. Now, update the coffee size using the “Medium” voice command

    3. Figure 18. “Medium” voice command updates the coffee size selection.

  3. Set the strength to the maximum using the “Strong” voice command

    4. Figure 19. “Strong” voice command updates the coffee strength selection.

  4. Finally, start the brewing animation again using the “Start” voice command

    5. Figure 20. Brewing screen displays the updated coffee selection.

  5. Once the animation is completed, and the final screen is reached, you are prompted again to save your new order (replacing the previous order)

    6. Figure 21. Final screen displays the updated coffee selection.

Responding to the prompt with “Confirm” overwrites your previously saved order with your latest selection. Now, the next time your face is recognized it automatically uses your updated order.

Figure 22. Modified coffee selection.

1.5 Deleting a Saved Face

Now that we have demonstrated saving a face and updating a coffee order, let’s finally demonstrate deleting our face from the local face database.

To do so, reawaken the board from Standby mode using the “Hey NXP” wake word and/or by touching the screen.

After the display turns back on, align your face with the onscreen bounding box. Once your face is recognized, use the “Delete” voice command to delete your face from the database. Alternatively, you can click the red X icon in the top-left corner of the screen to remove the recognized face.

Figure 23. Deleting a saved face

After successfully saving a new coffee machine user, modifying and deleting the order, we have completed our introduction of the TLHMI coffee machine out-of-box demo experience.

Now that you are familiar with the basics, see the Smart HMI User Guide for a comprehensive list of all the outof- box features available in SLN-TLHMI-IOT. The guide also includes the additional demo applications available such as the smart elevator and smart panel applications.

NextGet Software

2.Get Software

2.1 Getting MCUXpresso IDE

The MCUXpresso IDE brings developers an easy-to-use eclipse-based development environment for NXP MCUs based on Arm Cortex-M cores, including its general-purpose crossover and wireless-enabled MCUs.

The MCUXpresso IDE offers advanced editing, compiling and debugging features. It also offers MCU-specific debugging views, code trace and profiling, multicore debugging and integrated configuration tools.

Note:There is a bug in versions before 11.7.0 that does not allow proper build in multicore projects. Therefore, version 11.7.0 (or higher) is required.

To download MCUXpresso IDE, follow the steps below:

  1. Go to MCUXpresso IDE v11.7.0 [Build 9198] [2023-01-17] and click the Downloads button

    2. Figure 24. MCUXpresso IDE home page.

  2. The Downloads page appears. Next, click the Download button

    3. Figure 25. Downloads page.

  3. The Sign In page appears. Login with your credentials

    4. Figure 26. Sign In page.

  4. Once you have signed in, select the Previous tab

    5. Figure 27. Selecting the previous tab.

  5. Select v11.7.0 (See the above note)

    6. Figure 28. Selecting v11.7.0.

  6. The Software Terms and Conditionspage appears. Read the conditions and click the I Agree button

    7. Figure 29. Software terms and conditions page

  7. The Product Download page appears from where you can download the MCUXpresso IDE. Download the appropriate version for your system

    8. Figure 30. Product download page.

  8. Open the downloaded application and follow the instructions found in the installer. After successful completion of the download and installation procedure, verify installation of the proper version by selecting the “About MCUXpresso IDE” from the Help tab at the top of the IDE window

Figure 31. About MCUXpresso IDE button.

Figure 32. MCUXpresso version information.

2.2 Installing the SDK

MCUXpresso SDK is a comprehensive software enablement package designed to simplify and accelerate application development with NXP's microcontrollers based on Arm Cortex-M cores. The MCUXpresso SDK includes production-grade software with integrated RTOS (optional), stacks and middleware, reference software and more. It is available in custom downloads based on user selections of MCU, evaluation board, and optional software components.

Before building the SLN-TLHMI-IOT SDK example projects, the target SDK must be imported into MCUXpresso IDE.

To install MCUXpresso SDK, follow the steps below:

  1. Open MCUXpresso IDE. Upon opening the application, the MCUXpresso IDE welcome screen appears, as shown in Figure 33

    2. Figure 33. MCUXpresso IDE welcome screen.

    Note: If you have already dismissed the welcome screen and are instead taken directly to the MCUXpresso IDE landing page, you can search for "welcome" from the IDE's search menu, as shown in Figure 34.

    Figure 34. Search "welcome" on search menu.

  2. Click the Download and Install SDKs link

    3. Figure 35. Download SDKs.

  3. You see a catalog of all the SDKs that can be downloaded through MCUXpresso, as shown in figure Figure 36. These SDKs provide device knowledge, drivers, middleware, and reference example applications for your development board or MCU

    4. Figure 36. SDK installation popup.

  4. Type 1170 in the filter section and select evkmimxrt1170, as shown in Figure 37

    5. Figure 37. Download evkmimxrt1170 SDK

  5. The Review Licenses prompt appears. Accept the license agreement and click the Finish button

    6. Figure 38. Check License and Download

    Note: Coffee Machine and Elevator APPs are built on SDK V2.11.1 and the Smart Panel APP is built on SDK V2.13.0.

  6. The MCUXpresso proceeds to download the SDK

    7. Note: Newer versions of the SDK (2.12.x and greater) are currently incompatible with MCUXpresso IDE v11.5.0.

    Figure 39. Downloading SDK.

2.3 Downloading SLN-TLHMI-IOT Projects

The SLN-TLHMI-IOT out-of-box projects are published under the NXP GitHub page . You can either clone the repository using Git or download a zip folder containing the source code from here .

If you are unfamiliar with Git, it is recommended to download the zip folder for now.

Figure 40. SLN-TLHMI-IOT GitHub repository.

Note: If downloading a zipped archive, be sure to unzip this folder before proceeding to the next step.

Note: Coffee Machine and Elevator APPs are released on branch V1.1.2 and the smart panel APP is on branch V1.2.0.

PreviousPlug it In
NextBuild, Run

3. Build, Run

3.1 Importing SLN-TLHMI-IOT Projects

To import the projects we downloaded into the IDE, follow the steps below:

  1. Open the File menu in the top left corner of the IDE and select the Open Projects from File System … option
  2. From the screen displayed, navigate to the location where you unzipped the sln_tlhmi_iot source code. Now, import the files listed in Figure 41

    3. Figure 41. Open SLN-TLMHI-IOT Projects from filesystem

  3. Once successfully imported, you should see the projects open in the Project Explorer pane on the left side of the IDE

Figure 42. Finish importing SLN-TLHMI-IOT application projects.

3.2 Building the SLN-TLHMI-IOT Project

The SLN-TLHMI-IOT SDK allows you to build the coffee machine application directly. The application is made up of four subprojects: bootloader, dual core xxx_cm7 and xxx_cm4 projects for the chosen application demo (coffee machine in our case), and a Light and Versatile Graphics Library (LVGL) library project.

Figure 43. SLN-TLHMI-IOT projects

The bootloader project is a first-stage bootloader that manages to jump into the Coffee Machine, Elevator, and Smart Panel Applications. This application can be used for any additional bootloader functionality needed for the product. In the Project Explorer pane, select the bootloader project file and navigate to the QuickStart Panel. Select the Build option to start the compilation and linking of this application.

Figure 44. Build

Each of the SLN-TLHMI-IOT applications utilizes LVGL for its UI design. Recompiling the code associated with the UI can add significant time and cost to the build process. In MCUXpresso, this process must be done every time the cm7 project is built. This can be especially inconvenient when no UI code has been modified because the rebuilding of LVGL’s

To help alleviate this issue, a library project named lvgl_vglite_lib_coffee_machine has been created to allow more explicit control over when the UI files should be rebuilt. Essentially, the lvgl_vglite_lib_coffee_machine project only needs to be rebuilt whenever changes have been made specifically to the UI code, otherwise, the lvgl_vglite_lib_xxx project can safely be ignored. Whenever the library project is rebuilt, MCUXpresso IDE automatically links the newly created UI library code.

Figure 45. LVGL library projects

Note: For more information about the software layout and architecture of the SLN-TLHMI-IOT, be sure to check out the Smart HMI Software Development User Guide.

The sln_smart_TLHMI_coffee_machine_cm4/cm7 projects are the out-of-box applications we used earlier to demonstrate the SLN-TLHMI-IOT's face and voice recognition capabilities. These applications (in addition to the bootloader) are flashed onto your SLN-TLHMI-IOT kit by default.

The cm4 project acts as a subproject for the main cm7 project and handles things like the UI display, camera control and more. As a subproject to the cm7 project, building the cm7 master application automatically builds the cm4 and create a combined binary containing the code from both projects.

To build the coffee machine application, click the Build icon in the Quickstart Panel.

Building may take a few minutes to complete, but do not worry, this is normal for applications of this size. Once finished, a message like the following can be seen at the bottom of the IDE.

Figure 46. Build Successful

3.3 Flashing and Debugging SLN-TLHMI-IOT Projects

With the SLN-TLHMI-IOT application project compiled, it is now time to program its associated binaries into flash.

Flashing and debugging the SLN-TLHMI-IOT kit requires a SEGGER J-Link with a 9-pin Cortex-M adapter and V7.84a or newer of the J-Link Software and Documentation Pack which can be found on the SEGGER website.

Figure 47. J-Link 9-pin adapter

To flash the kit, follow the steps below:

  1. Attach your J-Link debug probe into the J204 header as shown in Figure 50 and connect your J-Link to your computer via USB

    2. Figure 48. Attach J-Link debugger

  2. Next, provide power to the kit by plugging a USB-C cable into the kit’s USB-C port and plug the other end into your laptop/PC

    3. Figure 49. Power on SLN-TLMHI-IOT kit

  3. Select the bootloader project in the Project Explorer pane
  4. To start the process of loading the binary into flash and begin debugging, choose the Debug option in the QuickStart Panel

    5. Figure 50. Debug bootloader

  5. Select the J-Link probe that is connected to your kit and click the OK button

    6. Figure 51. Probes discover

  6. This launches the flashing tool. Now, proceed to flash the binary associated with the currently selected project

    7. Figure 52. Flash bootloader

  7. Once you have flashed the Bootloader, repeat the same processes for the sln_smart_TLHMI_coffee_machine_cm7 project

    8. Figure 53. Flash application

  8. After successfully debugging the application, the program breaks at the cm7 master project main (found in sln_smart_TLHMI_cm7.cpp). The cm7 and cm4 projects then launch into the GDB debug process

    9. Figure 54. Debug cm7 master project

  9. To suspend the debug session, click the pause/play button on the top of the screen

    10. Figure 55. Single versus simultaneous core debugging

  10. To suspend the debug session, click the pause/play button on the top of the screen

    11. Figure 56. Debug dual core project

Note: If you notice issues after performing the steps in this section, ensure that you do not have any mismatched FW and/or face database versions remaining. To reset to a known good state, follow the instructions found in the Merged binary section of the Smart HMI Software Development User Guide.

3.4 Additional Resources

By making it this far, you have successfully completed the getting started experience for the SLN-TLHMI-IOT

For a comprehensive understanding of all the Out-of-Box Experience (OoBE) features, including the additional demo applications that come flashed with the kit, be sure to check out the SLN-TLHMI-IOT User Guide.

To start building your own applications and learn more about the software architecture, available developer tools, and more, head over to the Smart HMI Software Development User Guide.

PreviousGet Software
NextDesign Resources

Design Resources

Board Documents

Software

PreviousBuild, Run