Firmware

Manufacturing Test Firmware

Who: Inboard Technology
When: August 2018

Inboard contacted us to quickly create manufacturing test firmware for their new hardware platform. The engineer who wrote the original manufacturing test firmware had left the company, and the tests needed to be updated and expanded to support the new platform.

We ported existing tests to the new firmware platform and updated them to match the recent APIs. We also implemented additional tests for new hardware features. Along the way, we helped Inboard resolve hundreds of compiler warnings and a handful of customer-facing bugs.

After completing the manufacturing test firmware, we updated the factory test station sequence.

We provided overview documentation describing the logic behind each manufacturing test and instructions for updating and modifying the tests. We also created summary documentation describing the full manufacturing test flow, including the Windows application.

The project was completed in three weeks, enabling Inboard to begin testing their prototypes without delay.

Testimonial

Dan Casciato, Manufacturing & Test Engineer at Inboard Technology, had this to say:

It's simple: Embedded Artistry is very easy to do business with. Their deep industry knowledge and expertise makes them the ideal partner for any embedded project, no matter the size. Really, they're the best. 

Snapdragon Flight Driver Development

Who: RavenOps
When: 1/18-3/18
Where: San Francisco, CA
Languages: C, C++

RavenOps transitioned from the Crazyflie2 to the Snapdragon Flight platform. We assisted with the transition in the following ways:

  • Researching the platform and identifying an approach for writing custom device drivers for hardware components which integrate with the DSP
  • Writing a proof-of-concept driver and test application using a Snapdragon driver framework
  • Porting RavenOps's Time-of-Flight driver from the Crazyflie2 to the Snapdragon flight
  • Writing a demo application which demonstrates use of the driver
  • Documenting development environment setup and assisting with development environment setup
  • Documenting the driver framework and demonstration project, with notes on how to expand APIs and add drivers

The following articles were published as a result of our learnings on this project:

AWS IoT Migration

Who: Petzila
When: 5/2017-8/2017
Where: San Francisco, CA
Languages: C

When we first met Petzi, they were developing their second generation treat camera. We assisted with new product development in the following ways:

  • Ported existing platform from MC200 to MW300
  • Migrated to new SDK
  • Bring-up and debugging of new hardware design
  • Added support for AWS IoT infrastructure
  • Upgraded authentication and server interactions to work with new AWS-based backend
  • Implemented and validated new OTA logic, with fallback support for the old server environment
  • Resolved issues with on-boarding and provisioning flow
  • Tuned firmware settings for new hardware

The following article was published as a result of our learnings on this project:

Rylo Camera

Who: Rylo
When: 9/2016-4/2017
Where: San Francisco, CA
Languages: C++, CMake, Groovy, Jenkins Pipeline

We supported Rylo's first product development effort with the following services:

  • Schematic review
  • Build system & build server setup
  • Creation of product development roadmap & schedule
  • Creation of manufacturing test plan
  • Managed multiple external vendors
  • Build support in China
  • Bring-up of manufacturing test firmware
  • Bring-up of customer camera firmware behavior

RearVision

Who: Pearl Automation
When: 2014-2016
Where: Scotts Valley, CA
Languages: C, C++, ARM assembly, Thrift, Python

Pearl's RearVision consists of two separate embedded devices - the camera system, mounted on the car's license plate, as well as an in-car OBD-II powered system.

Phillip was an early hire at Pearl - #12, hired right after series A closed. He was the primary developer for the camera system.

In the early stages of the company, Phillip performed initial bringup of the system and was responsible for:

  • SOC Bringup
  • RTOS Selection
  • Setting up the build system
  • Porting initial software from gcc to clang
  • Software / source tree architecture
  • Implementing the DFU framework to flash devices
  • Creating a USB CDC shell and defining commands for various drivers
  • Driver bringup for multiple devices
    • SPI
    • SPI-NOR
    • Solar IC
    • GasGauge
    • USB
    • EHCI (ported u-boot EHCI stack to our system)
    • Cameras
  • Defining factory test methods and processes

After the bringup stage, focus shifted to firmware support for the camera sub-system. Responsibilities included:

  • Implementing low-level C++ functionality in the RearVision frame firmware
  • Converting early libraries and drivers to C++
  • Conversion of C-based USB Host stack to C++
  • Ported vendor host-side Camera USB/UVC APIs from Linux to an RTOS
  • Managed interfaces for client software to initiate and configure video streams.
  • Worked closely with our ISP vendor to:
    • Identify and prioritize requirements and issues
    • Assist in debugging ISP, video, and memory issues
    • Optimize ISP pipeline performance to reduce latency and improve quality over BT/Wifi
  • Optimized video system memory usage
    • Created a buffer pool class to re-use large buffers in camera path
    • Eliminated all copies from USB->Camera path when getting new frames.
    • Converting malloc()/free() calls to use smart pointers
    • Debugging memory stompers

Further Reading:
PearlAuto Website

iPhone 6 & 6+

Who: Apple
When: 2013-2014
Where: Cupertino, CA; Shenzhen, CN
Languages: C, Lua

After the iPhone 5C, Phillip transitioned onto bringup for the iPhone 6/6+. A shortage of team members meant that Phillip managed both the iPhone 6 and iPhone 6+ projects. After EVT was completed, he transitioned to project management of the iPhone test lines. He traveled to most builds for both programs during this product cycle.

Factory SW:
During the prototype phase, Phillip was responsible for rapid bringup of new drivers for evaluating parts at prototype builds. He worked closely with the product development and reliability teams to make sure they had the ability to validate all the parts under consideration while at the builds. He also expanded our driver and factory test support to encompass new design changes.

Phillip spent significant time training and developing the CM software team. Unlike on the iPhone 5C, he managed to get the iPhone 6 CM team to help write software, allowing the firmware team to manage their immense workload by offloading tasks to the CM team. Phillip's strategy was to keep the hardest tasks for himself - once a task could be crystallized into a simple set of instructions, he would utilize the CM team. This allowed him to stay ahead of deadlines with an extremely challenging schedule, and helped to expand his communication and project management abilities. It is not simple to provide instructions across language barriers!

Current testing was expanded on the iPhone 6/6+ programs. Limits could now be set for different phones while keeping the same core software logic (previously, duplicate software was required). Also, much of the current testing coverage was pushed up to SMT to catch failures before final assembly.

Factory PM:
After transitioning to project management, Phillip's primary responsibility was the factory test and calibration lines. He was responsible for coordinating software and fixture readiness, ensuring that software, fixtures, and other critical deliverables were ready for each builds. He frequently briefed executives on program status and readiness for hardware builds.

Phillip became well-versed in crisis management, as there were always new failures, missing coverage, late deliverables, and missing support. Prioritization and triage were important in making sure build goals were met without further delays.

Phillip managed test plans, test coverage, and line flow for iPhone 6 and iPhone 6+ development, including these areas:

  • SMT
  • Subassembly
  • IQC
  • FATP
  • Rel
  • Packout

iPhone 5C

Who: Apple
When: 2013
Where: Cupertino, CA; Shanghai, CN
Languages: C, lua

The iPhone 5C was Phillip's first project at Apple. Phillip joined during the EVT stage of the project, and his primary focus was expanding test coverage, debugging failures, fixing issues, and driving down retest rates and cycle time. He traveled to each remaining development build until the product ramped (4 builds).

Prior to the development builds, Phillip worked with with a variety of external teams to drive and implement test requirements:

  • System HW
  • RF HW
  • Touch firmware/test
  • Sensor HW
  • Operations
  • Reliability

During a late-stage part switch, Phillip performed a rapid bringup of a new accelerometer - this was a high pressure situation, as the build material was being held until firmware support was completed.

At development builds, Phillip worked closely with the Apple System HW team to debug failing units. Together, they worked to distinguish hardware failures from software bugs and add new coverage to catch failures that were missed.

Phillip joined forces with the System HW and RF HW teams to expand the current testing suite run during burnin. This testing suite helped identify numerous software bugs, SMT issues, and caught a major reliability issue prior to MP. The success of current testing on this product resulted its expansion and use at SMT in later iPhone products.

Phillip also worked closely with the operations team to monitor retest rates and test cycle times, helping improve UPH on key test items.

Phillip taught the CM's software team to help triage and debug manufacturing test issues. During later iPhone builds, failure and retest quantities become too large for one person to filter through. Developing the skills of the CM software team helped the firmware team stay afloat and focus on new issues rather than duplicates.