While the Embedded Lab Lead I had the chance to work on setting up physically embedded systems from a variety of hardware architectures (Intel, ARM, MIPS, PowerPC, Spark) as well a wide range of Operating Systems, both standard and real-time (VxWorks,VxWorks 653, Integrity, Lynx, Lynx-178, AIX, Ubuntu, Debian, Wind River Linux, Yocto, RedHawk, Android, AOSP). 

This required:

• The ability to read and understand embedded manuals, communicate with hardware manufacturers, Board Support Package and OS suppliers and other vendors. In some circumstances there would be no exact BSP for the embedded target and therefore some changes were necessary. For example, the POSIX standard states that clock_setttime() should be called before clock_gettime(), and this is something that is generally done by the BSP before an embedded target is initialized, so that when an application starts, it can automatically call clock_gettime(), get the time, from Epoch or from last boot-up time. However, this is not always the case, and changing/modifying the BSP might be necessary.
  
• Ability to debug compiler bugs, to build and create embedded kernels with the right components. For example, small things like a normal compiler optimization in RTP mode for VxWorks might turn out to be lethal for DKM mode, or even for a different architecture. In MIPS and Linux, optimization at the compiler level would be totally fine for debug libraries but not ok for release libraries without debug information. It is in these moments where experience can be crucial to narrow down a problem to a specific compiler optimization rather than looking at a possible application, kernel, driver or hardware problem.

I was also responsible to giving tours to customers, partners and others to explain how an  embedded lab works, and all the front-end and back-end infrastructure needed to have different clusters for different purposes, like performance testing, isolated testing, virtual networking, embedded targets with remote console and power, etc.

While at the lab, I also upgraded the embedded system infrastructure and started using vertical clusters, which were a lot better for scalability, maintenance and troubleshootings.