In the wake of the e-commerce boom, warehouse throughput at fulfillment and distribution centers has become increasingly important. To fill and send orders, a wide range of items need to be handled accurately and at high speeds — and because both worker availability and floor space are limited resources, a premium is placed on fully automated workflows that function within a small footprint.
There are a variety of approaches to address these challenges, but perhaps the most innovative is the introduction of deep learning into the warehouse space. Deep learning, or DL, is a growing subset of machine-learning technologies that enables computers to learn in the same organic way as humans, which is by example. Training machines to recognize subtle differences between similar components allows them to become highly efficient at identifying and sorting them, which is ideally suited to the most fundamental of warehouse processes: bin picking.
Figure 1. An illustration of SICK’s PLB system performing a picking task. Source: SICK
Although robots have been incorporated into the bin-picking process for some time, they have worked against significant limitations. A vision-based robot guidance system may be able to locate, with a fair degree of success, objects it has been specifically programmed to recognize within a well-organized environment — yet any given bin might also contain unfamiliar objects with a completely random organization. From the robot’s point of view, moreover, the organization of the bin changes each time an object is picked. Add to this the fact that there may be other important picking considerations such as an object’s position and orientation, and the scope of the challenge becomes clear.
To be sure, secondary processes have been developed to address some of these issues. A bin-picking task requiring specific part orientation, for example, may make use of a separate 2D camera for verification. If the camera ascertains that the part is facing in the wrong direction, or is upside-down, the guidance system can instruct the robot to release and re-pick. Yet this approach also has the potential to slow down workflow, increase the cost of hardware and require additional floor space.
Changing the equation
Deep learning changes the equation for a vision-based robot guidance system, essentially by teaching it to understand more of what it is “seeing.” As part of what is often referred to as the Industrial Internet of Things (IIoT) or Industry 4.0, a concept for an evolving industrial landscape focused heavily on interconnectivity and smart automation, DL trains artificial neural networks (ANNs) to recognize representative features of real-world objects. These ANNs are based on the biological neural networks of real-world living brains.
Not surprisingly, academic literature is filled with innovative research on deep learning. Creating real-world industrial applications takes a different kind of innovation, however — essentially “connecting the dots” to solve problems that haven’t been solved before, often in areas where DL has yet to be applied. This is an apt characterization of the partnership forged by SICK, a producer of intelligent sensors for industrial automation, and Stäubli, a mechatronics company whose products include robots designed for bin picking. SICK’s DL web service, dStudio, is being used to train neural networks as the brains behind a 3D robot guidance system it calls part localization in bin picking, or PLB.
Features of the PLB
SICK’s PLB system is designed for precise localization of randomly oriented parts in bins and boxes. It works in multiple stages: image acquisition, part localization, verification of the ability to grip a given part and transfer of pick coordinates to the robot’s programmable logic controller (PLC).
An essential component of the system is integration with a high-performance 3D camera that enables it to “see” what it’s looking at. The camera is used to acquire dense 3D images independent of part features and external motion. Its high-performance characteristics also offer superior image quality, reliable results unaffected by ambient light conditions and insensitivity to contrast and color variations — an ideal feature for working with parts made of metal.
Acquired images are compared to the knowledge stored in the system’s neural network, which has been trained through input of CAD models of both the robot’s gripper and the types of parts it’s liable to encounter. Possible pick positions for the gripper are then output in robot coordinates. Flexible algorithms are also applied to recorded image data, allowing the verification of an available space for object placement and avoidance of gripper collision.
With a typical part localization time of 5-10 seconds, the system supports the handling of complex and overlapping shapes at a high production rate. SICK’s PLB system configuration also puts the ability to both locate and classify parts into a single sensor, enabling operations to be performed directly in bins without the need for secondary processes such as 2D camera verification.
All these features combined add up to the PLB’s ability to detect anomalies and differentiate between similar shapes and colors with a precision previously unseen in the warehousing industry.
Additional key features
Camera and robot integration
The PLB system’s ability to be integrated with a variety of 3D cameras allows the creation of tailored bin-picking solutions, solving challenging scenarios such as identification of reflective or shiny objects and sorting in layers or by complex geometries.
Laser-based triangulation cameras ensure noise minimization, while active illumination stereo cameras provide high resolution. A single camera can be configured to cover several bins simultaneously; configuration of multiple camera heads connected to a single computer is also supported. Camera calibration is made simple by SICK’s pre-calibrated setup, providing out-of-the-box data that can be exchanged by users.
Similarly, while Stäubli bin-picking robots offer an ideal interface for SICK’s PLB system, calibration tools for a variety of other robots and PLCs are also available.
Figure 2. Stäubli’s products include robots designed for bin picking. Source: Stäubli International
Intuitive user interface
A significant feature of SICK’s approach to incorporating deep learning into industrial processes is the intuitive user interface of its dStudio service. Extensive AI knowledge is not required to train neural networks with new scenarios for classifying images. Training progress and success is also graphically displayed so the ANN can be assessed before being deployed into a production space.
In the context of the PLB system, this intuitive interface extends into a variety of methods for teaching in new parts. The neural network can accept input from CAD drawings that will teach the system to localize parts that are arbitrarily oriented or concealed. It can also perform geometrical matching based on dimensional data, enabling precision position determination for rectangular- or cylindrical-shaped components and sorting according to defined sequences. Edge matching, based upon reference images marked up by an operator, can be used for localization of parts with irregular shapes and without pronounced surface variations. Additional plug-ins are available to enable detection of shapes such as predefined boxes, flat regions and circles.
Workforce resources
It’s often said that a company’s most valuable resource is its people, and the SICK PLB system offers human resource benefits even as it enables an automated workflow to fill in the gaps caused by staffing shortages. By putting robots on the front line of bin picking, the potential for injuries can be reduced; human workers can also be utilized for tasks of greater value.
At the same time, the robotic workforce’s ability to function on a 24-hour/7-day schedule both increases reliability and decreases labor costs.
Building the Industrial Internet of Things
Working in conjunction with one of several available 3D cameras and bin-picking robots like those available from Stäubli, the SICK PLB system represents a successful case study within the ongoing development of the IIoT. Its utility extends beyond warehouse applications into multiple industries including automotive, machine tooling and several others. In addition to bin picking, the PLB can be used for part depalletizing, component kitting and assembly, machine loading and other tasks. The common thread running through all of them is increased speed and efficiency for automated workflows within small spaces.
To learn more about how SICK’s strides in sensor intelligence, visit the SICK website.
