This is a guest post from Henri Gort, Platform Architect, and Mehmet Yalcinkaya, Data Lake Software Engineer from Konecranes. In their own words, “Konecranes is a world-leading group of Lifting Businesses™, serving a broad range of customers, including manufacturing and process industries, shipyards, ports, and terminals. The word ‘Konecranes’ includes the Finnish word ‘Kone,’ or machine, and ‘cranes,’ but in practice we provide productivity-enhancing lifting solutions as well as services for lifting equipment of all makes. We have approximately 16,000 employees in 50 countries.”
This post shares how Konecranes harnessed billions of records from sensors installed on our machines to improve the intelligence and security of our industrial machines.
Creating a data lake
First, we migrated our sensor data from SQL Servers to AWS. We did this by creating a data lake using the serverless AWS Glue service to efficiently move the data from SQL to Amazon S3 and Amazon DynamoDB, while performing batch ETL process on the fly. The ETL process covered the aggregation of multiple cranes’ historical data and joined them in a harmonized data structure, mining the data to derive the machine KPIs, and validating the data for operational usage in DynamoDB.
DynamoDB is the ideal database for storing our operational data. Its ability to scale, handle spikes in data traffic, and its overall performance is much better than our previous database architecture on Microsoft SQL Servers. Since the migration, we’ve seen our database speed increase and database administration costs lower.
Choosing AWS Glue
Adopting AWS Glue for ETL processing provided multiple benefits:
- No server maintenance.
- Cost savings by avoiding over- or under-provisioning resources.
- Support for multiple data sources, formats, and AWS storage services such as DynamoDB.
- Intelligent crawlers available out-of-the-box for many AWS services to infer the schema automatically.
- Developers can write classifiers to customize the crawlers and corresponding outputs.
Analyzing machine data in real time
With the migration complete, we now could improve our existing data processing infrastructure by using fully managed Amazon Kinesis and Amazon Kinesis Data Firehose to stream and aggregate the IoT data of over 20,000 cranes located around the world. The data producers on our machines push the data to Kinesis streaming endpoints as individual and batch messages.
The data routes to an AWS service for a different purpose; for example, each message is ingested to S3 via Kinesis Data Firehose for legacy storage, as well as consumed by AWS Lambda functions to derive and update the operational insights of new data to store in DynamoDB. The stream-processing pipeline powered with Kinesis dynamically updates Konecranes’s digital solutions within milliseconds.
Making our machines more safe and secure
As our fleet of machines and our customer base grow, we also need a new approach to IoT security. AWS improved the security of software running on IoT devices integrated with various lifting equipment. We also needed the capability to portably run the same software components on both the cloud and the IoT devices themselves, depending on the situation and use case. We achieved this by building our architecture on top of AWS IoT Greengrass with the security mechanisms using AWS STS, AWS Secrets Manager, AWS IoT Device Defender, and IAM roles. The results have been positive; we solved problems perpetuated by our legacy solutions and achieved lower CPU usage levels on some components when running the latest versions on top of AWS IoT Greengrass.
Conclusion
Since the migration from SQL Servers and our new architecture with AWS IoT Greengrass, API Gateway, DynamoDB, Kinesis, and Lambda, Konecranes can build cost-optimized, scalable, and low-latency infrastructure and serverless digital solutions for its customers. In addition, the management of such services with granular development stacks allowed the developers to focus on development while AWS handles maintaining the infrastructure management. The benefits of AWS for Konecranes have been remarkable: the AWS Cloud delivers performance, scalability, and low-cost and maintenance-free services, which provide reliable, low-latency digital services for our customers.
About the Authors
Henri Gort is a Platform Architect in Konecranes Digital Platform department, which is responsible for developing and delivering the digital services for the group’s customers. Henri has a background (Master’s Degree) in automation engineering & software development and he has over 10 years of experience working on service oriented architecture, serverless APIs, crane control systems, fieldbus technologies and other industrial communication concepts such as OPC-UA.
Mehmet Yalcinkaya (PhD in Information Technologies in Construction, Building Information Modelling) is a software engineer in Konecranes Global, experienced in machinery and construction industries focusing on digitalization and industrial internet of things (IIoT). Skilled in data engineering, big data, data analytics, backend development and micro-services.
