(ChinaIT.com News) In order to build a more complete IoT big data processing ecosystem and support simple and efficient migration from other time series databases to TDengine, we have developedtaosAdapter.TDengine is a big data platform specially designed and optimized by Taosi Data for the Internet of Things, Internet of Vehicles, Industrial Internet, IT Operation and Maintenance, etc. In addition to the core time series database functions that are more than 10 times faster, it also provides functions such as caching, data subscription, and streaming computing to minimize the complexity of R&D and operation and maintenance. The core code, including cluster functions, is all open source. Since TDengine announced the open source in July 2019, it has received very positive feedback on GitHub. More than 17,300 people gave star and more than 4,100 people fork the code. More and more users are beginning to adopt TDengine.TaosAdapter is a new independent program that can be included in TDengine 2.3.0.0 and above. The core starting point of taosAdapter is to solve the pain points of users and reduce migration costs.In fields such as the Internet of Things and operation and maintenance monitoring, some users are still using traditional solutions or older products to solve time series data processing problems, such as OpenTSDB. OpenTSDB is also an open source distributed time series database. It does not have its own storage engine and its related functions are completely based on HBase. Because of the earlier generation time, many operation and maintenance monitoring projects choose this system.bySF TechnologyAs an example, they adopted OpenTSDB+HBase as the storage solution for the full monitoring data of the big data monitoring platform. However, as the amount of data accessed by the platform becomes larger and larger, they have encountered many pain points, such as heavy system dependence, high cost of use, and unsatisfactory performance.
in particular:
Depends on many, poor stability: The big data monitoring platform is the underlying infrastructure. In terms of data storage, it relies on big data components such as Kafka, Spark, and HBase. In this way, the data processing link will be very long, and the longer the data link, the greater the challenge to ensure the reliability of the system. If there is a problem with the monitoring system itself, there is no way to find and locate the problem with the business system based on it.
High cost of use: The amount of written monitoring data is very large, and in order to trace historical problems, they need to save the full amount of monitoring data for more than half a year. Data storage costs remain high.
Performance cannot meet demand: OpenTSDB, as a full monitoring data storage solution, basically meets the requirements in terms of writing performance, but it can no longer meet the requirements in terms of daily large-span and high-frequency queries.
In order to solve these pain points, the engineers of SF Technology at that time believed that it was necessary to upgrade the full monitoring data storage solution. They investigated a number of time series database products and finally decided to choose TDengine. Afterwards, they modified the system based on TDengine. After the transformation is completed, the TDengine cluster can easily handle the full amount of monitoring data writing, and the current operation is stable.The improvement brought by this transformation is very eye-catching: the server-side physical machine has been reduced from 21 to 3, and the daily storage space required is only about 1/10 of OpenTSDB+HBase under the same conditions, which greatly reduces the hardware cost. In terms of query performance, in the case of using the pre-calculated function, the query p99 is within 0.7 seconds, which can already meet most of the daily query requirements; in the case of large-span (6 months) non-pre-calculated query, the first query cost When the time is about 10 seconds, the time consumption of subsequent similar queries will be greatly reduced (2-3s).Ruixin IoT PlatformThey also encountered similar problems. They previously used OpenTSDB to store time series data, which was functionally able to meet the needs; however, due to the complexity of the OpenTSDB architecture and heavy volume, it brought a lot of work to development and testing, installation and deployment, and operation and maintenance management. The trouble, with the development of business scale, the problem becomes more and more serious. After the same research, they also chose TDengine. In the process of upgrading, they need to retain historical data, so they need to migrate historical data from OpenTSDB to TDengine. To this end, they also specially developed a data migration tool and conducted detailed tests.The needs of users are the driving force for product evolution. TDengine’s R&D team began to think about this question: Since many users have this kind of migration needs, can they officially give a unified solution?
taosAdapter is our answer. taosAdapter mainly has the following functions:
It is compatible with OpenTSDB’s Telnet/JSON writing protocol. For operation and maintenance monitoring services, users can directly push the data collected by collectd and StatsD to TDengine through taosAdapter. After the data can be normally written to TDengine, Grafana can be adjusted and adapted to visualize the data written to TDengine. TDengine also provides connection plug-ins for Grafana. If you want to migrate historical data, Taosi Data has also developed a data synchronization tool DataX plug-in, which can help users automatically write data into TDengine. Users do not need to modify any line of code, only need to modify a few configurations, and the migration can be seamless.In the next step, taosAdapter will continue to improve and support the migration from more platforms to TDengine.
Articles on the website are limited to providing more information and do not represent the standpoint of this website. For reprint, please indicate the source. The reprinted article comes from the Internet. If you have any copyright issues, please contact us: content@chinait.com.