Subscribe to the industry RSS CSDN home> The industry

BDTC 2015 - HUAWEI big data technology in Beijing, the smooth ending

Published in15:21 2015-12-18| Time reading| sourceCSDN| ZeroArticle comments| authorFu Jiang

Abstract:December 11th afternoon, 2015 China big data technology conference (BDTC 2015) - HUAWEI big data technology in Beijing, the successful end of the special.

December 11th afternoon, 2015 China big data technology conference (BDTC 2015) - HUAWEI big data technology in Beijing, the successful end of the special. Huawei as the leading big data applications in combat on behalf of, in the big data industry landing has been a lot of practical examples, this year, Huawei sent a number of elite technical experts, big data technology innovation and business value, and the recommendation of the prediction technology and its application, financial data, Huawei data platform business practice topic to share insights and case studies.

HUAWEI Zhang Dazhen as host

HUAWEI big data cloud solutions chief architect Yu Hu

- "FusionInsight: business driven open big data platform"

Yu Hu to bring the theme of "FusionInsight: business driven open big data platform" speech. His speech content mainly includes four aspects: (1) the industry trends and challenges; (2) business driven data platform architecture design; (3) open, credible and agility of data platform; (4) the successful practice.

Big data has become a "new normal", from the stage of speculation entered a substantive stage, people are from "what is the big data" to "how do I from data acquisition value", "how do I build the my data strategy." Big data technology in the enterprise gradually get the practice, the ecosystem is becoming more mature. Modern enterprises need to deal with the challenges from three aspects: business understanding, data science, and information technology.The key to deal with big data challenges lies in: large data analysis applications, analysis modeling and algorithms, architecture and platform technology.

However, fusioninsight Huawei's R & D is a based on open source kernel to create a unified data processing platform, provide a mixed load handling capacity, support from batch and interactive query, data mining, to real-time flow calculation and query various large data scenarios. Among the innovative Carbon data storage in the analysis of technology, breaking the performance bottleneck of large data OLAP analysis, performance improvement 5-30X. All components are through FusionInsight Manager to provide a plug-in framework for on-demand installation.

In the open source contribution, HUAWEI actively contribute to the community, has entered the world's first lineup. In October 16th, HUAWEI's latest foreign open source StreamCQL, a query language built on the distributed stream processing platform, architecture support built on a variety of streaming processing engine, currently the main adapter Apache Storm, to provide the data stream filtering, conversion, split, merge and other functions, as well as window based statistics, association, the user uses the class SQL mode development business, can significantly reduce the complexity of the development.

HUAWEI Big Data Modeling Research Director Engineer Wang Fangshan

The discussion and application of incremental / online learning technology

Wang Fangshan shared a large scale of machine learning in the application of model delay and management issues; incremental / online learning starting point and goal; SAILS adaptive large-scale incremental learning system.

At present, the model of large-scale machine learning in the application of delay and management challenges, the mainstream model is a good model for off-line training, online or offline algorithm results, the results of online use. Major challenges in the following aspects:

  • Because offline is generally based on the number of days or longer period, the model and the actual business delay;
  • In the case of large amount of data, each new data if all the merger to the historical data for training, resource consumption, time and effort;
  • Online decision-making and training isolation, can not use the user feedback to adjust the model in a timely manner;
  • The life cycle and validity of the model can be adapted to the validity of the model;

So the question is whether it can be based on the new data, combined with the user feedback, the incremental updating model of the fast low cost, to adapt to the needs of business change.

The sails adaptive large-scale incremental learning system, Fang Shan Wang concluded that for non online applications, through relatively lightweight wrapper method to construct an incremental learning system, without reducing the effect at the same time to greatly reduce the computational complexity; for online applications, optimization algorithms and systems, in the distributed case, good convergence, can greatly enhance the effect, play big data value. In addition, Wang Fangshan also compared the batch mode and online mode in principle description, mainly adapts to the scene and the advantages and disadvantages of the difference.

HUAWEI big data system architect Li Kun

-- "real time analysis and innovation based on Carbon: Spark, Astro"

Li Kun's speech includes several aspects: Spark introduction, HUAWEI big data business, Spark Based Innovation: Carbon- high performance OLAP query; Astro- real time data analysis.

Hadoop Native Carbon:, for OLAP analysis and optimization of the storage. It is a good solution to the large data OLAP analysis of pain points: (1) ORC/Parquet do OLAP analysis of short board; (2) can not be a good tool for docking BI.

As a new file format and OLAP query calculation, Carbon is mainly characterized by:

  • According to the data model in advance to organize the data, with the pre processing time in exchange for query time: stored, distributed multi-dimensional index, inverted index, intelligent pre convergence, etc.
  • With SQL Spark query optimization to achieve more calculation under pressure, second level high performance ad-hoc query, improve query performance 5~10 times
  • Convenient docking BI tool, SparkSQL according to the conditions of the SQL statements to dynamically select the best query path.

Astro: Spark SQL into the ecology of on HBase.

Support for the use of SQL HBase query and data update and delete, and Spark ecological seamless docking, to support the results of the query for DataFrame, ML, Graphx and other libraries do depth analysis.

HUAWEI personalized recommendation chief architect Zhang Xu

- the application of recommended and Forecast Technology

Zhang Xu first introduced the overall architecture of HUAWEI's big data product platform: 4+2. 2 represents data governance and knowledge base, and two core support for several major applications: recommendation, prediction, marketing, and business intelligence.Zhang Xu believes that the platform is a knowledge base is a part of the wisdom of the center. The knowledge base includes user portrait, product library, algorithm base, rule base and so on. It can be extended on the platform. Recommended platform is the core of the recommendation engine, by the batch processing and real-time engine communication.

Examples of the user analysis program. Business appeal is based on the user's behavior trajectory, the identification of the user family type (whether there is an elderly, whether there is a child, etc.) to further identify potential user groups, enhance the penetration rate. The system can through off-line analysis (video data users) + SMS / call / offline marketing (customer side) way.

HUAWEI big data architect and financial sector chief technical adviser Sheng Yun

-- the commercial practice of HUAWEI big data platform

Sheng Yun focuses on the HUAWEI big data solutions in operators, public security, transportation, education, health care, banking, insurance, securities and other fields of wide application.

The overall rise of the mobile Internet, finance and telecommunications and other traditional industries in transition demand is particularly evident. The use of big data, can help these traditional industries to quickly complete the business transformation, accelerate business innovation, and to ensure that enterprises continue to maintain strong competitiveness.

To the bank as an example, the resolve scheme through the deployment of Huawei fusioninsight enterprise class data, historical data management, real-time credit, real-time event marketing, small micro credit won customer forecast, financial products, precision recommendation and other financial business innovation launched one after another, significantly improved in terms of business accuracy, real-time, and customer satisfaction.

For the future, HUAWEI will continue to invest in the field of big data, and hope that the establishment of the ecological system a "a riot of colour". HUAWEI is committed to build enterprise class big data support platform, so that it has become the industry's innovative soil, really let the data "wisdom" to speak, so as to support a variety of applications in various industries.

HUAWEI big data analysis technology expert Liu Shikai

-- Application of automatic modeling

Liu Shikai's speech includes: a review and Prospect of data mining, the application of automated modeling scenarios, the practice of automated modeling.

In which the practice of automated modeling, Liu Shikai introduced the supervised and unsupervised automatic modeling:

A supervised automatic modeling: This article mainly introduces how to carry on the data processing, feature engineering, algorithm and parameters selection, by parallel data processing to optimize execution speed, through the AUC data processing method; by gbdt, depth of learning, such as screening, feature combination; through GridSearch algorithm search parameters,.

Unsupervised automatic modeling: mainly introduces the method of fast anomaly detection algorithm, automatic clustering analysis and clustering parameter search, and the similarity measure of feature.

Activity site

Interactive Q & A

Prize draw Award

step on
  • CSDN official micro channel
  • Scan two-dimensional code, to CSDN Tucao
  • Micro signal: CSDNnews
Programmer mobile terminal subscription Download

Microblogging attention

Related popular articles