return Sign in

Programmers April 2016: Spark core technologies and practices

readTwenty-five thousand and six

Picture description

Spark is the most popular open source data memory computing framework, using the Scala language and by the UC Berkeley amplab laboratory development (2009) and in 2010 open source, in 2014 become top project of the Apache foundation. From 2014 to 2015, spark has experienced rapid development, databricks 2015 spark investigation report shows: September September 2014 to 2015, there have been more than 600 spark source contributions, and the number of 12 months prior to the only 315, spark beyond the Hadoop, indisputably become large numbers according to the most active in the field of open source projects. In addition, there are more than 200 companies for the Spark dedication to the source code, so that the Spark community has become so far the most developers to participate in the community.

This issue of the cover report, brought the following practices to share:

  • Spark Learning Guide (Zhou Zhihu, music group data center platform architect, data development supervisor)
  • DataFrame Streaming: unlimited growth form (Zhu Shixiong, Databricks Software Development Engineer)
  • Hierarchical storage: the end of the high cost of I/O Spark bottleneck (Yu Yucai, Intel big data team Engineer)
  • Practice of Spark in the United States (Ceng Linxi, Li Xuerui, Qin Siyuan, Bi Yan, Huang Zhong)
  • To Spark fire: 1.6 version problem summary and one pit(Cen Yuhai, drops travel data architecture engineer)
  • Spark practice in (Ma Yonggang, data platform development engineer)
  • MLlib Spark 2 Outlook (Liang Yanbo, Ming and a little data technology partner)
  • IFLYTEK based on user retention operation analysis and technical implementation of Spark (Yu Jun, Wan Wenqiang, Liu Danyue)


  • Twelve biggest challenges in space exploration
  • Intelligent equipment in power: the next generation of embedded sensor golf clubs, tennis, baseball bats will be more intelligent than ever
  • Hope, fear and software confusion
  • RSA "journey to the west": Six into San Francisco to go to the event, assess the situation and explore the Scriptures (Yan Shaowen, Antian enterprise security line of business Deputy General Manager)
  • Through GDC 2016 on VR's present and future (Jia Tao, Rui Yue information Nibiru co founder)


Android market channel dispersion is not a new thing, but how to efficiently pack is still a headache for many developers. This article focuses on the current three packaged programs, and from the security aspects of the three programs to review, believe it will give developers a new boost.

  • Use Cocos to develop a simple 3D VR grab the money game (Bian, Flint town of CEO)

2016 is recognized as the first year of virtual reality, VR and the game has become the focus of attention of the industry. In this paper, a simple VR 3D grab money game about the use of Cocos engine to develop the specific methods of VR games.

  • Car home mobile master App server architecture changes (Tang Quan, car home mobile service server architect)

Car home mobile master App server architecture has experienced from the outsourcing of the concept of no architecture, to the traffic surge after the restructuring, restructuring, etc.. This paper mainly introduces the main challenges, solutions and implementation process of the evolution of the main App server architecture.

  • Sadness of study (Lin Min, Professor, Guangzhou Acmdemy of Fine Arts)

The author has been in contact with a number of enterprises in recent years, there is an unexpected discovery, design is not the most difficult to be understood by the enterprise, the research is. "This two word research", in the local enterprises in the meaning is whirling. The larger enterprises have their own "Research Institute", a small business may have their own research team". These trap a lot of highly educated personnel "Research Institute" and "research team", how much is really doing research?

Author joined Ali has been nearly 7 years, often to answer questions, in many answers, he will be the most valuable summary of this. The answer may not be all right, the authors say, but it is true. Some things they do not necessarily practice well, he is also learning, but must first have this kind of consciousness.

Docker acquisition current master Unikernel community dominant discourse right Unikernel systems, from docker create libcontainer become independent container engine, to create a swarm become independent clustering scheme, now docker started investment in gold and silver, stir in the Unikernel this piece is full of opportunities of blue ocean, the move to see the docker's vision.

Small and micro finance, financial scene such as emerging bank financial business need a new flexible architecture to deal with the business impact of high concurrency, high flow. At the same time, to meet the application of fast iterative version upgrade, agile operation and maintenance management needs. In this paper, how to use the Internet to share the BoCloud Boyun application architecture and Docker container technology to help banks to deal with "Internet plus" challenge, the construction of agile IT architecture based on PaaS platform.

  • CaffeOnSpark to solve the three major issues - dialogue YAHOO machine learning platform (reporter / Zhou Jianding CSDN editor)

Feng Apache is Storm Committer Andy, but also the vice president of the Yahoo Corp responsible for big data and machine learning platform. He led Yahoo machine learning team based on open source spark and Caffe developed the depth of learning framework CaffeOnSpark, to support the business of Yahoo team in Hadoop and spark cluster seamlessly complete data processing, traditional machine learning and deep learning tasks, and in CaffeOnSpark a more mature after the open source (Https:// Feng Andy to accept the "programmer" reporter interview, from the original intention of the research and development, design ideas, technical architecture, implementation and application of the CaffeOnSpark for the interpretation of the point of view.

  • About Docker you don't know about things - Machine Docker (Yan Dong, the co-founder of the wizard cloud)

In addition to the well known Engine Docker, Docker also has a very important open source project Machine Docker. This article will introduce the relevant information of Machine Docker.

This article is the first article of the series of unmanned technology articles. The success of an unmanned vehicle involves a number of techniques, such as high precision maps, real-time positioning, and obstacle detection, and these technologies are all in the light of optical radar (LiDAR). This paper will deeply analyze how the optical radar is widely used in the various technologies of unmanned vehicle. Firstly, the paper introduces the working principle of optical radar, including how the laser scanning point cloud; then a detailed explanation of optical and radar in unmanned technology application, including map rendering, localization and obstacle detection. Finally, the paper optical radar technology is currently facing challenges, including the external environment interference, large amount of data, the cost is high,. Subsequent articles will discuss ROS systems, high precision maps, path planning, the underlying control and visual recognition technology.

  • LBS engine architecture technology (Jia Shuangcheng, Alibaba, senior engineer)

The current design model is very popular, but the design pattern in the end is what? I do not know how many people can say clearly. This paper introduces the design pattern through a concise language, and focuses on the LBS engine architecture, including technical points, failure cases, design recommendations and specific design case.

  • Design and implementation of distributed file system (Xu Haifeng, chief architect of reading group)

Read the text is Tencent and Shanda literature integration group was established, their design and deployment of a distributed file system (DFS), running in Linux system at low cost, with good can be telescopic and massive data access ability, thought to the library to provide the storage and access ability. This paper mainly discusses the system architecture and features.

Knowledge base column

  • AngularJS knowledge base content selection
  • AngularJS Study Guide
  • The practice summary of YY game cloud platform on AngularJS(Li Zeyang (giantray), YY cloud platform game play or front-end development engineer)
  • In depth understanding of Angular:ngModel binding value does not change the resolution of the solution (snow wolf, breaking the wolf, Peng Hongwei)


Comics - where did you go? (Xi Qiao)

2016 programmers (including iPad, Android, and print) please visitHttp://

Subscription advisory:

Online consulting (QQ): 2251809102
Telephone consultation: 010-64351436
Email address: reader@PROG3.COM
More news, welcome attention"Programmer's editor"