搭建斯帕克(Spark)的单机版集群

澳门美高梅手机网站,一样、成立用户

Redis作为缓存系统的话要分外有价的,在死数量方向里,也是用来缓存系统的。一般可以设想tachyon或者redis,由于redis安装以及使用还简约,所以如故先考虑了它。那么在有些气象下以保证数据的可靠性,就用采纳集群的形式部署,因而本篇小说就冲Redis
Cluster的背景讲解下安排和中期的下。

# useradd spark

约莫会包括下边的始末:

# passwd spark

  • Redis单机版的安和表明
  • Redis集群版的安装及表达
  • 行使图形化工具访问Redis
  • 使用Jedis访问Redis
  • 使用JedisCluster访问Redis Cluster

 

自此会介绍一下,怎样在斯帕克(Spark)(Spark)中运用redis,敬请期待。

老二、下充斥软件

Redis单机版安装

JDK,Scala,SBT,Maven

1.1 下载

率先去官网下充斥想使的本:
https://redis.io/download

自我这边选了一个本子没这大的,省的变型最为怪,各种软件卓殊不了。于是挑个往日的稳定性版本下载,下载redis-3.2.10.tar.gz拷贝到对象机器。

版本信息如下:

1.2 安装

把压缩文件拷贝到指定的服务器上,执行解压命令:

tar -zxvf redis-3.2.10.tar.gz

安前需事先安装必要之保管,yum -y install gcc gcc-c++ tcl,这里装之且是相似遭遇的不当所需要安装之堆栈。

安装了后,进入redis-3.2.10目,执行编译命令:make MALLOC=libc,这样虽装了了

JDK jdk-7u79-linux-x64.gz

1.3 测试

安装完毕得平素开行测试一下:

  • 起初服务器,执行下的通令:./src/redis-server redis.conf
  • 启航客户端,执行下的命:./src/redis-cli

于控制杜阿拉,输入下边发号施令:

set foo bar
get foo

省是不是正规

澳门美高梅手机网站 1

Scala scala-2.10.5.tgz

redis集群安装

照下面的不二法门,在各令机械下边还装一下redis。由于redis
cluster集群必须要多少个master,假诺做一个备份的话,就需6单节点。所以自己这边以三玉机械及,安装redis,然后每台机器上启动两个redis即可。一会面说一下,咋样以同等令机器上,启动四只redis。

SBT sbt-0.13.7.zip

2.1 安装

先是安装一下ruby,因为集群的下论是ruby版的

yum -y install ruby ruby-devel rubygems rpm-build

接下来实施:

gem install redis

然后把redis-trip.rb拷贝到/usr/local/bin产,这样就是得以其它目录执行redis-trip命令了。

cp src/redis-trip.rb /usr/local/bin

Maven apache-maven-3.2.5-bin.tar.gz

2.2 修改配置

连通下去需要遵照集群的情,制造server启动的布置了,先创建redis_cluster目录,拷贝redis.conf

[root@localnode7 redis-3.2.10]# mkdir redis_cluster
[root@localnode7 redis-3.2.10]# cp redis.conf ./redis_cluster/redis.conf

然后修改redis.conf

[root@localnode7 redis-3.2.10]# vi ./redis_cluster/redis.conf

改的情有:

bind 0.0.0.0 //为了别人能访问,这里暴力的监听了所有的地址
port 6379
daemonize yes //redis后台运行
cluster-enabled yes //开启集群  把注释#去掉
cluster-config-file nodes_6379.conf //集群的配置  配置文件首次启动自动生成
cluster-node-timeout 15000 //请求超时  默认15秒,可自行设置
appendonly yes //这里跟redis持久化的机制有关系,有兴趣可以多关注一下redis的日志记录机制 

PS:因为redis.conf安排文件较老,内容多。由此我们以改动配置的上可应用vi的疾速键/想要搜索的内容,然后按enter,可以快稳定到想假诺修改的布高达,假若发生差不多只,可以重新用n键跳转至下一个郎才女貌结果。

留意:尽管单纯是安斯帕克(Spark)(Spark)环境,则只待JDK和Scala即可,SBT和Maven是为着继续的源码编译。

2.3 启动服务器

连下就是足以启动redis服务器

[root@localnode7 redis-3.2.10]# ./src/redis-server redis_cluster/redis.conf
[root@localnode7 redis-3.2.10]# ps -aux | grep redis
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root     29361  0.0  0.0 130436  2452 ?        Ssl  13:21   0:00 ./src/redis-server 127.0.0.1:6379 [cluster]
root     29381  0.0  0.0 103464  1080 pts/0    R+   13:21   0:00 grep redis
[root@localnode7 redis-3.2.10]#

 

2.4 单个机器开动五只server

假诺想只要以相同宝机器上而启动五只redis,可以一向还拷贝一份redis.conf,命名成redis2.conf,修改中的port参数和cluster-config-file参数即可。

遵照下边的配备修改成:

bind 0.0.0.0
port 6380  // <----只有这里发生变化
daemonize yes
cluster-enabled yes
cluster-config-file nodes_6380.conf // <--只有这里发生变化
cluster-node-timeout 15000
appendonly yes

然后启动的早晚,指定为之conf即可:

./src/redis-server redis_cluster/redis2.conf

还查询启动进程:

[root@localnode4 redis-3.2.10]# ps -aux | grep redis
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root      7452  0.0  0.0 131892  2984 ?        Ssl  15:59   0:08 ./src/redis-server 0.0.0.0:6379 [cluster]
root      9337  0.0  0.0 130564  2744 ?        Ssl  16:00   0:08 ./src/redis-server 0.0.0.0:6380 [cluster]
root     32298  0.0  0.0 103464  1084 pts/0    S+   19:27   0:00 grep redis
[root@localnode4 redis-3.2.10]#

老三、解压上述文件并展开环境变量配置

2.5 创造集群

注意:

  • 想要搭建集群,至少6独节点,不然执行下的命令会报错的
  • 当创建集群时,只好用ip地址,不可知为此主机名,不然会连续不上…不了然啊啥…

实践下的命启动集群成立:

[root@localnode6 redis-3.2.10]# redis-trib.rb  create  --replicas 1 10.10.10.104:6379 10.10.10.106:6379 10.10.10.107:6379 10.10.10.104:6380 10.
10.10.106:6380 10.10.10.107:6380
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.10.10.107:6379
10.10.10.106:6379
10.10.10.104:6379
Adding replica 10.10.10.106:6380 to 10.10.10.107:6379
Adding replica 10.10.10.107:6380 to 10.10.10.106:6379
Adding replica 10.10.10.104:6380 to 10.10.10.104:6379
M: e59449112f33dcb2dfad7a1ec32920470f589c32 10.10.10.104:6379
   slots:10923-16383 (5461 slots) master
M: 566762510d6b7b2e1b361a8a8d44e4a9d788a92f 10.10.10.106:6379
   slots:5461-10922 (5462 slots) master
M: 8984e7d3e53ddcec5dd59684f69a1976a4db33ef 10.10.10.107:6379
   slots:0-5460 (5461 slots) master
S: 8e222ceb6ad4a9ca48566bd467b0e1b6573c2fc0 10.10.10.104:6380
   replicates e59449112f33dcb2dfad7a1ec32920470f589c32
S: 0c9e17b5955be559a7edf2853bff02d7415ea72f 10.10.10.106:6380
   replicates 8984e7d3e53ddcec5dd59684f69a1976a4db33ef
S: a8e54df5776b412de65b904ac3928d0d308fdc15 10.10.10.107:6380
   replicates 566762510d6b7b2e1b361a8a8d44e4a9d788a92f
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 10.10.10.104:6379)
M: e59449112f33dcb2dfad7a1ec32920470f589c32 10.10.10.104:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 8984e7d3e53ddcec5dd59684f69a1976a4db33ef 10.10.10.107:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: a8e54df5776b412de65b904ac3928d0d308fdc15 10.10.10.107:6380
   slots: (0 slots) slave
   replicates 566762510d6b7b2e1b361a8a8d44e4a9d788a92f
M: 566762510d6b7b2e1b361a8a8d44e4a9d788a92f 10.10.10.106:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 0c9e17b5955be559a7edf2853bff02d7415ea72f 10.10.10.106:6380
   slots: (0 slots) slave
   replicates 8984e7d3e53ddcec5dd59684f69a1976a4db33ef
S: 8e222ceb6ad4a9ca48566bd467b0e1b6573c2fc0 10.10.10.104:6380
   slots: (0 slots) slave
   replicates e59449112f33dcb2dfad7a1ec32920470f589c32
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

中级用大家输入一个yes确定主从的分配。通过日记就可以看到master和slave的一个分配情形,以及slot的分红。这么些slots是和存储的时光hash有关的,即一个字符串先要经哈希,知道他应储存到深节点上,然后才会师储存到对应之server中。当我们为此平日的api去查询的时节,需要查阅好诚然存储的机械,才可以诵博到数。当然使用cluster
api就非碰面生出此题目了。

# cd /usr/local/

2.6 集群环境表达

下面大家证实下集群的效用:

首先查看转集群的状态,然后创立一个key,再转换另外一个节点登录,看看是否查询到。

[root@localnode6 redis-3.2.10]# ./src/redis-cli -h 10.10.10.104 -c -p 6379
10.10.10.104:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_sent:86
cluster_stats_messages_received:86
10.10.10.104:6379> get foo
(nil)
10.10.10.104:6379> set foo bar
OK
10.10.10.104:6379> get foo
"bar"
10.10.10.104:6379>
[root@localnode6 redis-3.2.10]# ./src/redis-cli -h 10.10.10.106 -c -p 6379
10.10.10.106:6379> get foo
-> Redirected to slot [12182] located at 10.10.10.104:6379
"bar"
10.10.10.104:6379>

诸如此类我们的集群环境就是充实建筑了了。

万里长征又基本上活动了扳平步~

# tar xvf /root/jdk-7u79-linux-x64.gz

安装监控软件

假设就此windows办公,还是出众多图形化的家伙,可以连接redis的,比如redis desktop:

率先得错过脚网址去下载对应的本:
https://redisdesktop.com/download

然后就是是无脑安装了,安装完毕登录到相应的机上即可。可是貌似是单节点登录,即你看不到集群的数量。可能是自身非会合用
🙁

澳门美高梅手机网站 2

# tar xvf /root/scala-2.10.5.tgz

使用Jedis API访问Redis Cluster

首先在pom.xml中引入Jedis的jar包:

<dependency>
   <groupId>redis.clients</groupId>
   <artifactId>jedis</artifactId>
   <version>2.9.0</version>
</dependency>

然后测试一下:

public class JedisTest {
    public static void main(String[] args) {
        Jedis redis = new Jedis("10.10.10.104",6379);
        String foo = redis.get("foo");
        System.out.println(foo);
    }
}

# tar xvf /root/apache-maven-3.2.5-bin.tar.gz

使用JedisCluster访问Redis Cluster

参照下就片小说就行,试验了了可用的:
http://www.cnblogs.com/shihaiming/p/5953956.html

# unzip /root/sbt-0.13.7.zip

参考

1
安装redis日常遭逢的题目:http://www.cnblogs.com/HKUI/p/4439575.html
2 make的时候error:
jemalloc/jemalloc.h报错:http://openskill.cn/article/151
3
linux常由此目录介绍:http://www.linuxidc.com/Linux/2016-08/134701.htm
4 redis集群环境搭建:http://www.cnblogs.com/wuxl360/p/5920330.html
5 redis安装:http://www.runoob.com/redis/redis-install.html
6 redis官方文档:https://redis.io/download
7 redis
client官方文档:https://github.com/uglide/RedisDesktopManager
8 redis desktop windows下载:https://redisdesktop.com/download

改环境变量的安排文件

# vim /etc/profile

export JAVA_HOME=/usr/local/jdk1.7.0_79
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export SCALA_HOME=/usr/local/scala-2.10.5
export MAVEN_HOME=/usr/local/apache-maven-3.2.5
export SBT_HOME=/usr/local/sbt
export PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN_HOME/bin:$SBT_HOME/bin

即便配置文件生效

# source /etc/profile

测试环境变量是否见效

# java –version

java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

# scala –version

Scala code runner version 2.10.5 -- Copyright 2002-2013, LAMP/EPFL

# mvn –version

Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-15T01:29:23+08:00)
Maven home: /usr/local/apache-maven-3.2.5
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/local/jdk1.7.0_79/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-229.el7.x86_64", arch: "amd64", family: "unix"

# sbt –version

sbt launcher version 0.13.7

 

季、主机名绑定

[root@spark01 ~]# vim /etc/hosts

192.168.244.147 spark01

 

五、配置spark

切换到spark用户下

下载hadoop和spark,可利用wget命令下载

spark-1.4.0
http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.6.tgz

Hadoop
http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

解压上述文件并举行环境变量配置

改spark用户环境变量的布局文件

[spark@spark01 ~]$ vim .bash_profile

export SPARK_HOME=$HOME/spark-1.4.0-bin-hadoop2.6
export HADOOP_HOME=$HOME/hadoop-2.6.0
export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop
export PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

如配置文件生效

[spark@spark01 ~]$ source .bash_profile

修改spark配置文件

[spark@spark01 ~]$ cd spark-1.4.0-bin-hadoop2.6/conf/

[spark@spark01 conf]$ cp spark-env.sh.template spark-env.sh

[spark@spark01 conf]$ vim spark-env.sh

在前面长如下内容:

export SCALA_HOME=/usr/local/scala-2.10.5
export SPARK_MASTER_IP=spark01
export SPARK_WORKER_MEMORY=1500m
export JAVA_HOME=/usr/local/jdk1.7.0_79

生规则的童鞋可拿SPARK_WORKER_MEMORY适当设大一点,因为我虚拟机内存是2G,所以仅受了1500m。

 

配置slaves

[spark@spark01 conf]$ cp slaves slaves.template

[spark@spark01 conf]$ vim slaves

将localhost修改为spark01

 

启动master

[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ sbin/start-master.sh

starting org.apache.spark.deploy.master.Master, logging to /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.master.Master-1-spark01.out

 

查上述日志的出口内容

[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ cd logs/

[spark@spark01 logs]$ cat
spark-spark-org.apache.spark.deploy.master.Master-1-spark01.out

Spark Command: /usr/local/jdk1.7.0_79/bin/java -cp /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../conf/:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/spark/spark-1.4.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/spark/hadoop-2.6.0/etc/hadoop/ -Xms512m -Xmx512m -XX:MaxPermSize=128m org.apache.spark.deploy.master.Master --ip spark01 --port 7077 --webui-port 8080
========================================
16/01/16 15:12:30 INFO master.Master: Registered signal handlers for [TERM, HUP, INT]
16/01/16 15:12:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/16 15:12:32 INFO spark.SecurityManager: Changing view acls to: spark
16/01/16 15:12:32 INFO spark.SecurityManager: Changing modify acls to: spark
16/01/16 15:12:32 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/01/16 15:12:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/01/16 15:12:33 INFO Remoting: Starting remoting
16/01/16 15:12:33 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@spark01:7077]
16/01/16 15:12:33 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
16/01/16 15:12:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/16 15:12:34 INFO server.AbstractConnector: Started SelectChannelConnector@spark01:6066
16/01/16 15:12:34 INFO util.Utils: Successfully started service on port 6066.
16/01/16 15:12:34 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
16/01/16 15:12:34 INFO master.Master: Starting Spark master at spark://spark01:7077
16/01/16 15:12:34 INFO master.Master: Running Spark version 1.4.0
16/01/16 15:12:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/16 15:12:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080
16/01/16 15:12:34 INFO util.Utils: Successfully started service 'MasterUI' on port 8080.
16/01/16 15:12:34 INFO ui.MasterWebUI: Started MasterWebUI at http://192.168.244.147:8080
16/01/16 15:12:34 INFO master.Master: I have been elected leader! New state: ALIVE

 

自日记被为不过张,master启动正常

下来探望master的 web管理界面,默认在8080端口

澳门美高梅手机网站 3

启动worker

[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ sbin/start-slaves.sh
spark://spark01:7077

spark01: Warning: Permanently added 'spark01,192.168.244.147' (ECDSA) to the list of known hosts.
spark@spark01's password:
spark01: starting org.apache.spark.deploy.worker.Worker, logging to /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-spark01.out

输入spark01上spark用户之密码

可是经过日记的音来认同workder是否正常启动,因音讯最多,在这边就未粘出了。

[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ cd
logs/

[spark@spark01 logs]$ cat
spark-spark-org.apache.spark.deploy.worker.Worker-1-spark01.out

 

启动**spark shell**

[spark@spark01 spark-1.4.0-bin-hadoop2.6]$
bin/spark-shell –master spark://spark01:7077

16/01/16 15:33:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/16 15:33:18 INFO spark.SecurityManager: Changing view acls to: spark
16/01/16 15:33:18 INFO spark.SecurityManager: Changing modify acls to: spark
16/01/16 15:33:18 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/01/16 15:33:18 INFO spark.HttpServer: Starting HTTP Server
16/01/16 15:33:18 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/16 15:33:18 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42300
16/01/16 15:33:18 INFO util.Utils: Successfully started service 'HTTP class server' on port 42300.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
16/01/16 15:33:30 INFO spark.SparkContext: Running Spark version 1.4.0
16/01/16 15:33:30 INFO spark.SecurityManager: Changing view acls to: spark
16/01/16 15:33:30 INFO spark.SecurityManager: Changing modify acls to: spark
16/01/16 15:33:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/01/16 15:33:31 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/01/16 15:33:31 INFO Remoting: Starting remoting
16/01/16 15:33:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.244.147:43850]
16/01/16 15:33:31 INFO util.Utils: Successfully started service 'sparkDriver' on port 43850.
16/01/16 15:33:31 INFO spark.SparkEnv: Registering MapOutputTracker
16/01/16 15:33:31 INFO spark.SparkEnv: Registering BlockManagerMaster
16/01/16 15:33:31 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-7b7bd4bd-ff20-4e3d-a354-61a4ca7c4b2f/blockmgr-0e855210-3609-4204-b5e3-151e0c096c15
16/01/16 15:33:31 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB
16/01/16 15:33:31 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-7b7bd4bd-ff20-4e3d-a354-61a4ca7c4b2f/httpd-56ac16d2-dd82-41cb-99d7-4d11ef36b42e
16/01/16 15:33:31 INFO spark.HttpServer: Starting HTTP Server
16/01/16 15:33:31 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/16 15:33:31 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:47633
16/01/16 15:33:31 INFO util.Utils: Successfully started service 'HTTP file server' on port 47633.
16/01/16 15:33:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/01/16 15:33:31 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/01/16 15:33:31 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/01/16 15:33:31 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/01/16 15:33:31 INFO ui.SparkUI: Started SparkUI at http://192.168.244.147:4040
16/01/16 15:33:32 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@spark01:7077/user/Master...
16/01/16 15:33:33 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160116153332-0000
16/01/16 15:33:33 INFO client.AppClient$ClientActor: Executor added: app-20160116153332-0000/0 on worker-20160116152314-192.168.244.147-58914 (192.168.244.147:58914) with 2 cores
16/01/16 15:33:33 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160116153332-0000/0 on hostPort 192.168.244.147:58914 with 2 cores, 512.0 MB RAM
16/01/16 15:33:33 INFO client.AppClient$ClientActor: Executor updated: app-20160116153332-0000/0 is now LOADING
16/01/16 15:33:33 INFO client.AppClient$ClientActor: Executor updated: app-20160116153332-0000/0 is now RUNNING
16/01/16 15:33:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33146.
16/01/16 15:33:34 INFO netty.NettyBlockTransferService: Server created on 33146
16/01/16 15:33:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/01/16 15:33:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.244.147:33146 with 265.4 MB RAM, BlockManagerId(driver, 192.168.244.147, 33146)
16/01/16 15:33:34 INFO storage.BlockManagerMaster: Registered BlockManager
16/01/16 15:33:34 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/01/16 15:33:34 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/01/16 15:33:38 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
16/01/16 15:33:43 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/01/16 15:33:43 INFO metastore.ObjectStore: ObjectStore, initialize called
16/01/16 15:33:44 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/01/16 15:33:44 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/01/16 15:33:44 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.244.147:46741/user/Executor#-2043358626]) with ID 0
16/01/16 15:33:44 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/01/16 15:33:45 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.244.147:33017 with 265.4 MB RAM, BlockManagerId(0, 192.168.244.147, 33017)
16/01/16 15:33:46 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/01/16 15:33:48 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/01/16 15:33:48 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
16/01/16 15:33:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/16 15:33:52 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/16 15:33:54 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/01/16 15:33:54 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/01/16 15:33:54 INFO metastore.ObjectStore: Initialized ObjectStore
16/01/16 15:33:54 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
16/01/16 15:33:55 INFO metastore.HiveMetaStore: Added admin role in metastore
16/01/16 15:33:55 INFO metastore.HiveMetaStore: Added public role in metastore
16/01/16 15:33:56 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
16/01/16 15:33:56 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
16/01/16 15:33:56 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala>

开辟spark shell将来,能够写一个略的次,say hello to the world

scala> println("helloworld")
helloworld

 

再来看望spark的web管理界面,可以见见,多了一个Workders和Running Applications的信息

澳门美高梅手机网站 4

 

由来,斯帕克(Spark)的伪分布式环境搭建完毕,

发出以下几点需要小心:

1.
上述被的Maven和SBT是匪要的,只是为着继承的源码编译,所以,假设一味是单的搭建斯帕克(Spark)(Spark)环境,可不用下载Maven和SBT。

2.
欠Spark(Spark)的伪分布式环境其实是集群的根基,只需要修改极少之地点,然后copy到slave节点上即可,鉴于篇幅有限,后文再表。 

 

发表评论

电子邮件地址不会被公开。 必填项已用*标注