spark 部署

发表于 分类于
部署 Spark 并演示 local 与 Spark on YARN 的 client、cluster 模式，含自定义 Python 环境与 Hive 集成。

环境信息

使用的 hadoop 完全分布式集群，节点限制，全安装在一起 用户 hadoop

1
2
3
192.168.2.241 hadoop01
192.168.2.242 hadoop02
192.168.2.243 hadoop03

spark on yarn， 当前只在 hadoop03 上 安装 spark

spark 安装

官网 https://spark.apache.org/downloads.html

hadoop03

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
wget --no-check-certificate  https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz

mkdir -p /opt/bigdata/spark
tar -zxf spark-3.2.1-bin-hadoop3.2.tgz -C /opt/bigdata/spark
cd /opt/bigdata/spark/
ln -s spark-3.2.1-bin-hadoop3.2 current

chown -R hadoop:hadoop /opt/bigdata/spark/

cat > /etc/profile.d/spark_env.sh<<-eof
export SPARK_HOME=/opt/bigdata/spark/current
export PATH=\$PATH:\$SPARK_HOME/bin
eof

## local 模式测试
spark-submit --master local[2]  --class org.apache.spark.examples.SparkPi ./examples/jars/spark-examples_2.12-3.2.1.jar 100

没问题 则可以 开始使用 spark-cluster 模式，或者 使用 spark on yarn

spark on yarn

  1. 修改 /etc/hadoop/conf/yarn-site.xml

按需修改

1
2
3
4
5
6
7
8
9
10
11
12
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,spark_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
      <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
      <value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>

重启 yarn, ResourceManager， NodeManager 等服务

注意，可能需要

1
cp ${SPARK_HOME}/yarn/spark-*-yarn-shuffle.jar ${HADOOP_HOME}/share/hadoop/yarn/lib/

不然会报错 

1
org.apache.spark.network.yarn.YarnShuffleService not found
  1. 开启 Spark 日志记录功能

/opt/bigdata/spark/current/conf/spark-env.sh

1
2
3
4
5
6
7
export JAVA_HOME=/opt/bigdata/java/current
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/bigdata/hadoop/current/lib/native
export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH
export SPARK_CLASSPATH=$SPARK_CLASSPATH
export HADOOP_HOME=/opt/bigdata/hadoop/current
export HADOOP_CONF_DIR=/opt/bigdata/hadoop/current/etc/hadoop
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=30 -Dspark.history.fs.logDirectory=hdfs://bigdata/spark-job-log"

/opt/bigdata/spark/current/conf/spark-defaults.conf （文件名一定是 defaults，带 s。写成 spark-default.conf 会被静默忽略，这里的 eventLog、historyServer、shuffle service 全都不生效，也不报错）

1
2
3
4
5
6
7
8
9
spark.shuffle.service.enabled true
spark.eventLog.enabled           true
spark.yarn.historyServer.address=hadoop03:18080
spark.history.ui.port=18080
spark.eventLog.dir               hdfs://bigdata/spark-job-log

# 选择一个就行
# spark.yarn.archive               hdfs://bigdata/libs/sparkjars.zip
spark.yarn.jars                    hdfs://bigdata/libs/spark-yarn-jars/*.jar
  1. 按配置准备好文件
1
2
3
hdfs dfs -mkdir /spark-job-log
hdfs dfs -mkdir /libs/spark-yarn-jars
hdfs dfs -put /opt/bigdata/spark/current/jars/*.jar /libs/spark-yarn-jars
  1. 启动并验证
1
2
3
4
5
6
7
8
9
10
cd /opt/bigdata/spark/current/sbin
./start-history-server.sh

# yarn client 模式，结果输出到屏幕

spark-submit --class org.apache.spark.examples.SparkPi --master  yarn /opt/bigdata/spark/current/examples/jars/spark-examples_2.12-3.2.1.jar

# yarn cluster 模式，结果不会输出到屏幕

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /opt/bigdata/spark/current/examples/jars/spark-examples_2.12-3.2.1.jar
  1. 网页查看

spark网页

yarn-网页结果

spark-结果

自定义 python 环境

测试代码 /opt/test.py. 都是 在 /opt 目录下执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import numpy as np
import jieba

print(np.version.version)
from pyspark.sql import SparkSession

# 创建 SparkSession 并启用 Hive 支持
spark = SparkSession.builder \
    .appName("ShowDatabasesExample") \
    .enableHiveSupport() \
    .getOrCreate()

# 执行 Hive 查询
databases_df = spark.sql("SHOW DATABASES")

# 显示查询结果
databases_df.show()

# 停止 SparkSession
spark.stop()

环境创建

1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建 Python 3.7.9 环境
conda create -n myenv python=3.7.9

# 激活环境
conda activate myenv

# 安装包
conda install numpy jieba


# 可以在非 myenv 环境执行
conda install -c conda-forge conda-pack
conda-pack -n myenv -o pyspark_env.tar.gz

本地模式

解压 

1
2
3
4
# tar -zxvf pyspark_env.tar.gz -c /opt/

bin/
bin/python

修改 spark-env.sh

1
2
3
4
5
6
export SPARK_SCALA_VERSION=2.12
export SPARK_PRINT_LAUNCH_COMMAND=1
# 写成 := 的形式，否则下面在 shell 里 export 的自定义解释器会被这里覆盖掉：
# spark-submit → spark-class → load-spark-env.sh 会在同一个 shell 里 source 本文件，
# 无条件 export 就等于每次都把外面设的值冲掉（这也是后面要用 --conf 兜底的原因）。
export PYSPARK_PYTHON=${PYSPARK_PYTHON:-/usr/bin/python3}

导入环境变量

1
2
export PYSPARK_PYTHON=/opt/bin/python3
export PYSPARK_DRIVER_PYTHON=/opt/bin/python3

然后执行

1
spark-submit --master local test.py

或者 直接指定

1
2
3
4
5
spark-submit \
  --master local \
  --conf spark.pyspark.python=/opt/bin/python3 \
  --conf spark.pyspark.driver.python=/opt/bin/python3 \
  test.py

yarn 模式

上传到 hdfs

1
hdfs dfs -put pyspark_env.tar.gz /user/test/python_env/

客户端模式只是 driver 跑在本地，executor 仍然在 YARN 容器里。
spark.pyspark.python 对 driver 和 executor 都生效，而 ./bin/python 是相对容器工作目录的路径——
不分发 archive 的话容器里没有这个文件。下面这条之所以”能跑通”，只因为 test.py 里只有 spark.sql、
没有启动 Python worker；一旦用上 UDF 或 RDD 的 map，就会 Cannot run program "./bin/python"。
所以 driver 端要用绝对路径，executor 端仍然得靠 --archives 分发（或者全节点预装同版本 Python）：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 客户端模式：driver 用本地绝对路径，executor 靠分发
spark-submit \
--master yarn \
--deploy-mode client \
--archives hdfs:///user/test/python_env/pyspark_env.tar.gz#pyspark_env \
--conf spark.pyspark.driver.python=/opt/bin/python3 \
--conf spark.executorEnv.PYSPARK_PYTHON=./pyspark_env/bin/python \
test.py

# 服务端模式需要 conf
spark-submit \
--master yarn \
--deploy-mode cluster \
--conf spark.yarn.dist.archives=hdfs:///user/test/python_env/pyspark_env.tar.gz#pyspark_env \
--conf spark.executorEnv.PYSPARK_PYTHON=./pyspark_env/bin/python \
--conf spark.executorEnv.PYSPARK_DRIVER_PYTHON=./pyspark_env/bin/python \
--conf spark.pyspark.python=./pyspark_env/bin/python \
--conf spark.pyspark.driver.python=./pyspark_env/bin/python \
test.py

其中 #pyspark_env 不能省略，是解压路径

这五个参数到底谁管谁

上面出现了五个看起来都在指定 Python 解释器的配置，很容易配到一堆互相覆盖、最后不知道生效的是哪个。它们的分工其实很清楚：

配置 作用对象 路径相对于
spark.pyspark.driver.python 只管 driver driver 进程的工作目录
spark.pyspark.python driver 和 executor 都管 各自的工作目录
spark.executorEnv.PYSPARK_PYTHON 只管 executor 容器工作目录
PYSPARK_DRIVER_PYTHON（环境变量） 只管 driver 提交命令所在目录
PYSPARK_PYTHON（环境变量） driver 和 executor 都管 各自的工作目录

优先级是 spark.pyspark.driver.python > spark.pyspark.python > PYSPARK_DRIVER_PYTHON > PYSPARK_PYTHON，越专用的越优先；spark.executorEnv.* 只在 executor 侧生效，和 driver 那条线互不干扰。

真正的坑在最后一列——“相对路径”相对于谁：

  • driver 在 client 模式下跑在提交机上，工作目录就是你敲命令的那个目录，./bin/python 指的是本地路径；
  • executor 永远跑在 YARN 容器里，工作目录是 NodeManager 分配的那个临时目录，./pyspark_env/bin/python 指的是 --archives 解压出来的软链接位置。

同一个 ./bin/python 写给两边，含义完全不同。所以 driver 端建议一律写绝对路径，executor 端才用相对路径配合 --archives。

验证的时候一定要带一个 Python UDF

这一节最容易被自己骗过去的地方：用 spark.sql 做验证是测不出问题的。

1
2
# 这样跑通了，什么都证明不了——全程没启动过 Python worker
spark.sql("select count(1) from ad1").show()

原因是纯 SQL 的执行完全在 JVM 里，executor 端根本不需要 Python 解释器。只有用到 Python UDF、RDD 的 map/filter、或者 pandas UDF 时，executor 才会去 fork 一个 Python worker 进程——那时才会真的按 PYSPARK_PYTHON 找解释器，找不到就 Cannot run program。

所以验证要这么写：

1
2
3
4
5
6
7
8
9
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

@udf(returnType=StringType())
def tag(x):
    import sys
    return sys.version           # 顺手把 executor 实际用的解释器版本带回来

spark.range(3).selectExpr("id").withColumn("v", tag("id")).show(truncate=False)

跑通并且 v 列显示的是你期望的那个 Python 版本，才算真的配对了。

shuffle service 配了但没生效

前面 spark on yarn 那两步配了 spark.shuffle.service.enabled=true 和 YARN 侧的 spark_shuffle aux-service，但这套东西单独开着收益是零——external shuffle service 的意义在于”executor 被回收之后，它产生的 shuffle 数据仍然能被其他 executor 读到”，而只有开了动态资源分配才会去回收 executor：

1
2
3
4
5
spark.shuffle.service.enabled    true
spark.dynamicAllocation.enabled  true          # 少了这一条，上面那条白配
spark.dynamicAllocation.minExecutors      1
spark.dynamicAllocation.maxExecutors      50
spark.dynamicAllocation.executorIdleTimeout  60s

另外有个多版本共存时的踩坑点：spark.shuffle.service.port（默认 7337）必须和 yarn-site.xml 里 spark.shuffle.service.port 一致。集群上同时跑两个 Spark 版本时，两个版本的 shuffle jar 抢同一个端口，后起的那个 NodeManager 会起不来；正确做法是给不同版本配不同的 aux-service 名字和端口（比如 spark_shuffle_3 / 7338）。

问题

适配 FI 遇到的问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "ename": "AnalysisException",
  "evaluate": "org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.spark.sql.hive.HiveSessionResourceLoader cannot be cast to org.apache.spark.sql.hive.HiveACLSessionResourceLoader",
  "execution_count": 0,
  "status": "error",
  "traceback": [
    "Traceback (most recent call last):",
    "  File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py\", line 723, in sql",
    "    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped",
    "  File \"/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py\", line 1305, in __call__",
    "    answer, self.gateway_client, self.target_id, self.name",
    "  File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 117, in deco",
    "    raise converted from None",
    "pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.spark.sql.hive.HiveSessionResourceLoader cannot be cast to org.apache.spark.sql.hive.HiveACLSessionResourceLoader"
  ]
}

处理方式

1
hive-site.xml配置  需要注释hive.security.metastore.authenticator.manager这个配置

spark-hive-yarn-cluster 运行日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
[test@hadoop01 ~]$ spark-submit   --master yarn   --deploy-mode cluster   --conf spark.yarn.dist.archives=hdfs:///user/test/pyspark_env.tar.gz#pyspark_env   --conf spark.executorEnv.PYSPARK_PYTHON=./pyspark_env/bin/python   --conf spark.executorEnv.PYSPARK_DRIVER_PYTHON=./pyspark_env/bin/python   --cnf spark.pyspark.python=./pyspark_env/bin/python   --conf spark.pyspark.driver.python=./pyspark_env/bin/python   test.py

Spark Command: /opt/bigdata/java/current/bin/java -cp /opt/bigdata/spark/current/conf/:/opt/bigdata/spark/current/jars/*:/opt/bigdata/hadoop/current/tc/hadoop/ org.apache.spark.deploy.SparkSubmit --master yarn --deploy-mode cluster --conf spark.executorEnv.PYSPARK_PYTHON=./pyspark_env/bin/python -conf spark.yarn.dist.archives=hdfs:///user/test/pyspark_env.tar.gz#pyspark_env --conf spark.executorEnv.PYSPARK_DRIVER_PYTHON=./pyspark_env/bin/python--conf spark.pyspark.driver.python=./pyspark_env/bin/python --conf spark.pyspark.python=./pyspark_env/bin/python test.py
========================================
2024-11-15 17:38:40,071 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2024-11-15 17:38:40,131 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
2024-11-15 17:38:40,983 INFO conf.Configuration: resource-types.xml not found
2024-11-15 17:38:40,984 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-11-15 17:38:41,004 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MBper container)
2024-11-15 17:38:41,005 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
2024-11-15 17:38:41,005 INFO yarn.Client: Setting up container launch context for our AM
2024-11-15 17:38:41,028 INFO yarn.Client: Setting up the launch environment for our AM container
2024-11-15 17:38:41,054 INFO yarn.Client: Preparing resources for our AM container
2024-11-15 17:38:41,302 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/HikariCP-2.5.1.jar
2024-11-15 17:38:41,435 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/JLargeArrays-1.5.jar
2024-11-15 17:38:41,440 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/JTransforms-3.1.jar
2024-11-15 17:38:41,445 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/RoaringBitmap-0.9.0.jar
2024-11-15 17:38:41,452 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/ST4-4.0.4.jar
2024-11-15 17:38:41,457 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/accessors-smart-1.2.jar
2024-11-15 17:38:41,461 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/activation-1.1.1.jar
2024-11-15 17:38:41,465 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/aircompressor-0.10.jar
2024-11-15 17:38:41,471 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/algebra_2.12-2.0.0-M2.jar
2024-11-15 17:38:41,476 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/animal-sniffer-annotations-1.17.jar
2024-11-15 17:38:41,482 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/antlr-runtime-3.5.2.jar
2024-11-15 17:38:41,487 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/antlr4-runtime-4.8-1.jar
2024-11-15 17:38:41,491 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/aopalliance-1.0.jar
2024-11-15 17:38:41,495 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/aopalliance-repackaged-2.6.1.jar
2024-11-15 17:38:41,501 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/arpack_combined_all-0.1.jar
2024-11-15 17:38:41,507 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/arrow-format-2.0.0.jar
2024-11-15 17:38:41,513 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/arrow-memory-core-2.0.0.jar
2024-11-15 17:38:41,517 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/arrow-memory-netty-2.0.0.jar
2024-11-15 17:38:41,521 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/arrow-vector-2.0.0.jar
2024-11-15 17:38:41,526 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/audience-annotations-0.5.0.jar
2024-11-15 17:38:41,531 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/automaton-1.11-8.jar
2024-11-15 17:38:41,536 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/avro-1.8.2.jar
2024-11-15 17:38:41,539 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/avro-ipc-1.8.2.jar
2024-11-15 17:38:41,544 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/avro-mapred-1.8.2-hadoop2.jar
2024-11-15 17:38:41,548 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/aws-java-sdk-bundle-1.11.563.jar
2024-11-15 17:38:41,554 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/azure-keyvault-core-1.0.0.jar
2024-11-15 17:38:41,558 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/azure-storage-7.0.0.jar
2024-11-15 17:38:41,562 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/bcpkix-jdk15on-1.60.jar
2024-11-15 17:38:41,566 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/bcprov-jdk15on-1.60.jar
2024-11-15 17:38:41,569 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/bonecp-0.8.0.RELEASE.jar
2024-11-15 17:38:41,573 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/breeze-macros_2.12-1.0.jar
2024-11-15 17:38:41,576 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/breeze_2.12-1.0.jar
2024-11-15 17:38:41,579 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/cats-kernel_2.12-2.0.0-M4.jar
2024-11-15 17:38:41,582 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/checker-qual-2.5.2.jar
2024-11-15 17:38:41,585 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/chill-java-0.9.5.jar
2024-11-15 17:38:41,588 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/chill_2.12-0.9.5.jar
2024-11-15 17:38:41,592 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-beanutils-1.9.4.jar
2024-11-15 17:38:41,595 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-cli-1.2.jar
2024-11-15 17:38:41,598 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-codec-1.10.jar
2024-11-15 17:38:41,601 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-collections-3.2.2.jar
2024-11-15 17:38:41,604 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-compiler-3.0.16.jar
2024-11-15 17:38:41,607 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-compress-1.20.jar
2024-11-15 17:38:41,610 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-configuration2-2.1.1.jar
2024-11-15 17:38:41,613 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-crypto-1.1.0.jar
2024-11-15 17:38:41,616 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-daemon-1.0.13.jar
2024-11-15 17:38:41,619 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-dbcp-1.4.jar
2024-11-15 17:38:41,623 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-httpclient-3.1.jar
2024-11-15 17:38:41,626 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-io-2.5.jar
2024-11-15 17:38:41,629 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-lang-2.6.jar
2024-11-15 17:38:41,632 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-lang3-3.10.jar
2024-11-15 17:38:41,636 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-logging-1.1.3.jar
2024-11-15 17:38:41,639 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-math3-3.4.1.jar
2024-11-15 17:38:41,642 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-net-3.1.jar
2024-11-15 17:38:41,645 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-pool-1.5.4.jar
2024-11-15 17:38:41,648 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/commons-text-1.6.jar
2024-11-15 17:38:41,651 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/compress-lzf-1.0.3.jar
2024-11-15 17:38:41,654 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/core-1.1.2.jar
2024-11-15 17:38:41,658 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/curator-client-2.13.0.jar
2024-11-15 17:38:41,661 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/curator-framework-2.13.0.jar
2024-11-15 17:38:41,664 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/curator-recipes-2.13.0.jar
2024-11-15 17:38:41,667 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/datanucleus-api-jdo-4.2.4.jar
2024-11-15 17:38:41,670 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/datanucleus-core-4.1.17.jar
2024-11-15 17:38:41,673 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/datanucleus-rdbms-4.1.19.jar
2024-11-15 17:38:41,676 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/derby-10.12.1.1.jar
2024-11-15 17:38:41,679 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/dnsjava-2.1.7.jar
2024-11-15 17:38:41,682 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
2024-11-15 17:38:41,685 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/ehcache-3.3.1.jar
2024-11-15 17:38:41,688 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/eigenbase-properties-1.1.5.jar
2024-11-15 17:38:41,691 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/error_prone_annotations-2.2.0.jar
2024-11-15 17:38:41,696 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/failureaccess-1.0.jar
2024-11-15 17:38:41,699 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/flatbuffers-java-1.9.0.jar
2024-11-15 17:38:41,702 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/generex-1.0.2.jar
2024-11-15 17:38:41,704 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/geronimo-jcache_1.0_spec-1.0-alpha-1.jar
2024-11-15 17:38:41,707 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/gmetric4j-1.0.10.jar
2024-11-15 17:38:41,709 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/gson-2.2.4.jar
2024-11-15 17:38:41,712 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/guava-27.0-jre.jar
2024-11-15 17:38:41,715 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/guice-4.0.jar
2024-11-15 17:38:41,717 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/guice-servlet-4.0.jar
2024-11-15 17:38:41,720 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-annotations-3.2.2.jar
2024-11-15 17:38:41,723 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-auth-3.2.2.jar
2024-11-15 17:38:41,725 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-aws-3.2.2.jar
2024-11-15 17:38:41,729 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-azure-3.2.2.jar
2024-11-15 17:38:41,732 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-client-3.2.2.jar
2024-11-15 17:38:41,735 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-common-3.2.2.jar
2024-11-15 17:38:41,739 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-hdfs-client-3.2.2.jar
2024-11-15 17:38:41,742 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-mapreduce-client-common-3.2.2.jar
2024-11-15 17:38:41,744 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-mapreduce-client-core-3.2.2.jar
2024-11-15 17:38:41,747 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-mapreduce-client-jobclient-3.2.2.jar
2024-11-15 17:38:41,750 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-openstack-3.2.2.jar
2024-11-15 17:38:41,753 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-yarn-api-3.2.2.jar
2024-11-15 17:38:41,756 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-yarn-client-3.2.2.jar
2024-11-15 17:38:41,759 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-yarn-common-3.2.2.jar
2024-11-15 17:38:41,762 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-yarn-registry-3.2.2.jar
2024-11-15 17:38:41,765 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-yarn-server-common-3.2.2.jar
2024-11-15 17:38:41,768 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hadoop-yarn-server-web-proxy-3.2.2.jar
2024-11-15 17:38:41,770 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-beeline-2.3.8.jar
2024-11-15 17:38:41,773 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-cli-2.3.8.jar
2024-11-15 17:38:41,775 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-common-2.3.8.jar
2024-11-15 17:38:41,778 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-exec-2.3.8-core.jar
2024-11-15 17:38:41,780 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-jdbc-2.3.8.jar
2024-11-15 17:38:41,783 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-llap-common-2.3.8.jar
2024-11-15 17:38:41,785 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-metastore-2.3.8.jar
2024-11-15 17:38:41,788 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-serde-2.3.8.jar
2024-11-15 17:38:41,791 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-service-rpc-3.1.2.jar
2024-11-15 17:38:41,793 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-shims-0.23-2.3.8.jar
2024-11-15 17:38:41,796 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-shims-2.3.8.jar
2024-11-15 17:38:41,798 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-shims-common-2.3.8.jar
2024-11-15 17:38:41,801 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-shims-scheduler-2.3.8.jar
2024-11-15 17:38:41,803 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-storage-api-2.7.2.jar
2024-11-15 17:38:41,806 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hive-vector-code-gen-2.3.8.jar
2024-11-15 17:38:41,808 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hk2-api-2.6.1.jar
2024-11-15 17:38:41,811 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hk2-locator-2.6.1.jar
2024-11-15 17:38:41,813 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/hk2-utils-2.6.1.jar
2024-11-15 17:38:41,816 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/htrace-core4-4.1.0-incubating.jar
2024-11-15 17:38:41,819 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/httpclient-4.5.6.jar
2024-11-15 17:38:41,822 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/httpcore-4.4.12.jar
2024-11-15 17:38:41,825 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/istack-commons-runtime-3.0.8.jar
2024-11-15 17:38:41,828 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/ivy-2.4.0.jar
2024-11-15 17:38:41,830 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/j2objc-annotations-1.1.jar
2024-11-15 17:38:41,833 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-annotations-2.10.0.jar
2024-11-15 17:38:41,835 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-core-2.10.0.jar
2024-11-15 17:38:41,839 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-core-asl-1.9.13.jar
2024-11-15 17:38:41,842 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-databind-2.10.0.jar
2024-11-15 17:38:41,844 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-dataformat-cbor-2.10.0.jar
2024-11-15 17:38:41,847 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-dataformat-yaml-2.10.0.jar
2024-11-15 17:38:41,849 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-datatype-jsr310-2.11.2.jar
2024-11-15 17:38:41,852 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-jaxrs-base-2.9.10.jar
2024-11-15 17:38:41,854 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-jaxrs-json-provider-2.9.10.jar
2024-11-15 17:38:41,857 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-mapper-asl-1.9.13.jar
2024-11-15 17:38:41,859 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-module-jaxb-annotations-2.10.0.jar
2024-11-15 17:38:41,862 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-module-paranamer-2.10.0.jar
2024-11-15 17:38:41,864 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jackson-module-scala_2.12-2.10.0.jar
2024-11-15 17:38:41,866 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.activation-api-1.2.1.jar
2024-11-15 17:38:41,869 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.annotation-api-1.3.5.jar
2024-11-15 17:38:41,871 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.inject-2.6.1.jar
2024-11-15 17:38:41,873 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.servlet-api-4.0.3.jar
2024-11-15 17:38:41,876 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.validation-api-2.0.2.jar
2024-11-15 17:38:41,878 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.ws.rs-api-2.1.6.jar
2024-11-15 17:38:41,881 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jakarta.xml.bind-api-2.3.2.jar
2024-11-15 17:38:41,883 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/janino-3.0.16.jar
2024-11-15 17:38:41,885 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/javassist-3.25.0-GA.jar
2024-11-15 17:38:41,888 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/javax.activation-api-1.2.0.jar
2024-11-15 17:38:41,890 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/javax.inject-1.jar
2024-11-15 17:38:41,892 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/javax.jdo-3.2.0-m3.jar
2024-11-15 17:38:41,895 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/javolution-5.5.1.jar
2024-11-15 17:38:41,897 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jaxb-api-2.2.11.jar
2024-11-15 17:38:41,899 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jaxb-runtime-2.3.2.jar
2024-11-15 17:38:41,902 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jcip-annotations-1.0-1.jar
2024-11-15 17:38:41,904 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jcl-over-slf4j-1.7.30.jar
2024-11-15 17:38:41,906 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jdo-api-3.0.1.jar
2024-11-15 17:38:41,909 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-client-2.30.jar
2024-11-15 17:38:41,911 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-common-2.30.jar
2024-11-15 17:38:41,914 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-container-servlet-2.30.jar
2024-11-15 17:38:41,916 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-container-servlet-core-2.30.jar
2024-11-15 17:38:41,919 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-hk2-2.30.jar
2024-11-15 17:38:41,921 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-media-jaxb-2.30.jar
2024-11-15 17:38:41,924 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jersey-server-2.30.jar
2024-11-15 17:38:41,926 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jetty-util-9.4.40.v20210413.jar
2024-11-15 17:38:41,928 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jetty-util-ajax-9.4.40.v20210413.jar
2024-11-15 17:38:41,931 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jline-2.14.6.jar
2024-11-15 17:38:41,933 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jniloader-1.1.jar
2024-11-15 17:38:41,935 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/joda-time-2.10.5.jar
2024-11-15 17:38:41,938 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jodd-core-3.5.2.jar
2024-11-15 17:38:41,940 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jpam-1.1.jar
2024-11-15 17:38:41,943 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/json-1.8.jar
2024-11-15 17:38:41,945 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/json-smart-2.3.jar
2024-11-15 17:38:41,947 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/json4s-ast_2.12-3.7.0-M5.jar
2024-11-15 17:38:41,950 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/json4s-core_2.12-3.7.0-M5.jar
2024-11-15 17:38:41,952 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/json4s-jackson_2.12-3.7.0-M5.jar
2024-11-15 17:38:41,954 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/json4s-scalap_2.12-3.7.0-M5.jar
2024-11-15 17:38:41,957 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jsp-api-2.1.jar
2024-11-15 17:38:41,959 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jsr305-3.0.0.jar
2024-11-15 17:38:41,962 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jta-1.1.jar
2024-11-15 17:38:41,964 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/jul-to-slf4j-1.7.30.jar
2024-11-15 17:38:41,967 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-admin-1.0.1.jar
2024-11-15 17:38:41,969 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-client-1.0.1.jar
2024-11-15 17:38:41,972 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-common-1.0.1.jar
2024-11-15 17:38:41,974 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-core-1.0.1.jar
2024-11-15 17:38:41,977 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-crypto-1.0.1.jar
2024-11-15 17:38:41,980 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-identity-1.0.1.jar
2024-11-15 17:38:41,983 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-server-1.0.1.jar
2024-11-15 17:38:41,985 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-simplekdc-1.0.1.jar
2024-11-15 17:38:41,988 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerb-util-1.0.1.jar
2024-11-15 17:38:41,990 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerby-asn1-1.0.1.jar
2024-11-15 17:38:41,992 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerby-config-1.0.1.jar
2024-11-15 17:38:41,995 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerby-pkix-1.0.1.jar
2024-11-15 17:38:41,997 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerby-util-1.0.1.jar
2024-11-15 17:38:42,000 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kerby-xdr-1.0.1.jar
2024-11-15 17:38:42,003 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kryo-shaded-4.0.2.jar
2024-11-15 17:38:42,006 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-client-4.12.0.jar
2024-11-15 17:38:42,008 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-admissionregistration-4.12.0.jar
2024-11-15 17:38:42,011 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-apiextensions-4.12.0.jar
2024-11-15 17:38:42,013 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-apps-4.12.0.jar
2024-11-15 17:38:42,016 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-autoscaling-4.12.0.jar
2024-11-15 17:38:42,018 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-batch-4.12.0.jar
2024-11-15 17:38:42,020 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-certificates-4.12.0.jar
2024-11-15 17:38:42,023 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-common-4.12.0.jar
2024-11-15 17:38:42,025 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-coordination-4.12.0.jar
2024-11-15 17:38:42,028 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-core-4.12.0.jar
2024-11-15 17:38:42,030 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-discovery-4.12.0.jar
2024-11-15 17:38:42,032 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-events-4.12.0.jar
2024-11-15 17:38:42,035 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-extensions-4.12.0.jar
2024-11-15 17:38:42,037 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-metrics-4.12.0.jar
2024-11-15 17:38:42,040 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-networking-4.12.0.jar
2024-11-15 17:38:42,042 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-policy-4.12.0.jar
2024-11-15 17:38:42,044 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-rbac-4.12.0.jar
2024-11-15 17:38:42,046 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-scheduling-4.12.0.jar
2024-11-15 17:38:42,048 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-settings-4.12.0.jar
2024-11-15 17:38:42,051 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/kubernetes-model-storageclass-4.12.0.jar
2024-11-15 17:38:42,053 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/leveldbjni-all-1.8.jar
2024-11-15 17:38:42,056 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/libfb303-0.9.3.jar
2024-11-15 17:38:42,058 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/libthrift-0.12.0.jar
2024-11-15 17:38:42,060 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
2024-11-15 17:38:42,063 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/log4j-1.2.17.jar
2024-11-15 17:38:42,065 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/logging-interceptor-3.12.12.jar
2024-11-15 17:38:42,067 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/lz4-java-1.7.1.jar
2024-11-15 17:38:42,070 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/machinist_2.12-0.6.8.jar
2024-11-15 17:38:42,072 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/macro-compat_2.12-1.1.1.jar
2024-11-15 17:38:42,074 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/mesos-1.4.0-shaded-protobuf.jar
2024-11-15 17:38:42,077 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/metrics-core-4.1.1.jar
2024-11-15 17:38:42,079 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/metrics-graphite-4.1.1.jar
2024-11-15 17:38:42,081 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/metrics-jmx-4.1.1.jar
2024-11-15 17:38:42,084 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/metrics-json-4.1.1.jar
2024-11-15 17:38:42,086 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/metrics-jvm-4.1.1.jar
2024-11-15 17:38:42,089 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/minlog-1.3.0.jar
2024-11-15 17:38:42,091 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/native_ref-java-1.1.jar
2024-11-15 17:38:42,094 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/native_system-java-1.1.jar
2024-11-15 17:38:42,096 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_ref-linux-armhf-1.1-natives.jar
2024-11-15 17:38:42,098 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_ref-linux-i686-1.1-natives.jar
2024-11-15 17:38:42,101 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_ref-linux-x86_64-1.1-natives.jar
2024-11-15 17:38:42,104 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_ref-osx-x86_64-1.1-natives.jar
2024-11-15 17:38:42,106 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_ref-win-i686-1.1-natives.jar
2024-11-15 17:38:42,109 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_ref-win-x86_64-1.1-natives.jar
2024-11-15 17:38:42,111 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_system-linux-armhf-1.1-natives.jar
2024-11-15 17:38:42,113 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_system-linux-i686-1.1-natives.jar
2024-11-15 17:38:42,116 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_system-linux-x86_64-1.1-natives.jar
2024-11-15 17:38:42,118 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_system-osx-x86_64-1.1-natives.jar
2024-11-15 17:38:42,121 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_system-win-i686-1.1-natives.jar
2024-11-15 17:38:42,123 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netlib-native_system-win-x86_64-1.1-natives.jar
2024-11-15 17:38:42,126 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/netty-all-4.1.51.Final.jar
2024-11-15 17:38:42,130 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/nimbus-jose-jwt-7.9.jar
2024-11-15 17:38:42,134 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/objenesis-2.6.jar
2024-11-15 17:38:42,136 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/okhttp-2.7.5.jar
2024-11-15 17:38:42,140 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/okhttp-3.12.12.jar
2024-11-15 17:38:42,143 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/okio-1.14.0.jar
2024-11-15 17:38:42,146 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/opencsv-2.3.jar
2024-11-15 17:38:42,149 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/orc-core-1.5.12.jar
2024-11-15 17:38:42,152 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/orc-mapreduce-1.5.12.jar
2024-11-15 17:38:42,155 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/orc-shims-1.5.12.jar
2024-11-15 17:38:42,160 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/oro-2.0.8.jar
2024-11-15 17:38:42,164 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/osgi-resource-locator-1.0.3.jar
2024-11-15 17:38:42,166 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/paranamer-2.8.jar
2024-11-15 17:38:42,169 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/parquet-column-1.10.1.jar
2024-11-15 17:38:42,171 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/parquet-common-1.10.1.jar
2024-11-15 17:38:42,173 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/parquet-encoding-1.10.1.jar
2024-11-15 17:38:42,176 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/parquet-format-2.4.0.jar
2024-11-15 17:38:42,178 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/parquet-hadoop-1.10.1.jar
2024-11-15 17:38:42,180 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/parquet-jackson-1.10.1.jar
2024-11-15 17:38:42,183 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/protobuf-java-2.5.0.jar
2024-11-15 17:38:42,185 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/py4j-0.10.9.jar
2024-11-15 17:38:42,187 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/pyrolite-4.30.jar
2024-11-15 17:38:42,189 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/re2j-1.1.jar
2024-11-15 17:38:42,191 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/remotetea-oncrpc-1.1.2.jar
2024-11-15 17:38:42,194 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/scala-collection-compat_2.12-2.1.1.jar
2024-11-15 17:38:42,196 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/scala-compiler-2.12.10.jar
2024-11-15 17:38:42,198 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/scala-library-2.12.10.jar
2024-11-15 17:38:42,200 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/scala-parser-combinators_2.12-1.1.2.jar
2024-11-15 17:38:42,202 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/scala-reflect-2.12.10.jar
2024-11-15 17:38:42,204 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/scala-xml_2.12-1.2.0.jar
2024-11-15 17:38:42,207 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/shapeless_2.12-2.3.3.jar
2024-11-15 17:38:42,209 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/shims-0.9.0.jar
2024-11-15 17:38:42,211 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/slf4j-api-1.7.30.jar
2024-11-15 17:38:42,213 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/slf4j-log4j12-1.7.30.jar
2024-11-15 17:38:42,216 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/snakeyaml-1.24.jar
2024-11-15 17:38:42,218 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/snappy-java-1.1.8.2.jar
2024-11-15 17:38:42,220 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-catalyst_2.12-3.1.2.jar
2024-11-15 17:38:42,222 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-core_2.12-3.1.2.jar
2024-11-15 17:38:42,224 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-ganglia-lgpl_2.12-3.1.2.jar
2024-11-15 17:38:42,226 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-graphx_2.12-3.1.2.jar
2024-11-15 17:38:42,230 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-hadoop-cloud_2.12-3.1.2.jar
2024-11-15 17:38:42,232 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-hive-thriftserver_2.12-3.1.2.jar
2024-11-15 17:38:42,234 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-hive_2.12-3.1.2.jar
2024-11-15 17:38:42,236 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-kubernetes_2.12-3.1.2.jar
2024-11-15 17:38:42,238 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-kvstore_2.12-3.1.2.jar
2024-11-15 17:38:42,241 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-launcher_2.12-3.1.2.jar
2024-11-15 17:38:42,243 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-mesos_2.12-3.1.2.jar
2024-11-15 17:38:42,245 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-mllib-local_2.12-3.1.2.jar
2024-11-15 17:38:42,247 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-mllib_2.12-3.1.2.jar
2024-11-15 17:38:42,249 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-network-common_2.12-3.1.2.jar
2024-11-15 17:38:42,251 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-network-shuffle_2.12-3.1.2.jar
2024-11-15 17:38:42,253 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-repl_2.12-3.1.2.jar
2024-11-15 17:38:42,256 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-sketch_2.12-3.1.2.jar
2024-11-15 17:38:42,258 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-sql_2.12-3.1.2.jar
2024-11-15 17:38:42,260 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-streaming_2.12-3.1.2.jar
2024-11-15 17:38:42,262 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-tags_2.12-3.1.2-tests.jar
2024-11-15 17:38:42,264 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-tags_2.12-3.1.2.jar
2024-11-15 17:38:42,266 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-unsafe_2.12-3.1.2.jar
2024-11-15 17:38:42,268 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spark-yarn_2.12-3.1.2.jar
2024-11-15 17:38:42,270 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spire-macros_2.12-0.17.0-M1.jar
2024-11-15 17:38:42,272 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spire-platform_2.12-0.17.0-M1.jar
2024-11-15 17:38:42,274 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spire-util_2.12-0.17.0-M1.jar
2024-11-15 17:38:42,276 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/spire_2.12-0.17.0-M1.jar
2024-11-15 17:38:42,278 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/stax-api-1.0.1.jar
2024-11-15 17:38:42,281 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/stax2-api-3.1.4.jar
2024-11-15 17:38:42,283 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/stream-2.9.6.jar
2024-11-15 17:38:42,285 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/super-csv-2.2.0.jar
2024-11-15 17:38:42,287 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/threeten-extra-1.5.0.jar
2024-11-15 17:38:42,289 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/token-provider-1.0.1.jar
2024-11-15 17:38:42,291 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/transaction-api-1.1.jar
2024-11-15 17:38:42,294 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/univocity-parsers-2.9.1.jar
2024-11-15 17:38:42,296 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/velocity-1.5.jar
2024-11-15 17:38:42,298 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/wildfly-openssl-1.0.7.Final.jar
2024-11-15 17:38:42,300 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/woodstox-core-5.0.3.jar
2024-11-15 17:38:42,302 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/xbean-asm7-shaded-4.15.jar
2024-11-15 17:38:42,304 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/xz-1.5.jar
2024-11-15 17:38:42,306 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/zjsonpatch-0.3.0.jar
2024-11-15 17:38:42,308 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/zookeeper-3.4.14.jar
2024-11-15 17:38:42,310 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://example-hdfs/user/test/spark/libs/spark-arn-jars/zstd-jni-1.4.8-1.jar
2024-11-15 17:38:42,327 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs:/user/test/pyspark_env.tar.gz#pyspark_env
2024-11-15 17:38:42,344 INFO yarn.Client: Uploading resource file:/home/test/test.py -> hdfs://example-hdfs/user/test/.sparkStaging/application_173088658438_0003/test.py
2024-11-15 17:38:42,861 INFO yarn.Client: Uploading resource file:/opt/bigdata/spark/current/python/lib/pyspark.zip -> hdfs://example-hdfs/user/test/.sarkStaging/application_1730886518438_0003/pyspark.zip
2024-11-15 17:38:42,950 INFO yarn.Client: Uploading resource file:/opt/bigdata/spark/current/python/lib/py4j-0.10.9-src.zip -> hdfs://example-hdfs/use/test/.sparkStaging/application_1730886518438_0003/py4j-0.10.9-src.zip
2024-11-15 17:38:43,419 INFO yarn.Client: Uploading resource file:/tmp/spark-0d4cb7de-c038-4966-a554-391cf5eb2814/__spark_conf__7713810412816413378.zp -> hdfs://example-hdfs/user/test/.sparkStaging/application_1730886518438_0003/__spark_conf__.zip
2024-11-15 17:38:43,563 INFO spark.SecurityManager: Changing view acls to: test
2024-11-15 17:38:43,564 INFO spark.SecurityManager: Changing modify acls to: test
2024-11-15 17:38:43,565 INFO spark.SecurityManager: Changing view acls groups to:
2024-11-15 17:38:43,567 INFO spark.SecurityManager: Changing modify acls groups to:
2024-11-15 17:38:43,568 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(test; groups with view permissions: Set(); users  with modify permissions: Set(test); groups with modify permissions: Set()
2024-11-15 17:38:43,690 INFO yarn.Client: Submitting application application_1730886518438_0003 to ResourceManager
2024-11-15 17:38:43,982 INFO impl.YarnClientImpl: Submitted application application_1730886518438_0003
2024-11-15 17:38:44,988 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:44,995 INFO yarn.Client:
         client token: N/A
         diagnostics: AM container is launched, waiting for AM container to Register with RM
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1731663523749
         final status: UNDEFINED
         tracking URL: http://hadoop03:8088/proxy/application_1730886518438_0003/
         user: test
2024-11-15 17:38:45,998 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:47,001 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:48,004 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:49,008 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:50,010 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:51,013 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:52,016 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:53,019 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:54,022 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:55,025 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:56,028 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:57,031 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:58,035 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:38:59,039 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:39:00,042 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:39:01,045 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:39:02,047 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:39:03,051 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:39:04,054 INFO yarn.Client: Application report for application_1730886518438_0003 (state: ACCEPTED)
2024-11-15 17:39:05,057 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:05,058 INFO yarn.Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: hadoop02
         ApplicationMaster RPC port: 43158
         queue: default
         start time: 1731663523749
         final status: UNDEFINED
         tracking URL: http://hadoop03:8088/proxy/application_1730886518438_0003/
         user: test
2024-11-15 17:39:06,061 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:07,064 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:08,067 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:09,070 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:10,072 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:11,075 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:12,079 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:13,081 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:14,084 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:15,087 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:16,090 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:17,093 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:18,096 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:19,099 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:20,102 INFO yarn.Client: Application report for application_1730886518438_0003 (state: RUNNING)
2024-11-15 17:39:21,105 INFO yarn.Client: Application report for application_1730886518438_0003 (state: FINISHED)
2024-11-15 17:39:21,106 INFO yarn.Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: hadoop02
         ApplicationMaster RPC port: 43158
         queue: default
         start time: 1731663523749
         final status: SUCCEEDED
         tracking URL: http://hadoop03:8088/proxy/application_1730886518438_0003/
         user: test
2024-11-15 17:39:21,135 INFO util.ShutdownHookManager: Shutdown hook called
2024-11-15 17:39:21,138 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-8bcc6fb4-6632-4593-8e86-445ccba65497
2024-11-15 17:39:21,149 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0d4cb7de-c038-4966-a554-391cf5eb2814