0

I am writing some unit test cases in Spark which then I am running in Eclipse.

Eclipse console id displaying the content of the data frames correctly but in the process is showing as well a lot lines related with the execution state, which I don't need:

[Stage 1:====>                                                   (17 + 1) / 200]
[Stage 1:======>                                                 (24 + 1) / 200]
[Stage 1:========>                                               (31 + 1) / 200]
[Stage 1:===========>                                            (40 + 1) / 200]
[Stage 1:=============>                                          (47 + 1) / 200]
[Stage 1:==============>                                         (53 + 1) / 200]
[Stage 1:=================>                                      (61 + 1) / 200]
[Stage 1:===================>                                    (70 + 1) / 200]
[Stage 1:======================>                                 (79 + 1) / 200]
[Stage 1:========================>                               (88 + 1) / 200]
[Stage 1:===========================>                            (97 + 1) / 200]
[Stage 1:=============================>                         (107 + 1) / 200]
[Stage 1:================================>                      (117 + 1) / 200]
[Stage 1:==================================>                    (127 + 1) / 200]
[Stage 1:=====================================>                 (137 + 2) / 200]
[Stage 1:========================================>              (148 + 1) / 200]
[Stage 1:===========================================>           (158 + 1) / 200]
[Stage 1:==============================================>        (168 + 1) / 200]
[Stage 1:================================================>      (177 + 1) / 200]
[Stage 1:===================================================>   (187 + 1) / 200]
[Stage 1:======================================================>(197 + 1) / 200]

There are so many of this lines that makes it hard to analyse the content of the dataframes after the multuple transformation

I have checked other questions (link1, link2) asking how to limit the logs. For example by adding the following lines:

import org.apache.log4j.Logger
import org.apache.log4j.Level
 
Logger.getLogger("org").setLevel(Level.ERROR)
Logger.getLogger("akka").setLevel(Level.ERROR)

However the stage information is still displayed. Is there any way to configure Eclipse or Spark to don't display all this information?

Ignacio Alorre
  • 7,307
  • 8
  • 57
  • 94
  • In _Window > Preferences: Run/Debug > Console_, is ticking the checkbox _Interpret ASCII control characters_ what are you looking for? – howlger Apr 05 '21 at 09:26
  • @howlger I dont see the option you mention. I have: *fix width console*, *limit console output*, *Show when program writes to standard out* and *Show when program writes to standard error* – Ignacio Alorre Apr 05 '21 at 09:36
  • 1
    Yes this is the right preference page and the checkbox is at the very bottom. The option has been around for a few versions. – howlger Apr 05 '21 at 09:41

1 Answers1

0

Can you set the logging level under spark session to ERROR and check once ? sparkSession.sparkContext.setLogLevel("ERROR")

You can refer to this doc for more information - http://spark.apache.org/docs/latest/configuration.html#configuring-logging

Shane
  • 588
  • 6
  • 20