Flink webui when running from IDE

Question

I am trying to see my job in the web ui.

I use createLocalEnvironmentWithWebUI, code is running well in IDE, but impossible to see my job in http://localhost:8081/#/overview

      val conf: Configuration = new Configuration()
      import org.apache.flink.configuration.ConfigConstants
      conf.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true)
      val env =  StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf)
      env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)


      val rides = env.addSource(
        new TaxiRideSource("nycTaxiRides.gz", 1,100))//60, 600))

      val filteredRides = rides
        .filter(r => GeoUtils.isInNYC(r.startLon, r.startLat) && GeoUtils.isInNYC(r.endLon, r.endLat))
        .map(r => (r.passengerCnt, 1))
        .keyBy(_._1)
        .window(TumblingTimeWindows.of(Time.seconds(5)))
        .sum(1)
        .map(r => (r._1.toString+"test", r._2))

      filteredRides.print()
      env.execute("Taxi Ride Cleansing")

Did I need to setup something else?

David Anderson · Accepted Answer · 2023-07-30T00:33:47.237

36

I was able to start the Flink webui from IntelliJ by adding flink-runtime-web to the dependencies for my project. I did this by adding this to my pom.xml file:

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-runtime-web</artifactId>
        <version>${flink.version}</version>
    </dependency>

You also then need to create a local execution environment that includes the WebUI:

    Configuration conf = new Configuration();
    env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

edited Jul 30 '23 at 00:33

answered Oct 29 '17 at 13:52

David Anderson

39,434
4
33
60

Thank you @alpinegizmo. I added this dependency to my pom.xml. It didn't work. Did you need to build a jar to have it working, as suggest @Amarjit? – GermainGum Oct 29 '17 at 14:41
1

No, I did not build a jar. That approach does work, but I thought your goal was to run the app entirely in the IDE. Did you check the logs? Now that it's working I get a line that says "16:20:25,425 INFO org.apache.flink.runtime.minicluster.FlinkMiniCluster - Starting JobManger web frontend". Before I added the dependency, I got an error instead. – David Anderson Oct 29 '17 at 20:22
Maybe your pom doesn't define flink.version, in which case put the version number instead (e.g., 1.3.2). – David Anderson Oct 29 '17 at 20:24
Indeed I have the same line as you, I had to stop from the terminal : '/bin/stop-cluster.sh'. Now it is working! Thx alpinegizmo. – GermainGum Oct 29 '17 at 21:52
See: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Can-t-access-Flink-Dashboard-at-8081-running-Flink-program-using-Eclipse-td8016.html – Georg Heiler Dec 02 '17 at 13:28
Thank you all, it does work for me. However, when the job finishes, the web UI disappears too. So is there a solution that I can check the web UI information after the job is finished from IntelliJ? – Chenghao Jan 03 '18 at 08:12
Thanks @DavidAnderson, it worked! This part really needs to be in the official docs. I couldn't find it anywhere... Any chance to add this to the documentation in: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/local_execution.html ? – Rafi Aroch Sep 12 '18 at 15:05
1

On Flink 1.15+ the dependency is now `flink-runtime-web` (no Scala binary version) – Nicus Jul 29 '23 at 06:42

score 5 · Answer 2 · edited Feb 08 '22 at 12:08

As of Flink version 1.5.0 adding the dependency mentioned before and using the following piece of code to start the StreamEnvironment worked for me:

Configuration config = new Configuration();
config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(config);

While the processing is running, the web UI is available under http://localhost:8081

score 0 · Answer 3 · answered Oct 29 '17 at 12:26

Yes, if you want to use WebUI Dashboard, then you need to create an executable jar and then submit this jar to Flink dashboard. I will explain you this step by step

Step 1: Creating the jar from IDE code

you may need to change your execution environment to

StreamExecutionEnvironment envrionment = StreamExecutionEnvironment.getExecutionEnvironment();

In case you have multiple jars, then set the main class in Main-Class: variable of Manifest.mf file
Then create a jar using build artifacts in your IDE

Step 2: Start flink-local cluster which will show you dashboard.

I will assume that you have not downloaded the Flink binary, you can easily download it here, if you have Macintosh, I will suggest you to use brew install apache-flink which will download the latest stable release which is 1.3.2 currently
Ok, now you have to go to path where flink is installed and start local cluster

Step# 3 : submitting the job

submit jar via submit new job option and then run it

Thanks @Amarjit, I would like to avoid to create the jar. I tried this solution but did not work for me : https://stackoverflow.com/questions/35138538/how-can-i-start-the-flink-job-manager-web-interface-when-running-flink-from-an-i — GermainGum, Oct 29 '17 at 14:39

Flink webui when running from IDE

3 Answers3

Linked