$ docker run -it -v ~/.aws:/home/glue_user/.aws -v $WORKSPACE_LOCATION:/home/glue_user/workspace/ -e AWS_PROFILE=$PROFILE_NAME -e DISABLE_SSL=true -rm -p 4040:4040 -p 18080:18080 -name glue_spark_submit amazon/aws-glue-libs:glue_libs_4.0.0_image_01 spark-submit /home/glue_user/workspace/src/$SCRIPT_FILE_NAME For more information, see Using interactive sessions with AWS Glue. Interactive sessions allow you to build and test applications from the environment of your choice. For more information, see the AWS Glue Studio User Guide. You can inspect the schema and data results in each step of the job. You can visually compose data transformation workflows and seamlessly run them on AWS Glue's Apache Spark-based serverless ETL engine. The AWS Glue Studio visual editor is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in AWS Glue. If you prefer local development without Docker, installing the AWS Glue ETL library directory locally is a good choice. This helps you to develop and test Glue job script anywhere you prefer without incurring AWS Glue cost. If you prefer local/remote development experience, the Docker image is a good choice. If you want to use your own local environment, interactive sessions is a good choice. For more information, see Using Notebooks with AWS Glue Studio and AWS Glue. If you prefer an interactive notebook experience, AWS Glue Studio notebook is a good choice. If you prefer no code or less code experience, the AWS Glue Studio visual editor is a good choice. You can choose any of the above options based on your requirements.
0 Comments
Leave a Reply. |