1

I am trying to start spark 2.4.5 using spark-ec2 script in my aws ubuntu ec2

I am using AWS Temporary Credentials consisting of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN as 

export AWS_ACCESS_KEY_ID=MyAccessKey
export AWS_SECRET_ACCESS_KEY=MySecretAccessKey
export AWS_SESSION_TOKEN=MySessionToken

/opt/spark-2.4.5-bin-hadoop2.7/ec2/spark-ec2 --key-pair=mykey--identity-file=./mykey.pem --region=us-east-1 launch my-spark-cluster

but whenever I run this I am getting an error that AWS was not able to validate the provided credentials

/opt/spark-2.4.5-bin-hadoop2.7/ec2/lib/boto-2.34.0/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/opt/spark-2.4.5-bin-hadoop2.7/ec2/lib/boto-2.34.0/boto/provider.py:197: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/ubuntu/.aws/credentials' mode='r' encoding='UTF-8'>
  self.shared_credentials.load_from_path(shared_path)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
ERROR:boto:401 Unauthorized
ERROR:boto:b'<?xml version="1.0" encoding="UTF-8"?>\n<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>***********************</RequestID></Response>'
Traceback (most recent call last):
  File "/opt/spark-2.4.5-bin-hadoop2.7/ec2/spark_ec2.py", line 1574, in <module>
    main()
  File "/opt/spark-2.4.5-bin-hadoop2.7/ec2/spark_ec2.py", line 1566, in main
    real_main()
  File "/opt/spark-2.4.5-bin-hadoop2.7/ec2/spark_ec2.py", line 1386, in real_main
    opts.zone = random.choice(conn.get_all_zones()).name
  File "/opt/spark-2.4.5-bin-hadoop2.7/ec2/lib/boto-2.34.0/boto/ec2/connection.py", line 1758, in get_all_zones
    return self.get_list('DescribeAvailabilityZones', params,
  File "/opt/spark-2.4.5-bin-hadoop2.7/ec2/lib/boto-2.34.0/boto/connection.py", line 1182, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>AWS was not able to validate the provided access credentials</Message></Error></Errors><RequestID>******************</RequestID></Response>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('********', *******), raddr=('********', ******)>

Any help is really appreciated.

Shourya
  • 11

0 Answers0