While working on the Cloud Resume Challenge, I encountered a common and frustrating error that many challengers face. This error is related to CORS and appears during steps 7-10 of the challenge when making an API request to your backend from your website. I'd like to share my 4-day experience in overcoming this error, what I learned, and how I resolved it.
The Objective
Nothing fancy. I just wanted to update a visitor counter on my website each time someone visited. Your front end shouldn't communicate directly with the back end. So you decouple them and use cloud services like Lambda and API Gateway to route requests and responses between them. The flow looks something like this:
In the end all that's supposed to happen is something like this:
Every time there is a visitor, meaning each time the DOM is loaded (not the best way to count, I know), the counter goes up. Simple, but not easy the first time around.
I set up the API Gateway, the Lambda Function, and the integration between the two using Terraform. This required reading some documentation and looking at a few examples. After that, I tested my Lambda code in the AWS web console with the provided test suite. I got this error: Object of type Decimal is not JSON serializable, indicating that the data returned from DynamoDB is not JSON serializable by Python. The fix was easy: just cast the Decimal data to int. I didn't need the precision of Decimal/float anyway. The test passed in the AWS web console, and the only thing left to do was write some JavaScript to make a request from my front end to my API.
I fired up my website and was greeted with the infamous CORS error.
I would then begin my annoying 4-day battle with these red boxes.
What is CORS?
This is lesson 1: understand what the error is even saying. Before trying to fix the problem, understand it. In that spirit, it's important to know what CORS is and what it does. CORS stands for Cross-Origin Resource Sharing and according to MDN, it "is an HTTP-header based mechanism that allows a server to indicate any origins (domain name, scheme, or port) other than its own from which a browser should permit loading resources." You will encounter a CORS error if an HTTP request is made from one origin to a different origin and the checks your browser does for CORS aren't passed.
The Battle Begins
The first and most immediate solution I came across is to just include this header:
return{
#... blah blah other important things to return
'headers':{
'Access-Control-Allow-Origin' : '*'
},
#... blah blah other important things to return
}
in the return block of my Lambda code. This is a really permissive solution that tells the browser that requests from any origin are allowed. Well, any origin also includes my website, so I slapped it on there. This worked! Kind of. It simplified the error, so it did something. Now I was only getting this:
I had no idea what a preflight request was, even though it was mentioned in the previous error. In line with the first lesson, look at the error and actually read it, all of it. It's not that scary and it doesn't take much time. A preflight request is one that the browser makes to the server to check if the real request is safe to send. This was saying that the response to this request didn't return the expected HTTP status. What sucks about this error is that it's not that specific; it's possible that a lot of things could cause this.
Here began the trial and error: modifying Lambda code, tweaking JavaScript, adjusting API Gateway configurations, consulting documentation, searching on Google, and watching YouTube videos. After two days of this, I made no progress. I just ended up learning more about CORS and API Gateway. That's a good thing, don't get me wrong, but it was time to be more systematic. The error told me that the response from API Gateway, and therefore my Lambda function, wasn't what the browser expected. But every time I tested it in the AWS web console, it seemed to work correctly.
I googled "Lambda internal server error 500" since this was the message (give or take a few words and curly braces) the browser was sending when it threw the preflight CORS error. I found this documentation: Troubleshooting issues with HTTP API Lambda integrations. After reading this I understood how to do lesson 2:
Setting up logs to get more specific information
I followed the instructions in the documentation and set up CloudWatch to log the integration error messages from Lambda. I got this:
the JSON object must be str, bytes or bytearray, not NoneType. Check your Lambda function code and try again.
I almost started fiddling with the Python code again, but I reread the error and realized that the JSON object being NoneType didn't jive with what I thought was going on. For reasons I'm still unsure of and investigating, the following line from my JavaScript code was causing issues with how the Lambda function was handling the input.
body: JSON.stringify({ stat_name: 'visitor_count' })
So, I wondered if there was another way to do it and remembered the tests I ran in the AWS web console.
I looked at how the test was passing input into the body, and it was different from how I did it in the JavaScript, so I wondered if maybe that would work. I tried it in my JavaScript code and voila! No more CORS errors.
Closing thoughts
That was quite the journey. I say that but, this wasn't that long a process. It was about 30 minutes to an hour every day for 4 days. It's just funnier to say I spent 4 days on it. If you stick it out, you will find the solution and I'm glad I did. Here are a few other things I learned:
If you activate CORS in API Gateway, make sure to modify the headers it includes and the content of those headers. API Gateway will ignore CORS headers your Lambda function sends in its response.
CORS was about as frustrating as people said it would be. I didn't believe this at first.
That was fun.
Happy cloud building