When CURL fails

Every once in a while, as a programmer, you seemingly come upon a problem that flummoxes you, irritates you and takes you through a wide array of emotions – and in the end when you solve it, makes you look back and think, why did this issue even take so much time to fix.

Here’s one from my diaries: I was working for a client which dealt with lots of financial data, building an app which dealt with peer to peer wallet system and phone recharges. The app’s backend API was built using a custom PHP framework.

After having built the system, I went to my client’s office to deploy the system on their homegrown servers. One thing that I should mention is, being a company that handles lot of financial data – the entire campus was very secure. Regular audits, Access only to authorised personnel. But what struck me odd was that even the computers were not connected to Internet – A team of 20 programmers shared two computers that connected to Internet, which they used when they wanted to research the issues they were working on.

Having given a machine, I installed PHP on the servers and deployed my app – so I can start testing. This is where all the troubles started for me. For one of the methods, I had written some code to call client’s API to recharge their phone – but just no matter what happened, the API wouldn’t work.

Looking at the code, the code was written in a simple CURL snippet

// create a new cURL resource$ch = curl_init();
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/");
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and execute it
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);

But now here’s the funny part – the same URL was accessible via the browser, postman clients and everywhere. But through the code, it would simply refuse to work.

After looking into the logs – and learning more about CURL and how it works – I added the curl_error function to see what is failing this bit of code.

Error 7 – Bad request. Could not connect to host. 

Ah. So now we were getting somewhere. But there could be multiple reasons why we could not connect to the host – network dropping out, Internet not accessible and so on.

That’s when it hit me like a flash of light – The entire computers at the client’s office were passing through a proxy. Now the URLs worked perfectly because the browsers were configured. I opened a terminal and typed

curl -v www.google.com 

And truly enough – I got Error 7 again now. With a bit of more research – I was able to get it working. Adding these two lines to my code with the correct credentials got the code working.

curl_setopt($ch, CURLOPT_PROXY, “proxyurl:proxyport”);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, “username:password”);

And I ended my 12 hours of debugging this issue – being the more wiser about proxies and a deep understanding of CURL.

REGEXP with MySQL

In one of the recent projects that I have been working on, I had to look for a word in a phrase – but the challenge was to rank an occurrence of the phrase higher when it’s an individual word as compared to when it was part of a word.

For ex : Food Blogger should rate higher than Ardent Foodie, because in the first example – Food is an individual word as compared to latter where it is part of a whole word.

At times like this, I find MySQL’s REGEXP quite useful, as I can use different regular expressions to fit my criteria. So here’s what I did here

I first looked for an individual occurrence of a word.

select * from table_name where search_term REGEXP '[[:<:]]food[[:>:]]';

Here, [[:<:]] & [[:>:]] are markers that stand for word boundaries. They match the beginning and end of words, respectively. Now, as a next step, I follow it by this query.

select * from table_name where search_term LIKE '%food%';

But when the first case happens, it gets an higher priority as compared to the second case – so I reworked the query to this

select table_name.title, (case when search_term REGEXP '[[:<:]]food[[:>:]]'  
then 100 when search_term LIKE '%food%' then 99 else 1 end) as priority  
from table_name order by `priority` desc;  

That way, results that have the word as a standalone rank higher up as compared to when they are part of a phrase.

Deploying with Deploybot

I vividly remember the first time, I messed up a production server. It was my early days of being a programmer, and we had got our first client.

Back then, my deployment strategy was basically to upload files using FTP and then run any commands on the server via the shell. During a routine deployment, I noticed a file which remained in the server, and in trying to remove it, I typed sudo rm -rf /.

On the Production.

I watched the next few minutes in horror as the entire client’s machine was wiped clean and the site went down. Fortunately, my client was understanding, and we had backups – so there was not much of damage – but I had to spend the next 3 days fixing the mess (and contemplating if I am really cut out for this job.)

My biggest learning from the incident was to be very careful when on Production. Over the time, I learned Git and other tools, which made deployments more easier and safer. As someone developing in Laravel, and leading a team of Laravel developers – I am always on the look out to make deployments easier.

And I have tried everything from custom bash scripts to git workflows, where we would git pull on server. None of them however stuck primarily due to the complexities they bought in

And after much experimentation – my team and I zeroed down to DeployBot.

DeployBot allows you to deploy code from anywhere. It takes your code from your Github / Bitbucket or self hosted Git repositories and deploys to any server. At QICE, we primarily use Digital Ocean and AWS – both of which are supported by DeployBot and make it an ease to integrate in our projects.

Here’s how DeployBot has helped us

Continuous Deployment

Over the day, we make 2-3 deployments to our sandboxes on certain projects. And these are fairly large commits. DeployBot seamlessly gets the new commits and automatically ( or manual for production setups ) deploys the latest files to the server.

My team now does not have to worry about deploying to server. All we have to do is push to a branch, and we know it will end up being on the server.

Rollbacks

Despite much preparation, there are moments, when things don’t work on the production for weird reasons. Deploybot has a rollback to a specific version feature, which is quite nifty at times like these.

Pre and Post Deployment Commands

After deployment, we run a few commands ex : Gulp, Migrations and Composer updates.

Deploybot allows us to specify what commands to run before and after deployment. That means, more developer peace and not worrying about switching to server and typing in each command on production machines.

Modifying Configuration Files

Even after all this, you may have to sometimes go to the server to edit your configuration files.

Deploybot eliminates this as well, by asking you to enter your configuration files. Just ensure all the changes are in your configuration file before you deploy, and they are deployed in your next deployment.

Notifications

Pretty much every web app these days has Slack / Email integration – and so does Deploybot. It notifies us everytime there is a deployment in our Slack channels.

No more informing the entire team that the production is done and they can resume their work.

Amazing Support & Reliability

This is something of importance to us. In the past one year, that we used Deploybot, we faced a downtime of exactly 1 minute, where we couldn’t deploy to production. We reached out to Support and got a reply back within the next minute telling us that the issue has been fixed.

Thanks to Deploybot, my team and I can now focus on building stuff than worrying about getting it to our customers. If you are into developing web apps, it is an invaluable part of your toolset and takes care of all your deployment worries.

Autosaving in XCode

While working on a Swift Project, I spent ~ 3-5 hours adjusting a few pop-ups and fine tuning design for a screen. As it so happens, while working in the zone, I missed any committing and committed the code after my entire sprint.

Now, the other day, I was helping out a colleague with git and explaining the perils of git checkout. As it turns out, my brain decided to remind myself of the command exactly while committing and I ended up typing

git checkout . (instead of git add .)
git commit -m "My Commit Message" 
git push origin feature-branch

I didn’t realize my mistake until I came back to XCode and notice all the 4+ hours of work totally undone. Panicked, my first reaction was to find out how to undo files from an accidental checkout. Turns out, you cannot. Unless you have ever used git add or git stash on the files, there’s nothing that can be done.

I was about to lose hope when I saw an answer on SO on looking through the IDE’s auto save files. A bit more searches later – turns out OS X does auto-save files.

Gladly, I was able to get back the entire code in an auto-saved version 2-3 hours back and the day was saved thanks to auto-save.

However this is a lesson on reinforcing strict programming / source control habits.

  • Always think once before you use . operator – And twice before using git checkout.
  • Make sure you add / stash your code in smaller task increments. finished a small task – add to the index. Going to work later?, stash it