Easier Webhook Testing with ngrok

Picture this scenario :

You have built a web app that has webhooks from other service. But everytime you want to test it out, you have to deploy your app, and keep a look out on your logs as the request trickles in. 

or

You are working with a mobile developer in-house, and you both want to test an API. You deploy the app, even if it is a minute change and then wait to see if all works properly. 

I am sure, as a developer, we have all been there. Deploying apps for a couple of small changes, and make it run through your entire CI/CD cycle can be exhausting. And so I was delighted, when I stumbled upon ngrok.

ngrok is a tool that allows you to access your localhost via Internet. This can save you from countless hours of frustrations by avoiding deploying code for minor changes.

Running ngrok is as simple as downloading the relevant files from the website. Once downloaded, on Mac / Linux – type

unzip /path/to/ngrok.zip

Once you have unzipped the contents, ensure your localhost is running and you know the port number. To get your server accessible via Internet, type

./ngrok http your-port-number

This would start ngrok, and now your web app is publicly accessible via the Internet. The terminal also shows the incoming request to your app.

The Tunnel URL in my case is http://b3991383.ngrok.io, which points to port 8000 on my device. You can now update the tunnel URL in your webhooks, or pass it on to your client to test the app.

One cool feature of ngrok is it’s live dashboard – which allows you to inspect the status of server as well as the requests that are coming in. You can access this dashboard by navigating to http://localhost:4040/inspect/http

Apart from using ngrok for development, you can also use it to host your own webmail or other apps. ngrok also offers a paid tier with support for custom domains and other features.

ngrok has been easily one of the tools that has saved me hours of development time, and the paid plan is just worth it.

When CURL fails

Every once in a while, as a programmer, you seemingly come upon a problem that flummoxes you, irritates you and takes you through a wide array of emotions – and in the end when you solve it, makes you look back and think, why did this issue even take so much time to fix.

Here’s one from my diaries: I was working for a client which dealt with lots of financial data, building an app which dealt with peer to peer wallet system and phone recharges. The app’s backend API was built using a custom PHP framework.

After having built the system, I went to my client’s office to deploy the system on their homegrown servers. One thing that I should mention is, being a company that handles lot of financial data – the entire campus was very secure. Regular audits, Access only to authorised personnel. But what struck me odd was that even the computers were not connected to Internet – A team of 20 programmers shared two computers that connected to Internet, which they used when they wanted to research the issues they were working on.

Having given a machine, I installed PHP on the servers and deployed my app – so I can start testing. This is where all the troubles started for me. For one of the methods, I had written some code to call client’s API to recharge their phone – but just no matter what happened, the API wouldn’t work.

Looking at the code, the code was written in a simple CURL snippet

// create a new cURL resource$ch = curl_init();
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/");
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and execute it
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);

But now here’s the funny part – the same URL was accessible via the browser, postman clients and everywhere. But through the code, it would simply refuse to work.

After looking into the logs – and learning more about CURL and how it works – I added the curl_error function to see what is failing this bit of code.

Error 7 – Bad request. Could not connect to host. 

Ah. So now we were getting somewhere. But there could be multiple reasons why we could not connect to the host – network dropping out, Internet not accessible and so on.

That’s when it hit me like a flash of light – The entire computers at the client’s office were passing through a proxy. Now the URLs worked perfectly because the browsers were configured. I opened a terminal and typed

curl -v www.google.com 

And truly enough – I got Error 7 again now. With a bit of more research – I was able to get it working. Adding these two lines to my code with the correct credentials got the code working.

curl_setopt($ch, CURLOPT_PROXY, “proxyurl:proxyport”);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, “username:password”);

And I ended my 12 hours of debugging this issue – being the more wiser about proxies and a deep understanding of CURL.

Deploying with Deploybot

I vividly remember the first time, I messed up a production server. It was my early days of being a programmer, and we had got our first client.

Back then, my deployment strategy was basically to upload files using FTP and then run any commands on the server via the shell. During a routine deployment, I noticed a file which remained in the server, and in trying to remove it, I typed sudo rm -rf /.

On the Production.

I watched the next few minutes in horror as the entire client’s machine was wiped clean and the site went down. Fortunately, my client was understanding, and we had backups – so there was not much of damage – but I had to spend the next 3 days fixing the mess (and contemplating if I am really cut out for this job.)

My biggest learning from the incident was to be very careful when on Production. Over the time, I learned Git and other tools, which made deployments more easier and safer. As someone developing in Laravel, and leading a team of Laravel developers – I am always on the look out to make deployments easier.

And I have tried everything from custom bash scripts to git workflows, where we would git pull on server. None of them however stuck primarily due to the complexities they bought in

And after much experimentation – my team and I zeroed down to DeployBot.

DeployBot allows you to deploy code from anywhere. It takes your code from your Github / Bitbucket or self hosted Git repositories and deploys to any server. At QICE, we primarily use Digital Ocean and AWS – both of which are supported by DeployBot and make it an ease to integrate in our projects.

Here’s how DeployBot has helped us

Continuous Deployment

Over the day, we make 2-3 deployments to our sandboxes on certain projects. And these are fairly large commits. DeployBot seamlessly gets the new commits and automatically ( or manual for production setups ) deploys the latest files to the server.

My team now does not have to worry about deploying to server. All we have to do is push to a branch, and we know it will end up being on the server.

Rollbacks

Despite much preparation, there are moments, when things don’t work on the production for weird reasons. Deploybot has a rollback to a specific version feature, which is quite nifty at times like these.

Pre and Post Deployment Commands

After deployment, we run a few commands ex : Gulp, Migrations and Composer updates.

Deploybot allows us to specify what commands to run before and after deployment. That means, more developer peace and not worrying about switching to server and typing in each command on production machines.

Modifying Configuration Files

Even after all this, you may have to sometimes go to the server to edit your configuration files.

Deploybot eliminates this as well, by asking you to enter your configuration files. Just ensure all the changes are in your configuration file before you deploy, and they are deployed in your next deployment.

Notifications

Pretty much every web app these days has Slack / Email integration – and so does Deploybot. It notifies us everytime there is a deployment in our Slack channels.

No more informing the entire team that the production is done and they can resume their work.

Amazing Support & Reliability

This is something of importance to us. In the past one year, that we used Deploybot, we faced a downtime of exactly 1 minute, where we couldn’t deploy to production. We reached out to Support and got a reply back within the next minute telling us that the issue has been fixed.

Thanks to Deploybot, my team and I can now focus on building stuff than worrying about getting it to our customers. If you are into developing web apps, it is an invaluable part of your toolset and takes care of all your deployment worries.

Autosaving in XCode

While working on a Swift Project, I spent ~ 3-5 hours adjusting a few pop-ups and fine tuning design for a screen. As it so happens, while working in the zone, I missed any committing and committed the code after my entire sprint.

Now, the other day, I was helping out a colleague with git and explaining the perils of git checkout. As it turns out, my brain decided to remind myself of the command exactly while committing and I ended up typing

git checkout . (instead of git add .)
git commit -m "My Commit Message" 
git push origin feature-branch

I didn’t realize my mistake until I came back to XCode and notice all the 4+ hours of work totally undone. Panicked, my first reaction was to find out how to undo files from an accidental checkout. Turns out, you cannot. Unless you have ever used git add or git stash on the files, there’s nothing that can be done.

I was about to lose hope when I saw an answer on SO on looking through the IDE’s auto save files. A bit more searches later – turns out OS X does auto-save files.

Gladly, I was able to get back the entire code in an auto-saved version 2-3 hours back and the day was saved thanks to auto-save.

However this is a lesson on reinforcing strict programming / source control habits.

  • Always think once before you use . operator – And twice before using git checkout.
  • Make sure you add / stash your code in smaller task increments. finished a small task – add to the index. Going to work later?, stash it