Mirrorbits is now on github!

Few months back I started working on a new project for the VideoLAN organization called Mirrorbits. Fast-forward, now in July 2014 it is finally online and open-sourced on Github!

Mirrorbits is a geographic download redirector written in Go. Let me rephrase that, it’s an application server that finds the best mirror to redirect a given user based on its location, pretty much like a CDN but using a pure software stack. For example, to distribute VLC we have to rely on a bunch of mirrors (around 70) located around the world and to make you able to download VLC faster we use mirrorbits to find the most suitable mirror for you. That means finding a mirror close to you, that is up and which has the file you requested. At first glance it can seem to be a pretty easy task to achieve but in reality there’s a lot of things to take care of. Take a look at the main features to find out all the things it can do.

The software is still in a early stage but is already used in production since April of this year, handling an average of 1 million downloads each day without a single hiccup and with a very low footprint. The system does not require a lot of attention and can run for months without human intervention since most of the usual tasks are fully automated. If a mirror goes down it will temporary be disabled until it recovers, all the scans of the mirrors (via rsync / ftp) are done in the background, …

Unlike other redirectors that only support the standard HTTP redirect, mirrorbits also has a JSON query API to be able to generate your own customized download page for showing alternate mirrors or a sponsor logo along with a countdown. On the control side it has a simple but yet powerful CLI to do the general maintenance.

Usage: mirrorbits [OPTIONS] COMMAND [arg...]

A smart download redirector.

Commands:
add        Add a new mirror
disable   Disable a mirror
edit        Edit a mirror
enable    Enable a mirror
export    Export the mirror database
list         List all mirrors
refresh   Refresh the local repository
reload    Reload configuration
remove   Remove a mirror
scan      (Re-)Scan a mirror
upgrade  Seamless binary upgrade
version   Print version informations

But that’s not all, it also has a web interface to get details about any served file, get real-time download statistics and see the number of downloads for each mirror.

Feel free to use mirrorbits for your own purpose but keep in mind that it’s still a very young project. And since I did not have the time to write a documentation yet you can contact me directly by email to get personal support :-)

To get the latest news about the project you can either star it on Github or follow the twitter account @mirrorbits.

From Gmail to Fastmail

Fastmail logoEmail is crucial to me, I can’t even remember the last day I didn’t check my inbox. It’s my preferred channel for communicating and interacting with other people “electronically”. Like many developers I also use emails to keep track of the activity of all the projects I contribute to. I also receive all sorts of emails from machines, sometime server alerts or even from my Arduino powered home automation. Finding a good email provider is a hard task. You need a company you can trust to receive, manage and deliver all your emails especially when you have more than 80k+ emails in total (like I do).

In this article I’m going to do a quick overview of the reasons I chose Fastmail as my new email provider.

A bit of history

I bought the l0cal.com domain name in 2003 and at the time I was building my first hosting company so I naturally started to self host all my emails on a server I owned. I remember starting with a qmailvpopmail, spamassassin and squirrelmail stack and it was fun. I eventually started to host friend’s emails too but after doing so for few years it really became clear that doing things right with emails is complicated. It requires a lot of work to keep spam away (RBL, greylisting, …) and to be nice with other SMTP servers that may basically refuse any messages originating from your server for no obvious reason.

The Gmail experience

While I started university around 2005 I had less and less time to spend managing my email server and keeping spam out of my inbox became even more complicated. By the time almost everyone I knew had already switched to Gmail and in early 2006 Google Apps was launched. The promise was simple: a Gmail experience with your own domain name, for free. A cool user interface with IMAP support and a spam-free inbox? I was sold.

Over time Google Apps gained ads, IMAP was working less reliably and I felt more like the product than the customer. I’ve tried to self-host my emails again quite a few times but these attempts failed for various reasons, the main one being that I was lazy.

Getting my data back under control

In the past months we’ve had a lot of scary revelations about privacy violations, the biggest being of course the NSA spying scandal. It became urgent to regain control of my data. I could have used GPG for exchanging privately with more people but encrypting everything is complicated especially considering the number of my contacts that are not used to it.

It was time to find a better email provider. At this point it was clear to me that choosing a non-free provider was the best deal. I would finally be able to enjoy a great service while not having the hassle of managing everything myself again. After few days of research for the best provider, my attention was drawn on Fastmail.

Fastmail.fm

Fastmail is nothing new. The company was founded in 1999 and they’re still alive today, and rather healthy. The company was bought by Opera in 2010 and Fastmail former employees bought the company back in 2013. What I really like about them is that they only do email and they do it well.

  • It’s blazing fast! Including IMAP, the webmail and even server-side search
  • First class SSL/TLS connections (with perfect forward secrecy)
  • 100% based on standards
  • They actively contribute to the open-source Cyrus IMAP server
  • Server side filtering rules (using Sieve scripts)
  • A lot of control for advanced users
  • Bonus: Awesome webmail

Their privacy policy is really short and comprehensible, far away from most providers. And since they are an Australian company they are subject to Australian law which has strong privacy laws in relation to email.

Yet they surprisingly run their main servers in the US and not in Australia but as stated in a post on their blog, the location of their server doesn’t make them subject to US laws because they are an Australian company. But this seems to be true only for their customers living in the USA. From my understanding their statement contradicts the Section 702 of the Foreign Intelligence Surveillance Act (FISA) that allows the NSA to conduct an investigation if the target user is not located in the USA. Yet it needs to be confirmed since I’m not a lawyer.
Anyway I really wish they opt to host some of their servers outside the USA in the future.

As noted in our recently updated privacy policy, we are an Australian company subject to Australian law. We are required to disclose information about specific individual accounts to properly authorised Australian law enforcement with the appropriate supporting documentation, which means a warrant signed by an Australian judge. We do not co-operate with any kind of blanket surveillance, monitoring or “fishing expeditions”, and we do not give out user information to anyone outside Australia.

Nevertheless they take privacy and security very seriously, SSL/TLS is mandatory, emails are replicated off-site in Iceland (where they own a bunch of servers) and they use full-disk encryption just in case. Pretty much like Lavabit but without some of the downsides. They even allow you to create alternative passwords (OTP, OTP 1h, SMS, Google Authenticator) which can be very useful when using unsecure computers.
But still, even with this level of security there is still room for unauthorized people to read your emails. I strongly urge you to use PGP if you really need to protect your conversations.

The webmail

Usually I prefer reading my emails using a desktop client with IMAP capabilities because I’ve not been impressed by any webmail I’ve used so far. But it’s nice to have one when you’re on the go.

I must admit this is the first time I was frankly impressed by a webmail. It really feels like it’s a desktop client, trust me. It’s responsive, supports tons of keyboard shortcut and it works equally well on mobile (but with gestures!). The look is professional and clean, it’s a pleasure to use.

Webmail screenshot

Taking the leap

I went for an Enhanced account with a two months free trial. The migration from Google Apps was painless even to try Fastmail out. Follow these simple steps to migrate seamlessly without losing any email in the process:

If you plan to signup for an account it would be awesome if you could use my referral link so I can maybe save few bucks during my next subscription!

Step 1: Redirecting everything

The first step was to create few personalities to handle all my email addresses (personal, work, …) and forward every new incoming email to my new Fastmail inbox using their provided alias (which is also your login). If you’re like me and you own your own domain, don’t forget to adjust your SPF records to be able to send emails from your Fastmail account in addition to your current email provider.

Step 2: Preparing your account

If you’re finally ready to use Fastmail as you main email provider you’ll need to configure your Virtual Domains so that Fastmail knows which domains to handle for you. This is easy and only take few seconds to setup.

Step 3: Changing your DNS

Change the MX records of your domain(s) to the Fastmail ones and adjust your SPF records (if any). This might take some time (depending of the TTL of your MX records) to be fully functional.
It’s a good practice to reduce the TTL before doing the actual switch.

Step 4: Moving all your emails

Use the built-in Migrate IMAP feature (available in the advanced settings) to copy all your emails from your old provider. This operation can take a very very long time. It took two days to copy my 5GB of emails out of Google Apps.

You might notice a lot of duplicated emails caused by Gmail’s non-standard way of handling folders and labels but don’t be afraid. Fastmail managed to provide a tool to do exactly that. Just launch the Mass delete/Download/Remove duplicates tool to deal with it.

Done! Now say goodbye to your old provider and enjoy Fastmail features that will make you more productive.

I hope you found this article useful and that it gave you an interesting overview of Fastmail. For more information you can read their blog, consult the status page and maybe follow them on twitter. There’s also a HN thread.

Rethinking the VLC’s mirrors infrastructure

Around 2005 when we started to gather some statistics about VLC, download numbers were around 150,000 downloads per day. Since then this number has increased significantly to reach more than 1M the good days. In the beginning we used few mirrors to handle the file distribution and it was an hassle to manage since back in the days it required a lot of human power to do a VLC release. Mostly because we had to wait several hours (if not days) until all mirrors were synchronized. Frustrated by the situation we moved to SourceForge.net during the month of April, 2010 to simplify the release process. We stayed there for 3 years until recently.

To better understand why we backed off let’s talk a bit about the SourceForge.net business model. Like any other company they have to make money to pay their bills and employees. No problem with that. The way they do it is to put ads on their downloads pages while your download starts. No problem with that either. Except when it comes to ads that are obviously designed to trick the user into believing they are part of the download procedure. Which is indeed bad and misleading. Let’s illustrate what I’m talking about.

VLC’s page on SF.net as taken on April 15, 2013 in France with an IE8 user agent.

Do you see these big buttons? Of course! They are even bigger than the real download link and you have absolutely no idea where they are linking to (Spoiler alert: it’s a scam). Obviously a lot of our users were tricked into clicking these ads and were downloading all kind of crapware. I don’t blame SourceForge for this, this is more a matter of how most advertising programs works on the web nowadays but anyway we care enough about our users to not continue this way. And yes, we asked SF.net many times to be more vigilant about the ads they are showing without much success. This is one of the reason why we (the VideoLAN organization) decided to move away from SourceForge and return to a more typical distribution channel.

Back to the mirrors

We went back to the traditional way of distributing files in the free software world: using mirrors. But we are no traditional software. We have millions of users to serve and tens of terabytes to transmit each day everywhere in the world in a reliable way. That’s not a trivial matter when you have no money for buying servers and bandwidth in every part of the world. So we had to rely on generous sponsors.

Finding the sponsors

Finding sponsors able to setup the mirrors and handle all the related costs (disk storage, bandwidth  maintenance) is nothing easy. I’ve sent hundreds of email to hosting providers, network operators and ISPs around the world and surprisingly most of them answered positively. One of the constraint we had to consider is where to put mirrors so that it reflects more or less our current user base in each country (dense areas tend to have more mirrors than others).

Every single server can (and will) fail

The situation of having a failing mirror is scary since you have no easy way to get this information soon enough to disable it without having too much users unable to download the requested files. There is no silver bullet but having good tools can help a lot in those situations. We opted for mirrorbrain, a full featured, battery included, open-source geographic load-balancer. Among its supported features mirrorbrain monitors each server, on a network and file level which is great for availability. If one of our mirror is misbehaving it will be disabled automatically, rerouting the requests to the closest available mirror in a matter of minutes and will be re-enabled as soon as it gets back online.

The setup

The first thing you need to know is that mirrorbrain only works as an Apache module. On a personal level I don’t like the Apache HTTP server, because the configuration is a pain and most of all it scales badly under pressure, hogging your CPU and memory quite fast when the traffic exceeds a certain amount of requests per second. Being scalable was not an option but a requirement so I achieved this by adding a fine-tuned nginx frontend.

Another requirement we had was to show a webpage during the download to show the logo of the selected mirror, a checksum of the file and few ads (we are currently supporting the open-source music player Tomahawk).

Putting things together this is how the actual platform looks like and what happen when you’re downloading VLC or any other software from the VideoLAN website:

Nginx is used as a frontend here, all the incoming requests are served through it. It provides static files (images, css, javascript) itself, forwarding download requests to a web application (the glue) in charge of querying mod_mirrorbrain for the best mirror for the given user and file. Eventually it generates the page containing the redirect, ads and checksum. Only few requests are directly forwarded to the Apache backend without passing by the web app but these are only used for monitoring and debugging purpose and are not part of the standard download process.

Conclusion

One month after we put the whole thing into production we are quite pleased by the result. We’re serving dozens of downloads (and VLC’s updates!) each second everywhere in the world in a reliable way from a total of 42 mirrors provided by awesome sponsors. And we even survived to a DDOS attack without a single downtime!

 

Worldwide network measurements with RIPE Atlas

The RIPE NCC is the Regional Internet Registry (RIR) for Europe, the Middle East and parts of Central Asia. It’s an administrative and technical organization in charge of allocation and registration of Internet number resources (AS numbers and IP adresses) in its region.

In december 2010 they started a new experimental project named RIPE Atlas. The aim of this project is to “produce a collection of live Internet maps with unprecedented detail” by distributing small probes across the region to collect details about the network from thousands of different locations. Anyone can participate, you can request your probe for free but notice that least equipped areas will get their probe first!

Few days ago I received mine, the hardware is an XPort® Pro from Lantronix customized by the RIPE with the MAC address of the device printed on a side.

The device contains a tiny 32-bit Freescale processor with 16MB of Flash, a 100Base-TX RJ45 connector and a USB port to give its power to the device.

The probe connected to my home router (Freebox)

Currently the predefined measurements are made against the first and second hops, a bunch of root servers and some RIPE subdomains. It is expected (with the next firmware update) that anyone with an up & running probe will earn points that could be converted at anytime to execute his own measurements within the Atlas network. It’s worth noting that the probe supports IPv4 and IPv6 as well.

The members’ area of the website is rather well-made but requires a lot of Javascript to work. For example they have made their own tabs for navigating between links. IMHO this is overkill but it’s definitely usable.

Once logged in you can access the graph generated by your own probe but also probes from others, this is not very useful at this time but it’s definitely great to see how others perform from different regions of the globe!

The Atlas network was designed with openness in mind thus the RIPE NCC also provides a very simple API to customize your graphs or access the raw data from your probe. For instance this is a graph showing the RTT from my probe located near Paris to the K-root server in IPv6:

Graph of the RTT from the probe to the K-root server in IPv6

This is the kind of project that will lead us to have a better comprehension of the inner behaviors of the networks and improves it for the sake of all.
Kudos to the RIPE NCC team for the initiative.

These companies that mislead our users

Some thoughts about the people who use the name of VLC media player to spread adware/spyware while infringing the VideoLAN‘s intellectual property and brand…

At VideoLAN we’re really fed up with all those websites/companies that are tricking our users to download malware and violate our IP by distributing misleading versions of VLC without conforming to the GPL license.

What bothers us the most is that many of them are bundling VLC with various crapware to monetize it in ways that mislead our users by thinking they’re downloading an original version. This is not acceptable. The result is a poor product that doesn’t work as intended, that can’t be uninstalled and that clearly abuses its users and their privacy. Not to mention that it also discredits our work as volunteers and that it’s time-consuming, time that is not invested in the development.

Of course this situation is not specific to VLC, other open source products are affected by this scourge and there’s not much we can do about it. They have the money to buy adwords, we don’t. Sadly, as a non-profit organization we don’t have the money to sue them.

We’re constantly trying to enforce our IP to protect our users, in the meantime tell people around you that VLC media player is and will always be free of any charge and for your own security always download it from the official website www.videolan.org.

FYI these are the two biggest companies who use VLC to distribute their crap:
- pinballcorp.com
- eorezo.com / tuto4pc.com

But there are many more:

  • http://vlc.us.com
  • http://www.eorezo.com/cgi-bin/download/direct/index?c_software=vlc
  • http://www.vlcdownload.org
  • http://www.softwaredownload.cc/?gclid=CMyGhoHrwJ8CFcpb4wodNHnJzg
  • http://www.iogiciel.com/l/index.php?option=com_content&view=article&id=53&Itemid=61
  • http://vlcplayer.2010-fr.com
  • http://www.mediaplayers-gratuits.com
  • http://www.downloadvlcplayer.net
  • http://vlc-media-player-blog.com
  • http://www.softesdown.com/fr/vlcmediaplayer/
  • http://www.getyoursoft.com/download/name/vlc-media-player/id_soft/18
  • http://supertelech.info
  • http://www.descargarvclmediaplayergratis.com
  • http://www.oficial-es.org/es
  • http://todotusoft.com/Video/Reproductor-Multimedia/1158/VLC-Media-Player.html
  • http://galleries.secure-softwaremanager.com/804e9dc7b4/854190c2bc1e
  • http://www.clickdownloadsoftware.com/player/
  • http://www.freefilesoft.net/VLCPlayer/
  • http://www.vlc.de
  • http://videolan.sk

 

Some screenshots of ads you’ll find when searching for VLC on Bing and Google:

Update #1: old links removed, new added
Update #2: added some screenshots of typical ads 

irssi: remotely attach/create a screen in one line

If you’re a big fan of irssi and you’re running it on a remote server over SSH and inside a screen, then I have a nice tip to share with you.

irssi logo

Just put the following line inside your .zshrc or .bashrc file:

alias rirssi='ssh user@your.server.com -t screen -dRUS irssi irssi'

This simple alias will allow you to:

  • Connect to your remote SSH server
  • Attach to your existing irssi screen named “irssi”
  • If the screen doesn’t already exist:
    • create a new screen
    • start irssi inside it
    • attach the newly created session

Pretty cool isn’t it? ;-)

Server migration: please welcome “Storm”!

It’s been a long time (more than a year now) since I wrote my latest post on this blog and I feel very sorry for people who were waiting for some updates! I will try to write more often and regularly…

But today, I finally moved my blog from my ageing server to a new – blazing fast – Xeon Quad Core aptly called: Storm. Some important changes occurred behind the scene; previously the website was running on a simple Apache + mod_php configuration that was overloaded very often, mostly because I’m hosting some high traffic websites like this one.

Nginx logo

After some research I went to the conclusion that the combination of Nginx + php-fpm was the way to go. So I spent much of the past week doing some benchmarks using ab and I can confirm that it is very efficient, fast and still pretty stable. I’ve also leaved Gentoo Linux in favor of Archlinux which is more straightforward to maintain on a day-by-day basis. These days Gentoo became such a pain in the ass to keep the system up-to-date without breaking everything… which is rather annoying for a server without a physical access.

We’ll see how it goes in the upcoming weeks. I still have some other domains and applications to transfer before I can definitely shutdown the old server.

EDIT: Aww… and you probably don’t care but the ipv6 access is no longer available due to various routing issues with my provider, but anyway it will be restored ASAP.

VLMC: schedule for the first release

Last weekend during the FOSDEM 2010, we planned to release our first technical preview of VLMC.
This version will be tagged as the 0.0.1 and will be made available for Windows and Unix.

The schedule:Mister cone

  • February 17th: Feature Freeze
  • March 3rd: String Freeze
  • March 24th: Release (estimated)

Some explanations about these terms:

  • Feature Freeze means that no new features will be accepted in the trunk after that date, only bug fixes and improvements of existing features will be merged.
  • String Freeze is the deadline where all the visible strings of the interface can not be changed. This ensures for the translators that they won’t have to rush during the hours that precedes the release.

Please note that this will be a preview and many key features will be missing (or even buggy).
In the meantime you can help us to squash bugs and send some patches !

VideoLAN Movie Creator: Cross-platform Video Editor

VLMC Logo

After 9 months of hard work, VideoLAN Movie Creator (aka VLMC) will be unveiled to the community during the VideoLAN Dev’days 2009 in Paris!

VLMC is a free, cross-platform, non-linear video editing software based on the famous VLC Media Player.

Made by a team of 6 students, the project started in March 2009 as an end of studies project and is now (almost) ready for its first public alpha release. Expected for early 2010, it will provide a new alternative for editing videos on Linux, Windows and even Mac.

Meet us and discover the new power of the cone on the 18th, 19th and 20th december 2009 in Paris.

Create an ebook using images on GNU/Linux

Today, my girlfriend had to create an Ebook (in pdf) using some images acquired from a scanner (using xsane). After searching a while on google how to do that I remembered that ImageMagick was the tool to manipulate images, so I gave it a try. Luckily (for me) it worked!

After this brief introduction, it’s time to share the two-steps solution with you.

Acquire the documents

The first step is to scan all of your documents using xsane. For a good quality use a DPI of 200 or more. You should name them accordingly to the order of inclusion. For example: 0001.png, 0002.png, …

Create the PDF

Now in a console, use this simple command to generate the ebook.

convert -define pdf:use-trimbox=true -compress jpeg -quality 95% *.png ebook.pdf

Remember that it can take some time if you have many pages.

That’s all folks !