Rethinking the VLC’s mirrors infrastructure

Around 2005 when we started to gather some statistics about VLC, download numbers were around 150,000 downloads per day. Since then this number has increased significantly to reach more than 1M the good days. In the beginning we used few mirrors to handle the file distribution and it was an hassle to manage since back in the days it required a lot of human power to do a VLC release. Mostly because we had to wait several hours (if not days) until all mirrors were synchronized. Frustrated by the situation we moved to SourceForge.net during the month of April, 2010 to simplify the release process. We stayed there for 3 years until recently.

To better understand why we backed off let’s talk a bit about the SourceForge.net business model. Like any other company they have to make money to pay their bills and employees. No problem with that. The way they do it is to put ads on their downloads pages while your download starts. No problem with that either. Except when it comes to ads that are obviously designed to trick the user into believing they are part of the download procedure. Which is indeed bad and misleading. Let’s illustrate what I’m talking about.

VLC’s page on SF.net as taken on April 15, 2013 in France with an IE8 user agent.

Do you see these big buttons? Of course! They are even bigger than the real download link and you have absolutely no idea where they are linking to (Spoiler alert: it’s a scam). Obviously a lot of our users were tricked into clicking these ads and were downloading all kind of crapware. I don’t blame SourceForge for this, this is more a matter of how most advertising programs works on the web nowadays but anyway we care enough about our users to not continue this way. And yes, we asked SF.net many times to be more vigilant about the ads they are showing without much success. This is one of the reason why we (the VideoLAN organization) decided to move away from SourceForge and return to a more typical distribution channel.

Back to the mirrors

We went back to the traditional way of distributing files in the free software world: using mirrors. But we are no traditional software. We have millions of users to serve and tens of terabytes to transmit each day everywhere in the world in a reliable way. That’s not a trivial matter when you have no money for buying servers and bandwidth in every part of the world. So we had to rely on generous sponsors.

Finding the sponsors

Finding sponsors able to setup the mirrors and handle all the related costs (disk storage, bandwidth  maintenance) is nothing easy. I’ve sent hundreds of email to hosting providers, network operators and ISPs around the world and surprisingly most of them answered positively. One of the constraint we had to consider is where to put mirrors so that it reflects more or less our current user base in each country (dense areas tend to have more mirrors than others).

Every single server can (and will) fail

The situation of having a failing mirror is scary since you have no easy way to get this information soon enough to disable it without having too much users unable to download the requested files. There is no silver bullet but having good tools can help a lot in those situations. We opted for mirrorbrain, a full featured, battery included, open-source geographic load-balancer. Among its supported features mirrorbrain monitors each server, on a network and file level which is great for availability. If one of our mirror is misbehaving it will be disabled automatically, rerouting the requests to the closest available mirror in a matter of minutes and will be re-enabled as soon as it gets back online.

The setup

The first thing you need to know is that mirrorbrain only works as an Apache module. On a personal level I don’t like the Apache HTTP server, because the configuration is a pain and most of all it scales badly under pressure, hogging your CPU and memory quite fast when the traffic exceeds a certain amount of requests per second. Being scalable was not an option but a requirement so I achieved this by adding a fine-tuned nginx frontend.

Another requirement we had was to show a webpage during the download to show the logo of the selected mirror, a checksum of the file and few ads (we are currently supporting the open-source music player Tomahawk).

Putting things together this is how the actual platform looks like and what happen when you’re downloading VLC or any other software from the VideoLAN website:

Nginx is used as a frontend here, all the incoming requests are served through it. It provides static files (images, css, javascript) itself, forwarding download requests to a web application (the glue) in charge of querying mod_mirrorbrain for the best mirror for the given user and file. Eventually it generates the page containing the redirect, ads and checksum. Only few requests are directly forwarded to the Apache backend without passing by the web app but these are only used for monitoring and debugging purpose and are not part of the standard download process.

Conclusion

One month after we put the whole thing into production we are quite pleased by the result. We’re serving dozens of downloads (and VLC’s updates!) each second everywhere in the world in a reliable way from a total of 42 mirrors provided by awesome sponsors. And we even survived to a DDOS attack without a single downtime!

 

Worldwide network measurements with RIPE Atlas

The RIPE NCC is the Regional Internet Registry (RIR) for Europe, the Middle East and parts of Central Asia. It’s an administrative and technical organization in charge of allocation and registration of Internet number resources (AS numbers and IP adresses) in its region.

In december 2010 they started a new experimental project named RIPE Atlas. The aim of this project is to “produce a collection of live Internet maps with unprecedented detail” by distributing small probes across the region to collect details about the network from thousands of different locations. Anyone can participate, you can request your probe for free but notice that least equipped areas will get their probe first!

Few days ago I received mine, the hardware is an XPort® Pro from Lantronix customized by the RIPE with the MAC address of the device printed on a side.

The device contains a tiny 32-bit Freescale processor with 16MB of Flash, a 100Base-TX RJ45 connector and a USB port to give its power to the device.

The probe connected to my home router (Freebox)

Currently the predefined measurements are made against the first and second hops, a bunch of root servers and some RIPE subdomains. It is expected (with the next firmware update) that anyone with an up & running probe will earn points that could be converted at anytime to execute his own measurements within the Atlas network. It’s worth noting that the probe supports IPv4 and IPv6 as well.

The members’ area of the website is rather well-made but requires a lot of Javascript to work. For example they have made their own tabs for navigating between links. IMHO this is overkill but it’s definitely usable.

Once logged in you can access the graph generated by your own probe but also probes from others, this is not very useful at this time but it’s definitely great to see how others perform from different regions of the globe!

The Atlas network was designed with openness in mind thus the RIPE NCC also provides a very simple API to customize your graphs or access the raw data from your probe. For instance this is a graph showing the RTT from my probe located near Paris to the K-root server in IPv6:

Graph of the RTT from the probe to the K-root server in IPv6

This is the kind of project that will lead us to have a better comprehension of the inner behaviors of the networks and improves it for the sake of all.
Kudos to the RIPE NCC team for the initiative.

These companies that mislead our users

Some thoughts about the people who use the name of VLC media player to spread adware/spyware while infringing the VideoLAN‘s intellectual property and brand…

At VideoLAN we’re really fed up with all those websites/companies that are tricking our users to download malware and violate our IP by distributing misleading versions of VLC without conforming to the GPL license.

What bothers us the most is that many of them are bundling VLC with various crapware to monetize it in ways that mislead our users by thinking they’re downloading an original version. This is not acceptable. The result is a poor product that doesn’t work as intended, that can’t be uninstalled and that clearly abuses its users and their privacy. Not to mention that it also discredits our work as volunteers and that it’s time-consuming, time that is not invested in the development.

Of course this situation is not specific to VLC, other open source products are affected by this scourge and there’s not much we can do about it. They have the money to buy adwords, we don’t. Sadly, as a non-profit organization we don’t have the money to sue them.

We’re constantly trying to enforce our IP to protect our users, in the meantime tell people around you that VLC media player is and will always be free of any charge and for your own security always download it from the official website www.videolan.org.

FYI these are the two biggest companies who use VLC to distribute their crap:
- pinballcorp.com
- eorezo.com / tuto4pc.com

But there are many more:

  • http://vlc.us.com
  • http://www.eorezo.com/cgi-bin/download/direct/index?c_software=vlc
  • http://www.vlcdownload.org
  • http://www.softwaredownload.cc/?gclid=CMyGhoHrwJ8CFcpb4wodNHnJzg
  • http://www.iogiciel.com/l/index.php?option=com_content&view=article&id=53&Itemid=61
  • http://vlcplayer.2010-fr.com
  • http://www.mediaplayers-gratuits.com
  • http://www.downloadvlcplayer.net
  • http://vlc-media-player-blog.com
  • http://www.softesdown.com/fr/vlcmediaplayer/
  • http://www.getyoursoft.com/download/name/vlc-media-player/id_soft/18
  • http://supertelech.info
  • http://www.descargarvclmediaplayergratis.com
  • http://www.oficial-es.org/es
  • http://todotusoft.com/Video/Reproductor-Multimedia/1158/VLC-Media-Player.html
  • http://galleries.secure-softwaremanager.com/804e9dc7b4/854190c2bc1e
  • http://www.clickdownloadsoftware.com/player/
  • http://www.freefilesoft.net/VLCPlayer/
  • http://www.vlc.de
  • http://videolan.sk

 

Some screenshots of ads you’ll find when searching for VLC on Bing and Google:

Update #1: old links removed, new added
Update #2: added some screenshots of typical ads 

irssi: remotely attach/create a screen in one line

If you’re a big fan of irssi and you’re running it on a remote server over SSH and inside a screen, then I have a nice tip to share with you.

irssi logo

Just put the following line inside your .zshrc or .bashrc file:

alias rirssi='ssh user@your.server.com -t screen -dRUS irssi irssi'

This simple alias will allow you to:

  • Connect to your remote SSH server
  • Attach to your existing irssi screen named “irssi”
  • If the screen doesn’t already exist:
    • create a new screen
    • start irssi inside it
    • attach the newly created session

Pretty cool isn’t it? ;-)

Server migration: please welcome “Storm”!

It’s been a long time (more than a year now) since I wrote my latest post on this blog and I feel very sorry for people who were waiting for some updates! I will try to write more often and regularly…

But today, I finally moved my blog from my ageing server to a new – blazing fast – Xeon Quad Core aptly called: Storm. Some important changes occurred behind the scene; previously the website was running on a simple Apache + mod_php configuration that was overloaded very often, mostly because I’m hosting some high traffic websites like this one.

Nginx logo

After some research I went to the conclusion that the combination of Nginx + php-fpm was the way to go. So I spent much of the past week doing some benchmarks using ab and I can confirm that it is very efficient, fast and still pretty stable. I’ve also leaved Gentoo Linux in favor of Archlinux which is more straightforward to maintain on a day-by-day basis. These days Gentoo became such a pain in the ass to keep the system up-to-date without breaking everything… which is rather annoying for a server without a physical access.

We’ll see how it goes in the upcoming weeks. I still have some other domains and applications to transfer before I can definitely shutdown the old server.

EDIT: Aww… and you probably don’t care but the ipv6 access is no longer available due to various routing issues with my provider, but anyway it will be restored ASAP.

VLMC: schedule for the first release

Last weekend during the FOSDEM 2010, we planned to release our first technical preview of VLMC.
This version will be tagged as the 0.0.1 and will be made available for Windows and Unix.

The schedule:Mister cone

  • February 17th: Feature Freeze
  • March 3rd: String Freeze
  • March 24th: Release (estimated)

Some explanations about these terms:

  • Feature Freeze means that no new features will be accepted in the trunk after that date, only bug fixes and improvements of existing features will be merged.
  • String Freeze is the deadline where all the visible strings of the interface can not be changed. This ensures for the translators that they won’t have to rush during the hours that precedes the release.

Please note that this will be a preview and many key features will be missing (or even buggy).
In the meantime you can help us to squash bugs and send some patches !

VideoLAN Movie Creator: Cross-platform Video Editor

VLMC Logo

After 9 months of hard work, VideoLAN Movie Creator (aka VLMC) will be unveiled to the community during the VideoLAN Dev’days 2009 in Paris!

VLMC is a free, cross-platform, non-linear video editing software based on the famous VLC Media Player.

Made by a team of 6 students, the project started in March 2009 as an end of studies project and is now (almost) ready for its first public alpha release. Expected for early 2010, it will provide a new alternative for editing videos on Linux, Windows and even Mac.

Meet us and discover the new power of the cone on the 18th, 19th and 20th december 2009 in Paris.

Create an ebook using images on GNU/Linux

Today, my girlfriend had to create an Ebook (in pdf) using some images acquired from a scanner (using xsane). After searching a while on google how to do that I remembered that ImageMagick was the tool to manipulate images, so I gave it a try. Luckily (for me) it worked!

After this brief introduction, it’s time to share the two-steps solution with you.

Acquire the documents

The first step is to scan all of your documents using xsane. For a good quality use a DPI of 200 or more. You should name them accordingly to the order of inclusion. For example: 0001.png, 0002.png, …

Create the PDF

Now in a console, use this simple command to generate the ebook.

convert -define pdf:use-trimbox=true -compress jpeg -quality 95% *.png ebook.pdf

Remember that it can take some time if you have many pages.

That’s all folks !

VLC 1.0.0 Goldeneye

VLC Media Player

VLC Media Player

After more than a year of work, the VideoLAN community is proud to announce that VLC 1.0.0 is finally out !

This release has been delayed many times since February (the original release date chosen during the VideoLAN dev’days 2008) but reached its goal today. More stable, tons of new features, this release is definitely the best and most advanced release of VLC.

Take a look at the impressive changelog, and give it a try !

A list of the most *bling* *bling* enhancements since 0.9.9:

  • Increased stability
  • Reduced memory usage
  • Live recording
  • Instant pausing and Frame-by-Frame support
  • New HD codecs (AES3, Dolby Digital Plus, TrueHD, Blu-Ray Linear PCM, Real Video 3.0 and 4.0, …)
  • New formats (Raw Dirac, M2TS, …) and majors improvements in many formats…
  • New Dirac encoder and MP3 fixed-point encoder
  • Video scaling in full-screen
  • RTSP Trickplay support
  • Zipped file playback
  • Customizable toolbars
  • Easier encoding GUI in Qt interface
  • Better integration in GTK environments
  • MTP devices on Linux
  • Air Tunes streaming
  • And much more…

Download it now !

First post ever!

… no, just kidding.

As you can imagine this is not really my first blog post ever. As every geek on this planet I already owned thousands of (unmaintained) blogs before. This is just one more :-D

Now it’s maybe time to briefly introduce myself to my fellow readers !
I’m Ludovic, a 24 years old guy living in Paris with his girlfriend and a cat (yes, that matters). I’m a free software activist and developer, taking part in some small and medium-sized projects; but this will be the subject of an upcoming post. Last but not least, I’m also a KDE and VIM user (FTW) !

kthxbye.