Using concurrent.futures to do Multithreading

Using concurrent.futures to download a list of urls and store the result in another list.

import concurrent.futures

import requests

def get_requests(url, data_list):

    resp = requests.get(url)


URLs = [”,”]


data_list = []

with concurrent.futures.ThreadPoolExecutor(max_workers=NUMBER_OF_THREADS) as executor:

    future_to_url = {executor.submit(get_requests, url, data_list):

                     url for url in URLs}

print data_list


How to Setup Static or Dynamic IP Addresses on Debian based Linux Systems

$ sudo vim /etc/network/interfaces

Set a dynamic IP address:

This is how a network having an dynamic IP address (got from a dhcp server) is configured.

auto eth0
iface eth0 inet dhcp


  • auto eth0 – enable at startup the eth0 interface
  • iface eth0 inet dhcp – consider that iface eth0 comes from interface eth0, inet tells you that the network configuration is IPv4 and dhcp that the dynamic ip is assigned by a dhcp server.

Set a static IP address

We have a little more work to do for configurationg a network interface with a static IP address. Edit your/etc/network/interfaces file again, so that it looks like this (these are my IP addresses, replace them with yours.) :

auto eth0
iface eth0 inet static


  • auto eth0 – enable at startup the eth0 interface
  • iface eth0 inet static– consider that iface eth0 comes from interface eth0, inet tells you that the network configuration is IPv4 and static that your network interface has static ip adresses.
  • address – the network’s IP address
  • netmask – the network’s mask address
  • network – the network’s address
  • broadcast – the broadcast address
  • gateway – the gateway address

It is not necesarry to insert the network and broadcast lines in the /etc/network/interfaces file because the system calculates them from the address and the netmask fields.

So, your network interface file could also look like this, for static IP configurations:

auto eth0
iface eth0 inet static

Set the network’s DNS server:

To set the DNS server address, you have to edit the /etc/resolv.conf file, with root priviledges:

$ sudo vim /etc/resolv.conf

The /etc/resolv.conf file, should look like this:

name server is Google’s private DNS address. You can replace with another DNS address, but everything works perfect with the Google DNS.

Reboot the networking service:

All the configurations that are made in configuration files, are persistent. They don’t apply before you reboot the service, but they will not be discarded after reboot. This is how you reboot the network service, to apply the changes:

$ sudo /etc/init.d/networking restart




How to Setup Static or Dynamic IP Addresses on Debian based Linux Systems

Basic Linux security


—–>  Keep your software upto date

Check if your software is up to date:

$sudo apt-get update

(This command updates your package source list. After this the system is aware what all software is available and the most recent versions )

$sudo apt-get upgrade

(This command would actually go ahead and update the software to the latest versions)

$sudo apt-get autoremove

(Removes not nedded software)

$sudo apt-install finger

(Install a software called finger.)


Install finger


(Gives all the user that are logged in currently)

$finger username

(Gives more info about the  user called username)

Finger gets most of the info from a file called ‘/etc/passwd’. This files is used to store user information.

$cat /etc/passwd |grep vagrant

vagrant:x:1000:1000:Vagrant user decscripton:/home/vagrant:/bin/bash

(Here I am getting info for a user called vagrant)

Each field in the output is separated by a ‘:’

vagrant –> username

x –> encrypted password of the user (Not used anymore)

1000 —> User id (0 for root)

1000 —> group id (0 for root)

Vagrant user decscripton —> User description

/home/vagrant —> home directory

/bin/bash –> shell

One should never be able ssh to a server as a root. 

You can create a user

$sudo adduser username

Give the new user sudo access. You can add the new user in the ‘/etc/sudoers’. But in ubuntu instead of directly adding it in this file which can be overwritten on an update you can also add the user to the ‘/etc/sudoers.d’ directory.

You can add your new user here. Here is a link for more info on sudoers

You can expire the password of this user so that he is forced to create a new secure one since now that he is added to the sudoers list.

$sudo passwd -e username


You should always use public private key authentication (RSA).

Generate a rsa key pair and always USE A PASSPHRASE!!

Disable ssh through password authentication.

$sudo nano /etc/ssh/sshd_config

In this file search for ‘passwordAuthentication yes’ change it to ‘passwordAuthentication no’. Restart ssh service.

File Permissions:

We use chmod to change file permission. But what exactly are file permission.

$ ls -al

-rw-r–r– 1 vagrant vagrant 3637 Apr  9  2014 .bashrc

We are interested in ‘rw-r–r–‘ (The first ‘-‘ represents wether it is a directory or a file)

These are basically divided in three groups

Owner : ‘rw-‘

Group : ‘r–‘

Everyone: ‘r–‘

This basically tells us which user or group of users is allowed to what with the particular file.

r: read permission

w: write permission

– : not permitted

x : allowed to execute this file

Octal permissions:

r –> 4

w –> 2

x –> 1

So to represent ‘rw-r–r–‘ in octal form we just add the values



$chmod 644 filename

Will make the filename permssions to be ‘rw-r–r–‘


Ubuntu comes with a firewall called ‘ufw’. You can check the status using

$sudo ufw status

Good rule of thumb is to start with denying all incoming traffic.

$ sudo ufw default deny incoming

Also we should allow all outgoing traffice

$sudo ufw default allow outgoing

Now open only the ports you need to use

$sudo ufw allow ssh

(This will open the port 22 to allow us to ssh to the server)

$sudo ufw allow www

(Open port 80 for http traffic)

Using Pep8 and autopep8

PEP 8:

Pep8 is a tool to check your Python code against some of the style conventions in PEP 8.

For more info check this out:



You can install pep8 using pip

$ pip install pep8



Here you are checking the file to see if it has any errors. This will only give you the pep8 error and the line on which the error occurs. The ‘–first’ will only show you the first occurrence of each error.

$ pep8 –first


If you also want to see the code which has the error along with the pep8 error. The file is being checked.

$ pep8 –show-source –show-pep8 testsuite/



Autopep8 automatically formats Python code to conform to the PEP 8 style guide. Autopep8 is capable of fixing most of the formatting issues that can be reported by pep8. Also autoppep8 is awesome (USE IT!!!!!)


$ pip install autopep8


$autopep8 –in-place –aggressive –aggressive <filename>


-i, --in-place        make changes to files in place
-a, --aggressive      enable non-whitespace changes; multiple -a result in
                      more aggressive changes


Implement LRU Cache

How to implement LRU caching scheme? What data structures should be used?

We are given total possible page numbers that can be referred. We are also given cache (or memory) size (Number of page frames that cache can hold at a time). The LRU caching scheme is to remove the least recently used frame when the cache is full and a new page is referenced which is not there in cache. Please see the Galvin book for more details (see the LRU page replacement slide here).

We use two data structures to implement an LRU Cache.

1. A Queue which is implemented using a doubly linked list. The maximum size of the queue will be equal to the total number of frames available (cache size).
The most recently used pages will be near front end and least recently pages will be near rear end.

2. A Hash with page number as key and address of the corresponding queue node as value.

When a page is referenced, the required page may be in the memory. If it is in the memory, we need to detach the node of the list and bring it to the front of the queue.
If the required page is not in the memory, we bring that in memory. In simple words, we add a new node to the front of the queue and update the corresponding node address in the hash. If the queue is full, i.e. all the frames are full, we remove a node from the rear of queue, and add the new node to the front of queue.

Note: Initially no page is in the memory.


How to run one last function before getting killed in Python?


The atexit module defines a single function to register cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination. atexit runs these functions in the reverse order in which they were registered; if you register A, B, and C, at interpreter termination time they will be run in the order C, B, A.

Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.

This is an alternate interface to the functionality provided by the sys.exitfunc() variable.

Note: This module is unlikely to work correctly when used with other code that sets sys.exitfunc. In particular, other core Python modules are free to use atexit without the programmer’s knowledge. Authors who use sys.exitfunc should convert their code to use atexit instead. The simplest way to convert code that sets sys.exitfunc is to import atexit and register the function that had been bound to sys.exitfunc.


import atexit

def print_count():

    print ‘This function is called on an OS exit or system exit’