This project makes use of my raspberry pi "server" sitting at home, to do real time monitoring of stock exchanges around the world.
- Monitor specific stocks, and according to a set of criteria (strategies) advise the right time to buy/sell a specific stock. Advices are sent via Telegram.
- Keep track of all movements (buy, sell, dividend, ...) of the stocks existing in a portfolio, and calculate total return of the investment. Receives commands to buy, sell, ... via Telegram.
- Test the investment strategies with historical data (aka "backtesting") to check the return of such strategies with past quotations
The project uses quotations from Google Finance, that allows real time quotation in many exchanges around the world. See current list of stock exchanges in this link
The project uses past quotations from Yahoo Finance for backtesting purposes.
The project includes a Telegram bot, that allows a user to interact with the raspberry pi, via exchanged messages. See example in attached photo.
For example, sending a message of "/returns" to the bot, will receive a reply with the current return of every stock in the portfolio.
The project accounts for different opening and closing times in markets worldwide.
It works with multiple time zones, and multiple currencies.
All data is stored in a local sqlite database.
Step 1: Prepare the Raspberry Pi
My particular setup is a Raspberry pi 3, with an external WD Pi Drive 314Gb disk. There is no need for the external disk though. The SD card should be enough, unless you have a few thousand stocks to monitor...
I will not show how to setup the raspberry pi to have it running normally.
I assume that you have already a running raspberry pi, with internet access.
Files for the project can be found in github here .Clone the files to your raspberry pi, issuing the command bellow, at the raspberry pi command line:
git clone https://github.com/kal001/stocks.git
You should have git installed in your raspberry pi for this to work. If it is not the case, install it first:
sudo apt-get install git-core
You should now have a "stocks" folder with all the files on it. List them with ls -la just to see if everything worked ok.
Step 2: Create Database
sudo apt-get install sqlite3
Run sqlite3 to create the database file (stockdata.sqlite):
At the sqlite prompt, create the table structure by reading the commands in sql file stocks_db_structure.sql:
sqlite> .read stocks_db_structure.sql
Check if the tables where created correctly and exit:
sqlite> .tables currencies exchanges movements portfolio stocks dividends hollidays options quotes strategies sqlite> .exit
You should now have a stockdata.sqlite file with the database.
All tables should be empty. You can populate them, to suit your needs. To do so, check the file database.html that list all the tables in the database, and the meaning of each field.
If you want to populate the tables with some sample data that I use, run sqlite again and this time read the commands from stocks_db_data.sql:
sqlite3 stockdata.sqlite sqlite> .read stocks_db_data.sql
To test if everything is ok, list the rows in stocks table, for example:
sqlite> select * from stocks; 1|Millenium BCP|ELI:BCP|BCP.LS|1|False|15|||1|stock 2|The Navigator Company|ELI:NVG|NVG.LS|1|True|15|2.88|2016-06-10T17:35:00+02:00|1|stock 3|EURO Dollar exchange ratio|EURUSD|EURUSD=X|3|True|60|1.1253|2016-06-12T14:41:38.815735+00:00||currency 4|Jeronimo Martins|ELI:JMT|JMT.LS|1|True|15|13.77|2016-06-10T17:36:00+02:00|1|stock 5|Galp Energia|ELI:GALP|GALP.LS|1|False|15|||1|stock 6|EDP Energias de Portugal|ELI:EDP|EDP.LS|1|False|15|||1|stock 7|REN Redes Energeticas Nacionais|ELI:RENE|RENE.LS|1|False|15|||1|stock 8|CTT Correios de Portugal SA|ELI:CTT|CTT.LS|1|False|15|||1|stock 9|Qlik Technologies, Inc.|NASDAQ:QLIK|QLIK|4|True|15|30.0|2016-06-10T16:00:00-04:00|2|stock
You can find a description of each database table, in the file database.html .
Most important tables are: stocks, where you describe the stocks you are interested in; exchanges where you define the stock exchanges where these will be traded (defining opening and close hours of the exchange for example)
Step 3: Install Requirements
Start by installing pip.
pip is a package installer for python. With pip installed, it is very easy to install python packages.
To install pip, at command line write:
sudo apt-get install python-pip
Now install all the required packages:
sudo pip install -r requirements.txt
Now you are ready to start using the several python scripts included.
- monitor.py - that monitors continuously the stock market
stock_telegrambot.py - that is a Telegram bot that will send and receive telegram messages to interact with your scripts/stocks
- ggetquote.py - script that prints the most recent quote value for all the stocks in the stocks table, that have the field Tracked = True
backtest.py - script that tests some investment strategies saved in the table strategies, against historical data, and returns the value of the stock according to that strategy
Each script is detailed in the next steps.
Step 4: Configure Telegram/creating a Bot
You have to install Telegram in your devices (phone, computer, tablet, ...), and create an account, if you have not done so yet.
Telegram is a very fast, safe, and flexible messenger application.It works in iOS, Android, Windows, Mac, Linux, ...
It allows you to create a "bot" that is a program that handles messages. In this case we will write a script in python that will interact with the users, and this script runs in the raspberry pi.
A) Create a bot
When telegram opens with a chat with botfather, write /newbot to create a new bot.
Telegram will ask you for a name to the bot. Name it whatever you like, for example stockserver .
Telegram will ask for a username that should end in bot. For example stockserver_bot .
Botfather then creates the bot, and should give you a key to access it in the future. Something like:
Done! Congratulations on your new bot. You will find it at telegram.me/stockserver_bot. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this.
Use this token to access the HTTP API:
For a description of the Bot API, see this page: https://core.telegram.org/bots/api
The key is the bold underlined 2341... above. You will need to insert that key in the file stocks.ini:
[Telegram] token = 2341...
I advise you to copy paste this message and insert it into a file called stockserver_bot.txt, and save it for future memory.
Now you can tell your bot which commands it will accept. This step is Optional. But if you do it, if will ease issuing commands in your Telegram clients.
Write /setcommands at bothfather, and choose your bot. Now write the commands that you bot should understand:
buy - Buy shares sell - Sell shares dividend - Set new dividend status - Check bot status portfolio - Get current portfolio returns - Get current portfolio returns movements - Get movements for a stock
B) Find your telegram user id
Your bot needs to know your userid that Telegram assigned you, in order to send you messages.
Go to telegram, and send a message with /startto your bot.
You will receive no answer, as the raspberry pi is not yet running the bot, to answer you.
Now run the script gettelegramuid.py. It will print the number of your user id in Telegram, that it got from the message /start that you sent him.
Copy this number (something like 123456789) to the file stocks.ini:
[Telegram] token = ... uid = 123456789
You can also check gettelegramuid.py just to get a flavor of how it works. It is only a few lines.
Step 5: Monitorstock.py
The monitor.py is a python script that:
- See if it is time to buy or sell stock, according to different strategies.
All strategies that have Active = True in the table strategies are checked to see if:
A) It is time to buy - when the stock closes down for a consecutive defined number of days, and opens down again
B) It is time to sell - if the minimum specified return is reached
A Telegram message is sent to alert the user that it is time to buy/sell a stock
Acquire tracked stock quotes, and store them in the database.
All stocks that have Tracked = True in the table stocks are acquired with the defined interval sample time. Quotes are stored in the quotes table.
The scipt checks if exchanges are open (taking into account opening hours, weekends and holidays of each stock exchange), before doing any of the precedent actions.
Step 6: Putting Monitorstock.py to Run Periodically and Start Stock_telegrambot.py
A) Putting monitorstock.py to run periodically
You should install monitostock.py to run periodically. To do that, in raspberry pi write crontab -e. You should see your predefined editor (nano in my case) displaying the list of tasks that your raspberry pi runs periodically.
Add at the end of this file:
MAILTO="" */1 * * * * cd ~/bolsa && python monitorstock.py 0 12 * * * /home/fernando/bolsa/restartbot.sh
Press ctrl+X to exit. Answer Y to save the chages, and press enter. You will be back at prompt.
The first line tells crontab not to send emails with the output of the execution. Otherwise you would receive an email per minute!!
The second line changes to the folder where the scripts were installed, and then tells crontab to run python monitorstock.py every minute (*/1). If you want to change the periodicity to 5 minutes for example, change the */1 to */5 .
The third line runs a bash script everyday at 12:00. The script is restartbot.sh. This script will stop the stock_telegrambot.py if it is running, and then immediately starts it again.
I found that for some unexplained reason after running for a few days, the script stock_telegrambot.py stops answering, even if it running apparently ok.
Stopping and starting it again everyday tries to avoid this behavior.
B) Starting stock_telegrambot.py
Write ./startbot.sh . Your bot should start, and you should get a message like this, confirming it in your Telegram:
Start stock_telegrambot.py v0.0.2-5-gf75a193 2016-06-12 21:33:57.458815
The bot will be a running task, running in the background. It will run forever.
If you want to find out the task, write ps aux . That will print a list of all the tasks running in the raspberry pi. Find the line saying "python stock_telegrambot.py".
If you want to stop the bot, write ./killbot.sh . Confirm that the task is not executing, again writing ps aux.
C) Putting stock_telegrambot.py to run when raspberry pi boots
If you want the bot to start every time that the raspberry pi boots, issue sudo nano /etc/rc.local
Add the command line /home/fernando/bolsa/startbot.sh in this file, before the last line with exit 0. Press ctrl+x to exit, and answer Y to save the file.
Step 7: Playing Around
Now that everything is installed, you can use telegram to:
- /buy - tells telegram to add some stocks to your portfolio
Example of buying 50 shares of ELI:JMT at 15€ per stock:
/buy 50 eli:jmt 15 Success. Bought 50.00 ELI:JMT @ 15.000. New quantity on hand 400.00. New average price 11.875
- /sell - sells some stocks from your portfolio
Example of selling 50 shares of ELI:JMT at 16€ per stock:
/sell 50 eli:jmt 16 Success. Sold 50.00. New quantity on hand 350.00.
- /portfolio - returns your current portfolio
/portfolio QTY STOCK PRICE 350.00 Jeronimo Martins (ELI:JMT) 11.875 30.00 Qlik Technologies, Inc. (NASDAQ:QLIK) 30.000
- /returns - prints the current return for every asset in the portfolio
/returns QTY STOCK RETURN % 350.00 Jeronimo Martins 12.8 30.00 Qlik Technologies, Inc. -0.1
- /status - checks the status of the running bot
/status Ok. Running v0.0.2-5-gf75a193
- /dividend - tells the bot about a new dividend for a stock
Example of setting a dividend of 0.7€ per share of ELI:JMT with ex-dividend date 2020-01-01:
/dividend eli:jmt 0.7 2020-01-01 Ok. Dividend set for ELI:JMT
And that's it!!
This scripts are still in early stage of development. Although they seem to work ok, I expect that some bugs may exist. If you have a problem, send me an email.