Infrastructure: Difference between revisions
Line 108: | Line 108: | ||
apt-get install nginx | apt-get install nginx | ||
===== nginx configuration ===== | |||
<pre> | |||
/etc/nginx/sites-available# more off | |||
## | |||
# You should look at the following URL's in order to grasp a solid understanding | |||
# of Nginx configuration files in order to fully unleash the power of Nginx. | |||
# http://wiki.nginx.org/Pitfalls | |||
# http://wiki.nginx.org/QuickStart | |||
# http://wiki.nginx.org/Configuration | |||
# | |||
# Generally, you will want to move this file somewhere, and start with a clean | |||
# file but keep this around for reference. Or just disable in sites-enabled. | |||
# | |||
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples. | |||
## | |||
# Default server configuration | |||
# | |||
server { | |||
listen 80 default_server; | |||
listen [::]:80 default_server; | |||
server_name openfoodfacts.eu *.openfoodfacts.eu; | |||
# SSL configuration | |||
# | |||
# listen 443 ssl default_server; | |||
# listen [::]:443 ssl default_server; | |||
# | |||
# Self signed certs generated by the ssl-cert package | |||
# Don't use them in a production server! | |||
# | |||
# include snippets/snakeoil.conf; | |||
root /home/off/html; | |||
# Add index.php to the list if you are using PHP | |||
index index.html index.htm index.nginx-debian.html; | |||
location ~ ^/(images|js|rss|data|files|resources|foundation)/ { | |||
# First attempt to serve request as file, then | |||
# as directory, then fall back to displaying a 404. | |||
try_files $uri $uri/ =404; | |||
} | |||
location = /robots.txt { | |||
try_files $uri $uri/ =404; | |||
} | |||
location / { | |||
proxy_set_header Host $host; | |||
proxy_set_header X-Real-IP $remote_addr; | |||
proxy_pass http://127.0.0.1:8001/cgi/display.pl?; | |||
} | |||
location /cgi/ { | |||
proxy_set_header Host $host; | |||
proxy_set_header X-Real-IP $remote_addr; | |||
proxy_pass http://127.0.0.1:8001; | |||
} | |||
# deny access to .htaccess files, if Apache's document root | |||
# concurs with nginx's one | |||
# | |||
#location ~ /\.ht { | |||
# deny all; | |||
#} | |||
} | |||
</pre> | |||
/etc/nginx/sites-enabled# ln -s /etc/nginx/sites-available/off off | |||
rm default | |||
service nginx restart | |||
To check for errors: | |||
systemctl -l status nginx.service | |||
===== Apache configuration ===== | |||
off.conf: | |||
<pre> | |||
/etc/apache2/sites-available# cat off.conf | |||
# LoadModule perl_module modules/mod_perl.so | |||
PerlWarn Off | |||
PerlRequire /home/off/cgi/startup.pl | |||
<Location /cgi> | |||
SetHandler perl-script | |||
PerlResponseHandler ModPerl::Registry | |||
PerlOptions +ParseHeaders | |||
Options +ExecCGI | |||
Require all granted | |||
</Location> | |||
<VirtualHost *> | |||
DocumentRoot /home/off/html | |||
ServerName openfoodfacts.eu | |||
ErrorLog /home/off/logs/error_log | |||
CustomLog /home/off/logs/access_log combined | |||
LogLevel debug | |||
ScriptAlias /cgi/ "/home/off/cgi/" | |||
<Directory /home/off/html> | |||
Require all granted | |||
</Directory> | |||
</VirtualHost> | |||
PerlPostReadRequestHandler My::ProxyRemoteAddr | |||
</pre> | |||
<pre> | |||
/etc/apache2/sites-enabled# ls -lrt | |||
total 0 | |||
lrwxrwxrwx 1 root root 35 Jun 13 22:12 000-default.conf -> ../sites-available/000-default.conf | |||
/etc/apache2/sites-enabled# rm 000-default.conf | |||
/etc/apache2/sites-enabled# ln -s ../sites-available/off.conf off.conf | |||
</pre> | |||
Port 8001 | |||
<pre> | |||
/etc/apache2# vi ports.conf | |||
#Listen 80 | |||
Listen 8001 | |||
</pre> | |||
service apache2 restart | |||
To check for errors: | |||
systemctl -l status apache2.service | |||
mkdir /home/off/logs | |||
==== Product Opener ==== | ==== Product Opener ==== | ||
Line 153: | Line 299: | ||
vi /home/cgi/Config2.pm | vi /home/cgi/Config2.pm | ||
-> put right values for server domain, home path, and mongodb database name | -> put right values for server domain, home path, and mongodb database name | ||
</pre> | |||
===== robots.txt ===== | |||
Since we will run a copy of OFF on a separate domain, add a line to forbid robots completely. | |||
<pre> | |||
/home/off/html# vi robots.txt | |||
User-agent: * | |||
Disallow: / | |||
Disallow: /cgi | |||
Disallow: /code | |||
~ | |||
</pre> | </pre> |
Revision as of 12:51, 17 June 2016
Infrastructure
This page describes the hardware/software infrastructure for the Open Food Facts + Open Beauty Facts projects.
New server install log
OFF and OBF have been hosted from 2012 to 2016 on a (now very old) OVH dedicated server that is also hosting other projects. On June 13th 2016, a new dedicated server has been ordered specifically for OFF and OBF.
Hardware
- E3-SAT-3 Intel Xeon E3 1245v2 4 c / 8 t 3.4 GHz+ 32 Go 2x 2 To SATA3 Soft 16* 40.00€ HT (SOIT 48.00€ TTC)
- http://www.soyoustart.com/fr/offres/e3-sat-3.xml
Server setup
- Order server and pay.
- Login to manager https://eu.soyoustart.com/manager
- Install server
- Debian 8.4 Stable Jessie
Server configuration
- uname -a
- Linux ns3362784.ip-37-187-74.eu 3.14.32-xxxx-grs-ipv6-64 #7 SMP Wed Jan 27 18:05:09 CET 2016 x86_64 GNU/Linux
- perl -v
- This is perl 5, version 20, subversion 2 (v5.20.2) built for x86_64-linux-gnu-thread-multi
Basic configuration
- apt-get update
- apt-get upgrade
- apt-get install fail2ban
- apt-get install sudo
- apt-get install build-essential
- apt-get install git
Users
- admin users with sudo access
- off user
DNS
Product Opener needs a domain, with a A record for the domain itself and another wildcard A record for all subdomains.
For testing the new server, we will be using openfoodfacts.eu
Product Opener dependencies
exim
- apt-get install exim4
- dpkg-reconfigure exim4-config
- Internet Site mail is sent by smtp
- 127.0.0.1
MongoDB
See https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/
apt-get install mongodb
MongoDB shell version: 2.4.10
service mongod stop mv /var/lib/mongodb /home/mongodb
vi /etc/mongod.conf
# dbPath: /var/lib/mongodb dbPath: /home/mongodb
service mongod start
Stars with some warnings:
mongo MongoDB shell version: 3.2.7 connecting to: test Server has startup warnings: 2016-06-13T19:34:08.245+0200 I CONTROL [initandlisten] 2016-06-13T19:34:08.246+0200 I CONTROL [initandlisten] ** WARNING: Cannot detect if NUMA interleaving is enabled. Failed to probe "/sys/devices/system/node/node1": Permission denied 2016-06-13T19:34:08.246+0200 W CONTROL [initandlisten] 2016-06-13T19:34:08.246+0200 W CONTROL [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied 2016-06-13T19:34:08.246+0200 W CONTROL [initandlisten] 2016-06-13T19:34:08.246+0200 W CONTROL [initandlisten] Failed to probe "/sys/kernel/mm/transparent_hugepage": Permission denied 2016-06-13T19:34:08.246+0200 I CONTROL [initandlisten]
Apache / mod_perl and nginx
Apache 2 + mod_perl serve the dynamically generated HTML pages from Product Opener.
nginx is installed on port 80 as a reverse proxy. It serves the static files (images, JS, CSS etc.) and proxies the dynamic requests to the Apache server on another port.
apt-get install apache2
- stop apache in order to be able to install nginx (default port 80)
service stop apache2
apt-get install nginx
nginx configuration
/etc/nginx/sites-available# more off ## # You should look at the following URL's in order to grasp a solid understanding # of Nginx configuration files in order to fully unleash the power of Nginx. # http://wiki.nginx.org/Pitfalls # http://wiki.nginx.org/QuickStart # http://wiki.nginx.org/Configuration # # Generally, you will want to move this file somewhere, and start with a clean # file but keep this around for reference. Or just disable in sites-enabled. # # Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples. ## # Default server configuration # server { listen 80 default_server; listen [::]:80 default_server; server_name openfoodfacts.eu *.openfoodfacts.eu; # SSL configuration # # listen 443 ssl default_server; # listen [::]:443 ssl default_server; # # Self signed certs generated by the ssl-cert package # Don't use them in a production server! # # include snippets/snakeoil.conf; root /home/off/html; # Add index.php to the list if you are using PHP index index.html index.htm index.nginx-debian.html; location ~ ^/(images|js|rss|data|files|resources|foundation)/ { # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. try_files $uri $uri/ =404; } location = /robots.txt { try_files $uri $uri/ =404; } location / { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:8001/cgi/display.pl?; } location /cgi/ { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_pass http://127.0.0.1:8001; } # deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} }
/etc/nginx/sites-enabled# ln -s /etc/nginx/sites-available/off off rm default
service nginx restart
To check for errors:
systemctl -l status nginx.service
Apache configuration
off.conf:
/etc/apache2/sites-available# cat off.conf # LoadModule perl_module modules/mod_perl.so PerlWarn Off PerlRequire /home/off/cgi/startup.pl <Location /cgi> SetHandler perl-script PerlResponseHandler ModPerl::Registry PerlOptions +ParseHeaders Options +ExecCGI Require all granted </Location> <VirtualHost *> DocumentRoot /home/off/html ServerName openfoodfacts.eu ErrorLog /home/off/logs/error_log CustomLog /home/off/logs/access_log combined LogLevel debug ScriptAlias /cgi/ "/home/off/cgi/" <Directory /home/off/html> Require all granted </Directory> </VirtualHost> PerlPostReadRequestHandler My::ProxyRemoteAddr
/etc/apache2/sites-enabled# ls -lrt total 0 lrwxrwxrwx 1 root root 35 Jun 13 22:12 000-default.conf -> ../sites-available/000-default.conf /etc/apache2/sites-enabled# rm 000-default.conf /etc/apache2/sites-enabled# ln -s ../sites-available/off.conf off.conf
Port 8001
/etc/apache2# vi ports.conf #Listen 80 Listen 8001
service apache2 restart
To check for errors:
systemctl -l status apache2.service
mkdir /home/off/logs
Product Opener
Libraries
- apt-get install zlib1g-dev
Perl modules
apt-get install libwww-perl libimage-magick-perl libxml-encoding-perl libtext-unaccent-perl libmime-lite-perl libcache-memcached-fast-perl libjson-perl libclone-perl libgraphviz-perl libmime-lite-perl libcrypt-passwdmd5-perl libencode-detect-perl libgraphics-color-perl libbarcode-zbar-perl libxml-feedpp-perl libmongodb-perl liburi-find-perl libxml-simple-perl
Some modules seem not to have Debian packages and must be built using CPAN:
cpan install URI::Escape::XS install Encode::Punycode install GraphViz2 install HTML::Defang install Algorithm::CheckDigits install Geo::IP install Image::OCR::Tesseract install DateTime::Format::Mail install DateTime::Format::CLDR install DateTime::Locale
Symbolic links in cgi directory =
ls -lrt |grep -- "->"
Make sure all links are pointing to the right path.
/home/off/cgi# rm Blogs /home/off/cgi# ln -s /home/off/cgi Blogs ln -s SiteLang_off.pm SiteLang.pm vi /home/off/cgi/startup.pl # Needs to be configured use lib "/home/off/cgi/"; vi /home/cgi/Config2.pm -> put right values for server domain, home path, and mongodb database name
robots.txt
Since we will run a copy of OFF on a separate domain, add a line to forbid robots completely.
/home/off/html# vi robots.txt User-agent: * Disallow: / Disallow: /cgi Disallow: /code ~